Consolidated patch set.
cover Consolidated patch set.
This patch set is currently running on dev-ab-master.pdx.yoctodev.org:8010 for testing purposes.
The main bits of functionality here:
- fixes up some of the last issues with yoctogit.py that have cropped up.
- fixes how we use patchExists within yoctogit.py
- reimpliments some old release functionality that we lost with the UI changes.
- Cleans up some yoctogit.py code
- supports the name change of core-image-basic
- gives us some logging with the buildslave-janitor
- adds a new buildset to run builds on target
The following changes since commit ff48d4619a6910aa37473fd323d1865d8a32eb0c:
BuildSet.py: Indent error causes triggers with no nowait to fail (2014-03-19 08:47:16 -0700)
are available in the git repository at:
git://git.yoctoproject.org/yocto-autobuilder eflanagan/yocto-autobuilder-dev
http://git.yoctoproject.org/cgit.cgi//log/?h=eflanagan/yocto-autobuilder-dev
Beth Flanagan (10):
yoctogit.py: Use self.srcdir, not local srcdir construct. yoctogit.py: movecopy should only move 'build' yoctogit.py: DRYing up some code
yoctogit.py: Fix timeouts.
yoctogit.py: Set workdir for the class
yoctogit.py: Set workdir within dovccmd yoctogit.py: change _srccopy fromdir to it's own construction PublishArtifacts.py: Exclude genericx86-64 pre-1.5 SetDest.py: Use new UI variables.
yoctogit.py: sync on mv
Elizabeth Flanagan (9):
yoctogit.py: Fix cp construction from source to build BuildImages.py: core-image-basic name change support yoctogit.py: movecopy doesn't care if it's updateable yoctogit.py: movecopy needs an incremental callback CheckOutLayers.py: Remove unset Props yoctogit.py: pathExists returns a defered value. CheckOutLayers.py: Remove srcdir yoctogit.py: Stop passing in srcdir. yoctogit.py: Check for mirror on incremental builds
Richard Purdie (2):
yocto-start-autobuilder: Make sure janitor logging goes to a log file
buildslave-janitor: Add in try/except handling to make calls robust
against failure
Stefan Stanacar (1):
buildsets: add a new buildset to run builds on target
bin/buildslave-janitor | 15 +- .../nightly-qa-targetbuilds.conf | 26 +++ .../site-packages/autobuilder/BuildSet.py | 2 - .../autobuilder/buildsteps/BuildImages.py | 9 ++ .../autobuilder/buildsteps/CheckOutLayers.py | 9 +- .../autobuilder/buildsteps/PublishArtifacts.py | 15 +- .../autobuilder/buildsteps/SetDest.py | 7 +- .../buildbot/steps/source/yoctogit.py | 177 ++++++++++----------- yocto-start-autobuilder | 2 +- 9 files changed, 145 insertions(+), 117 deletions(-) create mode 100644 buildset-config.master/nightly-qa-targetbuilds.conf
Signature · 1 line
1.8.1.2
PATCH 1/22 buildsets: add a new buildset to run builds on target
From: Stefan Stanacar <[email protected]>
Builds core-image-sato-sdk for each qemu arch and runs tests building sudoku/cvs/iptables.
These do take a while, so they shouldn't be added to nightly!
--- .../nightly-qa-targetbuilds.conf | 26 ++++++++++++++++++++++ 1 file changed, 26 insertions(+) create mode 100644 buildset-config.master/nightly-qa-targetbuilds.conf
buildset-config.master/nightly-qa-targetbuilds.conf new file +26 -0
@@ -0,0 +1,26 @@ +[nightly-qa-targetbuilds] +builders: 'builder1' +repos: [{'poky': + {'repourl':'git://git.yoctoproject.org/poky', + 'layerversion':{'core':'meta', 'yoctobsp':'meta-yocto-bsp'}, + 'branch':'master'}}] +steps: [{'SetDest':{}}, + {'CheckOutLayers': {}}, + {'RunPreamble': {}}, + {'GetDistroVersion' : {'distro': 'poky'}}, + {'CreateAutoConf': {'machine': 'qemux86-64', 'SDKMACHINE' : 'x86_64', 'distro': 'poky'}}, + {'CreateBBLayersConf': {'buildprovider' : 'yocto'}}, + {'BuildImages': {'images': 'core-image-sato-sdk'}}, + {'RunSanityTests': {'images': 'core-image-sato-sdk', 'suites' : 'ping ssh buildsudoku buildcvs buildiptables'}}, + {'CreateAutoConf': {'machine': 'qemux86', 'SDKMACHINE' : 'x86_64', 'distro': 'poky'}}, + {'BuildImages': {'images': 'core-image-sato-sdk'}}, + {'RunSanityTests': {'images': 'core-image-sato-sdk', 'suites' : 'ping ssh buildsudoku buildcvs buildiptables'}}, + {'CreateAutoConf': {'machine': 'qemuarm', 'SDKMACHINE' : 'x86_64', 'distro': 'poky'}}, + {'BuildImages': {'images': 'core-image-sato-sdk'}}, + {'RunSanityTests': {'images': 'core-image-sato-sdk', 'suites' : 'ping ssh buildsudoku buildcvs buildiptables'}}, + {'CreateAutoConf': {'machine': 'qemuppc', 'SDKMACHINE' : 'x86_64', 'distro': 'poky'}}, + {'BuildImages': {'images': 'core-image-sato-sdk'}}, + {'RunSanityTests': {'images': 'core-image-sato-sdk', 'suites' : 'ping ssh buildsudoku buildcvs buildiptables'}}, + {'CreateAutoConf': {'machine': 'qemumips', 'SDKMACHINE' : 'x86_64', 'distro': 'poky'}}, + {'BuildImages': {'images': 'core-image-sato-sdk'}}, + {'RunSanityTests': {'images': 'core-image-sato-sdk', 'suites' : 'ping ssh buildsudoku buildcvs buildiptables'}}]
Signature · 1 line
1.8.1.2
PATCH 2/22 yoctogit.py: Fix cp construction from source to build
In reply to Elizabeth FlanaganThis commit removes the mkdir from movecopy as it is not needed. It also changes the workdir in barecopy and movecopy to the source dir, as we can't alwas be sure that the destination dir even exists.
This also ensures that we are in fact constructing our cp string correctly, by striping any trailing "/" from self.repourl
As repourl is used to construct the cp from source to build, that trailing slash, if it exists, causes the directory structure to not be correct (e.g. cp -R ./foo/ ./bar and cp -R ./foo ./bar don't do the same thing, obviously)
--- .../buildbot/steps/source/yoctogit.py | 35 +++++++--------------- 1 file changed, 11 insertions(+), 24 deletions(-)
lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py +11 -24
@@ -121,7 +121,7 @@ class YoctoGit(Source): self.branch = branch self.method = method self.prog = progress - self.repourl = repourl + self.repourl = repourl.rstrip('/') self.retryFetch = retryFetch self.submodules = submodules self.shallow = shallow
@@ -255,13 +255,11 @@ class YoctoGit(Source): if "poky" in self.layername or \ "oecore" in self.layername or \ "eclipse" in self.layername: - todir = "." - self.workdir = "build" + todir = "build" + self.workdir = 'source/'+self.repourl else: - todir = "./" + self.layername - self.workdir = "build/" + self.layername - - + todir = "build/" + self.layername + self.workdir = 'source/'+self.repourl d.addCallback(lambda _: self.incremental()) def copy(_): cmd = buildstep.RemoteCommand('cpdir',
@@ -278,7 +276,7 @@ class YoctoGit(Source): d.addCallback(lambda _: evaluateCommand(cmd)) return d d.addCallback(copy) - + def resetWorkdir(_): if "poky" in self.layername or \ "oecore" in self.layername or \
@@ -316,26 +314,15 @@ class YoctoGit(Source): log.msg("Source step failed while running command %s" % cmd) raise buildstep.BuildStepFailed() d.addCallback(lambda _: evaluateCommand(cmd)) - - if self.layername == "poky" or self.layername == "oecore": + if "poky" in self.layername or \ + "oecore" in self.layername or \ + "eclipse" in self.layername: todir = "build" + self.workdir = 'source/'+self.repourl.rstrip('/') else: todir = "build/" + self.layername + self.workdir = 'source/'+self.repourl - def mkdir(_): - cmd = buildstep.RemoteCommand('mkdir', {'dir': 'build', - 'logEnviron': self.logEnviron, - 'timeout': self.timeout,}) - cmd.useLog(self.stdio_log, False) - d = self.runCommand(cmd) - def evaluateCommand(cmd): - if cmd.didFail(): - log.msg("Source step failed while running command %s" % cmd) - raise buildstep.BuildStepFailed() - d.addCallback(lambda _: evaluateCommand(cmd)) - return d - d.addCallback(mkdir) - d.addCallback(lambda _: self.incremental()) def copy(_): cmd = buildstep.RemoteCommand('cpdir', {'fromdir': srcdir,
Signature · 1 line
1.8.1.2
Re: [AB PATCH 02/22] yoctogit.py: Fix cp construction from source to build
lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py class YoctoGit(Source): · line 319
Quoted 3 lines · Elizabeth Flanagan
lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py +3
-0+ if "poky" in self.layername or \ + "oecore" in self.layername or \ + "eclipse" in self.layername:
Do you actually need to do substring comparisons (in vs is) there? I suspect not, so I prefer this idiom as it involves less repetition:
if self.layername in ('poky', 'oecore', 'eclipse')
Ross
Re: [AB PATCH 02/22] yoctogit.py: Fix cp construction from source to build
Quoted 9 lines · Ross
lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py class YoctoGit(Source): · line 319
Quoted 3 lines · Elizabeth Flanagan
lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py +3
-0+ if "poky" in self.layername or \ + "oecore" in self.layername or \ + "eclipse" in self.layername:Do you actually need to do substring comparisons (in vs is) there? I suspect not, so I prefer this idiom as it involves less repetition:
if self.layername in ('poky', 'oecore', 'eclipse')
In this case, yeah, we need a substring comparision, as we need to support the layer being named eclipse-poky-<branch-name> and poky/poky-contrib. We need to eventually figure out a better, more standardized way of indicating the base layer, but until we do and propagate that change out (as it changes configs), doing it this way both supports the way people have come to understand what "base" is.
-b
Quoted 12 lines
Ross
---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.
Signature · 3 lines
Elizabeth Flanagan Yocto Project Build and Release
PATCH 3/22 BuildImages.py: core-image-basic name change support
In reply to Elizabeth FlanaganSupports http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=b7f1cca517bbd4191828c6bae32e0c5041f1ff19
--- .../site-packages/autobuilder/buildsteps/BuildImages.py | 9 +++++++++ 1 file changed, 9 insertions(+)
lib/python2.7/site-packages/autobuilder/buildsteps/BuildImages.py +9 -0
@@ -34,7 +34,16 @@ class BuildImages(ShellCommand): def start(self): self.layerversion_yoctobsp = self.getProperty("layerversion_yoctobsp") + self.layerversion_core = self.getProperty("layerversion_core") self.machine = self.getProperty("MACHINE") + # core-image-basic rename + # See: http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=b7f1cca517bbd4191828c6bae32e0c5041f1ff19 + # I hate making people change their configs, so support both. + if self.layerversion_core < "4": + self.images=self.images.replace("core-image-full-cmdline", "core-image-basic") + elif self.layerversion_core > "3": + self.images=self.images.replace("core-image-basic", "core-image-full-cmdline") + if self.layerversion_yoctobsp is not None and int(self.layerversion_yoctobsp) < 2 and self.machine is not None and self.machine == "genericx86-64": self.command = "echo 'Skipping Step.'" else:
Signature · 1 line
1.8.1.2
Re: [AB PATCH 03/22] BuildImages.py: core-image-basic name change support
lib/python2.7/site-packages/autobuilder/buildsteps/BuildImages.py class BuildImages(ShellCommand): · line 45
Quoted 4 lines · Elizabeth Flanagan
lib/python2.7/site-packages/autobuilder/buildsteps/BuildImages.py +4
-0+ if self.layerversion_core < "4": + self.images=self.images.replace("core-image-full-cmdline", "core-image-basic") + elif self.layerversion_core > "3": + self.images=self.images.replace("core-image-basic", "core-image-full-cmdline")
I had to read this logic carefully to make sure I wasn't missing any nuances, why can't the elif be a simple "else"?
Ross
PATCH 4/22 yoctogit.py: movecopy doesn't care if it's updateable
In reply to Elizabeth Flanaganmovecopy really shouldn't care if it's updatable or not. It should have an incremental callback in it to take care of that (it doesn't though, which is part of the problem!)
--- .../buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py +1 -1
@@ -184,7 +184,7 @@ class YoctoGit(Source): elif self.method == 'barecopy': yield self.barecopy() return - elif updatable and self.method == 'movecopy': + elif self.method == 'movecopy': yield self.movecopy() return
Signature · 1 line
1.8.1.2
PATCH 5/22 yoctogit.py: movecopy needs an incremental callback
In reply to Elizabeth FlanaganWe need to do an incremental before we copy or else, we're essentially copying nothing.
--- .../buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py | 1 + 1 file changed, 1 insertion(+)
lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py +1 -0
@@ -322,6 +322,7 @@ class YoctoGit(Source): else: todir = "build/" + self.layername self.workdir = 'source/'+self.repourl + d.addCallback(lambda _: self.incremental()) def copy(_): cmd = buildstep.RemoteCommand('cpdir',
Signature · 1 line
1.8.1.2
Re: [AB PATCH 05/22] yoctogit.py: movecopy needs an incremental callback
Quoted 21 lines · Elizabeth Flanagan
We need to do an incremental before we copy or else, we're essentially copying nothing.
Signed-off-by Elizabeth Flanagan <[email protected]>--- .../buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py | 1 + 1 file changed, 1 insertion(+)lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py +1
-1@@ -322,6 +322,7 @@ class YoctoGit(Source): else: todir = "build/" + self.layername self.workdir = 'source/'+self.repourl + d.addCallback(lambda _: self.incremental()) def copy(_): cmd = buildstep.RemoteCommand('cpdir', --
Quoted 1 line
1.8.1.2
This is fixing a bug in 2/22 so should be squashed into that commit.
Ross
PATCH 6/22 CheckOutLayers.py: Remove unset Props
In reply to Elizabeth FlanaganWe no longer set these trigger* props, so let's remove them.
--- lib/python2.7/site-packages/autobuilder/buildsteps/CheckOutLayers.py | 4 ---- 1 file changed, 4 deletions(-)
lib/python2.7/site-packages/autobuilder/buildsteps/CheckOutLayers.py +0 -4
@@ -82,10 +82,6 @@ class CheckOutLayers(YoctoGit): self.getProperty('commit_resolvedhead_'+self.layername) is not None: log.msg("Using Resolved HEAD") self.commit = self.getProperty('commit_resolvedhead_'+self.layername) - elif self.getProperty('commit_trigger_'+self.layername+"_"+self.scheduler): - self.commit = self.getProperty('commit_trigger_'+self.layername+"_"+self.scheduler) - elif self.getProperty('commit_triggerer_'+self.layername+"_"+self.scheduler) : - self.commit = self.getProperty('commit_triggerer_'+self.layername+"_"+self.scheduler) elif self.getProperty('commit_'+self.layername): self.commit = self.getProperty('commit_'+self.layername) else:
Signature · 1 line
1.8.1.2
PATCH 7/22 yoctogit.py: pathExists returns a defered value.
In reply to Elizabeth FlanaganWe can't use pathExists this way to figure out if the path actually exists as it will always evalutate to true as it returns a generator object and not T/F.
In order to actually figure out if the mirrordir + repourl exists, we need to call a function that returns the deferred generator object outside of any of fetch/movecopy/barecopy functions (which also return defereds and you can't yield a defered in a defered)
So, yield the generator object at the beginning, and save everyone the headache.
--- .../buildbot/steps/source/yoctogit.py | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-)
lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py +11 -4
@@ -177,6 +177,7 @@ class YoctoGit(Source): @defer.inlineCallbacks def full(self): updatable = yield self._sourcedirIsUpdatable() + self.mirrorexists = yield self._mirrorExists() if self.method == 'clobber': yield self.clobber()
@@ -248,9 +249,9 @@ class YoctoGit(Source): d.addCallback(lambda _: evaluateCommand(cmd)) if self.mirrordir: mirror = self.mirrordir + "/" + self.repourl.replace("git://", "") + "/" - if self.pathExists(mirror): + if self.mirrorexists: srcdir=mirror - else: + else: srcdir='source/'+self.repourl if "poky" in self.layername or \ "oecore" in self.layername or \
@@ -493,8 +494,10 @@ class YoctoGit(Source): mirror = None if self.mirrordir: mirror = self.mirrordir + "/" + self.repourl.replace("git://", "") + "/" - if self.pathExists(mirror): - command = ['clone'] + ["-s", "-n"] + [mirror, '.'] + if self.mirrorexists: + command = ['clone'] + ["-s", "-n"] + [mirror, '.'] + else: + command = ['clone'] + ["-s", "-n"] + [self.repourl, '.'] #Fix references if self.prog:
@@ -560,6 +563,10 @@ class YoctoGit(Source): def _sourcedirIsUpdatable(self): return self.pathExists(self.build.path_module.join(self.workdir, '.git')) + def _mirrorExists(self): + mirror = self.mirrordir + "/" + self.repourl.replace("git://", "") + "/" + return self.pathExists(self.build.path_module.join(mirror, '.git')) + def _updateSubmodule(self, _): if self.submodules: return self._dovccmd(['submodule', 'update', '--recursive'])
Signature · 1 line
1.8.1.2
PATCH 8/22 CheckOutLayers.py: Remove srcdir
In reply to Elizabeth FlanaganWe're removing srcdir from yoctogit.py. It causes way too many issues.
--- lib/python2.7/site-packages/autobuilder/BuildSet.py | 2 -- lib/python2.7/site-packages/autobuilder/buildsteps/CheckOutLayers.py | 5 ++--- 2 files changed, 2 insertions(+), 5 deletions(-)
lib/python2.7/site-packages/autobuilder/BuildSet.py +0 -2
@@ -73,7 +73,6 @@ class BuildSet(): mirrordir=None method = 'clobber' mode='full' - srcdir = 'source' if str(os.environ.get('OPTIMIZED_GIT_CLONE')) == "True": if 'poky' in layername or 'oecore' in layername or 'eclipse' in layername: method='movecopy'
@@ -93,7 +92,6 @@ class BuildSet(): func = getattr(m, step) factoryFN(func(locals()['f'+name], layername=layername, - srcdir=srcdir, mode=mode, scheduler=name, method=method,
lib/python2.7/site-packages/autobuilder/buildsteps/CheckOutLayers.py +2 -3
@@ -35,12 +35,11 @@ from buildbot.process import buildstep class CheckOutLayers(YoctoGit): def __init__(self, factory, scheduler=None, layername=None, mode='full', storedir=None, mirrordir=None, - method='clobber', submodules=False, shallow=False, srcdir='source', workdir='build', + method='clobber', submodules=False, shallow=False, workdir='build', timeout=100000, progress=True, retryFetch=True, clobberOnFailure=False, getDescription=True, argdict=None, **kwargs): self.scheduler = scheduler - self.srcdir=srcdir self.workdir=workdir self.branch = "" self.commit = ""
@@ -63,7 +62,7 @@ class CheckOutLayers(YoctoGit): for k, v in argdict.iteritems(): setattr(self, k, v) YoctoGit.__init__(self, repourl=self.repourl, branch=self.branch, mode=self.mode, - method=self.method, storedir=self.storedir, mirrordir=mirrordir, submodules=self.submodules, srcdir=self.srcdir+"/"+self.repourl, + method=self.method, storedir=self.storedir, mirrordir=mirrordir, submodules=self.submodules, workdir=self.workdir, shallow=self.shallow, progress=self.progress, retryFetch=self.retryFetch, clobberOnFailure=self.clobberOnFailure, getDescription=self.getDescription, layername=self.layername, **kwargs)
Signature · 1 line
1.8.1.2
PATCH 9/22 yoctogit.py: Stop passing in srcdir.
In reply to Elizabeth FlanaganDon't pass in srcdir. Set it depending on mirrordir/method as a class level property.
--- .../buildbot/steps/source/yoctogit.py | 20 +++++++++++++++----- 1 file changed, 15 insertions(+), 5 deletions(-)
lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py +15 -5
@@ -63,7 +63,7 @@ class YoctoGit(Source): def __init__(self, repourl=None, branch='HEAD', mode='incremental', method=None, submodules=False, shallow=False, progress=False, retryFetch=False, clobberOnFailure=False, getDescription=False, - workdir='build', layername=None, srcdir='sources', storedir="/tmp/junk", mirrordir=None, + workdir='build', layername=None, storedir="/tmp/junk", mirrordir=None, config=None, **kwargs): """ @type repourl: string
@@ -98,9 +98,6 @@ class YoctoGit(Source): @type getDescription: boolean or dict @param getDescription: Use 'git describe' to describe the fetched revision - @type srcdir: string - @param srcdir: Used to pass in where git clones should occur - @type workdir: string @param workdir: Used to indicate the destination of the copy of srcdir
@@ -125,7 +122,6 @@ class YoctoGit(Source): self.retryFetch = retryFetch self.submodules = submodules self.shallow = shallow - self.srcdir = srcdir self.workdir = workdir self.storedir = storedir self.mirrordir = mirrordir
@@ -135,6 +131,20 @@ class YoctoGit(Source): self.mode = mode self.getDescription = getDescription self.config = config + # We no longer pass srcdir in. There are way too many ways to hang + # yourself by doing that. + if self.mirrordir: + if self.method in ['barecopy', 'movecopy']: + self.srcdir='source/'+self.repourl + else: + if "poky" in self.layername or \ + "oecore" in self.layername or \ + "eclipse" in self.layername: + self.srcdir = "build" + else: + self.srcdir = 'build/'+self.layername + else: + self.srcdir = 'source' Source.__init__(self, **kwargs) if self.mode not in ['incremental', 'full']:
Signature · 1 line
1.8.1.2
PATCH 10/22 yoctogit.py: Check for mirror on incremental builds
In reply to Elizabeth FlanaganWe're assuming people are getting to incremental from full. This is a bad assumption. So, check for mirror in both.
--- .../buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py | 1 + 1 file changed, 1 insertion(+)
lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py +1 -0
@@ -212,6 +212,7 @@ class YoctoGit(Source): @defer.inlineCallbacks def incremental(self): updatable = yield self._sourcedirIsUpdatable() + self.mirrorexists = yield self._mirrorExists() # if not updateable, do a full checkout if not updatable:
Signature · 1 line
1.8.1.2
PATCH 11/22 yoctogit.py: Use self.srcdir, not local srcdir construct.
In reply to Elizabeth FlanaganFrom: Beth Flanagan <[email protected]>
Since we're now constructing the objects srcdir property, there is no need to use a local srcdir prop.
--- .../buildbot/steps/source/yoctogit.py | 12 ++---------- 1 file changed, 2 insertions(+), 10 deletions(-)
lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py +2 -10
@@ -258,12 +258,6 @@ class YoctoGit(Source): log.msg("Source step failed while running command %s" % cmd) raise buildstep.BuildStepFailed() d.addCallback(lambda _: evaluateCommand(cmd)) - if self.mirrordir: - mirror = self.mirrordir + "/" + self.repourl.replace("git://", "") + "/" - if self.mirrorexists: - srcdir=mirror - else: - srcdir='source/'+self.repourl if "poky" in self.layername or \ "oecore" in self.layername or \ "eclipse" in self.layername:
@@ -275,7 +269,7 @@ class YoctoGit(Source): d.addCallback(lambda _: self.incremental()) def copy(_): cmd = buildstep.RemoteCommand('cpdir', - {'fromdir': srcdir, + {'fromdir': self.srcdir, 'todir': todir, 'logEnviron': self.logEnviron, 'timeout': self.timeout,})
@@ -311,8 +305,6 @@ class YoctoGit(Source): # method probably won't work correctly crossing filesystems, so utilize # movecopy with caution. - srcdir='source/'+self.repourl - cmd = buildstep.RemoteCommand('mv', {'fromdir': self.workdir, 'todir': self.storedir + '/' + self.getProperty("DEST") + '/' + str(int(time.time())) + '-' +
@@ -338,7 +330,7 @@ class YoctoGit(Source): def copy(_): cmd = buildstep.RemoteCommand('cpdir', - {'fromdir': srcdir, + {'fromdir': self.srcdir, 'todir': todir, 'logEnviron': self.logEnviron, 'timeout': self.timeout,})
Signature · 1 line
1.8.1.2
PATCH 12/22 yoctogit.py: movecopy should only move 'build'
In reply to Elizabeth FlanaganFrom: Beth Flanagan <[email protected]>
Pin what movecopy moves to be 'build' and not workdir. This keeps people from doing stupid stuff like passing in a weird workdir.
--- .../buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-)
lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py +1 -3
@@ -305,7 +305,7 @@ class YoctoGit(Source): # method probably won't work correctly crossing filesystems, so utilize # movecopy with caution. - cmd = buildstep.RemoteCommand('mv', {'fromdir': self.workdir, + cmd = buildstep.RemoteCommand('mv', {'fromdir': 'build', 'todir': self.storedir + '/' + self.getProperty("DEST") + '/' + str(int(time.time())) + '-' + str(random.randrange(100, 100000, 2)) +
@@ -322,10 +322,8 @@ class YoctoGit(Source): "oecore" in self.layername or \ "eclipse" in self.layername: todir = "build" - self.workdir = 'source/'+self.repourl.rstrip('/') else: todir = "build/" + self.layername - self.workdir = 'source/'+self.repourl d.addCallback(lambda _: self.incremental()) def copy(_):
Signature · 1 line
1.8.1.2
PATCH 13/22 yoctogit.py: DRYing up some code
In reply to Elizabeth FlanaganFrom: Beth Flanagan <[email protected]>
Remove duplicate code from movecopy and barecopy and add it to a new callback _srccopy.
--- .../buildbot/steps/source/yoctogit.py | 73 +++++++++------------- 1 file changed, 30 insertions(+), 43 deletions(-)
lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py +30 -43
@@ -194,9 +194,11 @@ class YoctoGit(Source): return elif self.method == 'barecopy': yield self.barecopy() + yield self._srccopy() return elif self.method == 'movecopy': yield self.movecopy() + yield self._srccopy() return if not updatable:
@@ -267,32 +269,6 @@ class YoctoGit(Source): todir = "build/" + self.layername self.workdir = 'source/'+self.repourl d.addCallback(lambda _: self.incremental()) - def copy(_): - cmd = buildstep.RemoteCommand('cpdir', - {'fromdir': self.srcdir, - 'todir': todir, - 'logEnviron': self.logEnviron, - 'timeout': self.timeout,}) - cmd.useLog(self.stdio_log, False) - d = self.runCommand(cmd) - def evaluateCommand(cmd): - if cmd.didFail(): - log.msg("Source step failed while running command %s" % cmd) - raise buildstep.BuildStepFailed() - d.addCallback(lambda _: evaluateCommand(cmd)) - return d - d.addCallback(copy) - - def resetWorkdir(_): - if "poky" in self.layername or \ - "oecore" in self.layername or \ - "eclipse" in self.layername: - self.workdir = 'build' - else: - self.workdir = "build/" + self.layername - return 0 - - d.addCallback(resetWorkdir) return d def movecopy(self):
@@ -325,22 +301,35 @@ class YoctoGit(Source): else: todir = "build/" + self.layername d.addCallback(lambda _: self.incremental()) + return d - def copy(_): - cmd = buildstep.RemoteCommand('cpdir', - {'fromdir': self.srcdir, - 'todir': todir, - 'logEnviron': self.logEnviron, - 'timeout': self.timeout,}) - cmd.useLog(self.stdio_log, False) - d = self.runCommand(cmd) - def evaluateCommand(cmd): - if cmd.didFail(): - log.msg("Source step failed while running command %s" % cmd) - raise buildstep.BuildStepFailed() - d.addCallback(lambda _: evaluateCommand(cmd)) - return d - d.addCallback(copy) + # Buildbot's standard copy has been removed from yoctogit. + # We use _srccopy from either movecopy or barecopy as a callback + # this uses self.srcdir to copy to the build dir. + + def _srccopy(self): + if "poky" in self.layername or \ + "oecore" in self.layername or \ + "eclipse" in self.layername: + todir = "build" + else: + todir = "build/" + self.layername + + cmd = buildstep.RemoteCommand('cpdir', + {'fromdir': self.srcdir, + 'todir': todir, + 'logEnviron': self.logEnviron, + 'timeout': self.timeout,}) + cmd.useLog(self.stdio_log, False) + d = self.runCommand(cmd) + + def evaluateCommand(cmd): + if cmd.didFail(): + log.msg("Source step failed while running command %s" % cmd) + raise buildstep.BuildStepFailed() + + d.addCallback(lambda _: evaluateCommand(cmd)) + return d def resetWorkdir(_): if "poky" in self.layername or \
@@ -353,8 +342,6 @@ class YoctoGit(Source): d.addCallback(resetWorkdir) return d - # Buildbot's standard copy has been removed from yoctogit. - def finish(self, res): d = defer.succeed(res) def _gotResults(results):
Signature · 1 line
1.8.1.2
PATCH 14/22 yoctogit.py: Fix timeouts.
In reply to Elizabeth FlanaganFrom: Beth Flanagan <[email protected]>
We should really never need this big of a timeout, but poky-contrib (despite my continual kvetching) can be a large clone, or an autobuilder can have a large build directory and the mv across filesystems *which you should never do!* can go slow.
So let's set this sufficently high enough that we don't have to have failures from a timeout, but sufficently low enough that if things go south we don't wait forever to know about it.
--- .../buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py +3 -1
@@ -131,6 +131,7 @@ class YoctoGit(Source): self.mode = mode self.getDescription = getDescription self.config = config + self.timeout = 10*60 # We no longer pass srcdir in. There are way too many ways to hang # yourself by doing that. if self.mirrordir:
@@ -286,7 +287,8 @@ class YoctoGit(Source): str(int(time.time())) + '-' + str(random.randrange(100, 100000, 2)) + self.layername, - 'logEnviron':self.logEnviron,}) + 'logEnviron':self.logEnviron, + 'timeout': self.timeout,}) cmd.useLog(self.stdio_log, False) d = self.runCommand(cmd) def evaluateCommand(cmd):
Signature · 1 line
1.8.1.2
PATCH 15/22 yoctogit.py: Set workdir for the class
In reply to Elizabeth FlanaganFrom: Beth Flanagan <[email protected]>
Stop utilizing resetWorkdir, it just confuses things and is really not needed.
Just set self.workdir for the class. This ignores the self.workdir passed in (which I intend on removing soon.)
--- .../buildbot/steps/source/yoctogit.py | 31 ++++------------------ 1 file changed, 5 insertions(+), 26 deletions(-)
lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py +5 -26
@@ -146,6 +146,11 @@ class YoctoGit(Source): self.srcdir = 'build/'+self.layername else: self.srcdir = 'source' + if self.method in ['barecopy', 'movecopy']: + self.workdir='source/'+self.repourl + else: + self.workdir = 'build' + Source.__init__(self, **kwargs) if self.mode not in ['incremental', 'full']:
@@ -261,14 +266,6 @@ class YoctoGit(Source): log.msg("Source step failed while running command %s" % cmd) raise buildstep.BuildStepFailed() d.addCallback(lambda _: evaluateCommand(cmd)) - if "poky" in self.layername or \ - "oecore" in self.layername or \ - "eclipse" in self.layername: - todir = "build" - self.workdir = 'source/'+self.repourl - else: - todir = "build/" + self.layername - self.workdir = 'source/'+self.repourl d.addCallback(lambda _: self.incremental()) return d
@@ -296,12 +293,6 @@ class YoctoGit(Source): log.msg("Source step failed while running command %s" % cmd) raise buildstep.BuildStepFailed() d.addCallback(lambda _: evaluateCommand(cmd)) - if "poky" in self.layername or \ - "oecore" in self.layername or \ - "eclipse" in self.layername: - todir = "build" - else: - todir = "build/" + self.layername d.addCallback(lambda _: self.incremental()) return d
@@ -329,21 +320,9 @@ class YoctoGit(Source): if cmd.didFail(): log.msg("Source step failed while running command %s" % cmd) raise buildstep.BuildStepFailed() - d.addCallback(lambda _: evaluateCommand(cmd)) return d - def resetWorkdir(_): - if "poky" in self.layername or \ - "oecore" in self.layername or \ - "eclipse" in self.layername: - self.workdir = 'build.' - else: - self.workdir = "build/" + self.layername - return 0 - d.addCallback(resetWorkdir) - return d - def finish(self, res): d = defer.succeed(res) def _gotResults(results):
Signature · 1 line
1.8.1.2
PATCH 16/22 yoctogit.py: Set workdir within dovccmd
In reply to Elizabeth FlanaganFrom: Beth Flanagan <[email protected]>
Ensure we're in the correct place by setting workdir in dovccmd.
--- .../buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py | 4 ++++ 1 file changed, 4 insertions(+)
lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py +4 -0
@@ -376,6 +376,10 @@ class YoctoGit(Source): full_command.append('-c') full_command.append('%s=%s' % (name, value)) full_command.extend(command) + if self.method in ['barecopy', 'movecopy']: + self.workdir='source/'+self.repourl + else: + self.workdir = 'build' cmd = buildstep.RemoteShellCommand(self.workdir, full_command, env=self.env,
Signature · 1 line
1.8.1.2
PATCH 17/22 yoctogit.py: change _srccopy fromdir to it's own construction
In reply to Elizabeth FlanaganFrom: Beth Flanagan <[email protected]>
fromdir was using self.srcdir, but lets be a bit safe and use a variable we construct in the method.
--- .../buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py +2 -1
@@ -307,9 +307,10 @@ class YoctoGit(Source): todir = "build" else: todir = "build/" + self.layername + fromdir='source/'+self.repourl cmd = buildstep.RemoteCommand('cpdir', - {'fromdir': self.srcdir, + {'fromdir': fromdir, 'todir': todir, 'logEnviron': self.logEnviron, 'timeout': self.timeout,})
Signature · 1 line
1.8.1.2
PATCH 18/22 PublishArtifacts.py: Exclude genericx86-64 pre-1.5
In reply to Elizabeth FlanaganFrom: Beth Flanagan <[email protected]>
We don't build it so we shouldn't publish it.
--- .../autobuilder/buildsteps/PublishArtifacts.py | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-)
lib/python2.7/site-packages/autobuilder/buildsteps/PublishArtifacts.py +10 -5
@@ -169,11 +169,16 @@ class PublishArtifacts(ShellCommand): command=command+"echo 'Skipping copy of " + artifact + ".'" else: artifact_name, deploy_image_dir = self.getDeployNames(artifact, buildername) - - command=command+"mkdir -p " + DEST + "/"+ MACHINE_PUBLISH_DIR +"/" + artifact_name + ";" - command=command+"cp -R --no-dereference --preserve=links " + \ - deploy_image_dir + \ - "/*"+artifact+"* " + DEST + "/" + MACHINE_PUBLISH_DIR +"/" + artifact_name + ";" + self.layerversion_yoctobsp = self.getProperty("layerversion_yoctobsp") + if self.layerversion_yoctobsp is not None and \ + int(self.layerversion_yoctobsp) < 2 and \ + "genericx86-64" in artifacts: + command = command+"echo 'Skipping copy of genericx86-64.'; " + else: + command=command+"mkdir -p " + DEST + "/"+ MACHINE_PUBLISH_DIR +"/" + artifact_name + ";" + command=command+"cp -R --no-dereference --preserve=links " + \ + deploy_image_dir + \ + "/*"+artifact+"* " + DEST + "/" + MACHINE_PUBLISH_DIR +"/" + artifact_name + ";" self.command = command
Signature · 1 line
1.8.1.2
PATCH 19/22 SetDest.py: Use new UI variables.
In reply to Elizabeth FlanaganFrom: Beth Flanagan <[email protected]>
How we name releases is a lot more complex than the old UI could handle. This alligns us a bit more with what we actually do.
--- lib/python2.7/site-packages/autobuilder/buildsteps/SetDest.py | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)
lib/python2.7/site-packages/autobuilder/buildsteps/SetDest.py +5 -2
@@ -49,8 +49,11 @@ class SetDest(ShellCommand): else: buildername=self.getProperty("buildername") self.workdir=os.path.join(self.slaveworkdir, buildername) - if str(self.getProperty("custom_release_me_" + buildername)) == "True": - rel_name = self.getProperty("custom_release_name_" + buildername) + if str(self.getProperty("custom_release_me")) == "True": + poky_name = self.getProperty("custom_poky_name") + poky_number = self.getProperty("custom_poky_number") + yocto_number = self.getProperty("custom_yocto_number") + rel_name = 'yocto-'+ yocto_number if not rel_name or rel_name == "": rel_name == self.getProperty(commit-description) DEST=os.path.normpath(os.path.join(
Signature · 1 line
1.8.1.2
PATCH 20/22 yocto-start-autobuilder: Make sure janitor logging goes to a log file
In reply to Elizabeth FlanaganFrom: Richard Purdie <[email protected]>
Currently logging from the janitor gets lost. Ensure it gets to a log file instead including anything to stderr.
--- yocto-start-autobuilder | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
yocto-start-autobuilder +1 -1
@@ -79,7 +79,7 @@ if os.path.isfile(os.path.join(AB_BASE, ".setupdone")): os.chdir(AB_BASE) if os.environ["OPTIMIZED_GIT_CLONE"] == "True": os.chdir(AB_BASE) - subprocess.Popen("python bin/buildslave-janitor " + os.path.join(AB_BASE, "config/autobuilder.conf"), + subprocess.Popen("python bin/buildslave-janitor " + os.path.join(AB_BASE, "config/autobuilder.conf") + ">> yocto-slave/janitor.log 2>&1", shell=True, stdin=None, stdout=None, stderr=None, close_fds=True)
Signature · 1 line
1.8.1.2
PATCH 21/22 buildslave-janitor: Add in try/except handling to make calls robust against failure
In reply to Elizabeth FlanaganFrom: Richard Purdie <[email protected]>
There is a suspicion that the command can fail causing the janitor trash removal process to exit. This adds in some exception handling to deal with this. It include a timeout to stop it entering nasty loops.
This combined with the previous patch should let us figure out what error conditions its hitting.
--- bin/buildslave-janitor | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-)
bin/buildslave-janitor +10 -5
@@ -66,11 +66,16 @@ def trash_processor(trashdir): print("Not prepared to use a trashdir of /") return while True: - files = os.listdir(trashdir) - if files: - os.system("ionice -c 3 rm %s -rf" % trashdir) - else: - time.sleep(30*60) # 30 minutes + try: + files = os.listdir(trashdir) + if files: + os.system("ionice -c 3 rm %s -rf" % trashdir) + else: + time.sleep(30*60) # 30 minutes + except Exception as e: + print("Exception %s in trash cleaner" % str(e)) + time.sleep(60) # 1 minute timeout to prevent crazy looping + pass return def mirror_processor(mirrordir):
Signature · 1 line
1.8.1.2
PATCH 22/22 yoctogit.py: sync on mv
In reply to Elizabeth FlanaganFrom: Beth Flanagan <[email protected]>
On systems with very high load sometimes the system still thinks the build dir is there. I've only seen this once, but a sync here ensures that the system knows that build has been moved.
--- .../buildbot/steps/source/yoctogit.py | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+)
lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py +18 -0
@@ -204,6 +204,7 @@ class YoctoGit(Source): return elif self.method == 'movecopy': yield self.movecopy() + yield self._sync() yield self._srccopy() return
@@ -269,6 +270,23 @@ class YoctoGit(Source): d.addCallback(lambda _: self.incremental()) return d + def _sync(self): + cmd = buildstep.RemoteShellCommand(self.workdir, + 'sync', + env=self.env, + logEnviron=self.logEnviron, + timeout=self.timeout) + cmd.useLog(self.stdio_log, False) + log.msg("syncing filesystem") + d = self.runCommand(cmd) + def evaluateCommand(cmd): + if cmd.didFail(): + log.msg("Source step failed while running command %s" % cmd) + raise buildstep.BuildStepFailed() + d.addCallback(lambda _: evaluateCommand(cmd)) + return d + + def movecopy(self): # This relies on an external method of cleaning up the todir, like a cron job # it can be rather disk space intensive.
Signature · 1 line
1.8.1.2
Re: [AB PATCH 22/22] yoctogit.py: sync on mv
Quoted 3 lines · Elizabeth Flanagan
On systems with very high load sometimes the system still thinks the build dir is there. I've only seen this once, but a sync here ensures that the system knows that build has been moved.
That smells like a race in the AB code instead of the kernel being confused over the state of the file system, and the best a sync can do is reduce the window of failure at the cost of a potentially very heavy IO burst.
I'd say drop this and if the problem occurs again, attempt to diagnose where the race is.
Ross