fixes for use with Angstrom
cover fixes for use with Angstrom
From: Stefan Agner <[email protected]>
Hi folks,
We plan to make use of the Yocto Autobuilder at Toradex. I successfully deployed an installation, however since we distribute an Anstrom based BSP I had to adjust some files. With those patches and a custom layer configuration (not included since quite ugly) I'm able to build our BSP.
Would be good to have those changes upstream. Especially the new way to preserve the order of builders is more generic, however maybe not yet perfect. I needed this especially since I build some other stuff using POBBC (plain old build bot configuration :-)).
Just as a general question: What is planned with the autobuilder in general? Is there an integration planned with Toaster? As far as I can see, since the build directory is recreated each time, this is not really possible right now. It would need adjustment to the autobuilder to start the toaster web interface for each builder, and clean/restart builds in a way that doesn't interfere with Toaster. Any thoughts on this?
--
Stefan
Stefan Agner (4):
GetDistroVersion.py support for Angstrom distro version
README: add high level overview, fix build slave description master.py/botmaster.py: use builders list for builders order GetDeployDir.py/PublishArtifacts.py: get deploy dir from bitbake
README | 8 ++-- README-NEW-AUTOBUILDER | 11 +++++ .../site-packages/autobuilder/Autobuilder.py | 17 ++++---- .../autobuilder/buildsteps/GetDeployDir.py | 47 +++++++++++++++++++++ .../autobuilder/buildsteps/GetDistroVersion.py | 7 +++- .../autobuilder/buildsteps/PublishArtifacts.py | 48 ++++++++++++---------- .../buildbot/process/botmaster.py | 3 +- .../buildbot/status/master.py | 12 +----- 8 files changed, 106 insertions(+), 47 deletions(-) create mode 100644 lib/python2.7/site-packages/autobuilder/buildsteps/GetDeployDir.py
Signature · 1 line
1.9.0
PATCH 1/4 GetDistroVersion.py support for Angstrom distro version
From: Stefan Agner <[email protected]>
Ångström distribution has a different location for distro version, support "angstrom" as distro too.
--- .../site-packages/autobuilder/buildsteps/GetDistroVersion.py | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)
lib/python2.7/site-packages/autobuilder/buildsteps/GetDistroVersion.py +5 -2
@@ -32,10 +32,13 @@ class GetDistroVersion(ShellCommand): def start(self): buildername=self.getProperty("buildername") + distroconf=self.slaveworkdir + "/" + buildername if 'poky' in self.distro: - distroconf=self.slaveworkdir + "/" + buildername + "/build/meta-yocto/conf/distro/poky.conf|grep 'DISTRO_VERSION = \"'" + distroconf+="/build/meta-yocto/conf/distro/poky.conf|grep 'DISTRO_VERSION = \"'" elif 'oecore' in self.distro: - distroconf=self.slaveworkdir + "/" + buildername + "/build/meta/conf/distro/include/default-distrovars.inc|grep 'DISTRO_VERSION '" + distroconf+="/build/meta/conf/distro/include/default-distrovars.inc|grep 'DISTRO_VERSION '" + elif 'angstrom' in self.distro: + distroconf+="/build/meta-angstrom/conf/distro/angstrom-*.conf|grep 'DISTRO_VERSION = \"'" else: self.finished(FAILURE) cmd = 'cat ' + distroconf
Signature · 1 line
1.9.0
PATCH 2/4 README: add high level overview, fix build slave description
In reply to [email protected]From: Stefan Agner <[email protected]>
Replace builders with build slaves, since this are different things in BuildBot world. Also add a section with some high level view what Autobuilder actually is.
--- README | 8 ++++---- README-NEW-AUTOBUILDER | 11 +++++++++++ 2 files changed, 15 insertions(+), 4 deletions(-)
README +4 -4
@@ -26,7 +26,7 @@ Setup connection that I haven't figured out yet. Manually connecting seems to set up the session correctly.) -2. Adding additional builders +2. Adding additional build slaves The production yocto autobuilder uses a cluster of build slaves, all sharing the same SSTATE_DIR and DL_DIR via an NFS4 mounted NAS. The main nightly
@@ -36,7 +36,7 @@ Setup In theory you could also run your build slaves with NO_NETWORK to enforce a single point of populating DL_DIR. - Running multiple builders is fairly simple, but does require some setup. + Running multiple build slaves is fairly simple, but does require some setup. 1. Ensure the autobuilder.conf for each build slave is appropriate. As certain variables are set within this file that are local to each builder
@@ -45,7 +45,7 @@ Setup 2. Within yocto-master/master.cfg make the following changes substituing <PASS> for your debug password and max_builds for the number of builds you wish to run on each builder. Your main builder that will run the - nightly trigger buildset should have at least 2 builders running. + nightly trigger buildset should have at least 2 build slaves running. c['slaves'] = [BuildSlave("builder1", "<PASS>", max_builds=3),
@@ -53,7 +53,7 @@ c['slaves'] = [BuildSlave("builder1", "<PASS>", max_builds=3), BuildSlave("builder3", "<PASS>", max_builds=3), BuildSlave("builder4", "<PASS>", max_builds=3)] - 3. For each buildslave change yocto-slave/buildbot.tac to point to the + 3. For each build slave change yocto-slave/buildbot.tac to point to the correct build master and port. The default setup points to local host. buildmaster_host = 'builder1.myproject.org'
README-NEW-AUTOBUILDER +11 -0
@@ -23,6 +23,17 @@ TODO - deploy stuff - killing triggers. this has been a long standing issue. look into +What is it +------------ +Yocto Autobuilder is essentially some extension to the vanilla build bot. +This extension mainly addresses configuration file handling and Yocto specific +build steps. +For better maintainability, the Autobuilder (see Autobuilder.py located at +lib/python2.7/site-packages/autobuilder/), handles configuration from multiple +files. +Additional build steps such as CheckOutLayers.py or CreateBBLayersConf are +Yocto specific and simplify the bulders configuration. + Setup ------------ Nothing here has changed much.
Signature · 1 line
1.9.0
PATCH 3/4 master.py/botmaster.py: use builders list for builders order
In reply to [email protected]From: Stefan Agner <[email protected]>
This patch uses the order of the list builderNames when requesting builder names. This order is relevant for the Waterfall order.
However, the order of builderNames is not meaningful by default, hence the change in botmaster.py makes sure the builderNames is taken form the configuration files list of builder.
Hence we can get rid of the environment work around (which also did not honor non Autobuilder builders).
--- lib/python2.7/site-packages/autobuilder/Autobuilder.py | 17 +++++++++-------- .../buildbot/process/botmaster.py | 3 ++- .../buildbot-0.8.8-py2.7.egg/buildbot/status/master.py | 12 ++---------- 3 files changed, 13 insertions(+), 19 deletions(-)
lib/python2.7/site-packages/autobuilder/Autobuilder.py +9 -8
@@ -38,20 +38,21 @@ class Autobuilder: self.cfile = "./buildset-config/yoctoAB.conf" def createBuildsets(self): - beenHere=[] + buildersSorted=[] sort = True if self.config.has_option('BuildSets', 'order'): sort = False for set in ast.literal_eval(self.config.get('BuildSets', 'order')): - beenHere.append(set) + buildersSorted.append(set) + for key in self.configdict: - if key not in beenHere and key != "BuildSets": - beenHere.append(key) + if key not in buildersSorted and key != "BuildSets": + buildersSorted.append(key) + if sort: - beenHere.sort() - # REALLY crappy way to do this, but better than setting globals. - os.environ["YOCTO_SORTED_BUILDERS"]= str(beenHere) - for buildset in beenHere: + util.naturalSort(buildersSorted) + + for buildset in buildersSorted: self.schedprops = [] self.checkoutprops={} self.set_props = {}
lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/process/botmaster.py +2 -1
@@ -262,7 +262,8 @@ class BotMaster(config.ReconfigurableServiceMixin, service.MultiService): builder.master = self.master builder.setServiceParent(self) - self.builderNames = self.builders.keys() + # Use order according to configuration files builders list + self.builderNames = list([ (bc.name) for bc in new_config.builders ]) metrics.MetricCountEvent.log("num_builders", len(self.builders), absolute=True)
lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/status/master.py +2 -10
@@ -221,16 +221,8 @@ class Status(config.ReconfigurableServiceMixin, service.MultiService): def getBuilderNames(self, categories=None): if categories == None: - #YOCTO PROJECT ADDITION - import os - import ast - sortbuilders="" - sortbuilders=ast.literal_eval(os.environ.get('YOCTO_SORTED_BUILDERS')) - #I hate the way this is sorted. This is the laziest way to get this - #in the format I want. - return sortbuilders - #YOCTO PROJECT ADDITION - #return util.naturalSort(self.botmaster.builderNames) # don't let them break it + # assume this is already sorted... + return self.botmaster.builderNames l = [] # respect addition order
Signature · 1 line
1.9.0
PATCH 4/4 GetDeployDir.py/PublishArtifacts.py: get deploy dir from bitbake
In reply to [email protected]From: Stefan Agner <[email protected]>
Since the deploy dir is configureable (also through TMPDIR), we need to have a more generic variant to get that path. Also the Ångstrom distribution adds "-eglibc" to the TMPDIR. Hence, the safest way to get the deployment dir is to parse buildbots environment.
--- .../autobuilder/buildsteps/GetDeployDir.py | 47 +++++++++++++++++++++ .../autobuilder/buildsteps/PublishArtifacts.py | 48 ++++++++++++---------- 2 files changed, 73 insertions(+), 22 deletions(-) create mode 100644 lib/python2.7/site-packages/autobuilder/buildsteps/GetDeployDir.py
lib/python2.7/site-packages/autobuilder/buildsteps/GetDeployDir.py new file +47 -0
@@ -0,0 +1,47 @@ +''' +Created on Mai 8, 2014 + +__author__ = "Stefan Agner" +__copyright__ = "Copyright 2014, Toradex AG" +__credits__ = ["Stefan Agner"] +__license__ = "GPL" +__version__ = "2.0" +__maintainer__ = "Stefan Agner" +__email__ = "[email protected]" +''' +from buildbot.steps.shell import ShellCommand +from buildbot.status import progress +from buildbot.status.results import SUCCESS, FAILURE, WARNINGS +from buildbot.process.properties import WithProperties +from twisted.python import log +from autobuilder.config import * + +class GetDeployDir(ShellCommand): + haltOnFailure = False + flunkOnFailure = True + name = "GetDeployDir" + def __init__(self, factory, argdict=None, **kwargs): + for k, v in argdict.iteritems(): + setattr(self, k, v) + self.description = "Get Deploy Directory" + self.deploydir = None + ShellCommand.__init__(self, **kwargs) + + def start(self): + cmd = '. ./oe-init-build-env; bitbake -e | grep "^DEPLOY_DIR="' + self.command = cmd + ShellCommand.start(self) + + def commandComplete(self, cmd): + if cmd.didFail(): + return + result = cmd.logs['stdio'].getText() + if "=" not in result: + self.finished(FAILURE) + param, value = result.split("=") + deploydir = value.replace('"', '').strip() + self.setProperty('deploydir', deploydir, "Setting Deploy Directory") + self.finished(SUCCESS) + + def getText(self, cmd, results): + return ShellCommand.getText(self, cmd, results)
lib/python2.7/site-packages/autobuilder/buildsteps/PublishArtifacts.py +26 -22
132 lines not on this page · raw message
Signature · 1 line
1.9.0
Re: [yocto-autobuilder][PATCH v1 4/4] GetDeployDir.py/PublishArtifacts.py: get deploy dir from bitbake
Ok, so looking at this a bit more and I think we need to look at how we set up the deploydir in PublishArtifacts a bit more.
I think what we can do here is pull the functionality that we have in getDeployNames out, and put it into GetDeployDir (layerversion here matters as the default deploy for oe-core changes at LAYERVERSION_core =3) . So, yeah, if you could resubmit that I think we can pull the last patch in. The other three are under test right now.
Thanks,
-b
Quoted 213 lines · On Thu, May 8, 2014 at 9:44 AM, <[email protected]> wrote:
From: Stefan Agner <[email protected]>
Since the deploy dir is configureable (also through TMPDIR), we need to have a more generic variant to get that path. Also the Ångstrom distribution adds "-eglibc" to the TMPDIR. Hence, the safest way to get the deployment dir is to parse buildbots environment.
Signed-off-by Stefan Agner <[email protected]>--- .../autobuilder/buildsteps/GetDeployDir.py | 47 +++++++++++++++++++++ .../autobuilder/buildsteps/PublishArtifacts.py | 48 ++++++++++++---------- 2 files changed, 73 insertions(+), 22 deletions(-) create mode 100644 lib/python2.7/site-packages/autobuilder/buildsteps/GetDeployDir.pylib/python2.7/site-packages/autobuilder/buildsteps/GetDeployDir.py new file +47
-0@@ -0,0 +1,47 @@ +''' +Created on Mai 8, 2014 + +__author__ = "Stefan Agner" +__copyright__ = "Copyright 2014, Toradex AG" +__credits__ = ["Stefan Agner"] +__license__ = "GPL" +__version__ = "2.0" +__maintainer__ = "Stefan Agner" +__email__ = "[email protected]" +''' +from buildbot.steps.shell import ShellCommand +from buildbot.status import progress +from buildbot.status.results import SUCCESS, FAILURE, WARNINGS +from buildbot.process.properties import WithProperties +from twisted.python import log +from autobuilder.config import * + +class GetDeployDir(ShellCommand): + haltOnFailure = False + flunkOnFailure = True + name = "GetDeployDir" + def __init__(self, factory, argdict=None, **kwargs): + for k, v in argdict.iteritems(): + setattr(self, k, v) + self.description = "Get Deploy Directory" + self.deploydir = None + ShellCommand.__init__(self, **kwargs) + + def start(self): + cmd = '. ./oe-init-build-env; bitbake -e | grep "^DEPLOY_DIR="' + self.command = cmd + ShellCommand.start(self) + + def commandComplete(self, cmd): + if cmd.didFail(): + return + result = cmd.logs['stdio'].getText() + if "=" not in result: + self.finished(FAILURE) + param, value = result.split("=") + deploydir = value.replace('"', '').strip() + self.setProperty('deploydir', deploydir, "Setting Deploy Directory") + self.finished(SUCCESS) + + def getText(self, cmd, results): + return ShellCommand.getText(self, cmd, results)lib/python2.7/site-packages/autobuilder/buildsteps/PublishArtifacts.py +26
-23@@ -19,10 +19,10 @@ class PublishArtifacts(ShellCommand): haltOnFailure = False flunkOnFailure = True - name = "Publishing Artifacts" + name = "PublishArtifacts" + description = "Publishing Artifacts" def __init__(self, factory, argdict=None, **kwargs): self.factory = factory - self.description = "Publishing Artifacts" self.slavedir=os.path.join(os.path.join(YOCTO_ABBASE, "yocto-slave")) for k, v in argdict.iteritems(): if type(v) is bool:@@ -42,10 +42,14 @@ class PublishArtifacts(ShellCommand): self.layerversion_yoctobsp = self.getProperty("layerversion_yoctobsp") self.layerversion_yocto = self.getProperty("layerversion_yocto") self.layerversion_core = self.getProperty("layerversion_core") + self.deploydir = self.getProperty("deploydir") log.msg(layerversion_yoctobsp) - self.basedir=os.path.join(os.path.join( - self.slavedir, buildername), - "build/build/") + self.basedir = os.path.join(self.slavedir, buildername, "build/build/") + + # Deploydir might not be set in case the step did not run, fallback to default + if self.deploydir is None: + self.deploydir = os.path.join(self.basedir, "tmp/deploy/") + command="" DATESTAMP=datetime.datetime.now().strftime("%Y%m%d") if str(os.environ.get('PUBLISH_BUILDS')) == "True":@@ -70,17 +74,17 @@ class PublishArtifacts(ShellCommand): if artifact == "adt-installer": command=command+"mkdir -p " + os.path.join(DEST, ADT_INST_PUBLISH_DIR) + ";" command=command+"cp -R --no-dereference --preserve=links " + \ - os.path.join(self.basedir, "tmp/deploy/sdk/") + \ + os.path.join(self.deploydir, "sdk/") + \ "*adt* " + os.path.join(DEST, ADT_INST_PUBLISH_DIR) + ";" elif artifact == "adt-installer-QA": command=command+"mkdir -p " + os.path.join(DEST, ADTQA_INST_PUBLISH_DIR) + ";" command=command+"cp -R --no-dereference --preserve=links " + \ - os.path.join(self.basedir, "tmp/deploy/sdk/") + \ + os.path.join(self.deploydir, "sdk/") + \ "*adt* " + os.path.join(DEST, ADTQA_INST_PUBLISH_DIR)+ ";" elif artifact == "adtrepo-dev": adt_dev_dest= os.environ.get("ADTREPO_DEV_PATH") + "/" + distroversion + "-" + got_revision_poky + '-' + self.getProperty("branch_poky") command=command+"mkdir -p " + adt_dev_dest + "/adt-ipk;" - command=command+"rsync -av " + os.path.join(self.basedir, "tmp/deploy/ipk/") + " " + adt_dev_dest + "/adt-ipk;" + command=command+"rsync -av " + os.path.join(self.deploydir, "ipk/") + " " + adt_dev_dest + "/adt-ipk;" command=command+"rm -rf " + adt_dev_dest + "/rootfs; mkdir -p " + adt_dev_dest + "/rootfs;" command=command+"for x in `ls " + DEST + "/machines/qemu/|grep -v tiny`; do ln -s " + DEST + "/machines/qemu/$x " + adt_dev_dest + "/rootfs/$x; done;" if self.layerversion_yocto is not None and int(self.layerversion_yocto) < 2:@@ -101,49 +105,49 @@ class PublishArtifacts(ShellCommand): command=command+"mkdir -p " + DEST + "/" + BA_PUBLISH_DIR + ";" if self.layerversion_core is not None and int(self.layerversion_core) > 2: command=command+"cp -R --no-dereference --preserve=links " + \ - os.path.join(self.basedir, "tmp/deploy/images/qemux86-64/*.zip") + \ + os.path.join(self.deploydir, "images/qemux86-64/*.zip") + \ " " + DEST + "/" + BA_PUBLISH_DIR + ";" else: command=command+"cp -R --no-dereference --preserve=links " + \ - os.path.join(self.basedir, "tmp/deploy/images/*.zip") + \ + os.path.join(self.deploydir, "images/*.zip") + \ " " + DEST + "/" + BA_PUBLISH_DIR + ";" elif artifact == "buildtools-tarball": command=command+"mkdir -p " + DEST + "/buildtools;" command=command+"cp -R --no-dereference --preserve=links " + \ - os.path.join(self.basedir, "tmp/deploy/sdk/*.sh") + \ + os.path.join(self.deploydir, "sdk/*.sh") + \ " " + DEST + "/buildtools;" elif artifact == "rpm": command=command+"mkdir -p " + os.path.join(DEST, RPM_PUBLISH_DIR) + ";" command=command+"cp -R --no-dereference --preserve=links " + \ - os.path.join(self.basedir, "tmp/deploy/rpm/* ") + \ + os.path.join(self.deploydir, "rpm/* ") + \ os.path.join(DEST, RPM_PUBLISH_DIR) + ";" elif artifact == "deb": command=command+"mkdir -p " + os.path.join(DEST, DEB_PUBLISH_DIR) + ";" command=command+"cp -R --no-dereference --preserve=links " + \ - os.path.join(self.basedir, "tmp/deploy/deb/* ") + \ + os.path.join(self.deploydir, "deb/* ") + \ os.path.join(DEST, DEB_PUBLISH_DIR) + ";" elif artifact == "ipk": command=command+"mkdir -p " + os.path.join(DEST, IPK_PUBLISH_DIR) + ";" command=command+"cp -R --no-dereference --preserve=links " + \ - os.path.join(self.basedir, "tmp/deploy/ipk/* ") + \ + os.path.join(self.deploydir, "ipk/* ") + \ os.path.join(DEST, IPK_PUBLISH_DIR) + ";" elif artifact == "toolchain": command=command+"mkdir -p " + os.path.join(DEST, X86TC_PUBLISH_DIR) + ";" command=command+"cp -R --no-dereference --preserve=links " + \ - os.path.join(self.basedir, "tmp/deploy/sdk/poky-eglibc-i686* ") + \ + os.path.join(self.deploydir, "sdk/poky-eglibc-i686* ") + \ os.path.join(DEST, X86TC_PUBLISH_DIR) + ";" command=command+"mkdir -p " + os.path.join(DEST, X8664TC_PUBLISH_DIR) + ";" command=command+"cp -R --no-dereference --preserve=links " + \ - os.path.join(self.basedir, "tmp/deploy/sdk/poky-eglibc-x86_64* ") + \ + os.path.join(self.deploydir, "sdk/poky-eglibc-x86_64* ") + \ os.path.join(DEST, X8664TC_PUBLISH_DIR) + ";" elif artifact == "oe-toolchain": command=command+"mkdir -p " + os.path.join(DEST, X86TC_PUBLISH_DIR) + ";" command=command+"cp -R --no-dereference --preserve=links " + \ - os.path.join(self.basedir, "tmp/deploy/sdk/oecore-i686* ") + \ + os.path.join(self.deploydir, "sdk/oecore-i686* ") + \ os.path.join(DEST, X86TC_PUBLISH_DIR) + ";" command=command+"mkdir -p " + os.path.join(DEST, X8664TC_PUBLISH_DIR) + ";" command=command+"cp -R --no-dereference --preserve=links " + \ - os.path.join(self.basedir, "tmp/deploy/sdk/oecore-x86_64* ") + \ + os.path.join(self.deploydir, "sdk/oecore-x86_64* ") + \ os.path.join(DEST, X8664TC_PUBLISH_DIR) + ";" elif "qemu" in artifact:@@ -162,11 +166,11 @@ class PublishArtifacts(ShellCommand): command=command+"mkdir -p " + DEST + "/" + QEMU_PUBLISH_DIR + "/qemu-tiny;" if self.layerversion_core is not None and int(self.layerversion_core) > 2: command=command+"cp -R --no-dereference --preserve=links " + \ - os.path.join(self.basedir, "tmp/deploy/images/qemux86/* ") + \ + os.path.join(self.deploydir, "images/qemux86/* ") + \ " " + DEST + "/" + QEMU_PUBLISH_DIR + "/qemu-tiny;" else: command=command+"cp -R --no-dereference --preserve=links " + \ - os.path.join(self.basedir, "tmp/deploy/images/*") + \ + os.path.join(self.deploydir, "images/*") + \ DEST + "/" + QEMU_PUBLISH_DIR + "/qemu-tiny;" elif artifact == "conf": artifact_name, deploy_image_dir = self.getDeployNames(artifact, buildername)@@ -208,9 +212,9 @@ class PublishArtifacts(ShellCommand): if "-lsb" in buildername: artifact_name = artifact_name + "-lsb" if self.layerversion_core is not None and int(self.layerversion_core) > 2: - deploy_dir_image = os.path.join(os.path.join(self.basedir, "tmp/deploy/images/"), artifact) + deploy_dir_image = os.path.join(os.path.join(self.deploydir, "images/"), artifact) else: - deploy_dir_image = os.path.join(self.basedir, "tmp/deploy/images/") + deploy_dir_image = os.path.join(self.deploydir, "images/") return artifact_name, deploy_dir_image def describe(self, done=False): --1.9.0
Signature · 3 lines
Elizabeth Flanagan Yocto Project Build and Release
Re: [yocto-autobuilder][PATCH v1 4/4] GetDeployDir.py/PublishArtifacts.py: get deploy dir from bitbake
Hi Beth,
Am 2014-05-16 19:11, schrieb Flanagan, Elizabeth:
Quoted 8 lines
Ok, so looking at this a bit more and I think we need to look at how we set up the deploydir in PublishArtifacts a bit more.
I think what we can do here is pull the functionality that we have in getDeployNames out, and put it into GetDeployDir (layerversion here matters as the default deploy for oe-core changes at
LAYERVERSION_core=3) . So, yeah, if you could resubmit that I think we can pull the last patch in. The other three are under test right now.
Sounds reasonable, I agree. Note that, I will be in vacation for the next three weeks, hence it will be June until I will be able to resubmit that...
--
Stefan
Am 2014-05-08 18:44, schrieb [email protected]:
Quoted 14 lines
From: Stefan Agner <[email protected]>
Hi folks,
We plan to make use of the Yocto Autobuilder at Toradex. I successfully deployed an installation, however since we distribute an Anstrom based BSP I had to adjust some files. With those patches and a custom layer configuration (not included since quite ugly) I'm able to build our BSP.
Would be good to have those changes upstream. Especially the new way to preserve the order of builders is more generic, however maybe not yet perfect. I needed this especially since I build some other stuff using POBBC (plain old build bot configuration :-)).
Quoted 7 lines
Just as a general question: What is planned with the autobuilder in general? Is there an integration planned with Toaster? As far as I can see, since the build directory is recreated each time, this is not really possible right now. It would need adjustment to the autobuilder to start the toaster web interface for each builder, and clean/restart builds in a way that doesn't interfere with Toaster. Any thoughts on this?
Any thoughts on this Patchset?
Today I had troubles with our builder. After removing the whole source directory and restart the build, the autobuilder somehow checked out and resetted the autobuilder git! (the dora branch was checked out, which I tried to checkout from a source branch). So far I could not reconstruct what exactly happend, but the builder must have been in the wrong directory when issuing some commands... Ugly. Calls for sandboxes :-)
Anyway, I lost my custom master.cfg since git restored the version from the dora branch (always remember to backup!)...
Hey Stefan,
Sorry, this got caught somewhere in my email
Quoted 10 lines · Stefan Agner
Am 2014-05-08 18:44, schrieb [email protected]:
Quoted 9 lines
From: Stefan Agner <[email protected]>
Hi folks,
We plan to make use of the Yocto Autobuilder at Toradex. I successfully deployed an installation, however since we distribute an Anstrom based BSP I had to adjust some files. With those patches and a custom layer configuration (not included since quite ugly) I'm able to build our BSP.
if you want, we could include it in buildset-config.examples.
Quoted 7 lines
Quoted 7 lines
Would be good to have those changes upstream. Especially the new way to preserve the order of builders is more generic, however maybe not yet perfect. I needed this especially since I build some other stuff using POBBC (plain old build bot configuration :-)).
Just as a general question: What is planned with the autobuilder in general? Is there an integration planned with Toaster?
At the time, no.
Quoted 6 lines
Quoted 6 lines
As far as I can
see, since the build directory is recreated each time, this is not really possible right now. It would need adjustment to the autobuilder to start the toaster web interface for each builder, and clean/restart builds in a way that doesn't interfere with Toaster. Any thoughts on this?
I've honestly not looked into it too much. What would be interesting is utilizing buildbot slaves and the toaster front end.
Quoted 10 lines
Any thoughts on this Patchset?
Today I had troubles with our builder. After removing the whole source directory and restart the build, the autobuilder somehow checked out and resetted the autobuilder git! (the dora branch was checked out, which I tried to checkout from a source branch). So far I could not reconstruct what exactly happend, but the builder must have been in the wrong directory when issuing some commands... Ugly. Calls for sandboxes :-)
Ah. Please switch to master. There was a VERY VERY nasty bug that is probably still in dora caused by the yoctogit fetcher. I'll look at dora today and backport the patch.
Quoted 2 lines
Anyway, I lost my custom master.cfg since git restored the version from the dora branch (always remember to backup!)...
Yeah, when I was debugging it, I ended up scripting my setup just to figure out what was wrong.
Signature · 3 lines
Elizabeth Flanagan Yocto Project Build and Release
Hi Beth,
Am 2014-05-15 17:32, schrieb Flanagan, Elizabeth:
Quoted 11 lines
Quoted 7 lines
Today I had troubles with our builder. After removing the whole source directory and restart the build, the autobuilder somehow checked out and resetted the autobuilder git! (the dora branch was checked out, which I tried to checkout from a source branch). So far I could not reconstruct what exactly happend, but the builder must have been in the wrong directory when issuing some commands... Ugly. Calls for sandboxes :-)
Ah. Please switch to master. There was a VERY VERY nasty bug that is probably still in dora caused by the yoctogit fetcher. I'll look at dora today and backport the patch.
Actually I was running master of Autobuilder. However, somehow Autobuilder checked the branch dora out (its the branch I tried to build, hence all my layers had "dora" as branch name).
Quoted 6 lines
Quoted 3 lines
Anyway, I lost my custom master.cfg since git restored the version from the dora branch (always remember to backup!)...
Yeah, when I was debugging it, I ended up scripting my setup just to figure out what was wrong.
Inbetween I figured out what happened: A git repository (fsl-arm-extra) was somehow broken in my download folder. Hence the clone from the download folder to the source directory failed. However, the git fetch was done anyway, but since the clone failed that folder was not a valid git repository. But since the yocto-slave lives under the Yocto Autobuilder repository, git checked out the dora branch of the Yocto Autobuilder repository...
Its crucial to check the return status of the git clone command and don't call further commands if clone fails. If those two commands would be two BuildBot steps, wouldn't it do this automatically?
First command:
git clone -s -n/tmp/yocto-autobuilder/git/mirror/git.yoctoproject.org/meta-fsl-arm/ .
in dir
/build/linuxdev/yocto-autobuilder/yocto-slave/toradex/source/git://git.yoctoproject.org/meta-fsl-arm (timeout 100000 secs)
watching logfiles {}
argv: ['git', 'clone', '-s', '-n',
'/tmp/yocto-autobuilder/git/mirror/git.yoctoproject.org/meta-fsl-arm/',
'.']
Output:
Cloning into '.'...
error: refs/heads/master does not point to a valid object!
error: refs/remotes/origin/1.4_M3 does not point to a valid object!
error: refs/remotes/origin/1.4_M5 does not point to a valid object!
error: refs/remotes/origin/danny does not point to a valid object!
error: refs/remotes/origin/danny-next does not point to a valid object!
error: refs/remotes/origin/denzil does not point to a valid object!
error: refs/remotes/origin/dora does not point to a valid object!
error: refs/remotes/origin/dora-next does not point to a valid object!
error: refs/remotes/origin/dylan does not point to a valid object!
error: refs/remotes/origin/dylan-next does not point to a valid object!
error: refs/tags/1.2 does not point to a valid object!
error: refs/tags/1.2-rc1 does not point to a valid object!
error: refs/tags/1.3 does not point to a valid object!
error: refs/tags/1.4 does not point to a valid object!
error: refs/tags/1.5 does not point to a valid object!
done.
error: Trying to write ref HEAD with nonexistent object c91ef2d2a12cd51af20011a4de8d9438e2d4890e
fatal: Cannot update the ref 'HEAD'.
fatal: The remote end hung up unexpectedly
Then, the second command:
git fetch -t --prune git://git.yoctoproject.org/meta-fsl-arm
+refs/heads/dora:refs/remotes/dora
in dir
/build/linuxdev/yocto-autobuilder/yocto-slave/toradex/source/git://git.yoctoproject.org/meta-fsl-arm (timeout 100000 secs)
watching logfiles {}
argv: ['git', 'fetch', '-t', '--prune',
'git://git.yoctoproject.org/meta-fsl-arm',
'+refs/heads/dora:refs/remotes/dora']
...
This resetted my head of the Yocto Autobuilder to dora...
--
Stefan
Ok, these look ok. I'll apply them and run them on dev. Expect something tomorrow.
-b
Quoted 45 lines · On Thu, May 8, 2014 at 9:44 AM, <[email protected]> wrote:
From: Stefan Agner <[email protected]>
Hi folks,
We plan to make use of the Yocto Autobuilder at Toradex. I successfully deployed an installation, however since we distribute an Anstrom based BSP I had to adjust some files. With those patches and a custom layer configuration (not included since quite ugly) I'm able to build our BSP.
Would be good to have those changes upstream. Especially the new way to preserve the order of builders is more generic, however maybe not yet perfect. I needed this especially since I build some other stuff using POBBC (plain old build bot configuration :-)).
Just as a general question: What is planned with the autobuilder in general? Is there an integration planned with Toaster? As far as I can see, since the build directory is recreated each time, this is not really possible right now. It would need adjustment to the autobuilder to start the toaster web interface for each builder, and clean/restart builds in a way that doesn't interfere with Toaster. Any thoughts on this?
--
StefanStefan Agner (4):
GetDistroVersion.pysupport for Angstrom distro version
README: add high level overview, fix build slave descriptionmaster.py/botmaster.py: use builders list for builders orderGetDeployDir.py/PublishArtifacts.py: get deploy dir from bitbakeREADME | 8 ++-- README-NEW-AUTOBUILDER | 11 +++++ .../site-packages/autobuilder/Autobuilder.py | 17 ++++---- .../autobuilder/buildsteps/GetDeployDir.py | 47 +++++++++++++++++++++ .../autobuilder/buildsteps/GetDistroVersion.py | 7 +++- .../autobuilder/buildsteps/PublishArtifacts.py | 48 ++++++++++++---------- .../buildbot/process/botmaster.py | 3 +- .../buildbot/status/master.py | 12 +----- 8 files changed, 106 insertions(+), 47 deletions(-) create mode 100644 lib/python2.7/site-packages/autobuilder/buildsteps/GetDeployDir.py--
1.9.0
Signature · 3 lines
Elizabeth Flanagan Yocto Project Build and Release