Consolidate Pull Requests for Autobuilder
cover Consolidate Pull Requests for Autobuilder
Scott,
This pull request updates the buildbot installer to support BuildBot 0.8.3, which was released yesterday, as well as updated build helper scripts and a new pokyABConfig.py which adds a swabber testing target,implements all current autobuilder build arches and lastly, implements sanity testing.
The buildbot installer script has been re-written in python and an INSTALL file added to explain the new installation process. I've tested it on Ubuntu 10.10 and openSUSE 11.3, and would appreciate anyone who has a different system letting me know their experiences with it (good and bad).
I've also incorporated a fix to poky-autobuild that sources the correct env script and added some missing build helper scripts, based off of what is actually on the autobuilder.
-b
Pull URL: git://git.pokylinux.org/poky-contrib.git
Branch: eflanagan/bb-0.8.3
Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=eflanagan/bb-0.8.3
Thanks,
Beth Flanagan <[email protected]>
---
Beth Flanagan (3):
Upgrade to Buildbot 0.8.3: scripts/poky-setup-autobuilder Adding new scripts to reflect a real autobuilder Change to poky-autobuild to source correct file:
INSTALL | 55 +++ README | 6 +- scripts/poky-autobuild | 39 ++- scripts/poky-autobuild-cleanoutput | 6 +- scripts/poky-autobuild-copy-images | 458 +++++++++++++++++++ scripts/poky-autobuild-generate-release-external | 220 +++++++++ scripts/poky-autobuild-generate-release-internal | 232 ++++++++++ scripts/poky-autobuild-generate-sources-tarball | 54 +++ scripts/poky-autobuild-postprocess | 6 +- scripts/poky-autobuild-preamble | 58 +++ scripts/poky-autobuild-rename-release-images | 128 ++++++ scripts/poky-autobuild-sanitytest | 58 +++ scripts/poky-output-sort | 117 +++++ scripts/poky-preserve-work-dir | 26 +- scripts/poky-setup-autobuilder | 514 +++++++++++++++------- scripts/pokyABConfig.py | 469 +++++++++++++++----- scripts/run-on-buildset-success | 11 + 17 files changed, 2154 insertions(+), 303 deletions(-) create mode 100644 INSTALL create mode 100755 scripts/poky-autobuild-copy-images create mode 100755 scripts/poky-autobuild-generate-release-external create mode 100755 scripts/poky-autobuild-generate-release-internal create mode 100755 scripts/poky-autobuild-generate-sources-tarball create mode 100755 scripts/poky-autobuild-preamble create mode 100755 scripts/poky-autobuild-rename-release-images create mode 100755 scripts/poky-autobuild-sanitytest create mode 100755 scripts/poky-output-sort create mode 100755 scripts/run-on-buildset-success
PATCH 1/3 Upgrade to Buildbot 0.8.3: scripts/poky-setup-autobuilder
I've ported the buildbot install script to python and added an INSTALL file that describes all the changes. This should be run from the base directory:
./scripts/poky-setup-autobuilder [options] master|slave|both
It's fairly straightfoward. I did not include remote machine installs this time around. I also do not create the poky-build user nor set up proxy info. This should all be done on by the build system admin.
--- INSTALL | 55 +++++ README | 6 +- scripts/poky-setup-autobuilder | 514 +++++++++++++++++++++++++++------------- 3 files changed, 407 insertions(+), 168 deletions(-) create mode 100644 INSTALL
INSTALL new file +55 -0
@@ -0,0 +1,55 @@ +Basic Setup +============================= + +From the user account that will run the autobuilder: + +git clone git://git.pokylinux.org/poky-autobuilder +cd poky-autobuilder +./scripts/poky-setup-autobuilder [options] master|slave|both + +Options +============================= + +At minimum you should use the --builduser --buildpasswd options. If those are left blank, you'll be prompted for them. + + --buildpasswd=BBBUILDPASSWD + The BuildBot users password. If this is left empty, + we'll prompt you for it. + --builduser=BBBUILDUSER + The BuildBot users password. If this is left empty, + we'll prompt you for it. + --gitrepo=BBGITREPO The Git Repo used to get as poky-autobuilder controler + repo. It's generally safe to ignore this. + --masterdir=BBMASTERDIR + The directory you want the master/slave installed to. + If this is left empty, we'll use ~/poky-master. + --slavedir=BBSLAVEDIR + The directory you want the master/slave installed to. + If this is left empty, we'll use ~/poky-slave. + --outputdir=BBOUTPUTDIR + The directory you want the build output to go to. If + this is left empty, we'll use ~/poky-slave/output. + --sourcedir=BBSOURCEDIR + The directory you want the build source to go to. If + this is left empty, we'll use ~/poky-slave/source. + --pstagedir=BBPSTAGINGDIR + The directory you want the package staging to occur + in. If this is left empty, we'll use ~/poky- + slave/pstaging. + --controldir=BBCONTROLDIR + The directory you want to install poky autobuilder + control scripts into. If this is left empty, we'll use + ~/poky-autobuilder. + --maxbuilds=BBMAXBUILDS + The max number of builds you want saved. If this is + left empty, we'll set this to 1. + --maxlogs=BBMAXLOGS The max number of logs you want saved. If this is left + empty, we'll set this to 10. + --adminmail=BBADMINMAIL + The administrator email address. If left unset we set + it to current user @ localhost + +Master, Slave or Both +============================= + +You must specify if the installation is for a Build controller, a Builder or Both.
README +3 -3
@@ -5,10 +5,10 @@ A lot of the autobuilder setup is individual to the circumstances of the user, but this repository contains sample materials which can be used as a starting point. -scripts/poky-setup-autobuilder +scripts/poky-setup-autobuilder [options] master|slave|both The core script which when run, sets up the current user with a buildbot -install, and sample master and slave configurations. +install. Please see INSTALL for more details. scripts/pokyABConfig.py
@@ -36,5 +36,5 @@ server. Designed to be run from cron. scripts/poky-autobuild-upload-share -The coutnerpart to poky-autobuild-upload which shares the images (and +The counterpart to poky-autobuild-upload which shares the images (and ensures no partly uploaded image is shared). Designed to be run from cron.
scripts/poky-setup-autobuilder +349 -165
544 lines not on this page · raw message
Signature · 1 line
1.7.1
PATCH 2/3 Adding new scripts to reflect a real autobuilder
In reply to Beth FlanaganI've ported over recent script additions to the autobuilder that haven't been in prior commits. Specifically, I've added some of the generate-release scripts, as well as the script that generates the release tarball. I've added some changes to enable Swabber host contamination detection and added the scripts needed for sanity testing.
Also, I've updated the generic pokyABConfig.py to incorporate this new functionality
--- scripts/poky-autobuild | 37 ++- scripts/poky-autobuild-cleanoutput | 6 +- scripts/poky-autobuild-copy-images | 458 +++++++++++++++++++++ scripts/poky-autobuild-generate-release-external | 220 ++++++++++ scripts/poky-autobuild-generate-release-internal | 232 +++++++++++ scripts/poky-autobuild-generate-sources-tarball | 54 +++ scripts/poky-autobuild-postprocess | 6 +- scripts/poky-autobuild-preamble | 58 +++ scripts/poky-autobuild-rename-release-images | 128 ++++++ scripts/poky-autobuild-sanitytest | 58 +++ scripts/poky-output-sort | 117 ++++++ scripts/poky-preserve-work-dir | 26 +- scripts/pokyABConfig.py | 469 +++++++++++++++++----- scripts/run-on-buildset-success | 11 + 14 files changed, 1746 insertions(+), 134 deletions(-) create mode 100755 scripts/poky-autobuild-copy-images create mode 100755 scripts/poky-autobuild-generate-release-external create mode 100755 scripts/poky-autobuild-generate-release-internal create mode 100755 scripts/poky-autobuild-generate-sources-tarball create mode 100755 scripts/poky-autobuild-preamble create mode 100755 scripts/poky-autobuild-rename-release-images create mode 100755 scripts/poky-autobuild-sanitytest create mode 100755 scripts/poky-output-sort create mode 100755 scripts/run-on-buildset-success
scripts/poky-autobuild +26 -11
@@ -2,7 +2,7 @@ # # Poky Automated Build Server Enviroment Setup Script # -# Copyright (C) 2010 Intel +# Copyright (C) 2010 Intel Corp. # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by
@@ -31,13 +31,6 @@ cd $CURRDIR echo "Changed to $CURRDIR" -if [ "xpreamble" = "x$1" ]; then - mkdir -p ./build/tmp/deploy/images - rm -f ./build/tmp/deploy/images/images-complete - git log -1 > ./build/tmp/deploy/images/gitinfo - exit 0 -fi - BDIR="build" . ./scripts/poky-env-internal POSTPROCESS=`which poky-autobuild-postprocess`
@@ -58,15 +51,31 @@ if [ ! -e "$CONFFILE" ]; then if [ ! -d "./conf" ]; then mkdir -p "./conf" fi - echo 'PACKAGE_CLASSES = "package_ipk package_deb package_rpm"' > "$CONFFILE" - echo 'BB_NUMBER_THREADS = "4"' >> "$CONFFILE" - echo 'PARALLEL_MAKE = "-j 3"' >> "$CONFFILE" + echo 'PACKAGE_CLASSES = "package_rpm package_deb package_ipk"' > "$CONFFILE" + echo 'BB_NUMBER_THREADS = "10"' >> "$CONFFILE" + echo 'PARALLEL_MAKE = "-j 16"' >> "$CONFFILE" echo 'SDKMACHINE ?= "i586"' >> "$CONFFILE" echo "DL_DIR = \"$SOURCEDIR\"" >> "$CONFFILE" echo 'INHERIT += "poky-autobuild-notifier"' >> "$CONFFILE" echo 'INHERIT += "rm_work"' >> "$CONFFILE" fi +if [ "x$ENABLE_SWABBER" = "xtrue" ]; then + echo 'USER_CLASSES += "image-prelink image-swab"' >> "$CONFFILE" +fi + +# Additions needed to work behind a generic proxy +#SITECONF="./conf/site.conf" +# +#if [ ! -e "$SITECONF" ]; then +# echo 'SCONF_VERSION = "1"' > "$SITECONF" +# echo 'CVS_PROXY_HOST = "proxy.sitename"' >> "$SITECONF" +# echo 'CVS_PROXY_PORT = "proxyportnumber"' >> "$SITECONF" +# echo 'GIT_PROXY_HOST = "proxy.sitename"' >> "$SITECONF" +# echo 'GIT_PROXY_PORT = "proxyportnumber"' >> "$SITECONF" +# echo 'GIT_PROXY_COMMAND = "/usr/local/bin/git-proxy"' >> "$SITECONF" +#fi + bitbake $@ retval=$?
@@ -76,4 +85,10 @@ if [ "x$POSTPROCESS" != "x" -a -x "$POSTPROCESS" ]; then $POSTPROCESS incremental $ABTARGET $TARGETDIR fi +if [ $retval -ne 0 ]; then + # Create a build failure file flag, which can be used by the + # run-on-buildset-success script + touch BuildFailure +fi + exit $retval
scripts/poky-autobuild-cleanoutput +2 -4
@@ -1,5 +1,3 @@ #!/bin/bash -rm build/tmp/deploy/images/poky-image* || /bin/true -rm build/tmp/deploy/images/moblin-image* || /bin/true -rm build/tmp/deploy/images/rootfs* || /bin/true -rm build/tmp/deploy/sdk/* || /bin/true +rm build/tmp/deploy/images/* || /bin/true +rm build/tmp/deploy/sdk/* || /bin/true
scripts/poky-autobuild-copy-images new file +458 -0
458 lines not on this page · raw message
scripts/poky-autobuild-generate-release-external new file +220 -0
220 lines not on this page · raw message
scripts/poky-autobuild-generate-release-internal new file +232 -0
232 lines not on this page · raw message
scripts/poky-autobuild-generate-sources-tarball new file +54 -0
54 lines not on this page · raw message
scripts/poky-autobuild-postprocess +3 -3
@@ -125,7 +125,7 @@ if [ "x$ABTARGET" = "xpoky-full" ]; then `readlink images/zImage-qemuarm.bin` `readlink images/zImage-spitz.bin` `readlink images/bzImage-qemux86.bin` - `readlink images/bzImage-netbook.bin` + `readlink images/bzImage-atom-pc.bin` `readlink images/zImage-htcuniversal.bin` `readlink images/zImage-mx31ads.bin` `readlink images/zImage-zylonite.bin`
@@ -137,8 +137,8 @@ if [ "x$ABTARGET" = "xpoky-full" ]; then updater.sh.c7x0 updater.sh.spitz gnu-tar - poky-image-live-netbook-*.hddimg - poky-image-minimal-live-netbook-*.hddimg + poky-image-live-atom-pc-*.hddimg + poky-image-minimal-live-atom-pc-*.hddimg poky-image-sato-akita-*.rootfs.summary.jffs2 poky-image-sato-c7x0-*.rootfs.jffs2 poky-image-sato-spitz-*.rootfs.tar.gz
scripts/poky-autobuild-preamble new file +58 -0
58 lines not on this page · raw message
scripts/poky-autobuild-rename-release-images new file +128 -0
128 lines not on this page · raw message
scripts/poky-autobuild-sanitytest new file +58 -0
58 lines not on this page · raw message
scripts/poky-output-sort new file +117 -0
117 lines not on this page · raw message
scripts/poky-preserve-work-dir +20 -6
@@ -1,15 +1,16 @@ #!/bin/bash # -# Move the $WORKDIR into a safe area so the Buildbot SCM system can -# blow away the build directory during updates. +# Move the tmp dir and shared state cache dir into a safe area so the +# Buildbot SCM system can blow away the build directory during updates. CMD=$1 if [ "x$CMD" = "xbackup" ]; then if [ ! -d ./build/conf ]; then - echo "No build/conf directory found; assuming this is the first checkout" - # Still create a tmp.protected directory so that restore will work + echo "No build/conf directory found; assuming this is a build from scratch" + # Create empty directores so that restore will work unmodified mkdir ../tmp.protected + mkdir ../sstate-cache.protected exit 0 fi
@@ -20,10 +21,17 @@ if [ "x$CMD" = "xbackup" ]; then fi mv build/tmp ../tmp.protected - if [ ! -e ../tmp.protected ]; then + if [ ! -d ../tmp.protected ]; then echo "Error moving build/tmp to a protected area" exit 1 fi + + mv build/sstate-cache ../sstate-cache.protected + if [ ! -d ../sstate-cache.protected ]; then + echo "Error moving build/sstate-cache to a protected area" + exit 1 + fi + elif [ "x$CMD" == "xrestore" ]; then if [ ! -d ../tmp.protected ]; then PWD=`pwd`
@@ -33,10 +41,16 @@ elif [ "x$CMD" == "xrestore" ]; then mkdir build mv ../tmp.protected build/tmp - if [ ! -e build/tmp ]; then + if [ ! -d build/tmp ]; then echo "Error moving tmp.protected dir back into build/tmp" exit 1 fi + + mv ../sstate-cache.protected build/sstate-cache + if [ ! -d build/sstate-cache ]; then + echo "Error moving sstate-cache.protected dir back into build/sstate-cache" + exit 1 + fi else echo "Unknown command '$CMD'" echo "Commands are 'backup' or 'restore'"
scripts/pokyABConfig.py +359 -110
735 lines not on this page · raw message
scripts/run-on-buildset-success new file +11 -0
@@ -0,0 +1,11 @@ +#!/bin/bash +# +# Run the command given as arguments only if the build does not have +# a 'BuildFailure' file in its build directory. + +if [ -e "BuildFailure" ]; then + echo "Build failure detected in this build set - NOT running '$@'" + exit 0 +fi + +$@
Signature · 1 line
1.7.1
PATCH 3/3 Change to poky-autobuild to source correct file:
In reply to Beth FlanaganA change to poky-init-build-env in poky master broke the autobuilder. Switching this to source poky-init-build-env instead of poky-env-internal.
--- scripts/poky-autobuild | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
scripts/poky-autobuild +1 -1
@@ -32,7 +32,7 @@ cd $CURRDIR echo "Changed to $CURRDIR" BDIR="build" -. ./scripts/poky-env-internal +. ./scripts/poky-init-build-env POSTPROCESS=`which poky-autobuild-postprocess` if [ "xcomplete" = "x$1" ]; then
Signature · 1 line
1.7.1
Quoted 42 lines · Beth Flanagan
Scott,
This pull request updates the buildbot installer to support BuildBot 0.8.3, which was released yesterday, as well as updated build helper scripts and a new
pokyABConfig.pywhich adds a swabber testing target,implements all current autobuilder build arches and lastly, implements sanity testing.The buildbot installer script has been re-written in python and an INSTALL file added to explain the new installation process. I've tested it on Ubuntu 10.10 and openSUSE 11.3, and would appreciate anyone who has a different system letting me know their experiences with it (good and bad).
I've also incorporated a fix to poky-autobuild that sources the correct env script and added some missing build helper scripts, based off of what is actually on the autobuilder.
-b
Pull URL:
git://git.pokylinux.org/poky-contrib.git
Branch: eflanagan/bb-0.8.3
Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=eflanagan/bb-0.8.3Thanks,
Beth Flanagan<[email protected]>
---Beth Flanagan (3):
Upgrade to Buildbot 0.8.3:scripts/poky-setup-autobuilderAdding new scripts to reflect a real autobuilder Change to poky-autobuild to source correct file:INSTALL | 55 +++ README | 6 +- scripts/poky-autobuild | 39 ++- scripts/poky-autobuild-cleanoutput | 6 +- scripts/poky-autobuild-copy-images | 458 +++++++++++++++++++ scripts/poky-autobuild-generate-release-external | 220 +++++++++ scripts/poky-autobuild-generate-release-internal | 232 ++++++++++ scripts/poky-autobuild-generate-sources-tarball | 54 +++ scripts/poky-autobuild-postprocess | 6 +- scripts/poky-autobuild-preamble | 58 +++ scripts/poky-autobuild-rename-release-images | 128 ++++++ scripts/poky-autobuild-sanitytest | 58 +++ scripts/poky-output-sort | 117 +++++ scripts/poky-preserve-work-dir | 26 +- scripts/poky-setup-autobuilder | 514 +++++++++++++++------- scripts/pokyABConfig.py | 469 +++++++++++++++----- scripts/run-on-buildset-success | 11 + 17 files changed, 2154 insertions(+), 303 deletions(-)
Quoted 10 lines
create mode 100644 INSTALL
create mode 100755scripts/poky-autobuild-copy-imagescreate mode 100755scripts/poky-autobuild-generate-release-externalcreate mode 100755scripts/poky-autobuild-generate-release-internalcreate mode 100755scripts/poky-autobuild-generate-sources-tarballcreate mode 100755scripts/poky-autobuild-preamblecreate mode 100755scripts/poky-autobuild-rename-release-imagescreate mode 100755scripts/poky-autobuild-sanitytestcreate mode 100755scripts/poky-output-sort
create mode 100755scripts/run-on-buildset-success
After a few iterations with Beth, I have now pulled this into poky-autobuilder master.
Thanks Beth!
Scott
Signature · 2 lines
Scott Garman Embedded Linux Distro Engineer - Yocto Project