Skip to contentSkip to conversation

Consolidate Pull Requests for Autobuilder

PATCH In review Beth Flanagan 2 participants 5 messages Dec 21, 2010 – Dec 23, 2010

lore
Share & tools
lore
#0 lore raw

cover Consolidate Pull Requests for Autobuilder

Scott,

This pull request updates the buildbot installer to support BuildBot 0.8.3, which was released yesterday, as well as updated build helper scripts and a new pokyABConfig.py which adds a swabber testing target,implements all current autobuilder build arches and lastly, implements sanity testing.

The buildbot installer script has been re-written in python and an INSTALL file added to explain the new installation process. I've tested it on Ubuntu 10.10 and openSUSE 11.3, and would appreciate anyone who has a different system letting me know their experiences with it (good and bad).

I've also incorporated a fix to poky-autobuild that sources the correct env script and added some missing build helper scripts, based off of what is actually on the autobuilder.

-b

Pull URL: git://git.pokylinux.org/poky-contrib.git
Branch: eflanagan/bb-0.8.3
Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=eflanagan/bb-0.8.3

Thanks,
Beth Flanagan <[email protected]>
---

Beth Flanagan (3):
Upgrade to Buildbot 0.8.3: scripts/poky-setup-autobuilder Adding new scripts to reflect a real autobuilder Change to poky-autobuild to source correct file:

 INSTALL                                          |   55 +++
 README                                           |    6 +-
 scripts/poky-autobuild                           |   39 ++-
 scripts/poky-autobuild-cleanoutput               |    6 +-
 scripts/poky-autobuild-copy-images               |  458 +++++++++++++++++++
 scripts/poky-autobuild-generate-release-external |  220 +++++++++
 scripts/poky-autobuild-generate-release-internal |  232 ++++++++++
 scripts/poky-autobuild-generate-sources-tarball  |   54 +++
 scripts/poky-autobuild-postprocess               |    6 +-
 scripts/poky-autobuild-preamble                  |   58 +++
 scripts/poky-autobuild-rename-release-images     |  128 ++++++
 scripts/poky-autobuild-sanitytest                |   58 +++
 scripts/poky-output-sort                         |  117 +++++
 scripts/poky-preserve-work-dir                   |   26 +-
 scripts/poky-setup-autobuilder                   |  514 +++++++++++++++-------
 scripts/pokyABConfig.py                          |  469 +++++++++++++++-----
 scripts/run-on-buildset-success                  |   11 +
 17 files changed, 2154 insertions(+), 303 deletions(-)
 create mode 100644 INSTALL
 create mode 100755 scripts/poky-autobuild-copy-images
 create mode 100755 scripts/poky-autobuild-generate-release-external
 create mode 100755 scripts/poky-autobuild-generate-release-internal
 create mode 100755 scripts/poky-autobuild-generate-sources-tarball
 create mode 100755 scripts/poky-autobuild-preamble
 create mode 100755 scripts/poky-autobuild-rename-release-images
 create mode 100755 scripts/poky-autobuild-sanitytest
 create mode 100755 scripts/poky-output-sort
 create mode 100755 scripts/run-on-buildset-success
#1 lore raw

PATCH 1/3 Upgrade to Buildbot 0.8.3: scripts/poky-setup-autobuilder

I've ported the buildbot install script to python and added an INSTALL file that describes all the changes. This should be run from the base directory:

./scripts/poky-setup-autobuilder [options] master|slave|both

It's fairly straightfoward. I did not include remote machine installs this time around. I also do not create the poky-build user nor set up proxy info. This should all be done on by the build system admin.

---
 INSTALL                        |   55 +++++
 README                         |    6 +-
 scripts/poky-setup-autobuilder |  514 +++++++++++++++++++++++++++-------------
 3 files changed, 407 insertions(+), 168 deletions(-)
 create mode 100644 INSTALL
INSTALL new file +55 -0
new file mode 100644
index 0000000..b7da3e6
@@ -0,0 +1,55 @@
+Basic Setup
+=============================
+
+From the user account that will run the autobuilder:
+
+git clone git://git.pokylinux.org/poky-autobuilder
+cd poky-autobuilder
+./scripts/poky-setup-autobuilder [options] master|slave|both
+
+Options
+=============================
+
+At minimum you should use the --builduser --buildpasswd options. If those are left blank, you'll be prompted for them. 
+
+  --buildpasswd=BBBUILDPASSWD
+                        The BuildBot users password. If this is left empty,
+                        we'll prompt you for it.
+  --builduser=BBBUILDUSER
+                        The BuildBot users password. If this is left empty,
+                        we'll prompt you for it.
+  --gitrepo=BBGITREPO   The Git Repo used to get as poky-autobuilder controler
+                        repo. It's generally safe to ignore this.
+  --masterdir=BBMASTERDIR
+                        The directory you want the master/slave installed to.
+                        If this is left empty, we'll use ~/poky-master.
+  --slavedir=BBSLAVEDIR
+                        The directory you want the master/slave installed to.
+                        If this is left empty, we'll use ~/poky-slave.
+  --outputdir=BBOUTPUTDIR
+                        The directory you want the build output to go to. If
+                        this is left empty, we'll use ~/poky-slave/output.
+  --sourcedir=BBSOURCEDIR
+                        The directory you want the build source to go to. If
+                        this is left empty, we'll use ~/poky-slave/source.
+  --pstagedir=BBPSTAGINGDIR
+                        The directory you want the package staging to occur
+                        in. If this is left empty, we'll use ~/poky-
+                        slave/pstaging.
+  --controldir=BBCONTROLDIR
+                        The directory you want to install poky autobuilder
+                        control scripts into. If this is left empty, we'll use
+                        ~/poky-autobuilder.
+  --maxbuilds=BBMAXBUILDS
+                        The max number of builds you want saved. If this is
+                        left empty, we'll set this to 1.
+  --maxlogs=BBMAXLOGS   The max number of logs you want saved. If this is left
+                        empty, we'll set this to 10.
+  --adminmail=BBADMINMAIL
+                        The administrator email address. If left unset we set
+                        it to current user @ localhost
+
+Master, Slave or Both
+=============================
+
+You must specify if the installation is for a Build controller, a Builder or Both.
README +3 -3
index 41ef130..0c33923 100644
@@ -5,10 +5,10 @@ A lot of the autobuilder setup is individual to the circumstances of the
 user, but this repository contains sample materials which can be used as
 a starting point.
 
-scripts/poky-setup-autobuilder
+scripts/poky-setup-autobuilder [options] master|slave|both
 
 The core script which when run, sets up the current user with a buildbot
-install, and sample master and slave configurations.
+install. Please see INSTALL for more details.
 
 scripts/pokyABConfig.py
 
@@ -36,5 +36,5 @@ server. Designed to be run from cron.
 
 scripts/poky-autobuild-upload-share
 
-The coutnerpart to poky-autobuild-upload which shares the images (and
+The counterpart to poky-autobuild-upload which shares the images (and
 ensures no partly uploaded image is shared). Designed to be run from cron.
scripts/poky-setup-autobuilder +349 -165

544 lines not on this page · raw message

Signature · 1 line
1.7.1
#2 lore raw

PATCH 2/3 Adding new scripts to reflect a real autobuilder

In reply to Beth Flanagan

I've ported over recent script additions to the autobuilder that haven't been in prior commits. Specifically, I've added some of the generate-release scripts, as well as the script that generates the release tarball. I've added some changes to enable Swabber host contamination detection and added the scripts needed for sanity testing.

Also, I've updated the generic pokyABConfig.py to incorporate this new functionality

---
 scripts/poky-autobuild                           |   37 ++-
 scripts/poky-autobuild-cleanoutput               |    6 +-
 scripts/poky-autobuild-copy-images               |  458 +++++++++++++++++++++
 scripts/poky-autobuild-generate-release-external |  220 ++++++++++
 scripts/poky-autobuild-generate-release-internal |  232 +++++++++++
 scripts/poky-autobuild-generate-sources-tarball  |   54 +++
 scripts/poky-autobuild-postprocess               |    6 +-
 scripts/poky-autobuild-preamble                  |   58 +++
 scripts/poky-autobuild-rename-release-images     |  128 ++++++
 scripts/poky-autobuild-sanitytest                |   58 +++
 scripts/poky-output-sort                         |  117 ++++++
 scripts/poky-preserve-work-dir                   |   26 +-
 scripts/pokyABConfig.py                          |  469 +++++++++++++++++-----
 scripts/run-on-buildset-success                  |   11 +
 14 files changed, 1746 insertions(+), 134 deletions(-)
 create mode 100755 scripts/poky-autobuild-copy-images
 create mode 100755 scripts/poky-autobuild-generate-release-external
 create mode 100755 scripts/poky-autobuild-generate-release-internal
 create mode 100755 scripts/poky-autobuild-generate-sources-tarball
 create mode 100755 scripts/poky-autobuild-preamble
 create mode 100755 scripts/poky-autobuild-rename-release-images
 create mode 100755 scripts/poky-autobuild-sanitytest
 create mode 100755 scripts/poky-output-sort
 create mode 100755 scripts/run-on-buildset-success
scripts/poky-autobuild +26 -11
index b4a130b..5ea61bb 100755
@@ -2,7 +2,7 @@
 #
 # Poky Automated Build Server Enviroment Setup Script
 #
-# Copyright (C) 2010 Intel
+# Copyright (C) 2010 Intel Corp.
 #
 # This program is free software; you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -31,13 +31,6 @@ cd $CURRDIR
 
 echo "Changed to $CURRDIR"
 
-if [ "xpreamble" = "x$1" ]; then
-    mkdir -p ./build/tmp/deploy/images 
-    rm -f ./build/tmp/deploy/images/images-complete
-    git log -1 > ./build/tmp/deploy/images/gitinfo
-    exit 0
-fi
-
 BDIR="build"
 . ./scripts/poky-env-internal
 POSTPROCESS=`which poky-autobuild-postprocess`
@@ -58,15 +51,31 @@ if [ ! -e "$CONFFILE" ]; then
     if [ ! -d "./conf" ]; then
         mkdir -p "./conf"
     fi
-    echo 'PACKAGE_CLASSES = "package_ipk package_deb package_rpm"' > "$CONFFILE"
-    echo 'BB_NUMBER_THREADS = "4"' >> "$CONFFILE"
-    echo 'PARALLEL_MAKE = "-j 3"' >> "$CONFFILE"
+    echo 'PACKAGE_CLASSES = "package_rpm package_deb package_ipk"' > "$CONFFILE"
+    echo 'BB_NUMBER_THREADS = "10"' >> "$CONFFILE"
+    echo 'PARALLEL_MAKE = "-j 16"' >> "$CONFFILE"
     echo 'SDKMACHINE ?= "i586"' >> "$CONFFILE"
     echo "DL_DIR = \"$SOURCEDIR\"" >> "$CONFFILE"
     echo 'INHERIT += "poky-autobuild-notifier"' >> "$CONFFILE"
     echo 'INHERIT += "rm_work"' >> "$CONFFILE"
 fi
 
+if [ "x$ENABLE_SWABBER" = "xtrue" ]; then
+    echo 'USER_CLASSES += "image-prelink image-swab"' >> "$CONFFILE"
+fi
+
+# Additions needed to work behind a generic proxy
+#SITECONF="./conf/site.conf"
+#
+#if [ ! -e "$SITECONF" ]; then
+#	echo 'SCONF_VERSION = "1"' > "$SITECONF"
+#	echo 'CVS_PROXY_HOST = "proxy.sitename"' >> "$SITECONF"
+#	echo 'CVS_PROXY_PORT = "proxyportnumber"' >> "$SITECONF"
+#	echo 'GIT_PROXY_HOST = "proxy.sitename"' >> "$SITECONF"
+#	echo 'GIT_PROXY_PORT = "proxyportnumber"' >> "$SITECONF"
+#	echo 'GIT_PROXY_COMMAND = "/usr/local/bin/git-proxy"' >> "$SITECONF"
+#fi
+
 bitbake $@
 
 retval=$?
@@ -76,4 +85,10 @@ if [ "x$POSTPROCESS" != "x" -a -x "$POSTPROCESS" ]; then
     $POSTPROCESS incremental $ABTARGET $TARGETDIR
 fi
 
+if [ $retval -ne 0 ]; then
+    # Create a build failure file flag, which can be used by the
+    # run-on-buildset-success script
+    touch BuildFailure
+fi
+
 exit $retval
scripts/poky-autobuild-cleanoutput +2 -4
index 6c006f3..6b085fc 100755
@@ -1,5 +1,3 @@
 #!/bin/bash
-rm build/tmp/deploy/images/poky-image* || /bin/true
-rm build/tmp/deploy/images/moblin-image* || /bin/true
-rm build/tmp/deploy/images/rootfs* || /bin/true
-rm build/tmp/deploy/sdk/* || /bin/true
\ No newline at end of file
+rm build/tmp/deploy/images/* || /bin/true
+rm build/tmp/deploy/sdk/* || /bin/true
scripts/poky-autobuild-copy-images new file +458 -0

458 lines not on this page · raw message

scripts/poky-autobuild-generate-release-external new file +220 -0

220 lines not on this page · raw message

scripts/poky-autobuild-generate-release-internal new file +232 -0

232 lines not on this page · raw message

scripts/poky-autobuild-generate-sources-tarball new file +54 -0

54 lines not on this page · raw message

scripts/poky-autobuild-postprocess +3 -3
index 5314615..70c59a7 100755
@@ -125,7 +125,7 @@ if [ "x$ABTARGET" = "xpoky-full" ]; then
         `readlink images/zImage-qemuarm.bin`
         `readlink images/zImage-spitz.bin`
         `readlink images/bzImage-qemux86.bin`
-        `readlink images/bzImage-netbook.bin`
+        `readlink images/bzImage-atom-pc.bin`
         `readlink images/zImage-htcuniversal.bin`
         `readlink images/zImage-mx31ads.bin`
         `readlink images/zImage-zylonite.bin`
@@ -137,8 +137,8 @@ if [ "x$ABTARGET" = "xpoky-full" ]; then
         updater.sh.c7x0
         updater.sh.spitz
         gnu-tar
-        poky-image-live-netbook-*.hddimg
-        poky-image-minimal-live-netbook-*.hddimg
+        poky-image-live-atom-pc-*.hddimg
+        poky-image-minimal-live-atom-pc-*.hddimg
         poky-image-sato-akita-*.rootfs.summary.jffs2
         poky-image-sato-c7x0-*.rootfs.jffs2
         poky-image-sato-spitz-*.rootfs.tar.gz
scripts/poky-autobuild-preamble new file +58 -0

58 lines not on this page · raw message

scripts/poky-autobuild-rename-release-images new file +128 -0

128 lines not on this page · raw message

scripts/poky-autobuild-sanitytest new file +58 -0

58 lines not on this page · raw message

scripts/poky-output-sort new file +117 -0

117 lines not on this page · raw message

scripts/poky-preserve-work-dir +20 -6
index da11204..b88535b 100755
@@ -1,15 +1,16 @@
 #!/bin/bash
 #
-# Move the $WORKDIR into a safe area so the Buildbot SCM system can
-# blow away the build directory during updates.
+# Move the tmp dir and shared state cache dir into a safe area so the
+# Buildbot SCM system can blow away the build directory during updates.
 
 CMD=$1
 
 if [ "x$CMD" = "xbackup" ]; then
 	if [ ! -d ./build/conf ]; then
-		echo "No build/conf directory found; assuming this is the first checkout"
-		# Still create a tmp.protected directory so that restore will work
+		echo "No build/conf directory found; assuming this is a build from scratch"
+		# Create empty directores so that restore will work unmodified
 		mkdir ../tmp.protected
+		mkdir ../sstate-cache.protected
 		exit 0
 	fi
 
@@ -20,10 +21,17 @@ if [ "x$CMD" = "xbackup" ]; then
 	fi
 
 	mv build/tmp ../tmp.protected
-	if [ ! -e ../tmp.protected ]; then
+	if [ ! -d ../tmp.protected ]; then
 		echo "Error moving build/tmp to a protected area"
 		exit 1
 	fi
+
+	mv build/sstate-cache ../sstate-cache.protected
+	if [ ! -d ../sstate-cache.protected ]; then
+		echo "Error moving build/sstate-cache to a protected area"
+		exit 1
+	fi
+
 elif [ "x$CMD" == "xrestore" ]; then
 	if [ ! -d ../tmp.protected ]; then
 		PWD=`pwd`
@@ -33,10 +41,16 @@ elif [ "x$CMD" == "xrestore" ]; then
 
 	mkdir build
 	mv ../tmp.protected build/tmp
-	if [ ! -e build/tmp ]; then
+	if [ ! -d build/tmp ]; then
 		echo "Error moving tmp.protected dir back into build/tmp"
 		exit 1
 	fi
+
+	mv ../sstate-cache.protected build/sstate-cache
+	if [ ! -d build/sstate-cache ]; then
+		echo "Error moving sstate-cache.protected dir back into build/sstate-cache"
+		exit 1
+	fi
 else
 	echo "Unknown command '$CMD'"
 	echo "Commands are 'backup' or 'restore'"
scripts/pokyABConfig.py +359 -110

735 lines not on this page · raw message

scripts/run-on-buildset-success new file +11 -0
new file mode 100755
index 0000000..e1428da
@@ -0,0 +1,11 @@
+#!/bin/bash
+#
+# Run the command given as arguments only if the build does not have
+# a 'BuildFailure' file in its build directory.
+
+if [ -e "BuildFailure" ]; then
+	echo "Build failure detected in this build set - NOT running '$@'"
+	exit 0
+fi
+
+$@
Signature · 1 line
1.7.1
#3 lore raw

PATCH 3/3 Change to poky-autobuild to source correct file:

In reply to Beth Flanagan

A change to poky-init-build-env in poky master broke the autobuilder. Switching this to source poky-init-build-env instead of poky-env-internal.

---
 scripts/poky-autobuild |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
scripts/poky-autobuild +1 -1
index 5ea61bb..882df97 100755
@@ -32,7 +32,7 @@ cd $CURRDIR
 echo "Changed to $CURRDIR"
 
 BDIR="build"
-. ./scripts/poky-env-internal
+. ./scripts/poky-init-build-env
 POSTPROCESS=`which poky-autobuild-postprocess`
 
 if [ "xcomplete" = "x$1" ]; then
Signature · 1 line
1.7.1
#4 lore raw
In reply to Beth Flanagan
Quoted 42 lines · Beth Flanagan

Scott,

This pull request updates the buildbot installer to support BuildBot 0.8.3, which was released yesterday, as well as updated build helper scripts and a new pokyABConfig.py which adds a swabber testing target,implements all current autobuilder build arches and lastly, implements sanity testing.

The buildbot installer script has been re-written in python and an INSTALL file added to explain the new installation process. I've tested it on Ubuntu 10.10 and openSUSE 11.3, and would appreciate anyone who has a different system letting me know their experiences with it (good and bad).

I've also incorporated a fix to poky-autobuild that sources the correct env script and added some missing build helper scripts, based off of what is actually on the autobuilder.

-b

Pull URL: git://git.pokylinux.org/poky-contrib.git
Branch: eflanagan/bb-0.8.3
Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=eflanagan/bb-0.8.3

Thanks,
Beth Flanagan<[email protected]>
---

Beth Flanagan (3):
Upgrade to Buildbot 0.8.3: scripts/poky-setup-autobuilder Adding new scripts to reflect a real autobuilder Change to poky-autobuild to source correct file:

  INSTALL                                          |   55 +++
  README                                           |    6 +-
  scripts/poky-autobuild                           |   39 ++-
  scripts/poky-autobuild-cleanoutput               |    6 +-
  scripts/poky-autobuild-copy-images               |  458 +++++++++++++++++++
  scripts/poky-autobuild-generate-release-external |  220 +++++++++
  scripts/poky-autobuild-generate-release-internal |  232 ++++++++++
  scripts/poky-autobuild-generate-sources-tarball  |   54 +++
  scripts/poky-autobuild-postprocess               |    6 +-
  scripts/poky-autobuild-preamble                  |   58 +++
  scripts/poky-autobuild-rename-release-images     |  128 ++++++
  scripts/poky-autobuild-sanitytest                |   58 +++
  scripts/poky-output-sort                         |  117 +++++
  scripts/poky-preserve-work-dir                   |   26 +-
  scripts/poky-setup-autobuilder                   |  514 +++++++++++++++-------
  scripts/pokyABConfig.py                          |  469 +++++++++++++++-----
  scripts/run-on-buildset-success                  |   11 +
  17 files changed, 2154 insertions(+), 303 deletions(-)
Quoted 10 lines

create mode 100644 INSTALL
create mode 100755 scripts/poky-autobuild-copy-images create mode 100755 scripts/poky-autobuild-generate-release-external create mode 100755 scripts/poky-autobuild-generate-release-internal create mode 100755 scripts/poky-autobuild-generate-sources-tarball create mode 100755 scripts/poky-autobuild-preamble create mode 100755 scripts/poky-autobuild-rename-release-images create mode 100755 scripts/poky-autobuild-sanitytest create mode 100755 scripts/poky-output-sort
create mode 100755 scripts/run-on-buildset-success

After a few iterations with Beth, I have now pulled this into poky-autobuilder master.

Thanks Beth!

Scott

Signature · 2 lines
Scott Garman
Embedded Linux Distro Engineer - Yocto Project