Autobuilder fixes and added functionality

Some minor fixes to the Autobuilder. I've reverted to Buildbot 0.8.2 for the time being, as well as fixed some issues with poky-autobuild-generate-sources-tarball.

There is also a nasty issue when you have easy_install installed on a system. The BB installer will attempt to install via easy_install, however, if the install directory does not exist, it fails to install. I now create the install directories beforehand.

This is actually something that should be fixed on the BB side, and when I have some time I'll look at it.

I've also added a --installbase option which does away with having to set all of the --(*)dir= options when you are trying to relocate the install from ~.

Pull URL: git://git.pokylinux.org/poky-contrib.git
Branch: eflanagan/bb-0.8.3
Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=eflanagan/bb-0.8.3

Thanks,
Beth Flanagan <[email protected]>
---

Beth Flanagan (1):
Fix for easy_install, source tarball and 0.8.3

 scripts/poky-autobuild-generate-sources-tarball |    2 +-
 scripts/poky-setup-autobuilder                  |   27 +++++++++++++++-------
 2 files changed, 19 insertions(+), 10 deletions(-)

A few fixes here. First, reverting down to 0.8.2 for the time being since there was issues with 0.8.3's git poller. Also, fixing an issue with how bb setup.py works when setuptools is installed.

Fixing a known error in how generate sources tarball performs

---
 scripts/poky-autobuild-generate-sources-tarball |    2 +-
 scripts/poky-setup-autobuilder                  |   27 +++++++++++++++-------
 2 files changed, 19 insertions(+), 10 deletions(-)

index 11ebda1..c445f24 100755
@@ -19,7 +19,7 @@ if [[ -z "$RELEASE" || -z "$VERSION" ]]; then
 fi
 
 if [[ -z "$BRANCH" ]]; then
-	$BRANCH = "master"
+	BRANCH = "master"
 fi
 
 BASEDIR=poky-tarball

index 51e06eb..ad31fef 100755
@@ -140,22 +140,23 @@ def configureBot(buildtype):
             bbInstallDir = bbMasterDir
         elif buildtype == "slave":
             bbInstallDir = bbSlaveDir
-        cmd = "cd " + bbSourceDir + "; export PYTHONPATH=" + bbInstallDir + "/lib/python2.6/site-packages/:$PYTHONPATH; python ./setup.py install --prefix=" + bbInstallDir
+        cmd = "cd " + bbSourceDir + "; export PYTHONPATH=" + bbInstallDir + "/lib/python2.6/site-packages/:$PYTHONPATH; python ./setup.py build; python ./setup.py install --prefix=" + bbInstallDir
         os.system (cmd) 
     except:
         print "Issues Configuring "
         sys.exit(1)
 
 
-#BuildBot download parameters
-bbVersion = "0.8.3"
+# BuildBot download parameters
+#bbVersion = "0.8.3"
+# Pushing this back down to 0.8.2 until 0.8.3p1 comes out.
+bbVersion = "0.8.2"
 bbMasterDownloadUrl = "http://buildbot.googlecode.com/files/buildbot-" + bbVersion + ".tar.gz"
 bbSlaveDownloadUrl = "http://buildbot.googlecode.com/files/buildbot-slave-" + bbVersion + ".tar.gz"
 bbConfigDownloadUrl = None
 bbInstallerHome = os.getcwd()
 bbInstallFile = ""
-# We need the expanded path for ~ later
-bbHome = os.path.expanduser('~')
+bbHome = ""
 bbType = None
 bbSlaveDesc="Poky Autobuilder Example"
 

@@ -188,7 +189,8 @@ parser.add_option( "--maxlogs", help = "The max number of logs you want saved. I
                        action = "store", dest = "bbMaxLogs", default = "10" )                        
 parser.add_option( "--adminmail", help = "The administrator email address. If left unset we set it to current user @ localhost",
                        action = "store", dest = "bbAdminMail", default = "root@localhost" )                        
-
+parser.add_option( "--installbase", help = "The base install directory. If left unset we set we'll use --masterdir/slavedir/sourcedir/pstagedir/controldir or their defaults",
+                       action = "store", dest = "bbHome", default = os.path.expanduser('~') )          
 options, args = parser.parse_args( sys.argv )
 
 # We need to decide if we're doing a master or slave install or both

@@ -236,7 +238,10 @@ if bbPStagingDir == "":
 
 if bbType == "master" or bbType == "both":
     try:
-        os.mkdir(bbMasterDir)
+        # We do this here, because if setuputils is installed BuildBot wants to use it the easy_install way
+        # however, their easy_install way will fail if the dir does not exist. It's a bug in the BB installer
+        # that I'm going to have to figure out and push upstream.
+        os.makedirs(bbMasterDir + "/lib/python2.6/site-packages/")
     except:
         print bbMasterDir + " already exists."
         pass

@@ -332,7 +337,10 @@ c['projectURL'] = pokyABConfig.poky_projurl
 
 if bbType == "slave" or bbType == "both":
     try:
-        os.mkdir(bbSlaveDir) 
+        # We do this here, because if setuputils is installed BuildBot wants to use it the easy_install way
+        # however, their easy_install way will fail if the dir does not exist. It's a bug in the BB installer
+        # that I'm going to have to figure out and push upstream.
+        os.makedirs(bbSlaveDir + "/lib/python2.6/site-packages/") 
         os.mkdir(bbOutputDir) 
         os.mkdir(bbPStagingDir) 
 

@@ -400,7 +408,7 @@ PSTAGEDIR=%s
 # We should correct this for slave only/master only builds
 
 print """
-nstallation complete. Please review the output above for any errors.
+Installation complete. Please review the output above for any errors.
 Then edit the master.cfg file in %s and start the build master and
 build slave by running 'make start' in %s and %s directories.
 ---------------------------------------------------------------------

@@ -418,3 +426,4 @@ cd <poky-slave>; make start
 """ % (bbMasterDir, bbMasterDir, bbSlaveDir, bbSlaveDir, bbSlaveDir, bbSlaveDir)
 
 
+

1.7.1

FYI I've attempted to install the autobuilder on my Fedora build box, however I ran into a few issues:

1) If you git clone poky-autobuilder into your home directory the install script falls over because it tries to create that same named directory by default. Of course you can override this path or clone it somewhere else but it would be nice if it could work out of the box.

2) The install script doesn't handle Python 2.7, but only due to paths. I have a patch for this which I'll post when your latest patches are merged.

3) BuildBot 0.8.2 unfortunately suffers from buildbot bug #1697 on my system, making it not very useful (as you can't view any of the build logs through the web interface). It's fixed in 0.8.3. More details:

http://trac.buildbot.net/ticket/1697

4) Looking through the logs locally I found some disturbing output from poky, I guess I may have missed some configuration option, or otherwise something else is going badly wrong:

----- snip -----
NOTE: package quilt-native-0.48-r0: task do_fetch: Started
ERROR: Error, lockfile path is not writable!: /
NOTE: package gnu-config-native-0.1+cvs20080123-r1: task do_fetch: Started
NOTE: package m4-native-1.4.15-r0: task do_fetch: Started
ERROR: Error, lockfile path is not writable!: /
NOTE: package autoconf-native-2.65-r2: task do_fetch: Started
ERROR: Error, lockfile path is not writable!: /
NOTE: package automake-native-1.11.1-r1: task do_fetch: Started
ERROR: Error, lockfile path is not writable!: /
NOTE: package libtool-native-2.4-r0: task do_fetch: Started
ERROR: Error, lockfile path is not writable!: /
NOTE: package zlib-native-1.2.5-r0: task do_fetch: Started
ERROR: Error, lockfile path is not writable!: /
NOTE: package unifdef-native-2.6.18+git-r0: task do_unpack: Started
NOTE: package ncurses-native-5.7-r0: task do_fetch: Started
ERROR: Error, lockfile path is not writable!: /
ERROR: Function 'Unknown fetch Error: 1' failed
NOTE: package quilt-native-0.48-r0: task Unknown fetch Error: 1: Failed ----- snip -----

(Interesting also that the function / task names are being replaced with the result message, is this expected behaviour from poky?)

5) I did try with 0.8.3 earlier (i.e. without your patch), but I suspect I was hit by the git poller problem that 0.8.3p1 is supposed to fix as there were a lot of strange bitbake parsing errors. I also found that I needed to have python-jinja2 and python-setuptools installed, whereas with your latest patch using 0.8.2 it seemed to be installing local copies of these (along with twisted).

Cheers,
Paul

Paul,

Yes, that's an issue I know about. Not sure how to handle it other than renaming either the git directory or the default autobuilder scripts home.

Excellent. Ty.

I hadn't run into this, so thanks for pointing it out. I'm tempted to wait for 0.8.3p1 which should have the fix for this, rather than trying to pull the fix out to patch 0.8.2

I found that error yesterday and will have a fix for it today.

I don't recall needed setuptools, but that may be just because I tested on my desktop. I'll look into this when I push it to 0.8.3p1

-b

Hi Beth,

Do you want me to merge this patchset into master now, or should I wait for a respin based on Buildbot 0.8.3p1 (which should be out by the end of the week, I think)?

Scott

Scott Garman
Embedded Linux Distro Engineer - Yocto Project

Let's wait on all this. I'm running into a few issues with the AB that I want to fix that are unrelated to the upgrade and would rather do one big patchset that did the upgrade and fixed some of the issues I'm trying to fix.

I'll keep everyone posted. That said, we should probably discuss tagging/versioning process for non-poky projects within yocto. If I remember correctly, there was discussion about this going on already, and I'm a little uneasy leaving the ab master broken without a tag to a known good version. Anyone have any thoughts on this?

-b
________________________________________
From: [email protected] [[email protected]] On Behalf Of Scott Garman [[email protected]]
Sent: Wednesday, January 05, 2011 9:22 AM
To: [email protected]
Subject: Re: [yocto] [PATCH 0/1] Autobuilder fixes and added functionality

Hi Beth,

Do you want me to merge this patchset into master now, or should I wait for a respin based on Buildbot 0.8.3p1 (which should be out by the end of the week, I think)?

Scott

--
Scott Garman
Embedded Linux Distro Engineer - Yocto Project
_______________________________________________
yocto mailing list
[email protected]
https://lists.yoctoproject.org/listinfo/yocto