Skip to contentSkip to conversation

bitbake: fixes for dora

PATCH New Robert Yang 1 participant 6 messages Dec 10, 2013

lore
Share & tools
lore
#0 lore raw

cover bitbake: fixes for dora

The following changes since commit 53d2563ff13fcec74d4250bef5419e36169e55cc:

bitbake: perforce: Fix path subdirectory issues (2013-12-03 17:51:31 +0000)

are available in the git repository at:

git://git.pokylinux.org/poky-contrib robert/dora-next-bitbake
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=robert/dora-next-bitbake

Olof Johansson (1):
bitbake: monitordisk: lower inode check warning to note

Peter Seebach (1):
bitbake: build.py: add single-quotes around already-expanded
directory name

Richard Purdie (3):
bitbake: cooker/command: Add error state for the server and use for
pre_serve errors
bitbake: fetch2: Fix handling of SCM mirrors in MIRRORS
bitbake: runqueue: Fix hole in setsceneverify skipped task logic

 bitbake/lib/bb/build.py           |    2 +-
 bitbake/lib/bb/command.py         |    2 ++
 bitbake/lib/bb/cooker.py          |    3 ++-
 bitbake/lib/bb/fetch2/__init__.py |    4 ++++
 bitbake/lib/bb/monitordisk.py     |    2 +-
 bitbake/lib/bb/runqueue.py        |   21 +++++++++++++++++++--
 6 files changed, 29 insertions(+), 5 deletions(-)
Signature · 1 line
1.7.10.4
#1 lore raw

PATCH 1/5 bitbake: cooker/command: Add error state for the server and use for pre_serve errors

From: Richard Purdie <[email protected]>

Currently if errors occur when starting the PR service, there is a race that occurs since the UI runs various commands including starting builds before processing the CookerExit(). By adding the error state and refusing to run async commands in this mode, builds are prevented from starting and the UI reaches the exit code with the system shutting down cleanly.

(Bitbake master rev: 42fa34142ea685f91115a551e74416ca28ef1c91)

---
 bitbake/lib/bb/command.py |    2 ++
 bitbake/lib/bb/cooker.py  |    3 ++-
 2 files changed, 4 insertions(+), 1 deletion(-)
bitbake/lib/bb/command.py +2 -0
index 3ca27a6..a2795ce 100644
@@ -86,6 +86,8 @@ class Command:
 
     def runAsyncCommand(self):
         try:
+            if self.cooker.state == bb.cooker.state.error:
+                return False
             if self.currentAsyncCommand is not None:
                 (command, options) = self.currentAsyncCommand
                 commandmethod = getattr(CommandsAsync, command)
bitbake/lib/bb/cooker.py +2 -1
index ad36b34..b504f45 100644
@@ -61,7 +61,7 @@ class CollectionError(bb.BBHandledException):
     """
 
 class state:
-    initial, parsing, running, shutdown, forceshutdown, stopped = range(6)
+    initial, parsing, running, shutdown, forceshutdown, stopped, error = range(7)
 
 
 class SkippedPackage:
@@ -1321,6 +1321,7 @@ class BBCooker:
             self.prhost = prserv.serv.auto_start(self.data)
         except prserv.serv.PRServiceConfigError:
             bb.event.fire(CookerExit(), self.event_data)
+            self.state = state.error
         return
 
     def post_serve(self):
Signature · 1 line
1.7.10.4
#2 lore raw

PATCH 2/5 bitbake: monitordisk: lower inode check warning to note

In reply to Robert Yang

From: Olof Johansson <[email protected]>

Filesystems like btrfs and reiserfs sets the inode count to 0, since they don't have an inode concept. This is expected, and having a warning show up every time you run bitbake can cause undue concern.

(Bitbake master rev: f3ac2d3678f48c68a250a0a20c08cf8687322d38)

---
 bitbake/lib/bb/monitordisk.py |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
bitbake/lib/bb/monitordisk.py +1 -1
index 3e6ecbd..fca43ee 100644
@@ -243,7 +243,7 @@ class diskMonitor:
                     # zero, this is a feature of the fs, we disable the inode
                     # checking for such a fs.
                     if st.f_files == 0:
-                        logger.warn("Inode check for %s is unavaliable, will remove it from disk monitor" % path)
+                        logger.info("Inode check for %s is unavaliable, will remove it from disk monitor" % path)
                         self.devDict[k][2] = None
                         continue
                     # Always show warning, the self.checked would always be False if the action is WARN
Signature · 1 line
1.7.10.4
#3 lore raw

PATCH 3/5 bitbake: fetch2: Fix handling of SCM mirrors in MIRRORS

In reply to Robert Yang

From: Richard Purdie <[email protected]>

If an SCM mirror is in PREMIRRORS, the tarball is downloaded and then found by the "upstream" check and handled correctly.

If an SCM mirror is in MIRRORS, the tarball is downloaded but not used since there is no "upstream" run after MIRRORS completes. It therefore sits there useless and unused. This code change forces the upstream to run after a mirror tarball is found and fixes the usage of SCM mirrors in MIRRORS.

(Bitbake master rev: a66ee0994645aa5658b2f5ea134ed17d89f8751a)

---
 bitbake/lib/bb/fetch2/__init__.py |    4 ++++
 1 file changed, 4 insertions(+)
bitbake/lib/bb/fetch2/__init__.py +4 -0
index 451d104..b9f673c 100644
@@ -805,6 +805,10 @@ def try_mirror_url(newuri, origud, ud, ld, check = False):
             dest = os.path.join(dldir, os.path.basename(ud.localpath))
             if not os.path.exists(dest):
                 os.symlink(ud.localpath, dest)
+            if not os.path.exists(origud.donestamp) or origud.method.need_update(origud.url, origud, ld):
+                origud.method.download(origud.url, origud, ld)
+                if hasattr(ud.method,"build_mirror_data"):
+                    origud.method.build_mirror_data(origud.url, origud, ld)
             return None
         # Otherwise the result is a local file:// and we symlink to it
         if not os.path.exists(origud.localpath):
Signature · 1 line
1.7.10.4
#4 lore raw

PATCH 4/5 bitbake: build.py: add single-quotes around already-expanded directory name

In reply to Robert Yang

From: Peter Seebach <[email protected]>

If the computed name of a directory contains an undefined variable reference, bitbake dutifully creates a directory with a name that has ${...} in it. However, the actual task script created then tries to cd to that directory, and the cd command fails, because no such directory exists -- because the shell has helpfully removed the ${...} which did not match any actual variables.

Since we want the name to be used exactly-as-is, add single quotes around the name so this doesn't cause strange failures running tasks, which allows us to progress past such failures and get to a point where they can be diagnosed.

(Bitbake master rev: 2809c2e6f2f35f9b08058950be896947ab5a0284)

---
 bitbake/lib/bb/build.py |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
bitbake/lib/bb/build.py +1 -1
index 2e49a09..edab556 100644
@@ -285,7 +285,7 @@ set -e
         if bb.msg.loggerVerboseLogs:
             script.write("set -x\n")
         if cwd:
-            script.write("cd %s\n" % cwd)
+            script.write("cd '%s'\n" % cwd)
         script.write("%s\n" % func)
         script.write('''
 # cleanup
Signature · 1 line
1.7.10.4
#5 lore raw

PATCH 5/5 bitbake: runqueue: Fix hole in setsceneverify skipped task logic

In reply to Robert Yang

From: Richard Purdie <[email protected]>

We have do_bundle_initramfs which is a task inserted after compile and before build. It is not covered by sstate.

If we run a build with a valid sstate cache present, the setsceneverify function realises it will rerun the do_compile step (due to the bundle_initramfs task) and hence marks do_populate_sysroot to rerun. do_install, a dependency of do_populate_sysroot is left as marked as covered by sstate.

What we need to do is traverse the dependency tree for any setsceneverify invalided task and ensure any dependencies are also invalidated. We can stop at any point we reach another setscene task though.

This means the do_populate_sysroot task has the data from do_install available and doesn't crash.

(Bitbake master rev: f21910157d873c030b149c4cdc5b57c5062ab5a6)

---
 bitbake/lib/bb/runqueue.py |   21 +++++++++++++++++++--
 1 file changed, 19 insertions(+), 2 deletions(-)
bitbake/lib/bb/runqueue.py +19 -2
index c09cfd4..72c0208 100644
@@ -1204,6 +1204,8 @@ class RunQueueExecuteTasks(RunQueueExecute):
 
         self.stampcache = {}
 
+        initial_covered = self.rq.scenequeue_covered.copy()
+
         # Mark initial buildable tasks
         for task in xrange(self.stats.total):
             self.runq_running.append(0)
@@ -1257,13 +1259,28 @@ class RunQueueExecuteTasks(RunQueueExecute):
             except TypeError:
                 covered_remove = bb.utils.better_eval(call2, locs)
 
-        for task in covered_remove:
+        def removecoveredtask(task):
             fn = self.rqdata.taskData.fn_index[self.rqdata.runq_fnid[task]]
             taskname = self.rqdata.runq_task[task] + '_setscene'
             bb.build.del_stamp(taskname, self.rqdata.dataCache, fn)
-            logger.debug(1, 'Not skipping task %s due to setsceneverify', task)
             self.rq.scenequeue_covered.remove(task)
 
+        toremove = covered_remove
+        for task in toremove:
+            logger.debug(1, 'Not skipping task %s due to setsceneverify', task)
+        while toremove:
+            covered_remove = []
+            for task in toremove:
+                removecoveredtask(task)
+                for deptask in self.rqdata.runq_depends[task]:
+                    if deptask not in self.rq.scenequeue_covered:
+                        continue
+                    if deptask in toremove or deptask in covered_remove or deptask in initial_covered:
+                        continue
+                    logger.debug(1, 'Task %s depends on task %s so not skipping' % (task, deptask))
+                    covered_remove.append(deptask)
+            toremove = covered_remove
+
         logger.debug(1, 'Full skip list %s', self.rq.scenequeue_covered)
 
         event.fire(bb.event.StampUpdate(self.rqdata.target_pairs, self.rqdata.dataCache.stamp), self.cfgData)
Signature · 1 line
1.7.10.4