Skip to contentSkip to conversation

BitBake memory-resident mode fixes

PATCH New Paul Eggleton 1 participant 5 messages Jun 9, 2017

lore
Share & tools
lore
#0 lore raw

cover BitBake memory-resident mode fixes

Three fixes for memres and one optimisation that I noticed while working on these.

The following changes since commit 705ab252e631903e6d2e46202b419a9e8adcd861:

bitbake-layers: check layer dependencies before adding (2017-06-05 13:33:14 +0100)

are available in the git repository at:

git://git.yoctoproject.org/poky-contrib paule/bb-memres-fixes
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=paule/bb-memres-fixes

Paul Eggleton (4):
cookerdata: fix variable history not showing in bitbake -e with memres
cooker: fix file changes not triggering task re-execution with memres
cooker: set config valid flags in initConfigurationData()
cooker: add BB_CMDLINE to enable access to UI command line with memres

 lib/bb/command.py    |  3 ++-
 lib/bb/cooker.py     | 29 +++++++++++++++++++++--------
 lib/bb/cookerdata.py |  4 ++--
 3 files changed, 25 insertions(+), 11 deletions(-)
Signature · 1 line
2.9.4
#1 lore raw

PATCH 1/4 cookerdata: fix variable history not showing in bitbake -e with memres

CookerConfiguration sets the "tracking" option to True when -e is specified in order to have history tracking enabled in the datastore so that we can show variable history (which isn't enabled by default for performance reasons), however in memory resident mode this wasn't doing anything because it was happening on the UI side only. We do have a mechanism for updating the cooker configuration in the server, but the tracking option wasn't being included in the list of options we updated, so we just need to add this option to fix the issue.

Fixes [YOCTO #10730].

---
 lib/bb/cookerdata.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
lib/bb/cookerdata.py +1 -1
index 722d860..6b850d3 100644
@@ -76,7 +76,7 @@ class ConfigParameters(object):
         for o in ["abort", "tryaltconfigs", "force", "invalidate_stamp", 
                   "verbose", "debug", "dry_run", "dump_signatures", 
                   "debug_domains", "extra_assume_provided", "profile",
-                  "prefile", "postfile"]:
+                  "prefile", "postfile", "tracking"]:
             options[o] = getattr(self.options, o)
 
         ret, error = server.runCommand(["updateConfig", options, environment])
Signature · 1 line
2.9.4
#2 lore raw

PATCH 2/4 cooker: fix file changes not triggering task re-execution with memres

In reply to Paul Eggleton

If you build a recipe, modify one of the files referred to in SRC_URI, then build it again, it should re-run do_fetch and the tasks that depend upon it to incorporate the modified file. With memory resident mode this was not working unless you restarted the server, because the mtime of the file was cached and thus we never checked the actual file. Clear out the mtime caches in buildTargets() to ensure this isn't an issue. I did a minor refactoring to break this out to its own function since we were already clearing out the build mtime cache here.

(The test I used for this was . ./oe-init-build-env-memres, bitbake -c fetch mdadm, modify the comments at the top of one of the patches referred to in the recipe, and then bitbake -c fetch mdadm again and it should re-execute the fetch task ("... 0 didn't need to be re-run".)

Fixes [YOCTO #10732].

---
 lib/bb/cooker.py | 14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)
lib/bb/cooker.py +13 -1
index 8846690..651b9c0 100644
@@ -1313,6 +1313,18 @@ class BBCooker:
         self.data.setVar("DATE", time.strftime('%Y%m%d', t))
         self.data.setVar("TIME", time.strftime('%H%M%S', t))
 
+    def reset_mtime_caches(self):
+        """
+        Reset mtime caches - this is particularly important when memory resident as something
+        which is cached is not unlikely to have changed since the last invocation (e.g. a
+        file associated with a recipe might have been modified by the user).
+        """
+        build.reset_cache()
+        bb.fetch._checksum_cache.mtime_cache.clear()
+        siggen_cache = getattr(bb.parse.siggen, 'checksum_cache', None)
+        if siggen_cache:
+            bb.parse.siggen.checksum_cache.mtime_cache.clear()
+
     def matchFiles(self, bf):
         """
         Find the .bb files which match the expression in 'buildfile'.
@@ -1490,7 +1502,7 @@ class BBCooker:
                 return True
             return retval
 
-        build.reset_cache()
+        self.reset_mtime_caches()
         self.buildSetVars()
 
         # If we are told to do the None task then query the default task
Signature · 1 line
2.9.4
#3 lore raw

PATCH 3/4 cooker: set config valid flags in initConfigurationData()

In reply to Paul Eggleton

If we set these flags here then we can not only tidy up some of the code that calls initConfigurationData() (and reset() which simply calls the former), we also avoid calling initConfigurationData() an extra time unnecessarily during startup (since setting baseconfig_valid = False in updateConfigOpts() also resulted in initConfigurationData() being called from updateCacheSync() at the end of the command.)

---
 lib/bb/cooker.py | 8 ++------
 1 file changed, 2 insertions(+), 6 deletions(-)
lib/bb/cooker.py +2 -6
index 651b9c0..1283136 100644
@@ -213,9 +213,6 @@ class BBCooker:
 
         self.configuration.server_register_idlecallback(_process_inotify_updates, [self.confignotifier, self.notifier])
 
-        self.baseconfig_valid = True
-        self.parsecache_valid = False
-
         # Take a lock so only one copy of bitbake can run against a given build
         # directory at a time
         if not self.lockBitbake():
@@ -375,6 +372,8 @@ class BBCooker:
         self.data.renameVar("__depends", "__base_depends")
         self.add_filewatch(self.data.getVar("__base_depends", False), self.configwatcher)
 
+        self.baseconfig_valid = True
+        self.parsecache_valid = False
 
     def enableDataTracking(self):
         self.configuration.tracking = True
@@ -568,7 +567,6 @@ class BBCooker:
                 clean = False
         if not clean:
             logger.debug(1, "Base environment change, triggering reparse")
-            self.baseconfig_valid = False        
             self.reset()
 
     def runCommands(self, server, data, abort):
@@ -1623,8 +1621,6 @@ class BBCooker:
         if not self.baseconfig_valid:
             logger.debug(1, "Reloading base configuration data")
             self.initConfigurationData()
-            self.baseconfig_valid = True
-            self.parsecache_valid = False
 
     # This is called for all async commands when self.state != running
     def updateCache(self):
Signature · 1 line
2.9.4
#4 lore raw

PATCH 4/4 cooker: add BB_CMDLINE to enable access to UI command line with memres

In reply to Paul Eggleton

In OpenEmbedded's buildhistory class we want access to the bitbake command line that launched the build, and up to now we were simply using sys.argv from within the event handler to get that. Unfortunately that doesn't work in memory resident mode, since the event handler is naturally executing within the server and thus will give you the command that launched the bitbake server which is much less interesting. Add a dynamic variable BB_CMDLINE to provide access to this, set from sys.argv within the UI process in updateToServer().

(Note that BB_CMDLINE isn't currently passed through to the worker, so this is only really readable from event handlers plus any explicit getVariable calls - in theory an observe-only UI could read it for example.)

Part of the fix for [YOCTO #11634].

---
 lib/bb/command.py    | 3 ++-
 lib/bb/cooker.py     | 7 ++++++-
 lib/bb/cookerdata.py | 2 +-
 3 files changed, 9 insertions(+), 3 deletions(-)
lib/bb/command.py +2 -1
index ab51d8e..a919f58 100644
@@ -310,7 +310,8 @@ class CommandsSync:
     def updateConfig(self, command, params):
         options = params[0]
         environment = params[1]
-        command.cooker.updateConfigOpts(options, environment)
+        cmdline = params[2]
+        command.cooker.updateConfigOpts(options, environment, cmdline)
     updateConfig.needconfig = False
 
     def parseConfiguration(self, command, params):
lib/bb/cooker.py +6 -1
index 1283136..45b5a61 100644
@@ -192,6 +192,8 @@ class BBCooker:
         bb.parse.__mtime_cache = {}
         bb.parse.BBHandler.cached_statements = {}
 
+        self.ui_cmdline = None
+
         self.initConfigurationData()
 
         # we log all events to a file if so directed
@@ -360,6 +362,8 @@ class BBCooker:
         if consolelog:
             self.data.setVar("BB_CONSOLELOG", consolelog)
 
+        self.data.setVar('BB_CMDLINE', self.ui_cmdline)
+
         #
         # Copy of the data store which has been expanded.
         # Used for firing events and accessing variables where expansion needs to be accounted for
@@ -539,7 +543,8 @@ class BBCooker:
 
         self.handleCollections(self.data.getVar("BBFILE_COLLECTIONS"))
 
-    def updateConfigOpts(self, options, environment):
+    def updateConfigOpts(self, options, environment, cmdline):
+        self.ui_cmdline = cmdline
         clean = True
         for o in options:
             if o in ['prefile', 'postfile']:
lib/bb/cookerdata.py +1 -1
index 6b850d3..62aa055 100644
@@ -79,7 +79,7 @@ class ConfigParameters(object):
                   "prefile", "postfile", "tracking"]:
             options[o] = getattr(self.options, o)
 
-        ret, error = server.runCommand(["updateConfig", options, environment])
+        ret, error = server.runCommand(["updateConfig", options, environment, sys.argv])
         if error:
                 raise Exception("Unable to update the server configuration with local parameters: %s" % error)
 
Signature · 1 line
2.9.4