BitBake memory-resident mode fixes
cover BitBake memory-resident mode fixes
Three fixes for memres and one optimisation that I noticed while working on these.
The following changes since commit 705ab252e631903e6d2e46202b419a9e8adcd861:
bitbake-layers: check layer dependencies before adding (2017-06-05 13:33:14 +0100)
are available in the git repository at:
git://git.yoctoproject.org/poky-contrib paule/bb-memres-fixes
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=paule/bb-memres-fixes
Paul Eggleton (4):
cookerdata: fix variable history not showing in bitbake -e with memres
cooker: fix file changes not triggering task re-execution with memres
cooker: set config valid flags in initConfigurationData()
cooker: add BB_CMDLINE to enable access to UI command line with memres
lib/bb/command.py | 3 ++- lib/bb/cooker.py | 29 +++++++++++++++++++++-------- lib/bb/cookerdata.py | 4 ++-- 3 files changed, 25 insertions(+), 11 deletions(-)
Signature · 1 line
2.9.4
PATCH 1/4 cookerdata: fix variable history not showing in bitbake -e with memres
CookerConfiguration sets the "tracking" option to True when -e is specified in order to have history tracking enabled in the datastore so that we can show variable history (which isn't enabled by default for performance reasons), however in memory resident mode this wasn't doing anything because it was happening on the UI side only. We do have a mechanism for updating the cooker configuration in the server, but the tracking option wasn't being included in the list of options we updated, so we just need to add this option to fix the issue.
Fixes [YOCTO #10730].
--- lib/bb/cookerdata.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
lib/bb/cookerdata.py +1 -1
@@ -76,7 +76,7 @@ class ConfigParameters(object): for o in ["abort", "tryaltconfigs", "force", "invalidate_stamp", "verbose", "debug", "dry_run", "dump_signatures", "debug_domains", "extra_assume_provided", "profile", - "prefile", "postfile"]: + "prefile", "postfile", "tracking"]: options[o] = getattr(self.options, o) ret, error = server.runCommand(["updateConfig", options, environment])
Signature · 1 line
2.9.4
PATCH 2/4 cooker: fix file changes not triggering task re-execution with memres
In reply to Paul EggletonIf you build a recipe, modify one of the files referred to in SRC_URI, then build it again, it should re-run do_fetch and the tasks that depend upon it to incorporate the modified file. With memory resident mode this was not working unless you restarted the server, because the mtime of the file was cached and thus we never checked the actual file. Clear out the mtime caches in buildTargets() to ensure this isn't an issue. I did a minor refactoring to break this out to its own function since we were already clearing out the build mtime cache here.
(The test I used for this was . ./oe-init-build-env-memres, bitbake -c fetch mdadm, modify the comments at the top of one of the patches referred to in the recipe, and then bitbake -c fetch mdadm again and it should re-execute the fetch task ("... 0 didn't need to be re-run".)
Fixes [YOCTO #10732].
--- lib/bb/cooker.py | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-)
lib/bb/cooker.py +13 -1
@@ -1313,6 +1313,18 @@ class BBCooker: self.data.setVar("DATE", time.strftime('%Y%m%d', t)) self.data.setVar("TIME", time.strftime('%H%M%S', t)) + def reset_mtime_caches(self): + """ + Reset mtime caches - this is particularly important when memory resident as something + which is cached is not unlikely to have changed since the last invocation (e.g. a + file associated with a recipe might have been modified by the user). + """ + build.reset_cache() + bb.fetch._checksum_cache.mtime_cache.clear() + siggen_cache = getattr(bb.parse.siggen, 'checksum_cache', None) + if siggen_cache: + bb.parse.siggen.checksum_cache.mtime_cache.clear() + def matchFiles(self, bf): """ Find the .bb files which match the expression in 'buildfile'.
@@ -1490,7 +1502,7 @@ class BBCooker: return True return retval - build.reset_cache() + self.reset_mtime_caches() self.buildSetVars() # If we are told to do the None task then query the default task
Signature · 1 line
2.9.4
PATCH 3/4 cooker: set config valid flags in initConfigurationData()
In reply to Paul EggletonIf we set these flags here then we can not only tidy up some of the code that calls initConfigurationData() (and reset() which simply calls the former), we also avoid calling initConfigurationData() an extra time unnecessarily during startup (since setting baseconfig_valid = False in updateConfigOpts() also resulted in initConfigurationData() being called from updateCacheSync() at the end of the command.)
--- lib/bb/cooker.py | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-)
lib/bb/cooker.py +2 -6
@@ -213,9 +213,6 @@ class BBCooker: self.configuration.server_register_idlecallback(_process_inotify_updates, [self.confignotifier, self.notifier]) - self.baseconfig_valid = True - self.parsecache_valid = False - # Take a lock so only one copy of bitbake can run against a given build # directory at a time if not self.lockBitbake():
@@ -375,6 +372,8 @@ class BBCooker: self.data.renameVar("__depends", "__base_depends") self.add_filewatch(self.data.getVar("__base_depends", False), self.configwatcher) + self.baseconfig_valid = True + self.parsecache_valid = False def enableDataTracking(self): self.configuration.tracking = True
@@ -568,7 +567,6 @@ class BBCooker: clean = False if not clean: logger.debug(1, "Base environment change, triggering reparse") - self.baseconfig_valid = False self.reset() def runCommands(self, server, data, abort):
@@ -1623,8 +1621,6 @@ class BBCooker: if not self.baseconfig_valid: logger.debug(1, "Reloading base configuration data") self.initConfigurationData() - self.baseconfig_valid = True - self.parsecache_valid = False # This is called for all async commands when self.state != running def updateCache(self):
Signature · 1 line
2.9.4
PATCH 4/4 cooker: add BB_CMDLINE to enable access to UI command line with memres
In reply to Paul EggletonIn OpenEmbedded's buildhistory class we want access to the bitbake command line that launched the build, and up to now we were simply using sys.argv from within the event handler to get that. Unfortunately that doesn't work in memory resident mode, since the event handler is naturally executing within the server and thus will give you the command that launched the bitbake server which is much less interesting. Add a dynamic variable BB_CMDLINE to provide access to this, set from sys.argv within the UI process in updateToServer().
(Note that BB_CMDLINE isn't currently passed through to the worker, so this is only really readable from event handlers plus any explicit getVariable calls - in theory an observe-only UI could read it for example.)
Part of the fix for [YOCTO #11634].
--- lib/bb/command.py | 3 ++- lib/bb/cooker.py | 7 ++++++- lib/bb/cookerdata.py | 2 +- 3 files changed, 9 insertions(+), 3 deletions(-)
lib/bb/command.py +2 -1
@@ -310,7 +310,8 @@ class CommandsSync: def updateConfig(self, command, params): options = params[0] environment = params[1] - command.cooker.updateConfigOpts(options, environment) + cmdline = params[2] + command.cooker.updateConfigOpts(options, environment, cmdline) updateConfig.needconfig = False def parseConfiguration(self, command, params):
lib/bb/cooker.py +6 -1
@@ -192,6 +192,8 @@ class BBCooker: bb.parse.__mtime_cache = {} bb.parse.BBHandler.cached_statements = {} + self.ui_cmdline = None + self.initConfigurationData() # we log all events to a file if so directed
@@ -360,6 +362,8 @@ class BBCooker: if consolelog: self.data.setVar("BB_CONSOLELOG", consolelog) + self.data.setVar('BB_CMDLINE', self.ui_cmdline) + # # Copy of the data store which has been expanded. # Used for firing events and accessing variables where expansion needs to be accounted for
@@ -539,7 +543,8 @@ class BBCooker: self.handleCollections(self.data.getVar("BBFILE_COLLECTIONS")) - def updateConfigOpts(self, options, environment): + def updateConfigOpts(self, options, environment, cmdline): + self.ui_cmdline = cmdline clean = True for o in options: if o in ['prefile', 'postfile']:
lib/bb/cookerdata.py +1 -1
@@ -79,7 +79,7 @@ class ConfigParameters(object): "prefile", "postfile", "tracking"]: options[o] = getattr(self.options, o) - ret, error = server.runCommand(["updateConfig", options, environment]) + ret, error = server.runCommand(["updateConfig", options, environment, sys.argv]) if error: raise Exception("Unable to update the server configuration with local parameters: %s" % error)
Signature · 1 line
2.9.4