cooker: fixes event notifications
v2 cover cooker: fixes event notifications
Hi RP,
I think that I've answered your questions' in V1, do you have any comments on these patches, please ?
// Robert
The following changes since commit 1b7a9d4f63d07d61d53daac12da275e8ef2feb24:
buildhistory-diff: honour report_all flag (2018-02-24 10:35:59 +0000)
are available in the git repository at:
git://git.pokylinux.org/poky-contrib rbt/cooker
http://git.pokylinux.org/cgit.cgi//log/?h=rbt/cooker
Robert Yang (3):
bitbake: cooker: don't preserve BB_CONSOLELOG
bitbake: cooker: fix event notifications
bitbake: cooker: improve inotify handling
bitbake/lib/bb/cache.py | 9 ++++--- bitbake/lib/bb/cooker.py | 66 ++++++++++++++++++++++++------------------------ 2 files changed, 39 insertions(+), 36 deletions(-)
Signature · 1 line
2.7.4
PATCH v2 1/3 bitbake: cooker: don't preserve BB_CONSOLELOG
Fixed:
$ bitbake --server-only -T -1 Set MACHINE = "qemux86" in conf/local.conf $ bitbake quilt Set MACHINE = "qemuppc" in conf/local.conf $ bitbake quilt
The log still goes into tmp/log/cooker/qemux86 in the second run, this is incorrect (should be tmp/log/cooker/qemuppc). I checked the code, the ui initializes it every time when it starts, so let it use the up-to-date one rather than old one.
--- bitbake/lib/bb/cooker.py | 8 -------- 1 file changed, 8 deletions(-)
bitbake/lib/bb/cooker.py +0 -8
@@ -326,11 +326,6 @@ class BBCooker: self.state = state.initial self.caches_array = [] - # Need to preserve BB_CONSOLELOG over resets - consolelog = None - if hasattr(self, "data"): - consolelog = self.data.getVar("BB_CONSOLELOG") - if CookerFeatures.BASEDATASTORE_TRACKING in self.featureset: self.enableDataTracking()
@@ -358,9 +353,6 @@ class BBCooker: self.data_hash = self.databuilder.data_hash self.extraconfigdata = {} - if consolelog: - self.data.setVar("BB_CONSOLELOG", consolelog) - self.data.setVar('BB_CMDLINE', self.ui_cmdline) #
Signature · 1 line
2.7.4
PATCH v2 2/3 bitbake: cooker: fix event notifications
In reply to Robert YangThere are two kinds of events:
* directory
We always need handle the event since we don't know what inside the directory, for example:
$ bitbake --server-only -T -1
$ bitbake -p
$ mkdir ${TOPDIR}/classes
$ cp /path/to/base.bbclass classesThe "${TOPDIR}/classes" is a new directory, the event only reports that the directory is created, we don't know there is a base.bbclass unless the parser runs.
* file:
Use bb.parse.supports() to check whether bitbake supports it or not.
And now the bbwatchedfiles is not needed anymore.
--- bitbake/lib/bb/cooker.py | 34 ++++++++++++++++++---------------- 1 file changed, 18 insertions(+), 16 deletions(-)
bitbake/lib/bb/cooker.py +18 -16
@@ -177,14 +177,12 @@ class BBCooker: self.configwatcher = pyinotify.WatchManager() self.configwatcher.bbseen = [] - self.configwatcher.bbwatchedfiles = [] self.confignotifier = pyinotify.Notifier(self.configwatcher, self.config_notifications) self.watchmask = pyinotify.IN_CLOSE_WRITE | pyinotify.IN_CREATE | pyinotify.IN_DELETE | \ pyinotify.IN_DELETE_SELF | pyinotify.IN_MODIFY | pyinotify.IN_MOVE_SELF | \ pyinotify.IN_MOVED_FROM | pyinotify.IN_MOVED_TO self.watcher = pyinotify.WatchManager() self.watcher.bbseen = [] - self.watcher.bbwatchedfiles = [] self.notifier = pyinotify.Notifier(self.watcher, self.notifications) # If being called by something like tinfoil, we need to clean cached data
@@ -247,10 +245,15 @@ class BBCooker: self.baseconfig_valid = False bb.parse.clear_cache() return - if not event.pathname in self.configwatcher.bbwatchedfiles: - return - if not event.pathname in self.inotify_modified_files: - self.inotify_modified_files.append(event.pathname) + + # Check whether it is a supported type when it is a file + if not event.dir: + if not bb.parse.supports(event.pathname, data): + return + if not event.pathname in self.inotify_modified_files: + self.inotify_modified_files.append(event.pathname) + + bb.debug(1, "Get config inotify event: %s" % event) self.baseconfig_valid = False def notifications(self, event):
@@ -259,18 +262,21 @@ class BBCooker: self.parsecache_valid = False bb.parse.clear_cache() return - if event.pathname.endswith("bitbake-cookerdaemon.log") \ - or event.pathname.endswith("bitbake.lock"): - return - if not event.pathname in self.inotify_modified_files: - self.inotify_modified_files.append(event.pathname) + + # Check whether it is a supported type when it is a file + if not event.dir: + if not bb.parse.supports(event.pathname, data): + return + if not event.pathname in self.inotify_modified_files: + self.inotify_modified_files.append(event.pathname) + + bb.debug(1, "Get inotify event: %s" % event) self.parsecache_valid = False def add_filewatch(self, deps, watcher=None, dirs=False): if not watcher: watcher = self.watcher for i in deps: - watcher.bbwatchedfiles.append(i[0]) if dirs: f = i[0] else:
@@ -278,19 +284,15 @@ class BBCooker: if f in watcher.bbseen: continue watcher.bbseen.append(f) - watchtarget = None while True: # We try and add watches for files that don't exist but if they did, would influence # the parser. The parent directory of these files may not exist, in which case we need # to watch any parent that does exist for changes. try: watcher.add_watch(f, self.watchmask, quiet=False) - if watchtarget: - watcher.bbwatchedfiles.append(watchtarget) break except pyinotify.WatchManagerError as e: if 'ENOENT' in str(e): - watchtarget = f f = os.path.dirname(f) if f in watcher.bbseen: break
Signature · 1 line
2.7.4
PATCH v2 3/3 bitbake: cooker: improve inotify handling
In reply to Robert YangEach recipe calls bb_cache.add_info() and it calls add_filewatch() for every depend file (not only existed files, but also non-exsited/attempted ones), which caused the parsing and loading very slow. The watchers are global, so we can only pass new depend files to add_filewatch() to improve the performance.
Here is the testing of 54 layers before and after the patch.
Parsing Loading Before 35s 33s Now 11s 9s
24 seconds are saved for parsing, and 24s are saved for loading.
--- bitbake/lib/bb/cache.py | 9 ++++++--- bitbake/lib/bb/cooker.py | 24 +++++++++++++++--------- 2 files changed, 21 insertions(+), 12 deletions(-)
bitbake/lib/bb/cache.py +6 -3
@@ -713,12 +713,15 @@ class Cache(NoCache): def mtime(cachefile): return bb.parse.cached_mtime_noerror(cachefile) - def add_info(self, filename, info_array, cacheData, parsed=None, watcher=None): + def add_info(self, filename, info_array, cacheData, parsed=None, watcher=None, watch_manager=None): if isinstance(info_array[0], CoreRecipeInfo) and (not info_array[0].skipped): cacheData.add_from_recipeinfo(filename, info_array) - if watcher: - watcher(info_array[0].file_depends) + if watcher and watch_manager: + deps_set = set() + for deps in info_array[0].file_depends: + deps_set.add(deps[0]) + watcher(deps_set - watch_manager.bbseen) if not self.has_cache: return
bitbake/lib/bb/cooker.py +15 -9
@@ -176,13 +176,13 @@ class BBCooker: self.configuration = configuration self.configwatcher = pyinotify.WatchManager() - self.configwatcher.bbseen = [] + self.configwatcher.bbseen = set() self.confignotifier = pyinotify.Notifier(self.configwatcher, self.config_notifications) self.watchmask = pyinotify.IN_CLOSE_WRITE | pyinotify.IN_CREATE | pyinotify.IN_DELETE | \ pyinotify.IN_DELETE_SELF | pyinotify.IN_MODIFY | pyinotify.IN_MOVE_SELF | \ pyinotify.IN_MOVED_FROM | pyinotify.IN_MOVED_TO self.watcher = pyinotify.WatchManager() - self.watcher.bbseen = [] + self.watcher.bbseen = set() self.notifier = pyinotify.Notifier(self.watcher, self.notifications) # If being called by something like tinfoil, we need to clean cached data
@@ -278,12 +278,14 @@ class BBCooker: watcher = self.watcher for i in deps: if dirs: - f = i[0] + f = i else: - f = os.path.dirname(i[0]) + f = os.path.dirname(i) if f in watcher.bbseen: continue - watcher.bbseen.append(f) + # Add both files and directories to to bbseen + watcher.bbseen.add(f) + watcher.bbseen.add(i) while True: # We try and add watches for files that don't exist but if they did, would influence # the parser. The parent directory of these files may not exist, in which case we need
@@ -296,7 +298,7 @@ class BBCooker: f = os.path.dirname(f) if f in watcher.bbseen: break - watcher.bbseen.append(f) + watcher.bbseen.add(f) continue if 'ENOSPC' in str(e): providerlog.error("No space left on device or exceeds fs.inotify.max_user_watches?")
@@ -367,7 +369,10 @@ class BBCooker: self.disableDataTracking() self.data.renameVar("__depends", "__base_depends") - self.add_filewatch(self.data.getVar("__base_depends", False), self.configwatcher) + deps_set = set() + for deps in self.data.getVar("__base_depends", False): + deps_set.add(deps[0]) + self.add_filewatch(deps_set, self.configwatcher) self.baseconfig_valid = True self.parsecache_valid = False
@@ -1510,7 +1515,7 @@ class BBCooker: # Add inotify watches for directories searched for bb/bbappend files for dirent in searchdirs: - self.add_filewatch([[dirent]], dirs=True) + self.add_filewatch([dirent], dirs=True) self.parser = CookerParser(self, filelist, masked) self.parsecache_valid = True
@@ -2144,7 +2149,8 @@ class CookerParser(object): self.cooker.skiplist[virtualfn] = SkippedPackage(info_array[0]) (fn, cls, mc) = bb.cache.virtualfn2realfn(virtualfn) self.bb_cache.add_info(virtualfn, info_array, self.cooker.recipecaches[mc], - parsed=parsed, watcher = self.cooker.add_filewatch) + parsed=parsed, watcher=self.cooker.add_filewatch, + watch_manager=self.cooker.watcher) return True def reparse(self, filename):
Signature · 1 line
2.7.4
Hi RP,
Do you have any comments on this, please ?
// Robert
Quoted 25 lines · Robert Yang
Hi RP,
I think that I've answered your questions' in V1, do you have any comments on these patches, please ?
// Robert
The following changes since commit
1b7a9d4f63d07d61d53daac12da275e8ef2feb24:buildhistory-diff: honour report_all flag (2018-02-24 10:35:59 +0000)are available in the git repository at:
git://git.pokylinux.org/poky-contribrbt/cooker
http://git.pokylinux.org/cgit.cgi//log/?h=rbt/cookerRobert Yang (3):
bitbake: cooker: don't preserveBB_CONSOLELOG
bitbake: cooker: fix event notifications
bitbake: cooker: improve inotify handlingbitbake/lib/bb/cache.py | 9 ++++--- bitbake/lib/bb/cooker.py | 66 ++++++++++++++++++++++++------------------------ 2 files changed, 39 insertions(+), 36 deletions(-)