Skip to contentSkip to conversation

cooker: fixes event notifications

PATCH RESEND In review Robert Yang 1 participant 5 messages Feb 27, 2018 – Mar 15, 2018

lore
Share & tools
lore
#0 lore raw

v2 cover cooker: fixes event notifications

Hi RP,

I think that I've answered your questions' in V1, do you have any comments on these patches, please ?

// Robert

The following changes since commit 1b7a9d4f63d07d61d53daac12da275e8ef2feb24:

buildhistory-diff: honour report_all flag (2018-02-24 10:35:59 +0000)

are available in the git repository at:

git://git.pokylinux.org/poky-contrib rbt/cooker
http://git.pokylinux.org/cgit.cgi//log/?h=rbt/cooker

Robert Yang (3):
bitbake: cooker: don't preserve BB_CONSOLELOG
bitbake: cooker: fix event notifications
bitbake: cooker: improve inotify handling

 bitbake/lib/bb/cache.py  |  9 ++++---
 bitbake/lib/bb/cooker.py | 66 ++++++++++++++++++++++++------------------------
 2 files changed, 39 insertions(+), 36 deletions(-)
Signature · 1 line
2.7.4
#1 lore raw

PATCH v2 1/3 bitbake: cooker: don't preserve BB_CONSOLELOG

Fixed:

$ bitbake --server-only -T -1
Set MACHINE = "qemux86" in conf/local.conf
$ bitbake quilt
Set MACHINE = "qemuppc" in conf/local.conf
$ bitbake quilt

The log still goes into tmp/log/cooker/qemux86 in the second run, this is incorrect (should be tmp/log/cooker/qemuppc). I checked the code, the ui initializes it every time when it starts, so let it use the up-to-date one rather than old one.

---
 bitbake/lib/bb/cooker.py | 8 --------
 1 file changed, 8 deletions(-)
bitbake/lib/bb/cooker.py +0 -8
index af482f9..3f113ae 100644
@@ -326,11 +326,6 @@ class BBCooker:
         self.state = state.initial
         self.caches_array = []
 
-        # Need to preserve BB_CONSOLELOG over resets
-        consolelog = None
-        if hasattr(self, "data"):
-            consolelog = self.data.getVar("BB_CONSOLELOG")
-
         if CookerFeatures.BASEDATASTORE_TRACKING in self.featureset:
             self.enableDataTracking()
 
@@ -358,9 +353,6 @@ class BBCooker:
         self.data_hash = self.databuilder.data_hash
         self.extraconfigdata = {}
 
-        if consolelog:
-            self.data.setVar("BB_CONSOLELOG", consolelog)
-
         self.data.setVar('BB_CMDLINE', self.ui_cmdline)
 
         #
Signature · 1 line
2.7.4
#2 lore raw

PATCH v2 2/3 bitbake: cooker: fix event notifications

In reply to Robert Yang

There are two kinds of events:
* directory
We always need handle the event since we don't know what inside the directory, for example:

  $ bitbake --server-only -T -1
  $ bitbake -p
  $ mkdir ${TOPDIR}/classes
  $ cp /path/to/base.bbclass classes

The "${TOPDIR}/classes" is a new directory, the event only reports that the directory is created, we don't know there is a base.bbclass unless the parser runs.

* file:
Use bb.parse.supports() to check whether bitbake supports it or not.

And now the bbwatchedfiles is not needed anymore.

---
 bitbake/lib/bb/cooker.py | 34 ++++++++++++++++++----------------
 1 file changed, 18 insertions(+), 16 deletions(-)
bitbake/lib/bb/cooker.py +18 -16
index 3f113ae..ffda225 100644
@@ -177,14 +177,12 @@ class BBCooker:
 
         self.configwatcher = pyinotify.WatchManager()
         self.configwatcher.bbseen = []
-        self.configwatcher.bbwatchedfiles = []
         self.confignotifier = pyinotify.Notifier(self.configwatcher, self.config_notifications)
         self.watchmask = pyinotify.IN_CLOSE_WRITE | pyinotify.IN_CREATE | pyinotify.IN_DELETE | \
                          pyinotify.IN_DELETE_SELF | pyinotify.IN_MODIFY | pyinotify.IN_MOVE_SELF | \
                          pyinotify.IN_MOVED_FROM | pyinotify.IN_MOVED_TO
         self.watcher = pyinotify.WatchManager()
         self.watcher.bbseen = []
-        self.watcher.bbwatchedfiles = []
         self.notifier = pyinotify.Notifier(self.watcher, self.notifications)
 
         # If being called by something like tinfoil, we need to clean cached data
@@ -247,10 +245,15 @@ class BBCooker:
             self.baseconfig_valid = False
             bb.parse.clear_cache()
             return
-        if not event.pathname in self.configwatcher.bbwatchedfiles:
-            return
-        if not event.pathname in self.inotify_modified_files:
-            self.inotify_modified_files.append(event.pathname)
+
+        # Check whether it is a supported type when it is a file
+        if not event.dir:
+            if not bb.parse.supports(event.pathname, data):
+                return
+            if not event.pathname in self.inotify_modified_files:
+                self.inotify_modified_files.append(event.pathname)
+
+        bb.debug(1, "Get config inotify event: %s" % event)
         self.baseconfig_valid = False
 
     def notifications(self, event):
@@ -259,18 +262,21 @@ class BBCooker:
             self.parsecache_valid = False
             bb.parse.clear_cache()
             return
-        if event.pathname.endswith("bitbake-cookerdaemon.log") \
-                or event.pathname.endswith("bitbake.lock"):
-            return
-        if not event.pathname in self.inotify_modified_files:
-            self.inotify_modified_files.append(event.pathname)
+
+        # Check whether it is a supported type when it is a file
+        if not event.dir:
+            if not bb.parse.supports(event.pathname, data):
+                return
+            if not event.pathname in self.inotify_modified_files:
+                self.inotify_modified_files.append(event.pathname)
+
+        bb.debug(1, "Get inotify event: %s" % event)
         self.parsecache_valid = False
 
     def add_filewatch(self, deps, watcher=None, dirs=False):
         if not watcher:
             watcher = self.watcher
         for i in deps:
-            watcher.bbwatchedfiles.append(i[0])
             if dirs:
                 f = i[0]
             else:
@@ -278,19 +284,15 @@ class BBCooker:
             if f in watcher.bbseen:
                 continue
             watcher.bbseen.append(f)
-            watchtarget = None
             while True:
                 # We try and add watches for files that don't exist but if they did, would influence
                 # the parser. The parent directory of these files may not exist, in which case we need
                 # to watch any parent that does exist for changes.
                 try:
                     watcher.add_watch(f, self.watchmask, quiet=False)
-                    if watchtarget:
-                        watcher.bbwatchedfiles.append(watchtarget)
                     break
                 except pyinotify.WatchManagerError as e:
                     if 'ENOENT' in str(e):
-                        watchtarget = f
                         f = os.path.dirname(f)
                         if f in watcher.bbseen:
                             break
Signature · 1 line
2.7.4
#3 lore raw

PATCH v2 3/3 bitbake: cooker: improve inotify handling

In reply to Robert Yang

Each recipe calls bb_cache.add_info() and it calls add_filewatch() for every depend file (not only existed files, but also non-exsited/attempted ones), which caused the parsing and loading very slow. The watchers are global, so we can only pass new depend files to add_filewatch() to improve the performance.

Here is the testing of 54 layers before and after the patch.

              Parsing       Loading
Before        35s           33s
Now           11s           9s

24 seconds are saved for parsing, and 24s are saved for loading.

---
 bitbake/lib/bb/cache.py  |  9 ++++++---
 bitbake/lib/bb/cooker.py | 24 +++++++++++++++---------
 2 files changed, 21 insertions(+), 12 deletions(-)
bitbake/lib/bb/cache.py +6 -3
index 168a77a..7d65939 100644
@@ -713,12 +713,15 @@ class Cache(NoCache):
     def mtime(cachefile):
         return bb.parse.cached_mtime_noerror(cachefile)
 
-    def add_info(self, filename, info_array, cacheData, parsed=None, watcher=None):
+    def add_info(self, filename, info_array, cacheData, parsed=None, watcher=None, watch_manager=None):
         if isinstance(info_array[0], CoreRecipeInfo) and (not info_array[0].skipped):
             cacheData.add_from_recipeinfo(filename, info_array)
 
-            if watcher:
-                watcher(info_array[0].file_depends)
+            if watcher and watch_manager:
+                deps_set = set()
+                for deps in info_array[0].file_depends:
+                    deps_set.add(deps[0])
+                watcher(deps_set - watch_manager.bbseen)
 
         if not self.has_cache:
             return
bitbake/lib/bb/cooker.py +15 -9
index ffda225..cd98ba0 100644
@@ -176,13 +176,13 @@ class BBCooker:
         self.configuration = configuration
 
         self.configwatcher = pyinotify.WatchManager()
-        self.configwatcher.bbseen = []
+        self.configwatcher.bbseen = set()
         self.confignotifier = pyinotify.Notifier(self.configwatcher, self.config_notifications)
         self.watchmask = pyinotify.IN_CLOSE_WRITE | pyinotify.IN_CREATE | pyinotify.IN_DELETE | \
                          pyinotify.IN_DELETE_SELF | pyinotify.IN_MODIFY | pyinotify.IN_MOVE_SELF | \
                          pyinotify.IN_MOVED_FROM | pyinotify.IN_MOVED_TO
         self.watcher = pyinotify.WatchManager()
-        self.watcher.bbseen = []
+        self.watcher.bbseen = set()
         self.notifier = pyinotify.Notifier(self.watcher, self.notifications)
 
         # If being called by something like tinfoil, we need to clean cached data
@@ -278,12 +278,14 @@ class BBCooker:
             watcher = self.watcher
         for i in deps:
             if dirs:
-                f = i[0]
+                f = i
             else:
-                f = os.path.dirname(i[0])
+                f = os.path.dirname(i)
             if f in watcher.bbseen:
                 continue
-            watcher.bbseen.append(f)
+            # Add both files and directories to to bbseen
+            watcher.bbseen.add(f)
+            watcher.bbseen.add(i)
             while True:
                 # We try and add watches for files that don't exist but if they did, would influence
                 # the parser. The parent directory of these files may not exist, in which case we need
@@ -296,7 +298,7 @@ class BBCooker:
                         f = os.path.dirname(f)
                         if f in watcher.bbseen:
                             break
-                        watcher.bbseen.append(f)
+                        watcher.bbseen.add(f)
                         continue
                     if 'ENOSPC' in str(e):
                         providerlog.error("No space left on device or exceeds fs.inotify.max_user_watches?")
@@ -367,7 +369,10 @@ class BBCooker:
             self.disableDataTracking()
 
         self.data.renameVar("__depends", "__base_depends")
-        self.add_filewatch(self.data.getVar("__base_depends", False), self.configwatcher)
+        deps_set = set()
+        for deps in self.data.getVar("__base_depends", False):
+            deps_set.add(deps[0])
+        self.add_filewatch(deps_set, self.configwatcher)
 
         self.baseconfig_valid = True
         self.parsecache_valid = False
@@ -1510,7 +1515,7 @@ class BBCooker:
 
             # Add inotify watches for directories searched for bb/bbappend files
             for dirent in searchdirs:
-                self.add_filewatch([[dirent]], dirs=True)
+                self.add_filewatch([dirent], dirs=True)
 
             self.parser = CookerParser(self, filelist, masked)
             self.parsecache_valid = True
@@ -2144,7 +2149,8 @@ class CookerParser(object):
                 self.cooker.skiplist[virtualfn] = SkippedPackage(info_array[0])
             (fn, cls, mc) = bb.cache.virtualfn2realfn(virtualfn)
             self.bb_cache.add_info(virtualfn, info_array, self.cooker.recipecaches[mc],
-                                        parsed=parsed, watcher = self.cooker.add_filewatch)
+                                        parsed=parsed, watcher=self.cooker.add_filewatch,
+                                        watch_manager=self.cooker.watcher)
         return True
 
     def reparse(self, filename):
Signature · 1 line
2.7.4
#4 lore raw
In reply to Robert Yang

Hi RP,

Do you have any comments on this, please ?

// Robert

Quoted 25 lines · Robert Yang

Hi RP,

I think that I've answered your questions' in V1, do you have any comments on these patches, please ?

// Robert

The following changes since commit 1b7a9d4f63d07d61d53daac12da275e8ef2feb24:

   buildhistory-diff: honour report_all flag (2018-02-24 10:35:59 +0000)

are available in the git repository at:

git://git.pokylinux.org/poky-contrib rbt/cooker
http://git.pokylinux.org/cgit.cgi//log/?h=rbt/cooker

Robert Yang (3):
bitbake: cooker: don't preserve BB_CONSOLELOG
bitbake: cooker: fix event notifications
bitbake: cooker: improve inotify handling

  bitbake/lib/bb/cache.py  |  9 ++++---
  bitbake/lib/bb/cooker.py | 66 ++++++++++++++++++++++++------------------------
  2 files changed, 39 insertions(+), 36 deletions(-)