Skip to contentSkip to conversation

cooker: fixes and parsing performance improve

PATCH In review Robert Yang 2 participants 13 messages Feb 1, 2018 – Feb 8, 2018

lore
Share & tools
lore
#0 lore raw

cover cooker: fixes and parsing performance improve

Hi RP,

These patches contain fixes for memory resident bitbake, and parsing/loading improvement, please see the patches for details.

BTW., do we have a plan to switch to memory resident bitbake by default? I've tested it a lot with these patches, it works well.

// Robert

The following changes since commit 385944254d0ef88bd6450a221a54cbcb40020b42:

tcmode-default.inc: drop preferred version of gzip-native (2018-01-31 17:01:20 +0000)

are available in the git repository at:

git://git.pokylinux.org/poky-contrib rbt/inotify
http://git.pokylinux.org/cgit.cgi//log/?h=rbt/inotify

Robert Yang (8):
bitbake: cooker: skip when BBFILE_PATTERN is empty
bitbake: cooker: fix for BBFILE_PATTERN matches bbappend
bitbake: cooker: don't stop file notifier when cooker is shutdown
bitbake: cooker: don't preserve BB_CONSOLELOG
bitbake: cache: improve debug message
bitbake: parse: fixes for resolve_file()
bitbake: cooker: fix event notifications
bitbake: cooker: improve inotify handling

 bitbake/lib/bb/cache.py                      | 14 +++--
 bitbake/lib/bb/cooker.py                     | 79 +++++++++++++++-------------
 bitbake/lib/bb/parse/__init__.py             |  3 +-
 bitbake/lib/bb/parse/parse_py/BBHandler.py   |  3 --
 bitbake/lib/bb/parse/parse_py/ConfHandler.py |  3 --
 bitbake/lib/bb/server/process.py             |  2 +
 6 files changed, 55 insertions(+), 49 deletions(-)
Signature · 1 line
2.7.4
#1 lore raw

PATCH 1/8 bitbake: cooker: skip when BBFILE_PATTERN is empty

There is nothing to do when BBFILE_PATTERN is empty.

---
 bitbake/lib/bb/cooker.py | 1 +
 1 file changed, 1 insertion(+)
bitbake/lib/bb/cooker.py +1 -0
index cd365f7..f0dab97 100644
@@ -1170,6 +1170,7 @@ class BBCooker:
                 elif regex == "":
                     parselog.debug(1, "BBFILE_PATTERN_%s is empty" % c)
                     errors = False
+                    continue
                 else:
                     try:
                         cre = re.compile(regex)
Signature · 1 line
2.7.4
#2 lore raw

PATCH 2/8 bitbake: cooker: fix for BBFILE_PATTERN matches bbappend

In reply to Robert Yang

The old code couldn't handle nestled layers correctly, e.g.:
parent_layer/sub_layer/foo.bb

Note there are two layers, parent_layer and sub_layer. And in parent_layer/conf/layer.conf:
BBFILE_PATTERN_parent_layer = ""^${LAYERDIR}/"

This setting is incorrect since it also matches parent_layer/sub_layer/foo.bb, so it warns that no files matched sub_layer, this is the expected behavior, but it doesn't warn when there is a parent_layer/sub_layer/bar.bbappend, this was incorrect since the bbappend is also matched by BBFILE_PATTERN_parent_layer, it should warn and let the user fix the problem. Check the bbappend in already "matched set" before return it as matched by "unmatched set" can fix the problem.

---
 bitbake/lib/bb/cooker.py | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)
bitbake/lib/bb/cooker.py +7 -3
index f0dab97..f991c8f 100644
@@ -1808,21 +1808,25 @@ class CookerCollectFiles(object):
             realfn, cls, mc = bb.cache.virtualfn2realfn(p)
             priorities[p] = self.calc_bbfile_priority(realfn, matched)
 
-        # Don't show the warning if the BBFILE_PATTERN did match .bbappend files
         unmatched = set()
         for _, _, regex, pri in self.bbfile_config_priorities:
             if not regex in matched:
                 unmatched.add(regex)
 
-        def findmatch(regex):
+        # Don't show the warning if the BBFILE_PATTERN did match .bbappend files
+        def find_bbappend_match(regex):
             for b in self.bbappends:
                 (bbfile, append) = b
                 if regex.match(append):
+                    # If the bbappend is matched by already "matched set", return False
+                    for matched_regex in matched:
+                        if matched_regex.match(append):
+                            return False
                     return True
             return False
 
         for unmatch in unmatched.copy():
-            if findmatch(unmatch):
+            if find_bbappend_match(unmatch):
                 unmatched.remove(unmatch)
 
         for collection, pattern, regex, _ in self.bbfile_config_priorities:
Signature · 1 line
2.7.4
#3 lore raw

PATCH 3/8 bitbake: cooker: don't stop file notifier when cooker is shutdown

In reply to Robert Yang

It should be live/exited with server rather than cooker, fixed:

$ bitbake --server-only -T -1
Set MACHINE = "qemux86" in conf/local.conf
$ bitbake quilt
Set MACHINE = "qemuppc" in conf/local.conf
$ bitbake quilt
[snip]
ERROR: When reparsing /workspace1/lyang1/poky/meta/recipes-connectivity/openssl/openssl_1.0.2m.bb.do_package, the basehash value changed from c216f7f4fdd3cf4a0b10b975a636426c to d5a8e9431ab261381752d7a64c7b2fa9. The metadata is not deterministic and this needs to be fixed.
[snip]

This is because the server doesn't know local.conf is changed since the notifiers are stopped, so it doesn't reparse, and then we would get the errors, let the notifiers live/exited with server can fix the problem.

---
 bitbake/lib/bb/cooker.py         | 2 --
 bitbake/lib/bb/server/process.py | 2 ++
 2 files changed, 2 insertions(+), 2 deletions(-)
bitbake/lib/bb/cooker.py +0 -2
index f991c8f..af482f9 100644
@@ -1604,8 +1604,6 @@ class BBCooker:
 
         if self.parser:
             self.parser.shutdown(clean=not force, force=force)
-        self.notifier.stop()
-        self.confignotifier.stop()
 
     def finishcommand(self):
         self.state = state.initial
bitbake/lib/bb/server/process.py +2 -0
index 3d31355..828159e 100644
@@ -223,6 +223,8 @@ class ProcessServer(multiprocessing.Process):
 
         try: 
             self.cooker.shutdown(True)
+            self.cooker.notifier.stop()
+            self.cooker.confignotifier.stop()
         except:
             pass
 
Signature · 1 line
2.7.4
#4 lore raw

PATCH 4/8 bitbake: cooker: don't preserve BB_CONSOLELOG

In reply to Robert Yang

Fixed:

$ bitbake --server-only -T -1
Set MACHINE = "qemux86" in conf/local.conf
$ bitbake quilt
Set MACHINE = "qemuppc" in conf/local.conf
$ bitbake quilt

The log still goes into tmp/log/cooker/qemux86 in the second run, this is incorrect (should be tmp/log/cooker/qemuppc). I checked the code, the ui initializes it every time when it starts, so let it use the up-to-date one rather than old one.

---
 bitbake/lib/bb/cooker.py | 8 --------
 1 file changed, 8 deletions(-)
bitbake/lib/bb/cooker.py +0 -8
index af482f9..3f113ae 100644
@@ -326,11 +326,6 @@ class BBCooker:
         self.state = state.initial
         self.caches_array = []
 
-        # Need to preserve BB_CONSOLELOG over resets
-        consolelog = None
-        if hasattr(self, "data"):
-            consolelog = self.data.getVar("BB_CONSOLELOG")
-
         if CookerFeatures.BASEDATASTORE_TRACKING in self.featureset:
             self.enableDataTracking()
 
@@ -358,9 +353,6 @@ class BBCooker:
         self.data_hash = self.databuilder.data_hash
         self.extraconfigdata = {}
 
-        if consolelog:
-            self.data.setVar("BB_CONSOLELOG", consolelog)
-
         self.data.setVar('BB_CMDLINE', self.ui_cmdline)
 
         #
Signature · 1 line
2.7.4
#5 lore raw

Re: [PATCH 4/8] bitbake: cooker: don't preserve BB_CONSOLELOG

Quoted 15 lines · Robert Yang

Fixed:

$ bitbake --server-only -T -1
Set MACHINE = "qemux86" in conf/local.conf
$ bitbake quilt
Set MACHINE = "qemuppc" in conf/local.conf
$ bitbake quilt

The log still goes into tmp/log/cooker/qemux86 in the second run, this is
incorrect (should be tmp/log/cooker/qemuppc). I checked the code, the ui
initializes it every time when it starts, so let it use the up-to- date
one rather than old one.

Quoted 4 lines

---
 bitbake/lib/bb/cooker.py | 8 --------

 1 file changed, 8 deletions(-)

This effectively reverts:

http://git.yoctoproject.org/cgit.cgi/poky/commit/bitbake/lib/bb/cooker.py?id=3ebf7617d6c869f798807792918e1030b3ab66de

Sadly I didn't put a bug number in that commit message but I do remember there being a good reason to do this.

I think the problem is a "UI" can run multiple commands with reset events within that command stream and it would only have one console log. I also believe there may have been issues where toaster would have been unable to find the logs if they kept moving around.

The console log handling probably needs more thought as there are clearly multiple issues here. We may want to move it out of a directory which is machine specific?

Cheers,

Richard

Quoted 29 lines
bitbake/lib/bb/cooker.py +0 -0
index af482f9..3f113ae 100644
@@ -326,11 +326,6 @@ class BBCooker:

         self.state = state.initial
         self.caches_array = []

-        # Need to preserve BB_CONSOLELOG over resets
-        consolelog = None
-        if hasattr(self, "data"):
-            consolelog = self.data.getVar("BB_CONSOLELOG") -
         if CookerFeatures.BASEDATASTORE_TRACKING in self.featureset:
             self.enableDataTracking()

bitbake/lib/bb/cooker.py +0 -0
@@ -358,9 +353,6 @@ class BBCooker:

         self.data_hash = self.databuilder.data_hash self.extraconfigdata = {}

-        if consolelog:
-            self.data.setVar("BB_CONSOLELOG", consolelog) -
         self.data.setVar('BB_CMDLINE', self.ui_cmdline)
         #
--
2.7.4

#6 lore raw

Re: [PATCH 4/8] bitbake: cooker: don't preserve BB_CONSOLELOG

Hi RP,

Quoted 25 lines · Richard Purdie
Quoted 15 lines · Robert Yang

Fixed:

$ bitbake --server-only -T -1
Set MACHINE = "qemux86" in conf/local.conf
$ bitbake quilt
Set MACHINE = "qemuppc" in conf/local.conf
$ bitbake quilt

The log still goes into tmp/log/cooker/qemux86 in the second run, this is
incorrect (should be tmp/log/cooker/qemuppc). I checked the code, the ui
initializes it every time when it starts, so let it use the up-to- date
one rather than old one.

Quoted 4 lines
---
  bitbake/lib/bb/cooker.py | 8 --------
  1 file changed, 8 deletions(-)

This effectively reverts:

http://git.yoctoproject.org/cgit.cgi/poky/commit/bitbake/lib/bb/cooker.py?id=3ebf7617d6c869f798807792918e1030b3ab66de

Quoted 7 lines

Sadly I didn't put a bug number in that commit message but I do remember there being a good reason to do this.

I think the problem is a "UI" can run multiple commands with reset events within that command stream and it would only have one console log. I also believe there may have been issues where toaster would have been unable to find the logs if they kept moving around.

I've found the bug:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=8411

I think that it has been fixed by:

https://bugzilla.yoctoproject.org/show_bug.cgi?id=8373

http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=a0791c19db7d5b33ce7a4e3614a574ad4a319a84

I've tried the following commands with my patch:

$ . /workspace1/lyang1/poky/bitbake/bin/toaster start noweb
$ MACHINE=qemuppc bitbake quilt-native

The log is tmp/log/cooker/qemuppc/build_20180208_120722.942.log

$ MACHINE=qemux86 bitbake quilt-native

The log is tmp/log/cooker/qemux86/build_20180208_121605.382.log

And the log contents are correct.

So I think that we don't need preserve BB_CONSOLELOG anymore.

// Robert

Quoted 39 lines

The console log handling probably needs more thought as there are clearly multiple issues here. We may want to move it out of a directory which is machine specific?

Cheers,

Richard

Quoted 29 lines
bitbake/lib/bb/cooker.py +0 -8
index af482f9..3f113ae 100644
@@ -326,11 +326,6 @@ class BBCooker:
          self.state = state.initial
          self.caches_array = []
 
-        # Need to preserve BB_CONSOLELOG over resets
-        consolelog = None
-        if hasattr(self, "data"):
-            consolelog = self.data.getVar("BB_CONSOLELOG")
-
          if CookerFeatures.BASEDATASTORE_TRACKING in self.featureset:
              self.enableDataTracking()
 
@@ -358,9 +353,6 @@ class BBCooker:
          self.data_hash = self.databuilder.data_hash
          self.extraconfigdata = {}
 
-        if consolelog:
-            self.data.setVar("BB_CONSOLELOG", consolelog)
-
          self.data.setVar('BB_CMDLINE', self.ui_cmdline)
 
          #
Signature · 1 line
2.7.4
#7 lore raw

PATCH 5/8 bitbake: cache: improve debug message

In reply to Robert Yang

* Print message when cachefile is found/not can help debug.
* Update "Using cache in" to "Cache dir:" since it was the same as the debug message of "codeparser & file checksum caches", which caused confusion. And whether the cache file will be used or not is still unknown at that time, so just print the cache dir.

---
 bitbake/lib/bb/cache.py | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
bitbake/lib/bb/cache.py +4 -1
index 86ce0e7..168a77a 100644
@@ -395,7 +395,7 @@ class Cache(NoCache):
         self.has_cache = True
         self.cachefile = getCacheFile(self.cachedir, "bb_cache.dat", self.data_hash)
 
-        logger.debug(1, "Using cache in '%s'", self.cachedir)
+        logger.debug(1, "Cache dir: %s", self.cachedir)
         bb.utils.mkdirhier(self.cachedir)
 
         cache_ok = True
@@ -408,6 +408,8 @@ class Cache(NoCache):
             self.load_cachefile()
         elif os.path.isfile(self.cachefile):
             logger.info("Out of date cache found, rebuilding...")
+        else:
+            logger.debug(1, "Cache file %s not found, building..." % self.cachefile)
 
     def load_cachefile(self):
         cachesize = 0
@@ -424,6 +426,7 @@ class Cache(NoCache):
 
         for cache_class in self.caches_array:
             cachefile = getCacheFile(self.cachedir, cache_class.cachefile, self.data_hash)
+            logger.debug(1, 'Loading cache file: %s' % cachefile)
             with open(cachefile, "rb") as cachefile:
                 pickled = pickle.Unpickler(cachefile)
                 # Check cache version information
Signature · 1 line
2.7.4
#8 lore raw

PATCH 6/8 bitbake: parse: fixes for resolve_file()

In reply to Robert Yang

The resolve_file() calls mark_dependency(), so the one which calls resolve_file() doesn't need call mark_dependency() again.

---
 bitbake/lib/bb/parse/__init__.py             | 3 ++-
 bitbake/lib/bb/parse/parse_py/BBHandler.py   | 3 ---
 bitbake/lib/bb/parse/parse_py/ConfHandler.py | 3 ---
 3 files changed, 2 insertions(+), 7 deletions(-)
bitbake/lib/bb/parse/__init__.py +2 -1
index 2fc4002..5397d57 100644
@@ -134,8 +134,9 @@ def resolve_file(fn, d):
         if not newfn:
             raise IOError(errno.ENOENT, "file %s not found in %s" % (fn, bbpath))
         fn = newfn
+    else:
+        mark_dependency(d, fn)
 
-    mark_dependency(d, fn)
     if not os.path.isfile(fn):
         raise IOError(errno.ENOENT, "file %s not found" % fn)
 
bitbake/lib/bb/parse/parse_py/BBHandler.py +0 -3
index f89ad24..e5039e3 100644
@@ -131,9 +131,6 @@ def handle(fn, d, include):
 
     abs_fn = resolve_file(fn, d)
 
-    if include:
-        bb.parse.mark_dependency(d, abs_fn)
-
     # actual loading
     statements = get_statements(fn, abs_fn, base_name)
 
bitbake/lib/bb/parse/parse_py/ConfHandler.py +0 -3
index 97aa130..9d3ebe1 100644
@@ -134,9 +134,6 @@ def handle(fn, data, include):
     abs_fn = resolve_file(fn, data)
     f = open(abs_fn, 'r')
 
-    if include:
-        bb.parse.mark_dependency(data, abs_fn)
-
     statements = ast.StatementGroup()
     lineno = 0
     while True:
Signature · 1 line
2.7.4
#9 lore raw

PATCH 7/8 bitbake: cooker: fix event notifications

In reply to Robert Yang

There are two kinds of events:
* directory
We always need handle the event since we don't know what inside the directory, for example:

  $ bitbake --server-only -T -1
  $ bitbake -p
  $ mkdir ${TOPDIR}/classes
  $ cp /path/to/base.bbclass classes

The "${TOPDIR}/classes" is a new directory, the event only reports that the directory is created, we don't know there is a base.bbclass unless the parser runs.

* file:
Use bb.parse.supports() to check whether bitbake supports it or not.

And now the bbwatchedfiles is not needed anymore.

---
 bitbake/lib/bb/cooker.py | 34 ++++++++++++++++++----------------
 1 file changed, 18 insertions(+), 16 deletions(-)
bitbake/lib/bb/cooker.py +18 -16
index 3f113ae..ffda225 100644
@@ -177,14 +177,12 @@ class BBCooker:
 
         self.configwatcher = pyinotify.WatchManager()
         self.configwatcher.bbseen = []
-        self.configwatcher.bbwatchedfiles = []
         self.confignotifier = pyinotify.Notifier(self.configwatcher, self.config_notifications)
         self.watchmask = pyinotify.IN_CLOSE_WRITE | pyinotify.IN_CREATE | pyinotify.IN_DELETE | \
                          pyinotify.IN_DELETE_SELF | pyinotify.IN_MODIFY | pyinotify.IN_MOVE_SELF | \
                          pyinotify.IN_MOVED_FROM | pyinotify.IN_MOVED_TO
         self.watcher = pyinotify.WatchManager()
         self.watcher.bbseen = []
-        self.watcher.bbwatchedfiles = []
         self.notifier = pyinotify.Notifier(self.watcher, self.notifications)
 
         # If being called by something like tinfoil, we need to clean cached data
@@ -247,10 +245,15 @@ class BBCooker:
             self.baseconfig_valid = False
             bb.parse.clear_cache()
             return
-        if not event.pathname in self.configwatcher.bbwatchedfiles:
-            return
-        if not event.pathname in self.inotify_modified_files:
-            self.inotify_modified_files.append(event.pathname)
+
+        # Check whether it is a supported type when it is a file
+        if not event.dir:
+            if not bb.parse.supports(event.pathname, data):
+                return
+            if not event.pathname in self.inotify_modified_files:
+                self.inotify_modified_files.append(event.pathname)
+
+        bb.debug(1, "Get config inotify event: %s" % event)
         self.baseconfig_valid = False
 
     def notifications(self, event):
@@ -259,18 +262,21 @@ class BBCooker:
             self.parsecache_valid = False
             bb.parse.clear_cache()
             return
-        if event.pathname.endswith("bitbake-cookerdaemon.log") \
-                or event.pathname.endswith("bitbake.lock"):
-            return
-        if not event.pathname in self.inotify_modified_files:
-            self.inotify_modified_files.append(event.pathname)
+
+        # Check whether it is a supported type when it is a file
+        if not event.dir:
+            if not bb.parse.supports(event.pathname, data):
+                return
+            if not event.pathname in self.inotify_modified_files:
+                self.inotify_modified_files.append(event.pathname)
+
+        bb.debug(1, "Get inotify event: %s" % event)
         self.parsecache_valid = False
 
     def add_filewatch(self, deps, watcher=None, dirs=False):
         if not watcher:
             watcher = self.watcher
         for i in deps:
-            watcher.bbwatchedfiles.append(i[0])
             if dirs:
                 f = i[0]
             else:
@@ -278,19 +284,15 @@ class BBCooker:
             if f in watcher.bbseen:
                 continue
             watcher.bbseen.append(f)
-            watchtarget = None
             while True:
                 # We try and add watches for files that don't exist but if they did, would influence
                 # the parser. The parent directory of these files may not exist, in which case we need
                 # to watch any parent that does exist for changes.
                 try:
                     watcher.add_watch(f, self.watchmask, quiet=False)
-                    if watchtarget:
-                        watcher.bbwatchedfiles.append(watchtarget)
                     break
                 except pyinotify.WatchManagerError as e:
                     if 'ENOENT' in str(e):
-                        watchtarget = f
                         f = os.path.dirname(f)
                         if f in watcher.bbseen:
                             break
Signature · 1 line
2.7.4
#10 lore raw

Re: [PATCH 7/8] bitbake: cooker: fix event notifications

Quoted 19 lines · Robert Yang

There are two kinds of events:
* directory
  We always need handle the event since we don't know what inside the directory, for example:

  $ bitbake --server-only -T -1
  $ bitbake -p
  $ mkdir ${TOPDIR}/classes
  $ cp /path/to/base.bbclass classes

  The "${TOPDIR}/classes" is a new directory, the event only reports that the
  directory is created, we don't know there is a base.bbclass unless the parser
  runs.

* file:
  Use bb.parse.supports() to check whether bitbake supports it or not.

Quoted 6 lines

And now the bbwatchedfiles is not needed anymore.

---
 bitbake/lib/bb/cooker.py | 34 ++++++++++++++++++----------------

 1 file changed, 18 insertions(+), 16 deletions(-)

I'm not sure you can do this. For example, bitbake itself doesn't know/care about *.patch files, however if one is referenced in a SRC_URI and ends up on the watch list, would this change break a patch file changing causing task checksums to change?

Its looking like we need better tests for some of this code too :/.

Cheers,

Richard

#11 lore raw

Re: [PATCH 7/8] bitbake: cooker: fix event notifications

Hi RP,

Quoted 27 lines · Richard Purdie
Quoted 19 lines · Robert Yang

There are two kinds of events:
* directory
  We always need handle the event since we don't know what inside the directory, for example:

   $ bitbake --server-only -T -1
   $ bitbake -p
   $ mkdir ${TOPDIR}/classes
   $ cp /path/to/base.bbclass classes

  The "${TOPDIR}/classes" is a new directory, the event only reports
that the
  directory is created, we don't know there is a base.bbclass unless
the parser
  runs.

* file:
  Use bb.parse.supports() to check whether bitbake supports it or
not.

Quoted 6 lines

And now the bbwatchedfiles is not needed anymore.

---
  bitbake/lib/bb/cooker.py | 34 ++++++++++++++++++----------------
  1 file changed, 18 insertions(+), 16 deletions(-)
Quoted 4 lines

I'm not sure you can do this. For example, bitbake itself doesn't know/care about *.patch files, however if one is referenced in a SRC_URI and ends up on the watch list, would this change break a patch file changing causing task checksums to change?

I can confirm the the following 2 things:
1) The watcher only watches __depends (and __base_depends) which only contains files supported by bb.parse.supports(), so use bb.parse.supports() to check it is OK (or maybe it is the best way).

2) The *.patch is not part of of __depends, but file_checksums, and they are not watched by watchers. When a patch is updated, it would compare the saved mtime and current mtime (in lib/bb/checksum.py), and decide whether need recompute.

So I think that this patch is fine.

The testcase:

$ bitbake --server-only -T -1
$ bitbake quilt-native

# Update quilt-native's patch to test the rebuild:

$ echo "hello world" >> /path/to/meta/recipes-devtools/quilt/quilt/0001-tests-Allow-different-output-from-mv.patch

# Build quilt-native again

$ bitbake quilt-native

NOTE: Tasks Summary: Attempted 10 tasks of which 0 didn't need to be rerun and all succeeded.

We can see that it has been rebuilt.

// Robert

Quoted 8 lines

Its looking like we need better tests for some of this code too :/.

Cheers,

Richard

#12 lore raw

PATCH 8/8 bitbake: cooker: improve inotify handling

In reply to Robert Yang

Each recipe calls bb_cache.add_info() and it calls add_filewatch() for every depend file (not only existed files, but also non-exsited/attempted ones), which caused the parsing and loading very slow. The watchers are global, so we can only pass new depend files to add_filewatch() to improve the performance.

Here is the testing of 54 layers before and after the patch.

              Parsing       Loading
Before        35s           33s
Now           11s           9s

24 seconds are saved for parsing, and 24s are saved for loading.

---
 bitbake/lib/bb/cache.py  |  9 ++++++---
 bitbake/lib/bb/cooker.py | 24 +++++++++++++++---------
 2 files changed, 21 insertions(+), 12 deletions(-)
bitbake/lib/bb/cache.py +6 -3
index 168a77a..7d65939 100644
@@ -713,12 +713,15 @@ class Cache(NoCache):
     def mtime(cachefile):
         return bb.parse.cached_mtime_noerror(cachefile)
 
-    def add_info(self, filename, info_array, cacheData, parsed=None, watcher=None):
+    def add_info(self, filename, info_array, cacheData, parsed=None, watcher=None, watch_manager=None):
         if isinstance(info_array[0], CoreRecipeInfo) and (not info_array[0].skipped):
             cacheData.add_from_recipeinfo(filename, info_array)
 
-            if watcher:
-                watcher(info_array[0].file_depends)
+            if watcher and watch_manager:
+                deps_set = set()
+                for deps in info_array[0].file_depends:
+                    deps_set.add(deps[0])
+                watcher(deps_set - watch_manager.bbseen)
 
         if not self.has_cache:
             return
bitbake/lib/bb/cooker.py +15 -9
index ffda225..cd98ba0 100644
@@ -176,13 +176,13 @@ class BBCooker:
         self.configuration = configuration
 
         self.configwatcher = pyinotify.WatchManager()
-        self.configwatcher.bbseen = []
+        self.configwatcher.bbseen = set()
         self.confignotifier = pyinotify.Notifier(self.configwatcher, self.config_notifications)
         self.watchmask = pyinotify.IN_CLOSE_WRITE | pyinotify.IN_CREATE | pyinotify.IN_DELETE | \
                          pyinotify.IN_DELETE_SELF | pyinotify.IN_MODIFY | pyinotify.IN_MOVE_SELF | \
                          pyinotify.IN_MOVED_FROM | pyinotify.IN_MOVED_TO
         self.watcher = pyinotify.WatchManager()
-        self.watcher.bbseen = []
+        self.watcher.bbseen = set()
         self.notifier = pyinotify.Notifier(self.watcher, self.notifications)
 
         # If being called by something like tinfoil, we need to clean cached data
@@ -278,12 +278,14 @@ class BBCooker:
             watcher = self.watcher
         for i in deps:
             if dirs:
-                f = i[0]
+                f = i
             else:
-                f = os.path.dirname(i[0])
+                f = os.path.dirname(i)
             if f in watcher.bbseen:
                 continue
-            watcher.bbseen.append(f)
+            # Add both files and directories to to bbseen
+            watcher.bbseen.add(f)
+            watcher.bbseen.add(i)
             while True:
                 # We try and add watches for files that don't exist but if they did, would influence
                 # the parser. The parent directory of these files may not exist, in which case we need
@@ -296,7 +298,7 @@ class BBCooker:
                         f = os.path.dirname(f)
                         if f in watcher.bbseen:
                             break
-                        watcher.bbseen.append(f)
+                        watcher.bbseen.add(f)
                         continue
                     if 'ENOSPC' in str(e):
                         providerlog.error("No space left on device or exceeds fs.inotify.max_user_watches?")
@@ -367,7 +369,10 @@ class BBCooker:
             self.disableDataTracking()
 
         self.data.renameVar("__depends", "__base_depends")
-        self.add_filewatch(self.data.getVar("__base_depends", False), self.configwatcher)
+        deps_set = set()
+        for deps in self.data.getVar("__base_depends", False):
+            deps_set.add(deps[0])
+        self.add_filewatch(deps_set, self.configwatcher)
 
         self.baseconfig_valid = True
         self.parsecache_valid = False
@@ -1510,7 +1515,7 @@ class BBCooker:
 
             # Add inotify watches for directories searched for bb/bbappend files
             for dirent in searchdirs:
-                self.add_filewatch([[dirent]], dirs=True)
+                self.add_filewatch([dirent], dirs=True)
 
             self.parser = CookerParser(self, filelist, masked)
             self.parsecache_valid = True
@@ -2144,7 +2149,8 @@ class CookerParser(object):
                 self.cooker.skiplist[virtualfn] = SkippedPackage(info_array[0])
             (fn, cls, mc) = bb.cache.virtualfn2realfn(virtualfn)
             self.bb_cache.add_info(virtualfn, info_array, self.cooker.recipecaches[mc],
-                                        parsed=parsed, watcher = self.cooker.add_filewatch)
+                                        parsed=parsed, watcher=self.cooker.add_filewatch,
+                                        watch_manager=self.cooker.watcher)
         return True
 
     def reparse(self, filename):
Signature · 1 line
2.7.4