Skip to contentSkip to conversation

OE-side fixes for bitbake-diffsigs

PATCH New Paul Eggleton 1 participant 3 messages Apr 6, 2017

lore
Share & tools
lore

Paul Eggleton

author
#0 lore raw

cover OE-side fixes for bitbake-diffsigs

Fixes for lib/oe/sstatesig.py corresponding to the bitbake-diffsigs patchset I just sent to bitbake-devel.

The following changes since commit 901659a51cd53625a93f57a9c5865e90a07ec09d:

oeqa/runtime/utils/targetbuildproject: use parent classes defaults tmpdir (2017-04-06 10:13:34 +0100)

are available in the git repository at:

git://git.openembedded.org/openembedded-core-contrib paule/diffsigs-fixes-oe
http://cgit.openembedded.org/openembedded-core-contrib/log/?h=paule/diffsigs-fixes-oe

Paul Eggleton (2):
lib/oe/sstatesig: fix finding native siginfo files in sstate-cache lib/oe/sstatesig: avoid reporting duplicate siginfo files from sstate

 meta/lib/oe/sstatesig.py | 39 ++++++++++++++++++++-------------------
 1 file changed, 20 insertions(+), 19 deletions(-)
Signature · 1 line
2.9.3
author
#1 lore raw

PATCH 1/2 lib/oe/sstatesig: fix finding native siginfo files in sstate-cache

When comparing signatures with bitbake-diffsigs -t or bitbake -S printdiff, we use this find_siginfo() function implemented in this module to find the siginfo/sigdata files corresponding to the tasks we're looking for. However, native sstate files go into a NATIVELSBSTRING subdirectory and there was no handling for this when asking about native recipes.

I'm not even sure why we were walking SSTATE_DIR in order to find this - we don't need to, we just need to run glob.glob() on the filespec we calculate, which should be a little bit more efficient.

Signed-off-by Paul Eggleton <[email protected]>
---
 meta/lib/oe/sstatesig.py | 25 +++++++++----------------
 1 file changed, 9 insertions(+), 16 deletions(-)
meta/lib/oe/sstatesig.py +9 -16
index 13fd3bd..a76a031 100644
@@ -318,22 +318,15 @@ def find_siginfo(pn, taskname, taskhashlist, d):
             sstatename = taskname[3:]
             filespec = '%s_%s.*.siginfo' % (localdata.getVar('SSTATE_PKG'), sstatename)
 
-            if hashval != '*':
-                sstatedir = "%s/%s" % (d.getVar('SSTATE_DIR'), hashval[:2])
-            else:
-                sstatedir = d.getVar('SSTATE_DIR')
-
-            for root, dirs, files in os.walk(sstatedir):
-                for fn in files:
-                    fullpath = os.path.join(root, fn)
-                    if fnmatch.fnmatch(fullpath, filespec):
-                        if taskhashlist:
-                            hashfiles[hashval] = fullpath
-                        else:
-                            try:
-                                filedates[fullpath] = os.stat(fullpath).st_mtime
-                            except:
-                                continue
+            matchedfiles = glob.glob(filespec)
+            for fullpath in matchedfiles:
+                if taskhashlist:
+                    hashfiles[hashval] = fullpath
+                else:
+                    try:
+                        filedates[fullpath] = os.stat(fullpath).st_mtime
+                    except:
+                        continue
 
     if taskhashlist:
         return hashfiles
Signature · 1 line
2.9.3
author
#2 lore raw

PATCH 2/2 lib/oe/sstatesig: avoid reporting duplicate siginfo files from sstate

In reply to Paul Eggleton

In find_siginfo(), which is used by bitbake-diffsigs among other things, avoid adding a siginfo file from the sstate-cache where we've already collected a sigdata file from the stamps directory with the same hash. This avoids the possibility that the top two files (as picked by default using the bitbake-diffsigs -t option) are for the same signature and thus the tool would report no differences. In order to do that, just use the hashfiles dict that we already have - we just need to change the code to populate that even if we're collecting matching files without looking for a fixed set of hashes (i.e. taskhashlist isn't set).

This replaces previous code in bitbake-diffsigs that attempted to filter these out with limited success.

Signed-off-by Paul Eggleton <[email protected]>
---
 meta/lib/oe/sstatesig.py | 20 ++++++++++++++------
 1 file changed, 14 insertions(+), 6 deletions(-)
meta/lib/oe/sstatesig.py +14 -6
index a76a031..8d93013 100644
@@ -254,9 +254,6 @@ def find_siginfo(pn, taskname, taskhashlist, d):
     import fnmatch
     import glob
 
-    if taskhashlist:
-        hashfiles = {}
-
     if not taskname:
         # We have to derive pn and taskname
         key = pn

@@ -266,8 +263,15 @@ def find_siginfo(pn, taskname, taskhashlist, d):
         if key.startswith('virtual:native:'):
             pn = pn + '-native'
 
+    hashfiles = {}
     filedates = {}
 
+    def get_hashval(siginfo):
+        if siginfo.endswith('.siginfo'):
+            return siginfo.rpartition(':')[2].partition('_')[0]
+        else:
+            return siginfo.rpartition('.')[2]
+
     # First search in stamps dir
     localdata = d.createCopy()
     localdata.setVar('MULTIMACH_TARGET_SYS', '*')

@@ -297,6 +301,8 @@ def find_siginfo(pn, taskname, taskhashlist, d):
                 filedates[fullpath] = os.stat(fullpath).st_mtime
             except OSError:
                 continue
+            hashval = get_hashval(fullpath)
+            hashfiles[hashval] = fullpath
 
     if not taskhashlist or (len(filedates) < 2 and not foundall):
         # That didn't work, look in sstate-cache

@@ -320,9 +326,11 @@ def find_siginfo(pn, taskname, taskhashlist, d):
 
             matchedfiles = glob.glob(filespec)
             for fullpath in matchedfiles:
-                if taskhashlist:
-                    hashfiles[hashval] = fullpath
-                else:
+                actual_hashval = get_hashval(fullpath)
+                if actual_hashval in hashfiles:
+                    continue
+                hashfiles[hashval] = fullpath
+                if not taskhashlist:
                     try:
                         filedates[fullpath] = os.stat(fullpath).st_mtime
                     except:
Signature · 1 line
2.9.3