siggen.py: Fix diffsigs output for filename comparisions

PATCH In review Richard Purdie 2 participants 2 messages Nov 17, 2011

Share & tools

Richard Purdie

author

2011-11-17 14:08 UTC #0 lore raw

PATCH 1/1 siggen.py: Fix diffsigs output for filename comparisions

When comparing sig files, if the recipe locations had changed, the dependent tasks list would show as changed even if the actual hash had not changed. This updates the code to only compare the base part of the pathnames.

It also tweaks some of the output to add newlines to aid comparing two lists of variables as it makes the location of the difference clearer.

Signed-off-by Richard Purdie <[email protected]>

---

bitbake/lib/bb/siggen.py +15 -3

index 4ccfc7d..217f29b 100644
@@ -236,6 +236,16 @@ def compare_sigfiles(a, b):
         removed = sb - sa
         return changed, added, removed
 
+    def clean_basepaths(a):
+        b = {}
+        for x in a:
+            if x.startswith("virtual:"):
+                y = x.rsplit(":", 1)[0] + x.rsplit("/", 1)[1]
+            else:
+                y = x.rsplit("/", 1)[1]
+            b[y] = a[x]
+        return b
+
     if 'basewhitelist' in a_data and a_data['basewhitelist'] != b_data['basewhitelist']:
         print "basewhitelist changed from %s to %s" % (a_data['basewhitelist'], b_data['basewhitelist'])

@@ -243,7 +253,7 @@ def compare_sigfiles(a, b):
         print "taskwhitelist changed from %s to %s" % (a_data['taskwhitelist'], b_data['taskwhitelist'])
 
     if a_data['taskdeps'] != b_data['taskdeps']:
-        print "Task dependencies changed from %s to %s" % (sorted(a_data['taskdeps']), sorted(b_data['taskdeps']))
+        print "Task dependencies changed from:\n%s\nto:\n%s" % (sorted(a_data['taskdeps']), sorted(b_data['taskdeps']))
 
     if a_data['basehash'] != b_data['basehash']:
         print "basehash changed from %s to %s" % (a_data['basehash'], b_data['basehash'])

@@ -266,7 +276,9 @@ def compare_sigfiles(a, b):
             print "Variable %s value changed from %s to %s" % (dep, a_data['varvals'][dep], b_data['varvals'][dep])
 
     if 'runtaskhashes' in a_data and 'runtaskhashes' in b_data:
-        changed, added, removed = dict_diff(a_data['runtaskhashes'], b_data['runtaskhashes'])
+        a = clean_basepaths(a_data['runtaskhashes'])
+        b = clean_basepaths(b_data['runtaskhashes'])
+        changed, added, removed = dict_diff(a, b)
         if added:
             for dep in added:
                 print "Dependency on task %s was added" % (dep)

@@ -275,7 +287,7 @@ def compare_sigfiles(a, b):
                 print "Dependency on task %s was removed" % (dep)
         if changed:
             for dep in changed:
-                print "Hash for dependent task %s changed from %s to %s" % (dep, a_data['runtaskhashes'][dep], b_data['runtaskhashes'][dep])
+                print "Hash for dependent task %s changed from %s to %s" % (dep, a[dep], b[dep])
     elif 'runtaskdeps' in a_data and 'runtaskdeps' in b_data and sorted(a_data['runtaskdeps']) != sorted(b_data['runtaskdeps']):
         print "Tasks this task depends on changed from %s to %s" % (sorted(a_data['runtaskdeps']), sorted(b_data['runtaskdeps']))

McClintock Matthew-B29882

2011-11-17 16:42 UTC #1 lore raw

Quoted 10 lines · Richard Purdie

Signed-off-by Richard Purdie <[email protected]>
---
bitbake/lib/bb/siggen.py +0 -0
index 4ccfc7d..217f29b 100644
@@ -236,6 +236,16 @@ def compare_sigfiles(a, b):
removed = sb - sa
return changed, added, removed

Quoted 10 lines

+ def clean_basepaths(a):
+ b = {}
+ for x in a:
+ if x.startswith("virtual:"):
+ y = x.rsplit(":", 1)[0] + x.rsplit("/", 1)[1]
+ else:
+ y = x.rsplit("/", 1)[1]
+ b[y] = a[x]
+ return b
+

So you are stripping off the full path leading up to the recipe name? I added some stuff to strip off COREBASE at some point. Does this work with layers in any location? I would think so.

Quoted 25 lines

if 'basewhitelist' in a_data and a_data['basewhitelist'] != b_data['basewhitelist']:
print "basewhitelist changed from %s to %s" % (a_data['basewhitelist'], b_data['basewhitelist'])
bitbake/lib/bb/siggen.py +0 -0
@@ -243,7 +253,7 @@ def compare_sigfiles(a, b):
         print "taskwhitelist changed from %s to %s" % (a_data['taskwhitelist'], b_data['taskwhitelist'])
 
if a_data['taskdeps'] != b_data['taskdeps']:
- print "Task dependencies changed from %s to %s" % (sorted(a_data['taskdeps']), sorted(b_data['taskdeps']))
+ print "Task dependencies changed from:\n%s\nto:\n%s" % (sorted(a_data['taskdeps']), sorted(b_data['taskdeps']))
if a_data['basehash'] != b_data['basehash']:
print "basehash changed from %s to %s" % (a_data['basehash'], b_data['basehash'])
bitbake/lib/bb/siggen.py +0 -0
@@ -266,7 +276,9 @@ def compare_sigfiles(a, b):
             print "Variable %s value changed from %s to %s" % (dep, a_data['varvals'][dep], b_data['varvals'][dep])
 
if 'runtaskhashes' in a_data and 'runtaskhashes' in b_data:
- changed, added, removed = dict_diff(a_data['runtaskhashes'], b_data['runtaskhashes'])
+ a = clean_basepaths(a_data['runtaskhashes'])
+ b = clean_basepaths(b_data['runtaskhashes'])
+ changed, added, removed = dict_diff(a, b)
if added:
for dep in added:
print "Dependency on task %s was added" % (dep)
bitbake/lib/bb/siggen.py +0 -0
@@ -275,7 +287,7 @@ def compare_sigfiles(a, b):
                 print "Dependency on task %s was removed" % (dep)

Quoted 7 lines

if changed:
for dep in changed:
- print "Hash for dependent task %s changed from %s to %s" % (dep, a_data['runtaskhashes'][dep], b_data['runtaskhashes'][dep])
+ print "Hash for dependent task %s changed from %s to %s" % (dep, a[dep], b[dep])
elif 'runtaskdeps' in a_data and 'runtaskdeps' in b_data and sorted(a_data['runtaskdeps']) != sorted(b_data['runtaskdeps']):
print "Tasks this task depends on changed from %s to %s" % (sorted(a_data['runtaskdeps']), sorted(b_data['runtaskdeps']))

So now we get shown the tashhash is different even when the paths/filenames are different? This is helpful for sure - took me a bit to get to this realization.

Some other bitbake-diffsig improvements/comments:

1) For showing the difference between two sets, it would be helpful to also show the symmetrical_difference to help see what is different more easily.

2) For taskdeps, if the task depends is part of the same recipe, it would be helpful to show that. For example if strace.bb.do_package depends on strace.bb.do_install, can't we just show that instead?

3) Continuing on #2 it would be nice to import all sigs from a folder and traverse down the task dependencies, and then even show whats difference between two sets of sstate-cache. For example:

strace.bb.do_package -> strace.bb.do_install -> eglibc.bb.do_install |-> foo.bb.do_install |-> bar.bb.do_install

Showing the hash above as well. Then somehow do a diff between the two trees... but even just showing this tree is a good first step. Maybe there is a way to dump this without even involving sstate-cache but just via bitbake?

Maybe there are easy ways to do these things already? Feedback welcome =)

-M