Skip to contentSkip to conversation

bitbake-diffsigs fixes/improvements

PATCH In review Paul Eggleton 4 participants 22 messages Apr 6, 2017 – Apr 7, 2017

lore
Share & tools
lore
#0 lore raw

cover bitbake-diffsigs fixes/improvements

As part of an effort to add task signature recording and comparison to buildhistory in OE, I went digging into bitbake-diffsigs and the code that supports it and discovered that unfortunately it had a number of bugs - in particular the -t option despite being around for a while and having numerous band-aids applied in the past still wasn't really working properly. I have to take a big chunk of the responsibility for this as I wrote the thing in the first place. This patchset corrects most of the issues that I found and also makes a number of improvements to the readability of the output.

NOTE: there are a few corresponding changes required in OE-Core that I am about to send out, these should be applied at the same time.

The following changes since commit 751c9dc51fd01fa64a1ff37ba2638110335f71af:

fetch/local: Drop FILESDIR (2017-04-05 09:38:01 +0100)

are available in the git repository at:

git://git.yoctoproject.org/poky-contrib paule/bb-sigstuff
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=paule/bb-sigstuff

Paul Eggleton (11):
bitbake-diffsigs: fix -t picking wrong files to compare lib/bb/siggen: add missing path separator to cleaned paths
bitbake-diffsigs: properly report which signature is missing
bitbake-diffsigs: drop naive logic for removing duplicate files lib/bb/siggen: show a diff when dumping changes to multi-line values lib/bb/siggen: don't show unchanged runtaskdeps list
bitbake-diffsigs: change to use argparse
bitbake-diffsigs: add an option to find and compare specific signatures lib/bb/siggen: add collapsed mode to compare_sigfiles() lib/bb/siggen: show word-diff for single-line values containing spaces
bitbake-diffsigs: colourise output

 LICENSE                    |   2 +
 bin/bitbake-diffsigs       | 175 +++++++++++++++++++++------------------
 lib/bb/siggen.py           | 166 +++++++++++++++++++++++++++++--------
 lib/simplediff/LICENSE     |  22 +++++
 lib/simplediff/__init__.py | 198 +++++++++++++++++++++++++++++++++++++++++++++
 5 files changed, 453 insertions(+), 110 deletions(-)
 create mode 100644 lib/simplediff/LICENSE
 create mode 100644 lib/simplediff/__init__.py
Signature · 1 line
2.9.3
#1 lore raw

PATCH 1/11 bitbake-diffsigs: fix -t picking wrong files to compare

We weren't picking the right files to compare here - according to the order in which the list is sorted (by mtime), we need to be taking the last two items and not the first two.

---
 bin/bitbake-diffsigs | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
bin/bitbake-diffsigs +3 -3
index 1e3de09..b2ebe91 100755
@@ -84,9 +84,9 @@ def find_compare_task(bbhandler, pn, taskname):
             return recout
 
         # Recurse into signature comparison
-        logger.debug("Signature file (previous): %s" % latestfiles[0])
-        logger.debug("Signature file (latest): %s" % latestfiles[1])
-        output = bb.siggen.compare_sigfiles(latestfiles[0], latestfiles[1], recursecb)
+        logger.debug("Signature file (previous): %s" % latestfiles[-2])
+        logger.debug("Signature file (latest): %s" % latestfiles[-1])
+        output = bb.siggen.compare_sigfiles(latestfiles[-2], latestfiles[-1], recursecb)
         if output:
             print('\n'.join(output))
     sys.exit(0)
Signature · 1 line
2.9.3
#2 lore raw

PATCH 2/11 lib/bb/siggen: add missing path separator to cleaned paths

In reply to Paul Eggleton

Printing "pbzip2pbzip2_1.1.13.bb" is ugly, we need to add a separating slash so that we get "pbzip2/pbzip2_1.1.13.bb" instead.

---
 lib/bb/siggen.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
lib/bb/siggen.py +1 -1
index 3ceeef1..f47af6d 100644
@@ -355,7 +355,7 @@ def clean_basepath(a):
     mc = None
     if a.startswith("multiconfig:"):
         _, mc, a = a.split(":", 2)
-    b = a.rsplit("/", 2)[1] + a.rsplit("/", 2)[2]
+    b = a.rsplit("/", 2)[1] + '/' + a.rsplit("/", 2)[2]
     if a.startswith("virtual:"):
         b = b + ":" + a.rsplit(":", 1)[0]
     if mc:
Signature · 1 line
2.9.3
#3 lore raw

PATCH 3/11 bitbake-diffsigs: properly report which signature is missing

In reply to Paul Eggleton

If just one of the two signatures we want to compare aren't available, report that one rather than misleadingly claiming both are missing.

---
 bin/bitbake-diffsigs | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)
bin/bitbake-diffsigs +7 -3
index b2ebe91..f84188d 100755
@@ -75,11 +75,15 @@ def find_compare_task(bbhandler, pn, taskname):
             hashfiles = bb.siggen.find_siginfo(key, None, hashes, bbhandler.config_data)
 
             recout = []
-            if len(hashfiles) == 2:
+            if len(hashfiles) == 0:
+                recout.append("Unable to find matching sigdata for %s with hashes %s or %s" % (key, hash1, hash2))
+            elif not hash1 in hashfiles:
+                recout.append("Unable to find matching sigdata for %s with hash %s" % (key, hash1))
+            elif not hash2 in hashfiles:
+                recout.append("Unable to find matching sigdata for %s with hash %s" % (key, hash2))
+            else:
                 out2 = bb.siggen.compare_sigfiles(hashfiles[hash1], hashfiles[hash2], recursecb)
                 recout.extend(list('  ' + l for l in out2))
-            else:
-                recout.append("Unable to find matching sigdata for %s with hashes %s or %s" % (key, hash1, hash2))
 
             return recout
 
Signature · 1 line
2.9.3
#5 lore raw

PATCH 4/11 bitbake-diffsigs: drop naive logic for removing duplicate files

In reply to Paul Eggleton

This logic doesn't work in practice, certainly not with current versions where sigdata files are preserved in the stamps directory and therefore there will often be multiple sigdata files - you can now easily get files for the same signature from sstate and the stamps directory with the result that bitbake-diffsigs reports nothing has changed. Instead, let's change the find_siginfo function in OE-Core to simply not return duplicates so we don't have to filter them out here.

---
 bin/bitbake-diffsigs | 16 ----------------
 1 file changed, 16 deletions(-)
bin/bitbake-diffsigs +0 -16
index f84188d..5400e5b 100755
@@ -37,12 +37,6 @@ logger = bb.msg.logger_create('bitbake-diffsigs')
 def find_compare_task(bbhandler, pn, taskname):
     """ Find the most recent signature files for the specified PN/task and compare them """
 
-    def get_hashval(siginfo):
-        if siginfo.endswith('.siginfo'):
-            return siginfo.rpartition(':')[2].partition('_')[0]
-        else:
-            return siginfo.rpartition('.')[2]
-
     if not hasattr(bb.siggen, 'find_siginfo'):
         logger.error('Metadata does not support finding signature data files')
         sys.exit(1)
@@ -59,16 +53,6 @@ def find_compare_task(bbhandler, pn, taskname):
         logger.error('Only one matching sigdata file found for the specified task (%s %s)' % (pn, taskname))
         sys.exit(1)
     else:
-        # It's possible that latestfiles contain 3 elements and the first two have the same hash value.
-        # In this case, we delete the second element.
-        # The above case is actually the most common one. Because we may have sigdata file and siginfo
-        # file having the same hash value. Comparing such two files makes no sense.
-        if len(latestfiles) == 3:
-            hash0 = get_hashval(latestfiles[0])
-            hash1 = get_hashval(latestfiles[1])
-            if hash0 == hash1:
-                latestfiles.pop(1)
-
         # Define recursion callback
         def recursecb(key, hash1, hash2):
             hashes = [hash1, hash2]
Signature · 1 line
2.9.3
#6 lore raw

PATCH 5/11 lib/bb/siggen: show a diff when dumping changes to multi-line values

In reply to Paul Eggleton

When dumping changes to signatures e.g. output of bitbake -s printdiff, if for example a function has changed, it's much more readable to see a unified diff of the changes rather than just printing the old function followed by the new function, so use difflib to do that.

Note: I elected to keep to one item in the returned list per change, rather than one line per line of output, so that the caller can still look at changes individually if needed. Thus I've added some handling to bitbake-diffsigs to split the change into lines so that each line is displayed indented.

---
 bin/bitbake-diffsigs |  4 +++-
 lib/bb/siggen.py     | 12 +++++++++++-
 2 files changed, 14 insertions(+), 2 deletions(-)
bin/bitbake-diffsigs +3 -1
index 5400e5b..4ca085f 100755
@@ -67,7 +67,9 @@ def find_compare_task(bbhandler, pn, taskname):
                 recout.append("Unable to find matching sigdata for %s with hash %s" % (key, hash2))
             else:
                 out2 = bb.siggen.compare_sigfiles(hashfiles[hash1], hashfiles[hash2], recursecb)
-                recout.extend(list('  ' + l for l in out2))
+                for change in out2:
+                    for line in change.splitlines():
+                        recout.append('  ' + line)
 
             return recout
 
lib/bb/siggen.py +11 -1
index f47af6d..f497fb9 100644
@@ -5,6 +5,7 @@ import re
 import tempfile
 import pickle
 import bb.data
+import difflib
 from bb.checksum import FileChecksumCache
 
 logger = logging.getLogger('BitBake.SigGen')
@@ -462,7 +463,16 @@ def compare_sigfiles(a, b, recursecb = None):
     changed, added, removed = dict_diff(a_data['varvals'], b_data['varvals'])
     if changed:
         for dep in changed:
-            output.append("Variable %s value changed from '%s' to '%s'" % (dep, a_data['varvals'][dep], b_data['varvals'][dep]))
+            oldval = a_data['varvals'][dep]
+            newval = b_data['varvals'][dep]
+            if newval and oldval and ('\n' in oldval or '\n' in newval):
+                diff = difflib.unified_diff(oldval.splitlines(), newval.splitlines(), lineterm='')
+                # Cut off the first two lines, since we aren't interested in
+                # the old/new filename (they are blank anyway in this case)
+                difflines = list(diff)[2:]
+                output.append("Variable %s value changed:\n%s" % (dep, '\n'.join(difflines)))
+            else:
+                output.append("Variable %s value changed from '%s' to '%s'" % (dep, oldval, newval))
 
     if not 'file_checksum_values' in a_data:
          a_data['file_checksum_values'] = {}
Signature · 1 line
2.9.3
#7 lore raw

PATCH 6/11 lib/bb/siggen: don't show unchanged runtaskdeps list

In reply to Paul Eggleton

If the runtaskdeps list hasn't actually changed (but the signatures of some of the tasks did) then it doesn't make sense to print out the old and new lists as they are both the same and may be very long, e.g. for do_rootfs in OE.

---
 lib/bb/siggen.py | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)
lib/bb/siggen.py +6 -2
index f497fb9..c6b14c2 100644
@@ -506,10 +506,14 @@ def compare_sigfiles(a, b, recursecb = None):
                 changed.append("%s with hash %s\n changed to\n%s with hash %s" % (a, a_data['runtaskhashes'][a], b, b_data['runtaskhashes'][b]))
 
     if changed:
-        output.append("runtaskdeps changed from %s to %s" % (clean_basepaths_list(a_data['runtaskdeps']), clean_basepaths_list(b_data['runtaskdeps'])))
+        clean_a = clean_basepaths_list(a_data['runtaskdeps'])
+        clean_b = clean_basepaths_list(b_data['runtaskdeps'])
+        if clean_a != clean_b:
+            output.append("runtaskdeps changed from %s to %s" % (clean_a, clean_b))
+        else:
+            output.append("runtaskdeps changed:")
         output.append("\n".join(changed))
 
-
     if 'runtaskhashes' in a_data and 'runtaskhashes' in b_data:
         a = a_data['runtaskhashes']
         b = b_data['runtaskhashes']
Signature · 1 line
2.9.3
#8 lore raw

PATCH 7/11 bitbake-diffsigs: change to use argparse

In reply to Paul Eggleton

Argparse is a bit easier to deal with than optparse, and since we're about to add some options, migrate this script over.

---
 bin/bitbake-diffsigs | 64 +++++++++++++++++++++++++++-------------------------
 1 file changed, 33 insertions(+), 31 deletions(-)
bin/bitbake-diffsigs +33 -31
index 4ca085f..e3f848d 100755
@@ -3,7 +3,7 @@
 # bitbake-diffsigs
 # BitBake task signature data comparison utility
 #
-# Copyright (C) 2012-2013 Intel Corporation
+# Copyright (C) 2012-2013, 2017 Intel Corporation
 #
 # This program is free software; you can redistribute it and/or modify
 # it under the terms of the GNU General Public License version 2 as
@@ -22,7 +22,7 @@ import os
 import sys
 import warnings
 import fnmatch
-import optparse
+import argparse
 import logging
 import pickle
 
@@ -83,22 +83,27 @@ def find_compare_task(bbhandler, pn, taskname):
 
 
 
-parser = optparse.OptionParser(
-    description = "Compares siginfo/sigdata files written out by BitBake",
-    usage = """
-  %prog -t recipename taskname
-  %prog sigdatafile1 sigdatafile2
-  %prog sigdatafile1""")
+parser = argparse.ArgumentParser(
+    description="Compares siginfo/sigdata files written out by BitBake")
 
-parser.add_option("-D", "--debug",
-        help = "enable debug",
-        action = "store_true", dest="debug", default = False)
+parser.add_argument('-d', '--debug',
+                    help='Enable debug output',
+                    action='store_true')
 
-parser.add_option("-t", "--task",
-        help = "find the signature data files for last two runs of the specified task and compare them",
-        action="store", dest="taskargs", nargs=2, metavar='recipename taskname')
+parser.add_argument("-t", "--task",
+        help="find the signature data files for last two runs of the specified task and compare them",
+        action="store", dest="taskargs", nargs=2, metavar=('recipename', 'taskname'))
 
-options, args = parser.parse_args(sys.argv)
+parser.add_argument("sigdatafile1",
+        help="First signature file to compare (or signature file to dump, if second not specified). Not used when using -t/--task.",
+        action="store", nargs='?')
+
+parser.add_argument("sigdatafile2",
+        help="Second signature file to compare",
+        action="store", nargs='?')
+
+
+options = parser.parse_args()
 
 if options.debug:
     logger.setLevel(logging.DEBUG)
@@ -108,20 +113,17 @@ if options.taskargs:
         tinfoil.prepare(config_only=True)
         find_compare_task(tinfoil, options.taskargs[0], options.taskargs[1])
 else:
-    if len(args) == 1:
-        parser.print_help()
-    else:
-        try:
-            if len(args) == 2:
-                output = bb.siggen.dump_sigfile(sys.argv[1])
-            else:
-                output = bb.siggen.compare_sigfiles(sys.argv[1], sys.argv[2])
-        except IOError as e:
-            logger.error(str(e))
-            sys.exit(1)
-        except (pickle.UnpicklingError, EOFError):
-            logger.error('Invalid signature data - ensure you are specifying sigdata/siginfo files')
-            sys.exit(1)
+    try:
+        if options.sigdatafile1 and options.sigdatafile2:
+            output = bb.siggen.compare_sigfiles(options.sigdatafile1, options.sigdatafile2)
+        elif options.sigdatafile1:
+            output = bb.siggen.dump_sigfile(options.sigdatafile1)
+    except IOError as e:
+        logger.error(str(e))
+        sys.exit(1)
+    except (pickle.UnpicklingError, EOFError):
+        logger.error('Invalid signature data - ensure you are specifying sigdata/siginfo files')
+        sys.exit(1)
 
-        if output:
-            print('\n'.join(output))
+    if output:
+        print('\n'.join(output))
Signature · 1 line
2.9.3
#9 lore raw

PATCH 8/11 bitbake-diffsigs: add an option to find and compare specific signatures

In reply to Paul Eggleton

With the -t option which recurses to find the ultimate cause of a signature change, it was hardcoded to take the last two executions of the specified task. On the other hand, if you have two specific task hashes (say from bitbake output, or some other tool) then you'll want to pick those, so provide an option to specify those as well. (Note, the new -s option needs to be specified alongside -t rather than instead of it.)

---
 bin/bitbake-diffsigs | 95 ++++++++++++++++++++++++++++++++--------------------
 1 file changed, 59 insertions(+), 36 deletions(-)
bin/bitbake-diffsigs +59 -36
index e3f848d..e9fdb48 100755
@@ -34,7 +34,7 @@ import bb.msg
 
 logger = bb.msg.logger_create('bitbake-diffsigs')
 
-def find_compare_task(bbhandler, pn, taskname):
+def find_compare_task(bbhandler, pn, taskname, sig1=None, sig2=None):
     """ Find the most recent signature files for the specified PN/task and compare them """
 
     if not hasattr(bb.siggen, 'find_siginfo'):
@@ -44,41 +44,54 @@ def find_compare_task(bbhandler, pn, taskname):
     if not taskname.startswith('do_'):
         taskname = 'do_%s' % taskname
 
-    filedates = bb.siggen.find_siginfo(pn, taskname, None, bbhandler.config_data)
-    latestfiles = sorted(filedates.keys(), key=lambda f: filedates[f])[-3:]
-    if not latestfiles:
-        logger.error('No sigdata files found matching %s %s' % (pn, taskname))
-        sys.exit(1)
-    elif len(latestfiles) < 2:
-        logger.error('Only one matching sigdata file found for the specified task (%s %s)' % (pn, taskname))
-        sys.exit(1)
+    if sig1 and sig2:
+        sigfiles = bb.siggen.find_siginfo(pn, taskname, [sig1, sig2], bbhandler.config_data)
+        if len(sigfiles) == 0:
+            logger.error('No sigdata files found matching %s %s matching either %s or %s' % (pn, taskname, sig1, sig2))
+            sys.exit(1)
+        elif not sig1 in sigfiles:
+            logger.error('No sigdata files found matching %s %s with signature %s' % (pn, taskname, sig1))
+            sys.exit(1)
+        elif not sig2 in sigfiles:
+            logger.error('No sigdata files found matching %s %s with signature %s' % (pn, taskname, sig2))
+            sys.exit(1)
+        latestfiles = [sigfiles[sig1], sigfiles[sig2]]
     else:
-        # Define recursion callback
-        def recursecb(key, hash1, hash2):
-            hashes = [hash1, hash2]
-            hashfiles = bb.siggen.find_siginfo(key, None, hashes, bbhandler.config_data)
-
-            recout = []
-            if len(hashfiles) == 0:
-                recout.append("Unable to find matching sigdata for %s with hashes %s or %s" % (key, hash1, hash2))
-            elif not hash1 in hashfiles:
-                recout.append("Unable to find matching sigdata for %s with hash %s" % (key, hash1))
-            elif not hash2 in hashfiles:
-                recout.append("Unable to find matching sigdata for %s with hash %s" % (key, hash2))
-            else:
-                out2 = bb.siggen.compare_sigfiles(hashfiles[hash1], hashfiles[hash2], recursecb)
-                for change in out2:
-                    for line in change.splitlines():
-                        recout.append('  ' + line)
-
-            return recout
-
-        # Recurse into signature comparison
-        logger.debug("Signature file (previous): %s" % latestfiles[-2])
-        logger.debug("Signature file (latest): %s" % latestfiles[-1])
-        output = bb.siggen.compare_sigfiles(latestfiles[-2], latestfiles[-1], recursecb)
-        if output:
-            print('\n'.join(output))
+        filedates = bb.siggen.find_siginfo(pn, taskname, None, bbhandler.config_data)
+        latestfiles = sorted(filedates.keys(), key=lambda f: filedates[f])[-3:]
+        if not latestfiles:
+            logger.error('No sigdata files found matching %s %s' % (pn, taskname))
+            sys.exit(1)
+        elif len(latestfiles) < 2:
+            logger.error('Only one matching sigdata file found for the specified task (%s %s)' % (pn, taskname))
+            sys.exit(1)
+
+    # Define recursion callback
+    def recursecb(key, hash1, hash2):
+        hashes = [hash1, hash2]
+        hashfiles = bb.siggen.find_siginfo(key, None, hashes, bbhandler.config_data)
+
+        recout = []
+        if len(hashfiles) == 0:
+            recout.append("Unable to find matching sigdata for %s with hashes %s or %s" % (key, hash1, hash2))
+        elif not hash1 in hashfiles:
+            recout.append("Unable to find matching sigdata for %s with hash %s" % (key, hash1))
+        elif not hash2 in hashfiles:
+            recout.append("Unable to find matching sigdata for %s with hash %s" % (key, hash2))
+        else:
+            out2 = bb.siggen.compare_sigfiles(hashfiles[hash1], hashfiles[hash2], recursecb)
+            for change in out2:
+                for line in change.splitlines():
+                    recout.append('  ' + line)
+
+        return recout
+
+    # Recurse into signature comparison
+    logger.debug("Signature file (previous): %s" % latestfiles[-2])
+    logger.debug("Signature file (latest): %s" % latestfiles[-1])
+    output = bb.siggen.compare_sigfiles(latestfiles[-2], latestfiles[-1], recursecb)
+    if output:
+        print('\n'.join(output))
     sys.exit(0)
 
 
@@ -94,6 +107,10 @@ parser.add_argument("-t", "--task",
         help="find the signature data files for last two runs of the specified task and compare them",
         action="store", dest="taskargs", nargs=2, metavar=('recipename', 'taskname'))
 
+parser.add_argument("-s", "--signature",
+        help="With -t/--task, specify the signatures to look for instead of taking the last two",
+        action="store", dest="sigargs", nargs=2, metavar=('fromsig', 'tosig'))
+
 parser.add_argument("sigdatafile1",
         help="First signature file to compare (or signature file to dump, if second not specified). Not used when using -t/--task.",
         action="store", nargs='?')
@@ -111,8 +128,14 @@ if options.debug:
 if options.taskargs:
     with bb.tinfoil.Tinfoil() as tinfoil:
         tinfoil.prepare(config_only=True)
-        find_compare_task(tinfoil, options.taskargs[0], options.taskargs[1])
+        if options.sigargs:
+            find_compare_task(tinfoil, options.taskargs[0], options.taskargs[1], options.sigargs[0], options.sigargs[1])
+        else:
+            find_compare_task(tinfoil, options.taskargs[0], options.taskargs[1])
 else:
+    if options.sigargs:
+        logger.error('-s/--signature can only be used together with -t/--task')
+        sys.exit(1)
     try:
         if options.sigdatafile1 and options.sigdatafile2:
             output = bb.siggen.compare_sigfiles(options.sigdatafile1, options.sigdatafile2)
Signature · 1 line
2.9.3
#10 lore raw

PATCH 9/11 lib/bb/siggen: add collapsed mode to compare_sigfiles()

In reply to Paul Eggleton

If we just want to drill down to the actual differences then we don't need to see certain things in the output, e.g. basehash changing or the signature of dependent tasks. This will be used for comparing signatures within buildhistory-diff in OE-Core; the default mode as used by bitbake-diffsigs and bitbake -S printdiff remains unchanged for the moment.

---
 lib/bb/siggen.py | 51 ++++++++++++++++++++++++++++-----------------------
 1 file changed, 28 insertions(+), 23 deletions(-)
lib/bb/siggen.py +28 -23
index c6b14c2..3c5d862 100644
@@ -375,7 +375,7 @@ def clean_basepaths_list(a):
         b.append(clean_basepath(x))
     return b
 
-def compare_sigfiles(a, b, recursecb = None):
+def compare_sigfiles(a, b, recursecb=None, collapsed=False):
     output = []
 
     with open(a, 'rb') as f:
@@ -443,7 +443,7 @@ def compare_sigfiles(a, b, recursecb = None):
     if a_data['taskdeps'] != b_data['taskdeps']:
         output.append("Task dependencies changed from:\n%s\nto:\n%s" % (sorted(a_data['taskdeps']), sorted(b_data['taskdeps'])))
 
-    if a_data['basehash'] != b_data['basehash']:
+    if a_data['basehash'] != b_data['basehash'] and not collapsed:
         output.append("basehash changed from %s to %s" % (a_data['basehash'], b_data['basehash']))
 
     changed, added, removed = dict_diff(a_data['gendeps'], b_data['gendeps'], a_data['basewhitelist'] & b_data['basewhitelist'])
@@ -495,24 +495,25 @@ def compare_sigfiles(a, b, recursecb = None):
     if not 'runtaskdeps' in b_data:
          b_data['runtaskdeps'] = {}
 
-    if len(a_data['runtaskdeps']) != len(b_data['runtaskdeps']):
-        changed = ["Number of task dependencies changed"]
-    else:
-        changed = []
-        for idx, task in enumerate(a_data['runtaskdeps']):
-            a = a_data['runtaskdeps'][idx]
-            b = b_data['runtaskdeps'][idx]
-            if a_data['runtaskhashes'][a] != b_data['runtaskhashes'][b]:
-                changed.append("%s with hash %s\n changed to\n%s with hash %s" % (a, a_data['runtaskhashes'][a], b, b_data['runtaskhashes'][b]))
-
-    if changed:
-        clean_a = clean_basepaths_list(a_data['runtaskdeps'])
-        clean_b = clean_basepaths_list(b_data['runtaskdeps'])
-        if clean_a != clean_b:
-            output.append("runtaskdeps changed from %s to %s" % (clean_a, clean_b))
+    if not collapsed:
+        if len(a_data['runtaskdeps']) != len(b_data['runtaskdeps']):
+            changed = ["Number of task dependencies changed"]
         else:
-            output.append("runtaskdeps changed:")
-        output.append("\n".join(changed))
+            changed = []
+            for idx, task in enumerate(a_data['runtaskdeps']):
+                a = a_data['runtaskdeps'][idx]
+                b = b_data['runtaskdeps'][idx]
+                if a_data['runtaskhashes'][a] != b_data['runtaskhashes'][b]:
+                    changed.append("%s with hash %s\n changed to\n%s with hash %s" % (a, a_data['runtaskhashes'][a], b, b_data['runtaskhashes'][b]))
+
+        if changed:
+            clean_a = clean_basepaths_list(a_data['runtaskdeps'])
+            clean_b = clean_basepaths_list(b_data['runtaskdeps'])
+            if clean_a != clean_b:
+                output.append("runtaskdeps changed from %s to %s" % (clean_a, clean_b))
+            else:
+                output.append("runtaskdeps changed:")
+            output.append("\n".join(changed))
 
     if 'runtaskhashes' in a_data and 'runtaskhashes' in b_data:
         a = a_data['runtaskhashes']
@@ -540,13 +541,17 @@ def compare_sigfiles(a, b, recursecb = None):
                     output.append("Dependency on task %s was removed with hash %s" % (clean_basepath(dep), a[dep]))
         if changed:
             for dep in changed:
-                output.append("Hash for dependent task %s changed from %s to %s" % (clean_basepath(dep), a[dep], b[dep]))
+                if not collapsed:
+                    output.append("Hash for dependent task %s changed from %s to %s" % (clean_basepath(dep), a[dep], b[dep]))
                 if callable(recursecb):
-                    # If a dependent hash changed, might as well print the line above and then defer to the changes in 
-                    # that hash since in all likelyhood, they're the same changes this task also saw.
                     recout = recursecb(dep, a[dep], b[dep])
                     if recout:
-                        output = [output[-1]] + recout
+                        if collapsed:
+                            output.extend(recout)
+                        else:
+                            # If a dependent hash changed, might as well print the line above and then defer to the changes in 
+                            # that hash since in all likelyhood, they're the same changes this task also saw.
+                            output = [output[-1]] + recout
 
     a_taint = a_data.get('taint', None)
     b_taint = b_data.get('taint', None)
Signature · 1 line
2.9.3
#13 lore raw

PATCH 10/11 lib/bb/siggen: show word-diff for single-line values containing spaces

In reply to Paul Eggleton

If a variable value has changed and either the new or old value contains spaces, a word diff should be appropriate and may be a bit more readable. Import the "simplediff" module and use it to show a word diff (in the style of GNU wdiff and git diff --word-diff).

Also use a similar style diff to show changes in the runtaskhashes list. I didn't use an actual word-diff here since it's a little different - we can be sure that the list is a list and not simply a free-format string.

---
 LICENSE                    |   2 +
 lib/bb/siggen.py           |  38 ++++++++-
 lib/simplediff/LICENSE     |  22 +++++
 lib/simplediff/__init__.py | 198 +++++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 259 insertions(+), 1 deletion(-)
 create mode 100644 lib/simplediff/LICENSE
 create mode 100644 lib/simplediff/__init__.py
LICENSE +2 -0
index 5d4a4c2..7d4e5f4 100644
@@ -15,3 +15,5 @@ Foundation and individual contributors.
 * QUnit is redistributed under the MIT license.
 
 * Font Awesome fonts redistributed under the SIL Open Font License 1.1
+
+* simplediff is distributed under the zlib license.
lib/bb/siggen.py +37 -1
index 3c5d862..d40c721 100644
@@ -6,6 +6,7 @@ import tempfile
 import pickle
 import bb.data
 import difflib
+import simplediff
 from bb.checksum import FileChecksumCache
 
 logger = logging.getLogger('BitBake.SigGen')
@@ -352,6 +353,39 @@ def dump_this_task(outfile, d):
     referencestamp = bb.build.stamp_internal(task, d, None, True)
     bb.parse.siggen.dump_sigtask(fn, task, outfile, "customfile:" + referencestamp)
 
+def worddiff_str(oldstr, newstr):
+    diff = simplediff.diff(oldstr.split(' '), newstr.split(' '))
+    ret = []
+    for change, value in diff:
+        value = ' '.join(value)
+        if change == '=':
+            ret.append(value)
+        elif change == '+':
+            item = '{+%s+}' % value
+            ret.append(item)
+        elif change == '-':
+            item = '[-%s-]' % value
+            ret.append(item)
+    whitespace_note = ''
+    if oldstr != newstr and ' '.join(oldstr.split()) == ' '.join(newstr.split()):
+        whitespace_note = ' (whitespace changed)'
+    return '"%s"%s' % (' '.join(ret), whitespace_note)
+
+def list_inline_diff(oldlist, newlist):
+    diff = simplediff.diff(oldlist, newlist)
+    ret = []
+    for change, value in diff:
+        value = ' '.join(value)
+        if change == '=':
+            ret.append("'%s'" % value)
+        elif change == '+':
+            item = "+'%s'" % value
+            ret.append(item)
+        elif change == '-':
+            item = "-'%s'" % value
+            ret.append(item)
+    return '[%s]' % (', '.join(ret))
+
 def clean_basepath(a):
     mc = None
     if a.startswith("multiconfig:"):
@@ -471,6 +505,8 @@ def compare_sigfiles(a, b, recursecb=None, collapsed=False):
                 # the old/new filename (they are blank anyway in this case)
                 difflines = list(diff)[2:]
                 output.append("Variable %s value changed:\n%s" % (dep, '\n'.join(difflines)))
+            elif newval and oldval and (' ' in oldval or ' ' in newval):
+                output.append("Variable %s value changed:\n%s" % (dep, worddiff_str(oldval, newval)))
             else:
                 output.append("Variable %s value changed from '%s' to '%s'" % (dep, oldval, newval))
 
@@ -510,7 +546,7 @@ def compare_sigfiles(a, b, recursecb=None, collapsed=False):
             clean_a = clean_basepaths_list(a_data['runtaskdeps'])
             clean_b = clean_basepaths_list(b_data['runtaskdeps'])
             if clean_a != clean_b:
-                output.append("runtaskdeps changed from %s to %s" % (clean_a, clean_b))
+                output.append("runtaskdeps changed:\n%s" % list_inline_diff(clean_a, clean_b))
             else:
                 output.append("runtaskdeps changed:")
             output.append("\n".join(changed))
lib/simplediff/LICENSE new file +22 -0
new file mode 100644
index 0000000..8242dde
@@ -0,0 +1,22 @@
+Copyright (c) 2008 - 2013 Paul Butler and contributors
+
+This sofware may be used under a zlib/libpng-style license:
+
+This software is provided 'as-is', without any express or implied warranty. In
+no event will the authors be held liable for any damages arising from the use
+of this software.
+
+Permission is granted to anyone to use this software for any purpose, including
+commercial applications, and to alter it and redistribute it freely, subject to
+the following restrictions:
+
+1. The origin of this software must not be misrepresented; you must not claim
+that you wrote the original software. If you use this software in a product, an
+acknowledgment in the product documentation would be appreciated but is not
+required.
+
+2. Altered source versions must be plainly marked as such, and must not be
+misrepresented as being the original software.
+
+3. This notice may not be removed or altered from any source distribution.
+
lib/simplediff/__init__.py new file +198 -0

198 lines not on this page · raw message

Signature · 1 line
2.9.3
#14 lore raw

PATCH 11/11 bitbake-diffsigs: colourise output

In reply to Paul Eggleton

If the output is a TTY, add colour to the output in order to make it easier to read. At the moment this is fairly basic, just add colour to the "titles" of each change and to the diff output.

I tried to introduce this without changing the code too much - rather than moving everything over to the new python formatting style, I've introduced a color_format() function which takes care of the colour formatting, either accepting additional format arguments or alternatively leaving the caller to use the old-style formatting (%) to insert values.

---
 bin/bitbake-diffsigs |  18 ++++++---
 lib/bb/siggen.py     | 101 +++++++++++++++++++++++++++++++++++++--------------
 2 files changed, 85 insertions(+), 34 deletions(-)
bin/bitbake-diffsigs +12 -6
index e9fdb48..884be1e 100755
@@ -34,7 +34,7 @@ import bb.msg
 
 logger = bb.msg.logger_create('bitbake-diffsigs')
 
-def find_compare_task(bbhandler, pn, taskname, sig1=None, sig2=None):
+def find_compare_task(bbhandler, pn, taskname, sig1=None, sig2=None, color=False):
     """ Find the most recent signature files for the specified PN/task and compare them """
 
     if not hasattr(bb.siggen, 'find_siginfo'):
@@ -79,7 +79,7 @@ def find_compare_task(bbhandler, pn, taskname, sig1=None, sig2=None):
         elif not hash2 in hashfiles:
             recout.append("Unable to find matching sigdata for %s with hash %s" % (key, hash2))
         else:
-            out2 = bb.siggen.compare_sigfiles(hashfiles[hash1], hashfiles[hash2], recursecb)
+            out2 = bb.siggen.compare_sigfiles(hashfiles[hash1], hashfiles[hash2], recursecb, color=color)
             for change in out2:
                 for line in change.splitlines():
                     recout.append('  ' + line)
@@ -89,7 +89,7 @@ def find_compare_task(bbhandler, pn, taskname, sig1=None, sig2=None):
     # Recurse into signature comparison
     logger.debug("Signature file (previous): %s" % latestfiles[-2])
     logger.debug("Signature file (latest): %s" % latestfiles[-1])
-    output = bb.siggen.compare_sigfiles(latestfiles[-2], latestfiles[-1], recursecb)
+    output = bb.siggen.compare_sigfiles(latestfiles[-2], latestfiles[-1], recursecb, color=color)
     if output:
         print('\n'.join(output))
     sys.exit(0)
@@ -103,6 +103,10 @@ parser.add_argument('-d', '--debug',
                     help='Enable debug output',
                     action='store_true')
 
+parser.add_argument('--color',
+        help='Colorize output (where %(metavar)s is %(choices)s)',
+        choices=['auto', 'always', 'never'], default='auto', metavar='color')
+
 parser.add_argument("-t", "--task",
         help="find the signature data files for last two runs of the specified task and compare them",
         action="store", dest="taskargs", nargs=2, metavar=('recipename', 'taskname'))
@@ -125,20 +129,22 @@ options = parser.parse_args()
 if options.debug:
     logger.setLevel(logging.DEBUG)
 
+color = (options.color == 'always' or (options.color == 'auto' and sys.stdout.isatty()))
+
 if options.taskargs:
     with bb.tinfoil.Tinfoil() as tinfoil:
         tinfoil.prepare(config_only=True)
         if options.sigargs:
-            find_compare_task(tinfoil, options.taskargs[0], options.taskargs[1], options.sigargs[0], options.sigargs[1])
+            find_compare_task(tinfoil, options.taskargs[0], options.taskargs[1], options.sigargs[0], options.sigargs[1], color=color)
         else:
-            find_compare_task(tinfoil, options.taskargs[0], options.taskargs[1])
+            find_compare_task(tinfoil, options.taskargs[0], options.taskargs[1], color=color)
 else:
     if options.sigargs:
         logger.error('-s/--signature can only be used together with -t/--task')
         sys.exit(1)
     try:
         if options.sigdatafile1 and options.sigdatafile2:
-            output = bb.siggen.compare_sigfiles(options.sigdatafile1, options.sigdatafile2)
+            output = bb.siggen.compare_sigfiles(options.sigdatafile1, options.sigdatafile2, color=color)
         elif options.sigdatafile1:
             output = bb.siggen.dump_sigfile(options.sigdatafile1)
     except IOError as e:
lib/bb/siggen.py +73 -28

216 lines not on this page · raw message

Signature · 1 line
2.9.3
#11 lore raw

Re: [PATCH 09/11] lib/bb/siggen: add collapsed mode to compare_sigfiles()

In reply to Paul Eggleton
Quoted 6 lines · Paul Eggleton

If we just want to drill down to the actual differences then we don't need to see certain things in the output, e.g. basehash changing or the signature of dependent tasks. This will be used for comparing signatures within buildhistory-diff in OE-Core; the default mode as used by bitbake-diffsigs and bitbake -S printdiff remains unchanged for the moment.

This sounds interesting. The bitbake-diffsigs output I got for the yocto-compat-layer had such useless "basehash changed" comments. Should we add a mode parameter to bitbake-diffsigs that allows choosing this collapsed mode?

Signature · 6 lines
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.
#15 lore raw
In reply to Paul Eggleton
Quoted 6 lines · Paul Eggleton

As part of an effort to add task signature recording and comparison to buildhistory in OE, I went digging into bitbake-diffsigs and the code that supports it and discovered that unfortunately it had a number of bugs - in particular the -t option despite being around for a while and having numerous band-aids applied in the past still wasn't really working properly.

I'm not sure whether it is working now, even with your changes. My test, using poky master and your bitbake branch:

1. bitbake -c cleansstate m4
2. bitbake m4
3. patch m4:

(quoted diff) +0 -0

b/meta/recipes-devtools/m4/m4_1.4.18.bb
index b12c0adf3ad..0a8e9c10778 100644

meta/recipes-devtools/m4/m4_1.4.18.bb +4 -0
@@ -1,3 +1,7 @@
 require m4-${PV}.inc
 
+do_install_append () {
+    echo hello world
+}
+
 BBCLASSEXTEND = "nativesdk"

4. bitbake m4
5. bitbake-diffsigs -t m4 do_build
=> ERROR: No sigdata files found matching m4 do_build

There is one file for do_build in tmp/stamps/i586-poky-linux/m4 and several for do_install:

$ ls -l tmp/stamps/i586-poky-linux/m4/*do_build* tmp/stamps/i586-poky-linux/m4/*do_install*
-rw-r--r-- 1 pohly pohly     0 Apr  7 09:53 tmp/stamps/i586-poky-linux/m4/1.4.18-r0.do_build.aa7d33bbb2dfef67ef3ca1b5d3f2cde1
-rw-r--r-- 1 pohly pohly     0 Apr  7 09:53 tmp/stamps/i586-poky-linux/m4/1.4.18-r0.do_install.a0954ae23d8da50c4af22f719255411f
-rw-r--r-- 1 pohly pohly 20573 Apr  7 09:52 tmp/stamps/i586-poky-linux/m4/1.4.18-r0.do_install.sigdata.464e707d80ec982fb5b0c50a6526f05a
-rw-r--r-- 1 pohly pohly 19610 Apr  7 09:53 tmp/stamps/i586-poky-linux/m4/1.4.18-r0.do_install.sigdata.a0954ae23d8da50c4af22f719255411f

bitbake-diffsigs does not fail for do_install, but does not print anything either (= no output at all from "bitbake-diffsigs -t m4 do_install").

Hmm, does it pick the right files to compare?

$ bitbake-diffsigs -d -t m4 do_install
DEBUG: Signature file (previous): /fast/build/poky/x86/sstate-cache/a0/sstate:m4:i586-poky-linux:1.4.18:r0:i586:3:a0954ae23d8da50c4af22f719255411f_install.tgz.siginfo
DEBUG: Signature file (latest): /fast/build/poky/x86/tmp/stamps/i586-poky-linux/m4/1.4.18-r0.do_install.sigdata.a0954ae23d8da50c4af22f719255411f

Am I using the tool incorrectly or are my expectations wrong?

My expectation was that "bitbake-diffsigs -t m4 do_build" would tell me that m4 was rebuilt because some tasks that it depends on changed, and then I would drill down to those tasks, eventually arriving at do_install.

Signature · 6 lines
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.
#12 lore raw

Re: [PATCH 09/11] lib/bb/siggen: add collapsed mode to compare_sigfiles()

In reply to Patrick Ohly
Quoted 8 lines · NZST Patrick Ohly
Quoted 6 lines · Paul Eggleton

If we just want to drill down to the actual differences then we don't need to see certain things in the output, e.g. basehash changing or the signature of dependent tasks. This will be used for comparing signatures within buildhistory-diff in OE-Core; the default mode as used by bitbake-diffsigs and bitbake -S printdiff remains unchanged for the moment.

Quoted 4 lines

This sounds interesting. The bitbake-diffsigs output I got for the yocto-compat-layer had such useless "basehash changed" comments. Should we add a mode parameter to bitbake-diffsigs that allows choosing this collapsed mode?

We could, but to be effective the caller needs to handle the output in a particular way (see what I've done in the buildhistory-diff patchset I sent today).

FYI my plan for 2.4 is to evaluate whether this collapsed mode or something like it is better as a default display mode.

Cheers,
Paul

Signature · 2 lines
Paul Eggleton
Intel Open Source Technology Centre
#16 lore raw
In reply to Patrick Ohly

Hi Patrick,

Quoted 61 lines · NZST Patrick Ohly
Quoted 6 lines · Paul Eggleton

As part of an effort to add task signature recording and comparison to buildhistory in OE, I went digging into bitbake-diffsigs and the code that supports it and discovered that unfortunately it had a number of bugs - in particular the -t option despite being around for a while and having numerous band-aids applied in the past still wasn't really working properly.

I'm not sure whether it is working now, even with your changes. My test, using poky master and your bitbake branch:

1. bitbake -c cleansstate m4
2. bitbake m4
3. patch m4:

meta/recipes-devtools/m4/m4_1.4.18.bb +4 -0
index b12c0adf3ad..0a8e9c10778 100644
@@ -1,3 +1,7 @@
 require m4-${PV}.inc
 
+do_install_append () {
+    echo hello world
+}
+
 BBCLASSEXTEND = "nativesdk"

4. bitbake m4
5. bitbake-diffsigs -t m4 do_build
=> ERROR: No sigdata files found matching m4 do_build

There is one file for do_build in tmp/stamps/i586-poky-linux/m4 and several for do_install:

$ ls -l tmp/stamps/i586-poky-linux/m4/*do_build*
tmp/stamps/i586-poky-linux/m4/*do_install* -rw-r--r-- 1 pohly pohly     0
Apr  7 09:53
tmp/stamps/i586-poky-linux/m4/1.4.18-r0.do_build.aa7d33bbb2dfef67ef3ca1b5d3
f2cde1 -rw-r--r-- 1 pohly pohly     0 Apr  7 09:53
tmp/stamps/i586-poky-linux/m4/1.4.18-r0.do_install.a0954ae23d8da50c4af22f71
9255411f -rw-r--r-- 1 pohly pohly 20573 Apr  7 09:52
tmp/stamps/i586-poky-linux/m4/1.4.18-r0.do_install.sigdata.464e707d80ec982f
b5b0c50a6526f05a -rw-r--r-- 1 pohly pohly 19610 Apr  7 09:53
tmp/stamps/i586-poky-linux/m4/1.4.18-r0.do_install.sigdata.a0954ae23d8da50c
4af22f719255411f

bitbake-diffsigs does not fail for do_install, but does not print anything either (= no output at all from "bitbake-diffsigs -t m4 do_install").

Hmm, does it pick the right files to compare?

$ bitbake-diffsigs -d -t m4 do_install
DEBUG: Signature file (previous):
/fast/build/poky/x86/sstate-cache/a0/sstate:m4:i586-poky-linux:1.4.18:r0:i5
86:3:a0954ae23d8da50c4af22f719255411f_install.tgz.siginfo DEBUG: Signature
file (latest):
/fast/build/poky/x86/tmp/stamps/i586-poky-linux/m4/1.4.18-r0.do_install.sig
data.a0954ae23d8da50c4af22f719255411f
Quoted 6 lines

Am I using the tool incorrectly or are my expectations wrong?

My expectation was that "bitbake-diffsigs -t m4 do_build" would tell me that m4 was rebuilt because some tasks that it depends on changed, and then I would drill down to those tasks, eventually arriving at do_install.

Hmm, something has gone wrong here because this is the exact scenario it's supposed to handle. In my similar tests it did work, so I'm not sure why it didn't work in this case - I'll have to give it a try here.

Cheers,
Paul

Signature · 2 lines
Paul Eggleton
Intel Open Source Technology Centre
#17 lore raw
Quoted 75 lines · NZST Paul Eggleton

Hi Patrick,

Quoted 72 lines · NZST Patrick Ohly
Quoted 6 lines · Paul Eggleton

As part of an effort to add task signature recording and comparison to buildhistory in OE, I went digging into bitbake-diffsigs and the code that supports it and discovered that unfortunately it had a number of bugs - in particular the -t option despite being around for a while and having numerous band-aids applied in the past still wasn't really working properly.

I'm not sure whether it is working now, even with your changes. My test, using poky master and your bitbake branch:

1. bitbake -c cleansstate m4
2. bitbake m4
3. patch m4:

meta/recipes-devtools/m4/m4_1.4.18.bb +4 -0
index b12c0adf3ad..0a8e9c10778 100644
@@ -1,3 +1,7 @@
 
 require m4-${PV}.inc
 
+do_install_append () {
+    echo hello world
+}
+

BBCLASSEXTEND = "nativesdk"

4. bitbake m4
5. bitbake-diffsigs -t m4 do_build
=> ERROR: No sigdata files found matching m4 do_build

There is one file for do_build in tmp/stamps/i586-poky-linux/m4 and several for do_install:

$ ls -l tmp/stamps/i586-poky-linux/m4/*do_build*
tmp/stamps/i586-poky-linux/m4/*do_install* -rw-r--r-- 1 pohly pohly     0
Apr  7 09:53
tmp/stamps/i586-poky-linux/m4/1.4.18-r0.do_build.aa7d33bbb2dfef67ef3ca1b5d
3
f2cde1 -rw-r--r-- 1 pohly pohly     0 Apr  7 09:53
tmp/stamps/i586-poky-linux/m4/1.4.18-r0.do_install.a0954ae23d8da50c4af22f7
1
9255411f -rw-r--r-- 1 pohly pohly 20573 Apr  7 09:52
tmp/stamps/i586-poky-linux/m4/1.4.18-r0.do_install.sigdata.464e707d80ec982
f
b5b0c50a6526f05a -rw-r--r-- 1 pohly pohly 19610 Apr  7 09:53
tmp/stamps/i586-poky-linux/m4/1.4.18-r0.do_install.sigdata.a0954ae23d8da50
c
4af22f719255411f

bitbake-diffsigs does not fail for do_install, but does not print anything either (= no output at all from "bitbake-diffsigs -t m4 do_install").

Hmm, does it pick the right files to compare?

$ bitbake-diffsigs -d -t m4 do_install
DEBUG: Signature file (previous):
/fast/build/poky/x86/sstate-cache/a0/sstate:m4:i586-poky-linux:1.4.18:r0:i
5
86:3:a0954ae23d8da50c4af22f719255411f_install.tgz.siginfo DEBUG: Signature
file (latest):
/fast/build/poky/x86/tmp/stamps/i586-poky-linux/m4/1.4.18-r0.do_install.si
g
data.a0954ae23d8da50c4af22f719255411f

Am I using the tool incorrectly or are my expectations wrong?

Quoted 8 lines
Quoted 4 lines

My expectation was that "bitbake-diffsigs -t m4 do_build" would tell me that m4 was rebuilt because some tasks that it depends on changed, and then I would drill down to those tasks, eventually arriving at do_install.

Hmm, something has gone wrong here because this is the exact scenario it's supposed to handle. In my similar tests it did work, so I'm not sure why it didn't work in this case - I'll have to give it a try here.

I send and then I immediately realised what might be wrong - you said you're using poky master which implies that you haven't also applied the patches for lib/oe/sstatesig.py that I sent to the OE-Core list today. If you haven't, try doing that, it should fix the problem.

Cheers,
Paul

Signature · 2 lines
Paul Eggleton
Intel Open Source Technology Centre
#18 lore raw
Quoted 9 lines · Paul Eggleton
Quoted 8 lines · NZST Paul Eggleton
Quoted 6 lines · NZST Patrick Ohly

Am I using the tool incorrectly or are my expectations wrong?

My expectation was that "bitbake-diffsigs -t m4 do_build" would tell me that m4 was rebuilt because some tasks that it depends on changed, and then I would drill down to those tasks, eventually arriving at do_install.

Quoted 8 lines
Quoted 3 lines

Hmm, something has gone wrong here because this is the exact scenario it's supposed to handle. In my similar tests it did work, so I'm not sure why it didn't work in this case - I'll have to give it a try here.

I send and then I immediately realised what might be wrong - you said you're using poky master which implies that you haven't also applied the patches for lib/oe/sstatesig.py that I sent to the OE-Core list today. If you haven't, try doing that, it should fix the problem.

Indeed, I had missed that part. It's working now for do_install.

What about do_build? Should bitbake-diffsigs be usable with that?

Signature · 6 lines
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.
#19 lore raw
Quoted 17 lines · NZST Patrick Ohly
Quoted 16 lines · Paul Eggleton
Quoted 14 lines · NZST Paul Eggleton
Quoted 7 lines · NZST Patrick Ohly

Am I using the tool incorrectly or are my expectations wrong?

My expectation was that "bitbake-diffsigs -t m4 do_build" would tell me
that m4 was rebuilt because some tasks that it depends on changed, and then I would drill down to those tasks, eventually arriving at do_install.

Hmm, something has gone wrong here because this is the exact scenario it's
supposed to handle. In my similar tests it did work, so I'm not sure why it
didn't work in this case - I'll have to give it a try here.

Quoted 8 lines
Quoted 4 lines

I send and then I immediately realised what might be wrong - you said you're using poky master which implies that you haven't also applied the patches for lib/oe/sstatesig.py that I sent to the OE-Core list today. If you haven't, try doing that, it should fix the problem.

Indeed, I had missed that part. It's working now for do_install.

What about do_build? Should bitbake-diffsigs be usable with that?

If it changed (i.e. there are two signature files to compare) then yes, -t should be able to work back from those. For some reason though you don't seem to have sigdata files for do_build, which is odd - a quick look in my stamps directory here suggests that you should.

Cheers,
Paul

Signature · 2 lines
Paul Eggleton
Intel Open Source Technology Centre
#20 lore raw
In reply to Paul Eggleton
Quoted 18 lines

-----Original Message-----
From: [email protected] [mailto:bitbake- [email protected]] On Behalf Of Paul Eggleton
Sent: den 6 april 2017 23:52
To: [email protected]
Subject: [bitbake-devel] [PATCH 00/11] bitbake-diffsigs fixes/improvements

As part of an effort to add task signature recording and comparison to buildhistory in OE, I went digging into bitbake-diffsigs and the code that supports it and discovered that unfortunately it had a number of bugs - in particular the -t option despite being around for a while and having numerous band-aids applied in the past still wasn't really working properly. I have to take a big chunk of the responsibility for this as I wrote the thing in the first place. This patchset corrects most of the issues that I found and also makes a number of improvements to the readability of the output.

Quoted 2 lines

NOTE: there are a few corresponding changes required in OE-Core that I am about to send out, these should be applied at the same time.

Will you do the same for bitbake-dumpsig as I believe it has many problems in common with bitbake-diffsigs?

//Peter

#4 lore raw

Re: [PATCH 03/11] bitbake-diffsigs: properly report which signature is missing

In reply to Paul Eggleton
Quoted 2 lines · Paul Eggleton

If just one of the two signatures we want to compare aren't available, report that one rather than misleadingly claiming both are missing.

I just noticed that bitbake-diffsigs -t ... do_build recursively dumps differences in other tasks - cool, I don't know it could do that. It worked fine for all-arch ca-certificates, but not for

$ bitbake-diffsigs -d -t gdb-cross-x86_64 do_build -s 68a82197720db0f0dbad8955635bc1c0 5631f484d848541bdbf6a4c1f8117f79
DEBUG: Signature file (previous): /fast/build/nodistro/x86/tmp-glibc/stamps/x86_64-linux/gdb-cross-x86_64/7.12.1-r0.do_build.sigdata.68a82197720db0f0dbad8955635bc1c0
DEBUG: Signature file (latest): /fast/build/nodistro/x86/tmp-glibc/stamps/x86_64-linux/gdb-cross-x86_64/7.12.1-r0.do_build.sigdata.5631f484d848541bdbf6a4c1f8117f79
Hash for dependent task gdb/gdb-cross_7.12.1.bb.do_populate_sysroot changed from 6247000d87570135b704fcfd3e4e7cd4 to 554c1ce69bf0538a91436e5f7c256488
Unable to find matching sigdata for /fast/work/openembedded-core/meta/recipes-devtools/gdb/gdb-cross_7.12.1.bb.do_populate_sysroot with hashes 6247000d87570135b704fcfd3e4e7cd4 or 554c1ce69bf0538a91436e5f7c256488
$ bitbake-diffsigs -d -t gdb-cross-x86_64 do_populate_sysroot -s 6247000d87570135b704fcfd3e4e7cd4 554c1ce69bf0538a91436e5f7c256488
DEBUG: Signature file (previous): /fast/build/nodistro/x86/tmp-glibc/stamps/x86_64-linux/gdb-cross-x86_64/7.12.1-r0.do_populate_sysroot.sigdata.6247000d87570135b704fcfd3e4e7cd4
DEBUG: Signature file (latest): /fast/build/nodistro/x86/tmp-glibc/stamps/x86_64-linux/gdb-cross-x86_64/7.12.1-r0.do_populate_sysroot.sigdata.554c1ce69bf0538a91436e5f7c256488
Hash for dependent task gdb/gdb-cross_7.12.1.bb.do_install changed from 0786db89020b877d443da848548b3d1d to 746c2815e180fa2a4ff28608c7c1b9d5
Unable to find matching sigdata for /fast/work/openembedded-core/meta/recipes-devtools/gdb/gdb-cross_7.12.1.bb.do_install with hashes 0786db89020b877d443da848548b3d1d or 746c2815e180fa2a4ff28608c7c1b9d5

In other words, automatic diff fails to find files, while manual diff finds them?!

This situation can be triggered with a local patch of mine (to be posted soon) and using yocto-compat-layer.py -n --machines intel-corei7-64 intel-core2-32 qemux86 qemux86-64 -- /fast/work/meta-intel

The test uses MACHINE=... bitbake -S none world to generate the signatures for different machines and then looks for tasks that are shared between machines with different signatures.

Signature · 6 lines
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.
#21 lore raw
In reply to Peter Kjellerstedt
Quoted 26 lines · Peter Kjellerstedt
Quoted 26 lines

-----Original Message-----
From: [email protected] [mailto:bitbake- [email protected]] On Behalf Of Paul Eggleton
Sent: den 6 april 2017 23:52
To: [email protected]
Subject: [bitbake-devel] [PATCH 00/11] bitbake-diffsigs fixes/improvements

As part of an effort to add task signature recording and comparison to
buildhistory in OE, I went digging into bitbake-diffsigs and the code
that supports it and discovered that unfortunately it had a number of
bugs - in particular the -t option despite being around for a while and
having numerous band-aids applied in the past still wasn't really working properly. I have to take a big chunk of the responsibility for
this as I wrote the thing in the first place. This patchset corrects
most of the issues that I found and also makes a number of improvements
to the readability of the output.

Quoted 5 lines
Quoted 3 lines

NOTE: there are a few corresponding changes required in OE-Core that I
am about to send out, these should be applied at the same time.

Will you do the same for bitbake-dumpsig as I believe it has many problems in common with bitbake-diffsigs?

bitbake-dumpsig is near enough a symlink to diffsigs, its all the same code.

Cheers,

Richard