bitbake-diffsigs fixes/improvements
cover bitbake-diffsigs fixes/improvements
As part of an effort to add task signature recording and comparison to buildhistory in OE, I went digging into bitbake-diffsigs and the code that supports it and discovered that unfortunately it had a number of bugs - in particular the -t option despite being around for a while and having numerous band-aids applied in the past still wasn't really working properly. I have to take a big chunk of the responsibility for this as I wrote the thing in the first place. This patchset corrects most of the issues that I found and also makes a number of improvements to the readability of the output.
NOTE: there are a few corresponding changes required in OE-Core that I am about to send out, these should be applied at the same time.
The following changes since commit 751c9dc51fd01fa64a1ff37ba2638110335f71af:
fetch/local: Drop FILESDIR (2017-04-05 09:38:01 +0100)
are available in the git repository at:
git://git.yoctoproject.org/poky-contrib paule/bb-sigstuff
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=paule/bb-sigstuff
Paul Eggleton (11):
bitbake-diffsigs: fix -t picking wrong files to compare lib/bb/siggen: add missing path separator to cleaned paths
bitbake-diffsigs: properly report which signature is missing
bitbake-diffsigs: drop naive logic for removing duplicate files lib/bb/siggen: show a diff when dumping changes to multi-line values lib/bb/siggen: don't show unchanged runtaskdeps list
bitbake-diffsigs: change to use argparse
bitbake-diffsigs: add an option to find and compare specific signatures lib/bb/siggen: add collapsed mode to compare_sigfiles() lib/bb/siggen: show word-diff for single-line values containing spaces
bitbake-diffsigs: colourise output
LICENSE | 2 + bin/bitbake-diffsigs | 175 +++++++++++++++++++++------------------ lib/bb/siggen.py | 166 +++++++++++++++++++++++++++++-------- lib/simplediff/LICENSE | 22 +++++ lib/simplediff/__init__.py | 198 +++++++++++++++++++++++++++++++++++++++++++++ 5 files changed, 453 insertions(+), 110 deletions(-) create mode 100644 lib/simplediff/LICENSE create mode 100644 lib/simplediff/__init__.py
Signature · 1 line
2.9.3
PATCH 1/11 bitbake-diffsigs: fix -t picking wrong files to compare
We weren't picking the right files to compare here - according to the order in which the list is sorted (by mtime), we need to be taking the last two items and not the first two.
--- bin/bitbake-diffsigs | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
bin/bitbake-diffsigs +3 -3
@@ -84,9 +84,9 @@ def find_compare_task(bbhandler, pn, taskname): return recout # Recurse into signature comparison - logger.debug("Signature file (previous): %s" % latestfiles[0]) - logger.debug("Signature file (latest): %s" % latestfiles[1]) - output = bb.siggen.compare_sigfiles(latestfiles[0], latestfiles[1], recursecb) + logger.debug("Signature file (previous): %s" % latestfiles[-2]) + logger.debug("Signature file (latest): %s" % latestfiles[-1]) + output = bb.siggen.compare_sigfiles(latestfiles[-2], latestfiles[-1], recursecb) if output: print('\n'.join(output)) sys.exit(0)
Signature · 1 line
2.9.3
PATCH 2/11 lib/bb/siggen: add missing path separator to cleaned paths
In reply to Paul EggletonPrinting "pbzip2pbzip2_1.1.13.bb" is ugly, we need to add a separating slash so that we get "pbzip2/pbzip2_1.1.13.bb" instead.
--- lib/bb/siggen.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
lib/bb/siggen.py +1 -1
@@ -355,7 +355,7 @@ def clean_basepath(a): mc = None if a.startswith("multiconfig:"): _, mc, a = a.split(":", 2) - b = a.rsplit("/", 2)[1] + a.rsplit("/", 2)[2] + b = a.rsplit("/", 2)[1] + '/' + a.rsplit("/", 2)[2] if a.startswith("virtual:"): b = b + ":" + a.rsplit(":", 1)[0] if mc:
Signature · 1 line
2.9.3
PATCH 3/11 bitbake-diffsigs: properly report which signature is missing
In reply to Paul EggletonIf just one of the two signatures we want to compare aren't available, report that one rather than misleadingly claiming both are missing.
--- bin/bitbake-diffsigs | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-)
bin/bitbake-diffsigs +7 -3
@@ -75,11 +75,15 @@ def find_compare_task(bbhandler, pn, taskname): hashfiles = bb.siggen.find_siginfo(key, None, hashes, bbhandler.config_data) recout = [] - if len(hashfiles) == 2: + if len(hashfiles) == 0: + recout.append("Unable to find matching sigdata for %s with hashes %s or %s" % (key, hash1, hash2)) + elif not hash1 in hashfiles: + recout.append("Unable to find matching sigdata for %s with hash %s" % (key, hash1)) + elif not hash2 in hashfiles: + recout.append("Unable to find matching sigdata for %s with hash %s" % (key, hash2)) + else: out2 = bb.siggen.compare_sigfiles(hashfiles[hash1], hashfiles[hash2], recursecb) recout.extend(list(' ' + l for l in out2)) - else: - recout.append("Unable to find matching sigdata for %s with hashes %s or %s" % (key, hash1, hash2)) return recout
Signature · 1 line
2.9.3
PATCH 4/11 bitbake-diffsigs: drop naive logic for removing duplicate files
In reply to Paul EggletonThis logic doesn't work in practice, certainly not with current versions where sigdata files are preserved in the stamps directory and therefore there will often be multiple sigdata files - you can now easily get files for the same signature from sstate and the stamps directory with the result that bitbake-diffsigs reports nothing has changed. Instead, let's change the find_siginfo function in OE-Core to simply not return duplicates so we don't have to filter them out here.
--- bin/bitbake-diffsigs | 16 ---------------- 1 file changed, 16 deletions(-)
bin/bitbake-diffsigs +0 -16
@@ -37,12 +37,6 @@ logger = bb.msg.logger_create('bitbake-diffsigs') def find_compare_task(bbhandler, pn, taskname): """ Find the most recent signature files for the specified PN/task and compare them """ - def get_hashval(siginfo): - if siginfo.endswith('.siginfo'): - return siginfo.rpartition(':')[2].partition('_')[0] - else: - return siginfo.rpartition('.')[2] - if not hasattr(bb.siggen, 'find_siginfo'): logger.error('Metadata does not support finding signature data files') sys.exit(1)
@@ -59,16 +53,6 @@ def find_compare_task(bbhandler, pn, taskname): logger.error('Only one matching sigdata file found for the specified task (%s %s)' % (pn, taskname)) sys.exit(1) else: - # It's possible that latestfiles contain 3 elements and the first two have the same hash value. - # In this case, we delete the second element. - # The above case is actually the most common one. Because we may have sigdata file and siginfo - # file having the same hash value. Comparing such two files makes no sense. - if len(latestfiles) == 3: - hash0 = get_hashval(latestfiles[0]) - hash1 = get_hashval(latestfiles[1]) - if hash0 == hash1: - latestfiles.pop(1) - # Define recursion callback def recursecb(key, hash1, hash2): hashes = [hash1, hash2]
Signature · 1 line
2.9.3
PATCH 5/11 lib/bb/siggen: show a diff when dumping changes to multi-line values
In reply to Paul EggletonWhen dumping changes to signatures e.g. output of bitbake -s printdiff, if for example a function has changed, it's much more readable to see a unified diff of the changes rather than just printing the old function followed by the new function, so use difflib to do that.
Note: I elected to keep to one item in the returned list per change, rather than one line per line of output, so that the caller can still look at changes individually if needed. Thus I've added some handling to bitbake-diffsigs to split the change into lines so that each line is displayed indented.
--- bin/bitbake-diffsigs | 4 +++- lib/bb/siggen.py | 12 +++++++++++- 2 files changed, 14 insertions(+), 2 deletions(-)
bin/bitbake-diffsigs +3 -1
@@ -67,7 +67,9 @@ def find_compare_task(bbhandler, pn, taskname): recout.append("Unable to find matching sigdata for %s with hash %s" % (key, hash2)) else: out2 = bb.siggen.compare_sigfiles(hashfiles[hash1], hashfiles[hash2], recursecb) - recout.extend(list(' ' + l for l in out2)) + for change in out2: + for line in change.splitlines(): + recout.append(' ' + line) return recout
lib/bb/siggen.py +11 -1
@@ -5,6 +5,7 @@ import re import tempfile import pickle import bb.data +import difflib from bb.checksum import FileChecksumCache logger = logging.getLogger('BitBake.SigGen')
@@ -462,7 +463,16 @@ def compare_sigfiles(a, b, recursecb = None): changed, added, removed = dict_diff(a_data['varvals'], b_data['varvals']) if changed: for dep in changed: - output.append("Variable %s value changed from '%s' to '%s'" % (dep, a_data['varvals'][dep], b_data['varvals'][dep])) + oldval = a_data['varvals'][dep] + newval = b_data['varvals'][dep] + if newval and oldval and ('\n' in oldval or '\n' in newval): + diff = difflib.unified_diff(oldval.splitlines(), newval.splitlines(), lineterm='') + # Cut off the first two lines, since we aren't interested in + # the old/new filename (they are blank anyway in this case) + difflines = list(diff)[2:] + output.append("Variable %s value changed:\n%s" % (dep, '\n'.join(difflines))) + else: + output.append("Variable %s value changed from '%s' to '%s'" % (dep, oldval, newval)) if not 'file_checksum_values' in a_data: a_data['file_checksum_values'] = {}
Signature · 1 line
2.9.3
PATCH 6/11 lib/bb/siggen: don't show unchanged runtaskdeps list
In reply to Paul EggletonIf the runtaskdeps list hasn't actually changed (but the signatures of some of the tasks did) then it doesn't make sense to print out the old and new lists as they are both the same and may be very long, e.g. for do_rootfs in OE.
--- lib/bb/siggen.py | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)
lib/bb/siggen.py +6 -2
@@ -506,10 +506,14 @@ def compare_sigfiles(a, b, recursecb = None): changed.append("%s with hash %s\n changed to\n%s with hash %s" % (a, a_data['runtaskhashes'][a], b, b_data['runtaskhashes'][b])) if changed: - output.append("runtaskdeps changed from %s to %s" % (clean_basepaths_list(a_data['runtaskdeps']), clean_basepaths_list(b_data['runtaskdeps']))) + clean_a = clean_basepaths_list(a_data['runtaskdeps']) + clean_b = clean_basepaths_list(b_data['runtaskdeps']) + if clean_a != clean_b: + output.append("runtaskdeps changed from %s to %s" % (clean_a, clean_b)) + else: + output.append("runtaskdeps changed:") output.append("\n".join(changed)) - if 'runtaskhashes' in a_data and 'runtaskhashes' in b_data: a = a_data['runtaskhashes'] b = b_data['runtaskhashes']
Signature · 1 line
2.9.3
PATCH 7/11 bitbake-diffsigs: change to use argparse
In reply to Paul EggletonArgparse is a bit easier to deal with than optparse, and since we're about to add some options, migrate this script over.
--- bin/bitbake-diffsigs | 64 +++++++++++++++++++++++++++------------------------- 1 file changed, 33 insertions(+), 31 deletions(-)
bin/bitbake-diffsigs +33 -31
@@ -3,7 +3,7 @@ # bitbake-diffsigs # BitBake task signature data comparison utility # -# Copyright (C) 2012-2013 Intel Corporation +# Copyright (C) 2012-2013, 2017 Intel Corporation # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License version 2 as
@@ -22,7 +22,7 @@ import os import sys import warnings import fnmatch -import optparse +import argparse import logging import pickle
@@ -83,22 +83,27 @@ def find_compare_task(bbhandler, pn, taskname): -parser = optparse.OptionParser( - description = "Compares siginfo/sigdata files written out by BitBake", - usage = """ - %prog -t recipename taskname - %prog sigdatafile1 sigdatafile2 - %prog sigdatafile1""") +parser = argparse.ArgumentParser( + description="Compares siginfo/sigdata files written out by BitBake") -parser.add_option("-D", "--debug", - help = "enable debug", - action = "store_true", dest="debug", default = False) +parser.add_argument('-d', '--debug', + help='Enable debug output', + action='store_true') -parser.add_option("-t", "--task", - help = "find the signature data files for last two runs of the specified task and compare them", - action="store", dest="taskargs", nargs=2, metavar='recipename taskname') +parser.add_argument("-t", "--task", + help="find the signature data files for last two runs of the specified task and compare them", + action="store", dest="taskargs", nargs=2, metavar=('recipename', 'taskname')) -options, args = parser.parse_args(sys.argv) +parser.add_argument("sigdatafile1", + help="First signature file to compare (or signature file to dump, if second not specified). Not used when using -t/--task.", + action="store", nargs='?') + +parser.add_argument("sigdatafile2", + help="Second signature file to compare", + action="store", nargs='?') + + +options = parser.parse_args() if options.debug: logger.setLevel(logging.DEBUG)
@@ -108,20 +113,17 @@ if options.taskargs: tinfoil.prepare(config_only=True) find_compare_task(tinfoil, options.taskargs[0], options.taskargs[1]) else: - if len(args) == 1: - parser.print_help() - else: - try: - if len(args) == 2: - output = bb.siggen.dump_sigfile(sys.argv[1]) - else: - output = bb.siggen.compare_sigfiles(sys.argv[1], sys.argv[2]) - except IOError as e: - logger.error(str(e)) - sys.exit(1) - except (pickle.UnpicklingError, EOFError): - logger.error('Invalid signature data - ensure you are specifying sigdata/siginfo files') - sys.exit(1) + try: + if options.sigdatafile1 and options.sigdatafile2: + output = bb.siggen.compare_sigfiles(options.sigdatafile1, options.sigdatafile2) + elif options.sigdatafile1: + output = bb.siggen.dump_sigfile(options.sigdatafile1) + except IOError as e: + logger.error(str(e)) + sys.exit(1) + except (pickle.UnpicklingError, EOFError): + logger.error('Invalid signature data - ensure you are specifying sigdata/siginfo files') + sys.exit(1) - if output: - print('\n'.join(output)) + if output: + print('\n'.join(output))
Signature · 1 line
2.9.3
PATCH 8/11 bitbake-diffsigs: add an option to find and compare specific signatures
In reply to Paul EggletonWith the -t option which recurses to find the ultimate cause of a signature change, it was hardcoded to take the last two executions of the specified task. On the other hand, if you have two specific task hashes (say from bitbake output, or some other tool) then you'll want to pick those, so provide an option to specify those as well. (Note, the new -s option needs to be specified alongside -t rather than instead of it.)
--- bin/bitbake-diffsigs | 95 ++++++++++++++++++++++++++++++++-------------------- 1 file changed, 59 insertions(+), 36 deletions(-)
bin/bitbake-diffsigs +59 -36
@@ -34,7 +34,7 @@ import bb.msg logger = bb.msg.logger_create('bitbake-diffsigs') -def find_compare_task(bbhandler, pn, taskname): +def find_compare_task(bbhandler, pn, taskname, sig1=None, sig2=None): """ Find the most recent signature files for the specified PN/task and compare them """ if not hasattr(bb.siggen, 'find_siginfo'):
@@ -44,41 +44,54 @@ def find_compare_task(bbhandler, pn, taskname): if not taskname.startswith('do_'): taskname = 'do_%s' % taskname - filedates = bb.siggen.find_siginfo(pn, taskname, None, bbhandler.config_data) - latestfiles = sorted(filedates.keys(), key=lambda f: filedates[f])[-3:] - if not latestfiles: - logger.error('No sigdata files found matching %s %s' % (pn, taskname)) - sys.exit(1) - elif len(latestfiles) < 2: - logger.error('Only one matching sigdata file found for the specified task (%s %s)' % (pn, taskname)) - sys.exit(1) + if sig1 and sig2: + sigfiles = bb.siggen.find_siginfo(pn, taskname, [sig1, sig2], bbhandler.config_data) + if len(sigfiles) == 0: + logger.error('No sigdata files found matching %s %s matching either %s or %s' % (pn, taskname, sig1, sig2)) + sys.exit(1) + elif not sig1 in sigfiles: + logger.error('No sigdata files found matching %s %s with signature %s' % (pn, taskname, sig1)) + sys.exit(1) + elif not sig2 in sigfiles: + logger.error('No sigdata files found matching %s %s with signature %s' % (pn, taskname, sig2)) + sys.exit(1) + latestfiles = [sigfiles[sig1], sigfiles[sig2]] else: - # Define recursion callback - def recursecb(key, hash1, hash2): - hashes = [hash1, hash2] - hashfiles = bb.siggen.find_siginfo(key, None, hashes, bbhandler.config_data) - - recout = [] - if len(hashfiles) == 0: - recout.append("Unable to find matching sigdata for %s with hashes %s or %s" % (key, hash1, hash2)) - elif not hash1 in hashfiles: - recout.append("Unable to find matching sigdata for %s with hash %s" % (key, hash1)) - elif not hash2 in hashfiles: - recout.append("Unable to find matching sigdata for %s with hash %s" % (key, hash2)) - else: - out2 = bb.siggen.compare_sigfiles(hashfiles[hash1], hashfiles[hash2], recursecb) - for change in out2: - for line in change.splitlines(): - recout.append(' ' + line) - - return recout - - # Recurse into signature comparison - logger.debug("Signature file (previous): %s" % latestfiles[-2]) - logger.debug("Signature file (latest): %s" % latestfiles[-1]) - output = bb.siggen.compare_sigfiles(latestfiles[-2], latestfiles[-1], recursecb) - if output: - print('\n'.join(output)) + filedates = bb.siggen.find_siginfo(pn, taskname, None, bbhandler.config_data) + latestfiles = sorted(filedates.keys(), key=lambda f: filedates[f])[-3:] + if not latestfiles: + logger.error('No sigdata files found matching %s %s' % (pn, taskname)) + sys.exit(1) + elif len(latestfiles) < 2: + logger.error('Only one matching sigdata file found for the specified task (%s %s)' % (pn, taskname)) + sys.exit(1) + + # Define recursion callback + def recursecb(key, hash1, hash2): + hashes = [hash1, hash2] + hashfiles = bb.siggen.find_siginfo(key, None, hashes, bbhandler.config_data) + + recout = [] + if len(hashfiles) == 0: + recout.append("Unable to find matching sigdata for %s with hashes %s or %s" % (key, hash1, hash2)) + elif not hash1 in hashfiles: + recout.append("Unable to find matching sigdata for %s with hash %s" % (key, hash1)) + elif not hash2 in hashfiles: + recout.append("Unable to find matching sigdata for %s with hash %s" % (key, hash2)) + else: + out2 = bb.siggen.compare_sigfiles(hashfiles[hash1], hashfiles[hash2], recursecb) + for change in out2: + for line in change.splitlines(): + recout.append(' ' + line) + + return recout + + # Recurse into signature comparison + logger.debug("Signature file (previous): %s" % latestfiles[-2]) + logger.debug("Signature file (latest): %s" % latestfiles[-1]) + output = bb.siggen.compare_sigfiles(latestfiles[-2], latestfiles[-1], recursecb) + if output: + print('\n'.join(output)) sys.exit(0)
@@ -94,6 +107,10 @@ parser.add_argument("-t", "--task", help="find the signature data files for last two runs of the specified task and compare them", action="store", dest="taskargs", nargs=2, metavar=('recipename', 'taskname')) +parser.add_argument("-s", "--signature", + help="With -t/--task, specify the signatures to look for instead of taking the last two", + action="store", dest="sigargs", nargs=2, metavar=('fromsig', 'tosig')) + parser.add_argument("sigdatafile1", help="First signature file to compare (or signature file to dump, if second not specified). Not used when using -t/--task.", action="store", nargs='?')
@@ -111,8 +128,14 @@ if options.debug: if options.taskargs: with bb.tinfoil.Tinfoil() as tinfoil: tinfoil.prepare(config_only=True) - find_compare_task(tinfoil, options.taskargs[0], options.taskargs[1]) + if options.sigargs: + find_compare_task(tinfoil, options.taskargs[0], options.taskargs[1], options.sigargs[0], options.sigargs[1]) + else: + find_compare_task(tinfoil, options.taskargs[0], options.taskargs[1]) else: + if options.sigargs: + logger.error('-s/--signature can only be used together with -t/--task') + sys.exit(1) try: if options.sigdatafile1 and options.sigdatafile2: output = bb.siggen.compare_sigfiles(options.sigdatafile1, options.sigdatafile2)
Signature · 1 line
2.9.3
PATCH 9/11 lib/bb/siggen: add collapsed mode to compare_sigfiles()
In reply to Paul EggletonIf we just want to drill down to the actual differences then we don't need to see certain things in the output, e.g. basehash changing or the signature of dependent tasks. This will be used for comparing signatures within buildhistory-diff in OE-Core; the default mode as used by bitbake-diffsigs and bitbake -S printdiff remains unchanged for the moment.
--- lib/bb/siggen.py | 51 ++++++++++++++++++++++++++++----------------------- 1 file changed, 28 insertions(+), 23 deletions(-)
lib/bb/siggen.py +28 -23
@@ -375,7 +375,7 @@ def clean_basepaths_list(a): b.append(clean_basepath(x)) return b -def compare_sigfiles(a, b, recursecb = None): +def compare_sigfiles(a, b, recursecb=None, collapsed=False): output = [] with open(a, 'rb') as f:
@@ -443,7 +443,7 @@ def compare_sigfiles(a, b, recursecb = None): if a_data['taskdeps'] != b_data['taskdeps']: output.append("Task dependencies changed from:\n%s\nto:\n%s" % (sorted(a_data['taskdeps']), sorted(b_data['taskdeps']))) - if a_data['basehash'] != b_data['basehash']: + if a_data['basehash'] != b_data['basehash'] and not collapsed: output.append("basehash changed from %s to %s" % (a_data['basehash'], b_data['basehash'])) changed, added, removed = dict_diff(a_data['gendeps'], b_data['gendeps'], a_data['basewhitelist'] & b_data['basewhitelist'])
@@ -495,24 +495,25 @@ def compare_sigfiles(a, b, recursecb = None): if not 'runtaskdeps' in b_data: b_data['runtaskdeps'] = {} - if len(a_data['runtaskdeps']) != len(b_data['runtaskdeps']): - changed = ["Number of task dependencies changed"] - else: - changed = [] - for idx, task in enumerate(a_data['runtaskdeps']): - a = a_data['runtaskdeps'][idx] - b = b_data['runtaskdeps'][idx] - if a_data['runtaskhashes'][a] != b_data['runtaskhashes'][b]: - changed.append("%s with hash %s\n changed to\n%s with hash %s" % (a, a_data['runtaskhashes'][a], b, b_data['runtaskhashes'][b])) - - if changed: - clean_a = clean_basepaths_list(a_data['runtaskdeps']) - clean_b = clean_basepaths_list(b_data['runtaskdeps']) - if clean_a != clean_b: - output.append("runtaskdeps changed from %s to %s" % (clean_a, clean_b)) + if not collapsed: + if len(a_data['runtaskdeps']) != len(b_data['runtaskdeps']): + changed = ["Number of task dependencies changed"] else: - output.append("runtaskdeps changed:") - output.append("\n".join(changed)) + changed = [] + for idx, task in enumerate(a_data['runtaskdeps']): + a = a_data['runtaskdeps'][idx] + b = b_data['runtaskdeps'][idx] + if a_data['runtaskhashes'][a] != b_data['runtaskhashes'][b]: + changed.append("%s with hash %s\n changed to\n%s with hash %s" % (a, a_data['runtaskhashes'][a], b, b_data['runtaskhashes'][b])) + + if changed: + clean_a = clean_basepaths_list(a_data['runtaskdeps']) + clean_b = clean_basepaths_list(b_data['runtaskdeps']) + if clean_a != clean_b: + output.append("runtaskdeps changed from %s to %s" % (clean_a, clean_b)) + else: + output.append("runtaskdeps changed:") + output.append("\n".join(changed)) if 'runtaskhashes' in a_data and 'runtaskhashes' in b_data: a = a_data['runtaskhashes']
@@ -540,13 +541,17 @@ def compare_sigfiles(a, b, recursecb = None): output.append("Dependency on task %s was removed with hash %s" % (clean_basepath(dep), a[dep])) if changed: for dep in changed: - output.append("Hash for dependent task %s changed from %s to %s" % (clean_basepath(dep), a[dep], b[dep])) + if not collapsed: + output.append("Hash for dependent task %s changed from %s to %s" % (clean_basepath(dep), a[dep], b[dep])) if callable(recursecb): - # If a dependent hash changed, might as well print the line above and then defer to the changes in - # that hash since in all likelyhood, they're the same changes this task also saw. recout = recursecb(dep, a[dep], b[dep]) if recout: - output = [output[-1]] + recout + if collapsed: + output.extend(recout) + else: + # If a dependent hash changed, might as well print the line above and then defer to the changes in + # that hash since in all likelyhood, they're the same changes this task also saw. + output = [output[-1]] + recout a_taint = a_data.get('taint', None) b_taint = b_data.get('taint', None)
Signature · 1 line
2.9.3
PATCH 10/11 lib/bb/siggen: show word-diff for single-line values containing spaces
In reply to Paul EggletonIf a variable value has changed and either the new or old value contains spaces, a word diff should be appropriate and may be a bit more readable. Import the "simplediff" module and use it to show a word diff (in the style of GNU wdiff and git diff --word-diff).
Also use a similar style diff to show changes in the runtaskhashes list. I didn't use an actual word-diff here since it's a little different - we can be sure that the list is a list and not simply a free-format string.
--- LICENSE | 2 + lib/bb/siggen.py | 38 ++++++++- lib/simplediff/LICENSE | 22 +++++ lib/simplediff/__init__.py | 198 +++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 259 insertions(+), 1 deletion(-) create mode 100644 lib/simplediff/LICENSE create mode 100644 lib/simplediff/__init__.py
LICENSE +2 -0
@@ -15,3 +15,5 @@ Foundation and individual contributors. * QUnit is redistributed under the MIT license. * Font Awesome fonts redistributed under the SIL Open Font License 1.1 + +* simplediff is distributed under the zlib license.
lib/bb/siggen.py +37 -1
@@ -6,6 +6,7 @@ import tempfile import pickle import bb.data import difflib +import simplediff from bb.checksum import FileChecksumCache logger = logging.getLogger('BitBake.SigGen')
@@ -352,6 +353,39 @@ def dump_this_task(outfile, d): referencestamp = bb.build.stamp_internal(task, d, None, True) bb.parse.siggen.dump_sigtask(fn, task, outfile, "customfile:" + referencestamp) +def worddiff_str(oldstr, newstr): + diff = simplediff.diff(oldstr.split(' '), newstr.split(' ')) + ret = [] + for change, value in diff: + value = ' '.join(value) + if change == '=': + ret.append(value) + elif change == '+': + item = '{+%s+}' % value + ret.append(item) + elif change == '-': + item = '[-%s-]' % value + ret.append(item) + whitespace_note = '' + if oldstr != newstr and ' '.join(oldstr.split()) == ' '.join(newstr.split()): + whitespace_note = ' (whitespace changed)' + return '"%s"%s' % (' '.join(ret), whitespace_note) + +def list_inline_diff(oldlist, newlist): + diff = simplediff.diff(oldlist, newlist) + ret = [] + for change, value in diff: + value = ' '.join(value) + if change == '=': + ret.append("'%s'" % value) + elif change == '+': + item = "+'%s'" % value + ret.append(item) + elif change == '-': + item = "-'%s'" % value + ret.append(item) + return '[%s]' % (', '.join(ret)) + def clean_basepath(a): mc = None if a.startswith("multiconfig:"):
@@ -471,6 +505,8 @@ def compare_sigfiles(a, b, recursecb=None, collapsed=False): # the old/new filename (they are blank anyway in this case) difflines = list(diff)[2:] output.append("Variable %s value changed:\n%s" % (dep, '\n'.join(difflines))) + elif newval and oldval and (' ' in oldval or ' ' in newval): + output.append("Variable %s value changed:\n%s" % (dep, worddiff_str(oldval, newval))) else: output.append("Variable %s value changed from '%s' to '%s'" % (dep, oldval, newval))
@@ -510,7 +546,7 @@ def compare_sigfiles(a, b, recursecb=None, collapsed=False): clean_a = clean_basepaths_list(a_data['runtaskdeps']) clean_b = clean_basepaths_list(b_data['runtaskdeps']) if clean_a != clean_b: - output.append("runtaskdeps changed from %s to %s" % (clean_a, clean_b)) + output.append("runtaskdeps changed:\n%s" % list_inline_diff(clean_a, clean_b)) else: output.append("runtaskdeps changed:") output.append("\n".join(changed))
lib/simplediff/LICENSE new file +22 -0
@@ -0,0 +1,22 @@ +Copyright (c) 2008 - 2013 Paul Butler and contributors + +This sofware may be used under a zlib/libpng-style license: + +This software is provided 'as-is', without any express or implied warranty. In +no event will the authors be held liable for any damages arising from the use +of this software. + +Permission is granted to anyone to use this software for any purpose, including +commercial applications, and to alter it and redistribute it freely, subject to +the following restrictions: + +1. The origin of this software must not be misrepresented; you must not claim +that you wrote the original software. If you use this software in a product, an +acknowledgment in the product documentation would be appreciated but is not +required. + +2. Altered source versions must be plainly marked as such, and must not be +misrepresented as being the original software. + +3. This notice may not be removed or altered from any source distribution. +
lib/simplediff/__init__.py new file +198 -0
198 lines not on this page · raw message
Signature · 1 line
2.9.3
PATCH 11/11 bitbake-diffsigs: colourise output
In reply to Paul EggletonIf the output is a TTY, add colour to the output in order to make it easier to read. At the moment this is fairly basic, just add colour to the "titles" of each change and to the diff output.
I tried to introduce this without changing the code too much - rather than moving everything over to the new python formatting style, I've introduced a color_format() function which takes care of the colour formatting, either accepting additional format arguments or alternatively leaving the caller to use the old-style formatting (%) to insert values.
--- bin/bitbake-diffsigs | 18 ++++++--- lib/bb/siggen.py | 101 +++++++++++++++++++++++++++++++++++++-------------- 2 files changed, 85 insertions(+), 34 deletions(-)
bin/bitbake-diffsigs +12 -6
@@ -34,7 +34,7 @@ import bb.msg logger = bb.msg.logger_create('bitbake-diffsigs') -def find_compare_task(bbhandler, pn, taskname, sig1=None, sig2=None): +def find_compare_task(bbhandler, pn, taskname, sig1=None, sig2=None, color=False): """ Find the most recent signature files for the specified PN/task and compare them """ if not hasattr(bb.siggen, 'find_siginfo'):
@@ -79,7 +79,7 @@ def find_compare_task(bbhandler, pn, taskname, sig1=None, sig2=None): elif not hash2 in hashfiles: recout.append("Unable to find matching sigdata for %s with hash %s" % (key, hash2)) else: - out2 = bb.siggen.compare_sigfiles(hashfiles[hash1], hashfiles[hash2], recursecb) + out2 = bb.siggen.compare_sigfiles(hashfiles[hash1], hashfiles[hash2], recursecb, color=color) for change in out2: for line in change.splitlines(): recout.append(' ' + line)
@@ -89,7 +89,7 @@ def find_compare_task(bbhandler, pn, taskname, sig1=None, sig2=None): # Recurse into signature comparison logger.debug("Signature file (previous): %s" % latestfiles[-2]) logger.debug("Signature file (latest): %s" % latestfiles[-1]) - output = bb.siggen.compare_sigfiles(latestfiles[-2], latestfiles[-1], recursecb) + output = bb.siggen.compare_sigfiles(latestfiles[-2], latestfiles[-1], recursecb, color=color) if output: print('\n'.join(output)) sys.exit(0)
@@ -103,6 +103,10 @@ parser.add_argument('-d', '--debug', help='Enable debug output', action='store_true') +parser.add_argument('--color', + help='Colorize output (where %(metavar)s is %(choices)s)', + choices=['auto', 'always', 'never'], default='auto', metavar='color') + parser.add_argument("-t", "--task", help="find the signature data files for last two runs of the specified task and compare them", action="store", dest="taskargs", nargs=2, metavar=('recipename', 'taskname'))
@@ -125,20 +129,22 @@ options = parser.parse_args() if options.debug: logger.setLevel(logging.DEBUG) +color = (options.color == 'always' or (options.color == 'auto' and sys.stdout.isatty())) + if options.taskargs: with bb.tinfoil.Tinfoil() as tinfoil: tinfoil.prepare(config_only=True) if options.sigargs: - find_compare_task(tinfoil, options.taskargs[0], options.taskargs[1], options.sigargs[0], options.sigargs[1]) + find_compare_task(tinfoil, options.taskargs[0], options.taskargs[1], options.sigargs[0], options.sigargs[1], color=color) else: - find_compare_task(tinfoil, options.taskargs[0], options.taskargs[1]) + find_compare_task(tinfoil, options.taskargs[0], options.taskargs[1], color=color) else: if options.sigargs: logger.error('-s/--signature can only be used together with -t/--task') sys.exit(1) try: if options.sigdatafile1 and options.sigdatafile2: - output = bb.siggen.compare_sigfiles(options.sigdatafile1, options.sigdatafile2) + output = bb.siggen.compare_sigfiles(options.sigdatafile1, options.sigdatafile2, color=color) elif options.sigdatafile1: output = bb.siggen.dump_sigfile(options.sigdatafile1) except IOError as e:
lib/bb/siggen.py +73 -28
216 lines not on this page · raw message
Signature · 1 line
2.9.3
Re: [PATCH 09/11] lib/bb/siggen: add collapsed mode to compare_sigfiles()
In reply to Paul EggletonQuoted 6 lines · Paul Eggleton
If we just want to drill down to the actual differences then we don't need to see certain things in the output, e.g. basehash changing or the signature of dependent tasks. This will be used for comparing signatures within buildhistory-diff in OE-Core; the default mode as used by bitbake-diffsigs and bitbake -S printdiff remains unchanged for the moment.
This sounds interesting. The bitbake-diffsigs output I got for the yocto-compat-layer had such useless "basehash changed" comments. Should we add a mode parameter to bitbake-diffsigs that allows choosing this collapsed mode?
Signature · 6 lines
Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter.
Quoted 6 lines · Paul Eggleton
As part of an effort to add task signature recording and comparison to buildhistory in OE, I went digging into bitbake-diffsigs and the code that supports it and discovered that unfortunately it had a number of bugs - in particular the -t option despite being around for a while and having numerous band-aids applied in the past still wasn't really working properly.
I'm not sure whether it is working now, even with your changes. My test, using poky master and your bitbake branch:
1. bitbake -c cleansstate m4
2. bitbake m4
3. patch m4:
(quoted diff) +0 -0
b/meta/recipes-devtools/m4/m4_1.4.18.bb
index b12c0adf3ad..0a8e9c10778 100644
meta/recipes-devtools/m4/m4_1.4.18.bb +4 -0
@@ -1,3 +1,7 @@ require m4-${PV}.inc +do_install_append () { + echo hello world +} + BBCLASSEXTEND = "nativesdk"
4. bitbake m4
5. bitbake-diffsigs -t m4 do_build
=> ERROR: No sigdata files found matching m4 do_build
There is one file for do_build in tmp/stamps/i586-poky-linux/m4 and several for do_install:
$ ls -l tmp/stamps/i586-poky-linux/m4/*do_build* tmp/stamps/i586-poky-linux/m4/*do_install* -rw-r--r-- 1 pohly pohly 0 Apr 7 09:53 tmp/stamps/i586-poky-linux/m4/1.4.18-r0.do_build.aa7d33bbb2dfef67ef3ca1b5d3f2cde1 -rw-r--r-- 1 pohly pohly 0 Apr 7 09:53 tmp/stamps/i586-poky-linux/m4/1.4.18-r0.do_install.a0954ae23d8da50c4af22f719255411f -rw-r--r-- 1 pohly pohly 20573 Apr 7 09:52 tmp/stamps/i586-poky-linux/m4/1.4.18-r0.do_install.sigdata.464e707d80ec982fb5b0c50a6526f05a -rw-r--r-- 1 pohly pohly 19610 Apr 7 09:53 tmp/stamps/i586-poky-linux/m4/1.4.18-r0.do_install.sigdata.a0954ae23d8da50c4af22f719255411f
bitbake-diffsigs does not fail for do_install, but does not print anything either (= no output at all from "bitbake-diffsigs -t m4 do_install").
Hmm, does it pick the right files to compare?
$ bitbake-diffsigs -d -t m4 do_install DEBUG: Signature file (previous): /fast/build/poky/x86/sstate-cache/a0/sstate:m4:i586-poky-linux:1.4.18:r0:i586:3:a0954ae23d8da50c4af22f719255411f_install.tgz.siginfo DEBUG: Signature file (latest): /fast/build/poky/x86/tmp/stamps/i586-poky-linux/m4/1.4.18-r0.do_install.sigdata.a0954ae23d8da50c4af22f719255411f
Am I using the tool incorrectly or are my expectations wrong?
My expectation was that "bitbake-diffsigs -t m4 do_build" would tell me that m4 was rebuilt because some tasks that it depends on changed, and then I would drill down to those tasks, eventually arriving at do_install.
Signature · 6 lines
Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter.
Re: [PATCH 09/11] lib/bb/siggen: add collapsed mode to compare_sigfiles()
In reply to Patrick OhlyQuoted 8 lines · NZST Patrick Ohly
Quoted 6 lines · Paul Eggleton
If we just want to drill down to the actual differences then we don't need to see certain things in the output, e.g. basehash changing or the signature of dependent tasks. This will be used for comparing signatures within buildhistory-diff in OE-Core; the default mode as used by bitbake-diffsigs and bitbake -S printdiff remains unchanged for the moment.
Quoted 4 lines
This sounds interesting. The bitbake-diffsigs output I got for the yocto-compat-layer had such useless "basehash changed" comments. Should we add a mode parameter to bitbake-diffsigs that allows choosing this collapsed mode?
We could, but to be effective the caller needs to handle the output in a particular way (see what I've done in the buildhistory-diff patchset I sent today).
FYI my plan for 2.4 is to evaluate whether this collapsed mode or something like it is better as a default display mode.
Cheers,
Paul
Signature · 2 lines
Paul Eggleton Intel Open Source Technology Centre
Hi Patrick,
Quoted 61 lines · NZST Patrick Ohly
Quoted 6 lines · Paul Eggleton
As part of an effort to add task signature recording and comparison to buildhistory in OE, I went digging into bitbake-diffsigs and the code that supports it and discovered that unfortunately it had a number of bugs - in particular the -t option despite being around for a while and having numerous band-aids applied in the past still wasn't really working properly.
I'm not sure whether it is working now, even with your changes. My test, using poky master and your bitbake branch:
1. bitbake -c cleansstate m4
2. bitbake m4
3. patch m4:meta/recipes-devtools/m4/m4_1.4.18.bb +4
-0@@ -1,3 +1,7 @@ require m4-${PV}.inc +do_install_append () { + echo hello world +} + BBCLASSEXTEND = "nativesdk"4. bitbake m4
5. bitbake-diffsigs -t m4do_build
=> ERROR: No sigdata files found matching m4do_buildThere is one file for
do_buildin tmp/stamps/i586-poky-linux/m4 and several fordo_install:$ ls -l tmp/stamps/i586-poky-linux/m4/*do_build* tmp/stamps/i586-poky-linux/m4/*do_install* -rw-r--r-- 1 pohly pohly 0 Apr 7 09:53 tmp/stamps/i586-poky-linux/m4/1.4.18-r0.do_build.aa7d33bbb2dfef67ef3ca1b5d3 f2cde1 -rw-r--r-- 1 pohly pohly 0 Apr 7 09:53 tmp/stamps/i586-poky-linux/m4/1.4.18-r0.do_install.a0954ae23d8da50c4af22f71 9255411f -rw-r--r-- 1 pohly pohly 20573 Apr 7 09:52 tmp/stamps/i586-poky-linux/m4/1.4.18-r0.do_install.sigdata.464e707d80ec982f b5b0c50a6526f05a -rw-r--r-- 1 pohly pohly 19610 Apr 7 09:53 tmp/stamps/i586-poky-linux/m4/1.4.18-r0.do_install.sigdata.a0954ae23d8da50c 4af22f719255411fbitbake-diffsigs does not fail for
do_install, but does not print anything either (= no output at all from "bitbake-diffsigs -t m4do_install").Hmm, does it pick the right files to compare?
$ bitbake-diffsigs -d -t m4 do_install DEBUG: Signature file (previous): /fast/build/poky/x86/sstate-cache/a0/sstate:m4:i586-poky-linux:1.4.18:r0:i5 86:3:a0954ae23d8da50c4af22f719255411f_install.tgz.siginfo DEBUG: Signature file (latest): /fast/build/poky/x86/tmp/stamps/i586-poky-linux/m4/1.4.18-r0.do_install.sig data.a0954ae23d8da50c4af22f719255411f
Quoted 6 lines
Am I using the tool incorrectly or are my expectations wrong?
My expectation was that "bitbake-diffsigs -t m4
do_build" would tell me that m4 was rebuilt because some tasks that it depends on changed, and then I would drill down to those tasks, eventually arriving atdo_install.
Hmm, something has gone wrong here because this is the exact scenario it's supposed to handle. In my similar tests it did work, so I'm not sure why it didn't work in this case - I'll have to give it a try here.
Cheers,
Paul
Signature · 2 lines
Paul Eggleton Intel Open Source Technology Centre
Quoted 75 lines · NZST Paul Eggleton
Hi Patrick,
Quoted 72 lines · NZST Patrick Ohly
Quoted 6 lines · Paul Eggleton
As part of an effort to add task signature recording and comparison to buildhistory in OE, I went digging into bitbake-diffsigs and the code that supports it and discovered that unfortunately it had a number of bugs - in particular the -t option despite being around for a while and having numerous band-aids applied in the past still wasn't really working properly.
I'm not sure whether it is working now, even with your changes. My test, using poky master and your bitbake branch:
1. bitbake -c cleansstate m4
2. bitbake m4
3. patch m4:meta/recipes-devtools/m4/m4_1.4.18.bb +4
-0@@ -1,3 +1,7 @@ require m4-${PV}.inc +do_install_append () { + echo hello world +} +BBCLASSEXTEND = "nativesdk"
4. bitbake m4
5. bitbake-diffsigs -t m4do_build
=> ERROR: No sigdata files found matching m4do_buildThere is one file for
do_buildin tmp/stamps/i586-poky-linux/m4 and several fordo_install:$ ls -l tmp/stamps/i586-poky-linux/m4/*do_build* tmp/stamps/i586-poky-linux/m4/*do_install* -rw-r--r-- 1 pohly pohly 0 Apr 7 09:53 tmp/stamps/i586-poky-linux/m4/1.4.18-r0.do_build.aa7d33bbb2dfef67ef3ca1b5d 3 f2cde1 -rw-r--r-- 1 pohly pohly 0 Apr 7 09:53 tmp/stamps/i586-poky-linux/m4/1.4.18-r0.do_install.a0954ae23d8da50c4af22f7 1 9255411f -rw-r--r-- 1 pohly pohly 20573 Apr 7 09:52 tmp/stamps/i586-poky-linux/m4/1.4.18-r0.do_install.sigdata.464e707d80ec982 f b5b0c50a6526f05a -rw-r--r-- 1 pohly pohly 19610 Apr 7 09:53 tmp/stamps/i586-poky-linux/m4/1.4.18-r0.do_install.sigdata.a0954ae23d8da50 c 4af22f719255411fbitbake-diffsigs does not fail for
do_install, but does not print anything either (= no output at all from "bitbake-diffsigs -t m4do_install").Hmm, does it pick the right files to compare?
$ bitbake-diffsigs -d -t m4 do_install DEBUG: Signature file (previous): /fast/build/poky/x86/sstate-cache/a0/sstate:m4:i586-poky-linux:1.4.18:r0:i 5 86:3:a0954ae23d8da50c4af22f719255411f_install.tgz.siginfo DEBUG: Signature file (latest): /fast/build/poky/x86/tmp/stamps/i586-poky-linux/m4/1.4.18-r0.do_install.si g data.a0954ae23d8da50c4af22f719255411fAm I using the tool incorrectly or are my expectations wrong?
Quoted 8 lines
Quoted 4 lines
My expectation was that "bitbake-diffsigs -t m4
do_build" would tell me that m4 was rebuilt because some tasks that it depends on changed, and then I would drill down to those tasks, eventually arriving atdo_install.Hmm, something has gone wrong here because this is the exact scenario it's supposed to handle. In my similar tests it did work, so I'm not sure why it didn't work in this case - I'll have to give it a try here.
I send and then I immediately realised what might be wrong - you said you're using poky master which implies that you haven't also applied the patches for lib/oe/sstatesig.py that I sent to the OE-Core list today. If you haven't, try doing that, it should fix the problem.
Cheers,
Paul
Signature · 2 lines
Paul Eggleton Intel Open Source Technology Centre
Quoted 9 lines · Paul Eggleton
Quoted 8 lines · NZST Paul Eggleton
Quoted 6 lines · NZST Patrick Ohly
Am I using the tool incorrectly or are my expectations wrong?
My expectation was that "bitbake-diffsigs -t m4
do_build" would tell me that m4 was rebuilt because some tasks that it depends on changed, and then I would drill down to those tasks, eventually arriving atdo_install.
Quoted 8 lines
Quoted 3 lines
Hmm, something has gone wrong here because this is the exact scenario it's supposed to handle. In my similar tests it did work, so I'm not sure why it didn't work in this case - I'll have to give it a try here.
I send and then I immediately realised what might be wrong - you said you're using poky master which implies that you haven't also applied the patches for
lib/oe/sstatesig.pythat I sent to the OE-Core list today. If you haven't, try doing that, it should fix the problem.
Indeed, I had missed that part. It's working now for do_install.
What about do_build? Should bitbake-diffsigs be usable with that?
Signature · 6 lines
Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter.
Quoted 17 lines · NZST Patrick Ohly
Quoted 16 lines · Paul Eggleton
Quoted 14 lines · NZST Paul Eggleton
Quoted 7 lines · NZST Patrick Ohly
Am I using the tool incorrectly or are my expectations wrong?
My expectation was that "bitbake-diffsigs -t m4
do_build" would tell me
that m4 was rebuilt because some tasks that it depends on changed, and then I would drill down to those tasks, eventually arriving atdo_install.Hmm, something has gone wrong here because this is the exact scenario it's
supposed to handle. In my similar tests it did work, so I'm not sure why it
didn't work in this case - I'll have to give it a try here.
Quoted 8 lines
Quoted 4 lines
I send and then I immediately realised what might be wrong - you said you're using poky master which implies that you haven't also applied the patches for
lib/oe/sstatesig.pythat I sent to the OE-Core list today. If you haven't, try doing that, it should fix the problem.Indeed, I had missed that part. It's working now for
do_install.What about
do_build? Should bitbake-diffsigs be usable with that?
If it changed (i.e. there are two signature files to compare) then yes, -t should be able to work back from those. For some reason though you don't seem to have sigdata files for do_build, which is odd - a quick look in my stamps directory here suggests that you should.
Cheers,
Paul
Signature · 2 lines
Paul Eggleton Intel Open Source Technology Centre
Quoted 18 lines
-----Original Message-----
From: [email protected] [mailto:bitbake- [email protected]] On Behalf Of Paul Eggleton
Sent: den 6 april 2017 23:52
To: [email protected]
Subject: [bitbake-devel] [PATCH 00/11] bitbake-diffsigs fixes/improvementsAs part of an effort to add task signature recording and comparison to buildhistory in OE, I went digging into bitbake-diffsigs and the code that supports it and discovered that unfortunately it had a number of bugs - in particular the -t option despite being around for a while and having numerous band-aids applied in the past still wasn't really working properly. I have to take a big chunk of the responsibility for this as I wrote the thing in the first place. This patchset corrects most of the issues that I found and also makes a number of improvements to the readability of the output.
Quoted 2 lines
NOTE: there are a few corresponding changes required in OE-Core that I am about to send out, these should be applied at the same time.
Will you do the same for bitbake-dumpsig as I believe it has many problems in common with bitbake-diffsigs?
//Peter
Re: [PATCH 03/11] bitbake-diffsigs: properly report which signature is missing
In reply to Paul EggletonQuoted 2 lines · Paul Eggleton
If just one of the two signatures we want to compare aren't available, report that one rather than misleadingly claiming both are missing.
I just noticed that bitbake-diffsigs -t ... do_build recursively dumps differences in other tasks - cool, I don't know it could do that. It worked fine for all-arch ca-certificates, but not for
$ bitbake-diffsigs -d -t gdb-cross-x86_64 do_build -s 68a82197720db0f0dbad8955635bc1c0 5631f484d848541bdbf6a4c1f8117f79 DEBUG: Signature file (previous): /fast/build/nodistro/x86/tmp-glibc/stamps/x86_64-linux/gdb-cross-x86_64/7.12.1-r0.do_build.sigdata.68a82197720db0f0dbad8955635bc1c0 DEBUG: Signature file (latest): /fast/build/nodistro/x86/tmp-glibc/stamps/x86_64-linux/gdb-cross-x86_64/7.12.1-r0.do_build.sigdata.5631f484d848541bdbf6a4c1f8117f79 Hash for dependent task gdb/gdb-cross_7.12.1.bb.do_populate_sysroot changed from 6247000d87570135b704fcfd3e4e7cd4 to 554c1ce69bf0538a91436e5f7c256488 Unable to find matching sigdata for /fast/work/openembedded-core/meta/recipes-devtools/gdb/gdb-cross_7.12.1.bb.do_populate_sysroot with hashes 6247000d87570135b704fcfd3e4e7cd4 or 554c1ce69bf0538a91436e5f7c256488
$ bitbake-diffsigs -d -t gdb-cross-x86_64 do_populate_sysroot -s 6247000d87570135b704fcfd3e4e7cd4 554c1ce69bf0538a91436e5f7c256488 DEBUG: Signature file (previous): /fast/build/nodistro/x86/tmp-glibc/stamps/x86_64-linux/gdb-cross-x86_64/7.12.1-r0.do_populate_sysroot.sigdata.6247000d87570135b704fcfd3e4e7cd4 DEBUG: Signature file (latest): /fast/build/nodistro/x86/tmp-glibc/stamps/x86_64-linux/gdb-cross-x86_64/7.12.1-r0.do_populate_sysroot.sigdata.554c1ce69bf0538a91436e5f7c256488 Hash for dependent task gdb/gdb-cross_7.12.1.bb.do_install changed from 0786db89020b877d443da848548b3d1d to 746c2815e180fa2a4ff28608c7c1b9d5 Unable to find matching sigdata for /fast/work/openembedded-core/meta/recipes-devtools/gdb/gdb-cross_7.12.1.bb.do_install with hashes 0786db89020b877d443da848548b3d1d or 746c2815e180fa2a4ff28608c7c1b9d5
In other words, automatic diff fails to find files, while manual diff finds them?!
This situation can be triggered with a local patch of mine (to be posted soon) and using yocto-compat-layer.py -n --machines intel-corei7-64 intel-core2-32 qemux86 qemux86-64 -- /fast/work/meta-intel
The test uses MACHINE=... bitbake -S none world to generate the signatures for different machines and then looks for tasks that are shared between machines with different signatures.
Signature · 6 lines
Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter.
Quoted 26 lines · Peter Kjellerstedt
Quoted 26 lines
-----Original Message-----
From: [email protected] [mailto:bitbake- [email protected]] On Behalf Of Paul Eggleton
Sent: den 6 april 2017 23:52
To: [email protected]
Subject: [bitbake-devel] [PATCH 00/11] bitbake-diffsigs fixes/improvementsAs part of an effort to add task signature recording and comparison to
buildhistory in OE, I went digging into bitbake-diffsigs and the code
that supports it and discovered that unfortunately it had a number of
bugs - in particular the -t option despite being around for a while and
having numerous band-aids applied in the past still wasn't really working properly. I have to take a big chunk of the responsibility for
this as I wrote the thing in the first place. This patchset corrects
most of the issues that I found and also makes a number of improvements
to the readability of the output.
Quoted 5 lines
Quoted 3 lines
NOTE: there are a few corresponding changes required in OE-Core that I
am about to send out, these should be applied at the same time.Will you do the same for bitbake-dumpsig as I believe it has many problems in common with bitbake-diffsigs?
bitbake-dumpsig is near enough a symlink to diffsigs, its all the same code.
Cheers,
Richard