buildhistory analysis improvements
cover buildhistory analysis improvements
Collect and monitor some additional information about images, and report other (possibly non-monitored) changes when a monitored item changes to provide context.
Additionally a couple of minor issues where found in field handling for the package info files.
The following changes since commit a0f5dd25a37fe3b8664c2133e80b6214559f93f6:
package_rpm.bbclass: Add support for filenames with spaces (2012-01-17 16:20:46 +0000)
are available in the git repository at:
git://git.openembedded.org/openembedded-core-contrib paule/buildhistory-imginfo
http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=paule/buildhistory-imginfo
Paul Eggleton (4):
buildhistory: record additional image info
buildhistory_analysis: correctly handle whitespace when splitting
lists
buildhistory_analysis: improve field handling robustness
buildhistory_analysis: include related fields in output
meta/classes/buildhistory.bbclass | 18 +++++ meta/lib/oe/buildhistory_analysis.py | 127 ++++++++++++++++++++++++---------- 2 files changed, 109 insertions(+), 36 deletions(-)
Signature · 1 line
1.7.5.4
PATCH 1/4 buildhistory: record additional image info
Record some additional information about images - the uncompressed size of the final image as well as the values of various variables that may have influenced its contents. This is recorded in a machine-readable "image-info.txt" file similar in structure to the package history files.
Also add some code to analyse changes to these values. (Most of the variable values aren't monitored directly but will be used as contextual information when they change at the same time as the content of the image changing.)
--- meta/classes/buildhistory.bbclass | 18 ++++++++++++ meta/lib/oe/buildhistory_analysis.py | 51 +++++++++++++++++++-------------- 2 files changed, 47 insertions(+), 22 deletions(-)
meta/classes/buildhistory.bbclass +18 -0
@@ -312,6 +312,14 @@ buildhistory_get_imageinfo() { # This awk script is somewhat messy, but handles where the size is not printed for device files under pseudo ( cd ${IMAGE_ROOTFS} && find . -ls | awk '{ if ( $7 ~ /[0-9]/ ) printf "%s %10-s %10-s %10s %s %s %s\n", $3, $5, $6, $7, $11, $12, $13 ; else printf "%s %10-s %10-s %10s %s %s %s\n", $3, $5, $6, 0, $10, $11, $12 }' > ${BUILDHISTORY_DIR_IMAGE}/files-in-image.txt ) + # Record some machine-readable meta-information about the image + echo -n > ${BUILDHISTORY_DIR_IMAGE}/image-info.txt + cat >> ${BUILDHISTORY_DIR_IMAGE}/image-info.txt <<END +${@buildhistory_get_imagevars(d)} +END + imagesize=`du -ks ${IMAGE_ROOTFS} | awk '{ print $1 }'` + echo "IMAGESIZE = $imagesize" >> ${BUILDHISTORY_DIR_IMAGE}/image-info.txt + # Add some configuration information echo "${MACHINE}: ${IMAGE_BASENAME} configured for ${DISTRO} ${DISTRO_VERSION}" > ${BUILDHISTORY_DIR_IMAGE}/build-id
@@ -330,6 +338,16 @@ def buildhistory_get_layers(d): return layertext +def buildhistory_get_imagevars(d): + imagevars = "DISTRO DISTRO_VERSION USER_CLASSES IMAGE_CLASSES IMAGE_FEATURES IMAGE_LINGUAS IMAGE_INSTALL BAD_RECOMMENDATIONS ROOTFS_POSTPROCESS_COMMAND IMAGE_POSTPROCESS_COMMAND" + + ret = "" + for var in imagevars.split(): + value = d.getVar(var, True) or "" + ret += "%s = %s\n" % (var, value) + return ret.rstrip('\n') + + buildhistory_commit() { if [ ! -d ${BUILDHISTORY_DIR} ] ; then # Code above that creates this dir never executed, so there can't be anything to commit
meta/lib/oe/buildhistory_analysis.py +29 -22
89 lines not on this page · raw message
Signature · 1 line
1.7.5.4
PATCH 2/4 buildhistory_analysis: correctly handle whitespace when splitting lists
In reply to Paul EggletonDon't specify any argument to the split() function when handling changes to list type variables (e.g. PACKAGES) so that the values are split by any whitespace and only split once for a block of multiple whitespace characters.
--- meta/lib/oe/buildhistory_analysis.py | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-)
meta/lib/oe/buildhistory_analysis.py +2 -2
@@ -34,8 +34,8 @@ class ChangeRecord: def __str__(self): if self.fieldname in list_fields: - aitems = self.oldvalue.split(' ') - bitems = self.newvalue.split(' ') + aitems = self.oldvalue.split() + bitems = self.newvalue.split() removed = list(set(aitems) - set(bitems)) added = list(set(bitems) - set(aitems)) return '%s: %s:%s%s' % (self.path, self.fieldname, ' removed "%s"' % ' '.join(removed) if removed else '', ' added "%s"' % ' '.join(added) if added else '')
Signature · 1 line
1.7.5.4
PATCH 3/4 buildhistory_analysis: improve field handling robustness
In reply to Paul EggletonAvoid errors when comparing changes for KEY = value files (package info files and image-info.txt):
* Handle keys appearing and disappearing - this will help to handle PE in package info files (which is only written when it is not blank) and when we add additional fields in future.
* Handle when old value is 0 for numeric field (avoid division by zero)
* Report when numeric field was empty or missing rather than 0 (but still treat it as 0 for comparison purposes)
--- meta/lib/oe/buildhistory_analysis.py | 29 +++++++++++++++++++---------- 1 files changed, 19 insertions(+), 10 deletions(-)
meta/lib/oe/buildhistory_analysis.py +19 -10
@@ -40,10 +40,13 @@ class ChangeRecord: added = list(set(bitems) - set(aitems)) return '%s: %s:%s%s' % (self.path, self.fieldname, ' removed "%s"' % ' '.join(removed) if removed else '', ' added "%s"' % ' '.join(added) if added else '') elif self.fieldname in numeric_fields: - aval = int(self.oldvalue) - bval = int(self.newvalue) - percentchg = ((bval - aval) / float(aval)) * 100 - return '%s: %s changed from %d to %d (%s%d%%)' % (self.path, self.fieldname, aval, bval, '+' if percentchg > 0 else '', percentchg) + aval = int(self.oldvalue or 0) + bval = int(self.newvalue or 0) + if aval != 0: + percentchg = ((bval - aval) / float(aval)) * 100 + else: + percentchg = 100 + return '%s: %s changed from %s to %s (%s%d%%)' % (self.path, self.fieldname, self.oldvalue or "''", self.newvalue or "''", '+' if percentchg > 0 else '', percentchg) elif self.fieldname in img_monitor_files: out = 'Changes to %s (%s):\n ' % (self.path, self.fieldname) if self.filechanges:
@@ -194,16 +197,22 @@ def compare_dict_blobs(path, ablob, bblob, report_all): bdict = blob_to_dict(bblob) changes = [] - for key in adict: + keys = list(set(adict.keys()) | set(bdict.keys())) + for key in keys: if report_all or key in monitor_fields: - if adict[key] != bdict[key]: + astr = adict.get(key, '') + bstr = bdict.get(key, '') + if astr != bstr: if (not report_all) and key in numeric_fields: - aval = int(adict[key]) - bval = int(bdict[key]) - percentchg = ((bval - aval) / float(aval)) * 100 + aval = int(astr or 0) + bval = int(bstr or 0) + if aval != 0: + percentchg = ((bval - aval) / float(aval)) * 100 + else: + percentchg = 100 if percentchg < monitor_numeric_threshold: continue - chg = ChangeRecord(path, key, adict[key], bdict[key]) + chg = ChangeRecord(path, key, astr, bstr) changes.append(chg) return changes
Signature · 1 line
1.7.5.4
PATCH 4/4 buildhistory_analysis: include related fields in output
In reply to Paul EggletonSometimes, when a value changes in the buildhistory it is useful to know when a related (but not necessarily itself monitored) value changes as it can help explain the change. For example, when the list of installed packages for an image changes it could be caused by a change to one of the image-related variables.
Related field changes are recorded as sub-items of each change. Currently the only way to visualise these is via the buildhistory-diff tool, so an example would be:
Changes to images/qemux86/eglibc/core-image-minimal (installed-package-names.txt):
locale-base-de-de was added
procps was added
* IMAGE_LINGUAS: added "de-de"
* IMAGE_INSTALL: added "procps"
Here we see that two additional packages have been added to the image, and looking at the related changes to the two variables IMAGE_INSTALL and IMAGE_LINGUAS we have the explanation as to why.
--- meta/lib/oe/buildhistory_analysis.py | 89 ++++++++++++++++++++++++---------- 1 files changed, 64 insertions(+), 25 deletions(-)
meta/lib/oe/buildhistory_analysis.py +64 -25
@@ -23,22 +23,41 @@ monitor_fields = ['RDEPENDS', 'RRECOMMENDS', 'PACKAGES', 'FILELIST', 'PKGSIZE', monitor_numeric_threshold = 20 # Image files to monitor (note that image-info.txt is handled separately) img_monitor_files = ['installed-package-names.txt', 'files-in-image.txt'] +# Related context fields for reporting (note: PE, PV & PR are always reported for monitored package fields) +related_fields = {} +related_fields['RDEPENDS'] = ['DEPENDS'] +related_fields['RRECOMMENDS'] = ['DEPENDS'] +related_fields['FILELIST'] = ['FILES'] +related_fields['PKGSIZE'] = ['FILELIST'] +related_fields['files-in-image.txt'] = ['installed-package-names.txt', 'USER_CLASSES', 'IMAGE_CLASSES', 'ROOTFS_POSTPROCESS_COMMAND', 'IMAGE_POSTPROCESS_COMMAND'] +related_fields['installed-package-names.txt'] = ['IMAGE_FEATURES', 'IMAGE_LINGUAS', 'IMAGE_INSTALL', 'BAD_RECOMMENDATIONS'] + class ChangeRecord: - def __init__(self, path, fieldname, oldvalue, newvalue): + def __init__(self, path, fieldname, oldvalue, newvalue, monitored): self.path = path self.fieldname = fieldname self.oldvalue = oldvalue self.newvalue = newvalue + self.monitored = monitored + self.related = [] self.filechanges = None def __str__(self): + return self._str_internal(True) + + def _str_internal(self, pathprefix): + if pathprefix: + prefix = '%s: ' % self.path + else: + prefix = '' + if self.fieldname in list_fields: aitems = self.oldvalue.split() bitems = self.newvalue.split() removed = list(set(aitems) - set(bitems)) added = list(set(bitems) - set(aitems)) - return '%s: %s:%s%s' % (self.path, self.fieldname, ' removed "%s"' % ' '.join(removed) if removed else '', ' added "%s"' % ' '.join(added) if added else '') + out = '%s:%s%s' % (self.fieldname, ' removed "%s"' % ' '.join(removed) if removed else '', ' added "%s"' % ' '.join(added) if added else '') elif self.fieldname in numeric_fields: aval = int(self.oldvalue or 0) bval = int(self.newvalue or 0)
@@ -46,9 +65,11 @@ class ChangeRecord: percentchg = ((bval - aval) / float(aval)) * 100 else: percentchg = 100 - return '%s: %s changed from %s to %s (%s%d%%)' % (self.path, self.fieldname, self.oldvalue or "''", self.newvalue or "''", '+' if percentchg > 0 else '', percentchg) + out = '%s changed from %s to %s (%s%d%%)' % (self.fieldname, self.oldvalue or "''", self.newvalue or "''", '+' if percentchg > 0 else '', percentchg) elif self.fieldname in img_monitor_files: - out = 'Changes to %s (%s):\n ' % (self.path, self.fieldname) + if pathprefix: + prefix = 'Changes to %s ' % self.path + out = '(%s):\n ' % self.fieldname if self.filechanges: out += '\n '.join(['%s' % i for i in self.filechanges]) else:
@@ -57,10 +78,15 @@ class ChangeRecord: diff = difflib.unified_diff(alines, blines, self.fieldname, self.fieldname, lineterm='') out += '\n '.join(list(diff)) out += '\n --' - return out else: - return '%s: %s changed from "%s" to "%s"' % (self.path, self.self.fieldname, self.oldvalue, self.newvalue) + out = '%s changed from "%s" to "%s"' % (self.fieldname, self.oldvalue, self.newvalue) + if self.related: + for chg in self.related: + for line in chg._str_internal(False).splitlines(): + out += '\n * %s' % line + + return '%s%s' % (prefix, out) class FileChange: changetype_add = 'A'
@@ -199,21 +225,20 @@ def compare_dict_blobs(path, ablob, bblob, report_all): changes = [] keys = list(set(adict.keys()) | set(bdict.keys())) for key in keys: - if report_all or key in monitor_fields: - astr = adict.get(key, '') - bstr = bdict.get(key, '') - if astr != bstr: - if (not report_all) and key in numeric_fields: - aval = int(astr or 0) - bval = int(bstr or 0) - if aval != 0: - percentchg = ((bval - aval) / float(aval)) * 100 - else: - percentchg = 100 - if percentchg < monitor_numeric_threshold: - continue - chg = ChangeRecord(path, key, astr, bstr) - changes.append(chg) + astr = adict.get(key, '') + bstr = bdict.get(key, '') + if astr != bstr: + if (not report_all) and key in numeric_fields: + aval = int(astr or 0) + bval = int(bstr or 0) + if aval != 0: + percentchg = ((bval - aval) / float(aval)) * 100 + else: + percentchg = 100 + if percentchg < monitor_numeric_threshold: + continue + chg = ChangeRecord(path, key, astr, bstr, key in monitor_fields) + changes.append(chg) return changes
@@ -236,7 +261,7 @@ def process_changes(repopath, revision1, revision2 = 'HEAD', report_all = False) blines = d.b_blob.data_stream.read().splitlines() filechanges = compare_file_lists(alines,blines) if filechanges: - chg = ChangeRecord(path, filename, None, None) + chg = ChangeRecord(path, filename, None, None, True) chg.filechanges = filechanges changes.append(chg) elif filename == 'installed-package-names.txt':
@@ -244,13 +269,27 @@ def process_changes(repopath, revision1, revision2 = 'HEAD', report_all = False) blines = d.b_blob.data_stream.read().splitlines() filechanges = compare_lists(alines,blines) if filechanges: - chg = ChangeRecord(path, filename, None, None) + chg = ChangeRecord(path, filename, None, None, True) chg.filechanges = filechanges changes.append(chg) else: - chg = ChangeRecord(path, filename, d.a_blob.data_stream.read(), d.b_blob.data_stream.read()) + chg = ChangeRecord(path, filename, d.a_blob.data_stream.read(), d.b_blob.data_stream.read(), True) changes.append(chg) elif filename == 'image-info.txt': changes.extend(compare_dict_blobs(path, d.a_blob, d.b_blob, report_all)) - return changes + # Link related changes + for chg in changes: + if chg.monitored: + for chg2 in changes: + # (Check dirname in the case of fields from recipe info files) + if chg.path == chg2.path or os.path.dirname(chg.path) == chg2.path: + if chg2.fieldname in related_fields.get(chg.fieldname, []): + chg.related.append(chg2) + elif chg.path.startswith('packages/') and chg2.fieldname in ['PE', 'PV', 'PR']: + chg.related.append(chg2) + + if report_all: + return changes + else: + return [chg for chg in changes if chg.monitored]
Signature · 1 line
1.7.5.4
Richard Purdie
AppliedQuoted 15 lines · Paul Eggleton
Collect and monitor some additional information about images, and report other (possibly non-monitored) changes when a monitored item changes to provide context.
Additionally a couple of minor issues where found in field handling for the package info files.
The following changes since commit
a0f5dd25a37fe3b8664c2133e80b6214559f93f6:package_rpm.bbclass: Add support for filenames with spaces (2012-01-17 16:20:46 +0000)
are available in the git repository at:
git://git.openembedded.org/openembedded-core-contribpaule/buildhistory-imginfo
http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=paule/buildhistory-imginfo
Quoted 6 lines
Paul Eggleton (4):
buildhistory: record additional image info
buildhistory_analysis: correctly handle whitespace when splitting
lists
buildhistory_analysis: improve field handling robustness
buildhistory_analysis: include related fields in output
Merged to master, thanks.
Richard