buildhistory improvements
cover buildhistory improvements
A bunch of improvements for buildhistory, implementing some feature requests and todo items of my own, as well as tidying up some of the code.
The following changes since commit 404d2d490fc347203e89d274530c17fb5f0aa20f:
gcc-configure-target: Set native-system-header-dir for target gcc (2012-08-01 23:11:09 +0100)
are available in the git repository at:
git://git.openembedded.org/openembedded-core-contrib paule/buildhistory-fixes7
http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=paule/buildhistory-fixes7
Paul Eggleton (9):
classes/buildhistory: remove obsolete flat package history feature classes/buildhistory: ensure old package info is removed classes/buildhistory: remove unused functions classes/buildhistory: save preinst/postinst/prerm/postrm classes/buildhistory: record PKG/PKGE/PKGV/PKGR
scripts: add buildhistory-tag script
buildhistory_analysis: tidy up duplicate split_version function
buildhistory_analysis: ignore removal of self-dependencies classes/buildhistory: save metadata revisions
meta/classes/buildhistory.bbclass | 197 ++++++++++++++++++---------------- meta/lib/oe/buildhistory_analysis.py | 96 +++++++++++++---- scripts/buildhistory-tag | 42 ++++++++ 3 files changed, 221 insertions(+), 114 deletions(-) create mode 100755 scripts/buildhistory-tag
Signature · 1 line
1.7.9.5
PATCH 1/9 classes/buildhistory: remove obsolete flat package history feature
Remove the ability to set BUILDHISTORY_KEEP_VERSIONS = "1" to save the package history as flat versioned files rather than relying on git to keep previous versions of the package information. git has proven to work quite well in this capacity.
--- meta/classes/buildhistory.bbclass | 43 +++++++------------------------------ 1 file changed, 8 insertions(+), 35 deletions(-)
meta/classes/buildhistory.bbclass +8 -35
@@ -156,7 +156,6 @@ python buildhistory_emit_pkghistory() { rcpinfo.depends = sortlist(squashspaces(d.getVar('DEPENDS', True) or "")) rcpinfo.packages = packages write_recipehistory(rcpinfo, d) - write_latestlink(None, pe, pv, pr, d) # Apparently the version can be different on a per-package basis (see Python) pkgdest = d.getVar('PKGDEST', True)
@@ -199,8 +198,6 @@ python buildhistory_emit_pkghistory() { pkginfo.filelist = " ".join(filelist) write_pkghistory(pkginfo, d) - - write_latestlink(pkg, pe, pv, pr, d) }
@@ -212,8 +209,8 @@ def write_recipehistory(rcpinfo, d): if not os.path.exists(pkghistdir): os.makedirs(pkghistdir) - verfile = os.path.join(pkghistdir, "%s:%s-%s" % (rcpinfo.pe, rcpinfo.pv, rcpinfo.pr)) - f = open(verfile, "w") + infofile = os.path.join(pkghistdir, "latest") + f = open(infofile, "w") try: if rcpinfo.pe != "0": f.write("PE = %s\n" % rcpinfo.pe)
@@ -226,16 +223,16 @@ def write_recipehistory(rcpinfo, d): def write_pkghistory(pkginfo, d): - bb.debug(2, "Writing package history") + bb.debug(2, "Writing package history for package %s" % pkginfo.name) pkghistdir = d.getVar('BUILDHISTORY_DIR_PACKAGE', True) - verpath = os.path.join(pkghistdir, pkginfo.name) - if not os.path.exists(verpath): - os.makedirs(verpath) + pkgpath = os.path.join(pkghistdir, pkginfo.name) + if not os.path.exists(pkgpath): + os.makedirs(pkgpath) - verfile = os.path.join(verpath, "%s:%s-%s" % (pkginfo.pe, pkginfo.pv, pkginfo.pr)) - f = open(verfile, "w") + infofile = os.path.join(pkgpath, "latest") + f = open(infofile, "w") try: if pkginfo.pe != "0": f.write("PE = %s\n" % pkginfo.pe)
@@ -250,30 +247,6 @@ def write_pkghistory(pkginfo, d): f.close() -def write_latestlink(pkg, pe, pv, pr, d): - import shutil - - pkghistdir = d.getVar('BUILDHISTORY_DIR_PACKAGE', True) - - def rm_link(path): - try: - os.unlink(path) - except OSError: - return - - if pkg: - filedir = os.path.join(pkghistdir, pkg) - else: - filedir = pkghistdir - latest_file = os.path.join(filedir, "latest") - ver_file = os.path.join(filedir, "%s:%s-%s" % (pe, pv, pr)) - rm_link(latest_file) - if d.getVar('BUILDHISTORY_KEEP_VERSIONS', True) == '1': - shutil.copy(ver_file, latest_file) - else: - shutil.move(ver_file, latest_file) - - buildhistory_get_image_installed() { # Anything requiring the use of the packaging system should be done in here # in case the packaging files are going to be removed for this image
Signature · 1 line
1.7.9.5
PATCH 2/9 classes/buildhistory: ensure old package info is removed
In reply to Paul EggletonIf a package is removed from PACKAGES, ensure that the package info file and directory are removed from buildhistory so that we don't have stale data lying around.
--- meta/classes/buildhistory.bbclass | 18 ++++++++++++++---- 1 file changed, 14 insertions(+), 4 deletions(-)
meta/classes/buildhistory.bbclass +14 -4
@@ -149,6 +149,19 @@ python buildhistory_emit_pkghistory() { pr = d.getVar('PR', True) packages = squashspaces(d.getVar('PACKAGES', True)) + packagelist = packages.split() + if not os.path.exists(pkghistdir): + os.makedirs(pkghistdir) + else: + # Remove files for packages that no longer exist + for item in os.listdir(pkghistdir): + if item != "latest": + if item not in packagelist: + subdir = os.path.join(pkghistdir, item) + for subfile in os.listdir(subdir): + os.unlink(os.path.join(subdir, subfile)) + os.rmdir(subdir) + rcpinfo = RecipeInfo(pn) rcpinfo.pe = pe rcpinfo.pv = pv
@@ -159,7 +172,7 @@ python buildhistory_emit_pkghistory() { # Apparently the version can be different on a per-package basis (see Python) pkgdest = d.getVar('PKGDEST', True) - for pkg in packages.split(): + for pkg in packagelist: pe = getpkgvar(pkg, 'PE') or "0" pv = getpkgvar(pkg, 'PV') pr = getpkgvar(pkg, 'PR')
@@ -206,9 +219,6 @@ def write_recipehistory(rcpinfo, d): pkghistdir = d.getVar('BUILDHISTORY_DIR_PACKAGE', True) - if not os.path.exists(pkghistdir): - os.makedirs(pkghistdir) - infofile = os.path.join(pkghistdir, "latest") f = open(infofile, "w") try:
Signature · 1 line
1.7.9.5
PATCH 3/9 classes/buildhistory: remove unused functions
In reply to Paul Eggletongetlastrecipeversion and readRecipeInfo weren't called anywhere. Remove them so we don't have to keep them up-to-date with future additions to the recipe info file.
--- meta/classes/buildhistory.bbclass | 29 ----------------------------- 1 file changed, 29 deletions(-)
meta/classes/buildhistory.bbclass +0 -29
@@ -68,28 +68,6 @@ python buildhistory_emit_pkghistory() { return val - def readRecipeInfo(pn, histfile): - rcpinfo = RecipeInfo(pn) - f = open(histfile, "r") - try: - for line in f: - lns = line.split('=') - name = lns[0].strip() - value = lns[1].strip(" \t\r\n").strip('"') - if name == "PE": - rcpinfo.pe = value - elif name == "PV": - rcpinfo.pv = value - elif name == "PR": - rcpinfo.pr = value - elif name == "DEPENDS": - rcpinfo.depends = value - elif name == "PACKAGES": - rcpinfo.packages = value - finally: - f.close() - return rcpinfo - def readPackageInfo(pkg, histfile): pkginfo = PackageInfo(pkg) f = open(histfile, "r")
@@ -118,13 +96,6 @@ python buildhistory_emit_pkghistory() { f.close() return pkginfo - def getlastrecipeversion(pn): - try: - histfile = os.path.join(pkghistdir, "latest") - return readRecipeInfo(pn, histfile) - except EnvironmentError: - return None - def getlastpkgversion(pkg): try: histfile = os.path.join(pkghistdir, pkg, "latest")
Signature · 1 line
1.7.9.5
PATCH 4/9 classes/buildhistory: save preinst/postinst/prerm/postrm
In reply to Paul EggletonWrite the value of these package script variables into the packageinfo so that any changes to them can be tracked (in separate files since they are multi-line).
Inspired by an earlier patch from Andreas Müller <[email protected]>
--- meta/classes/buildhistory.bbclass | 24 ++++++++++------ meta/lib/oe/buildhistory_analysis.py | 50 +++++++++++++++++++++++++++++++++- 2 files changed, 65 insertions(+), 9 deletions(-)
meta/classes/buildhistory.bbclass +16 -8
@@ -57,6 +57,8 @@ python buildhistory_emit_pkghistory() { self.rrecommends = "" self.files = "" self.filelist = "" + # Variables that need to be written to their own separate file + self.filevars = dict.fromkeys(['pkg_preinst', 'pkg_postinst', 'pkg_prerm', 'pkg_postrm']) # Should check PACKAGES here to see if anything removed
@@ -167,6 +169,8 @@ python buildhistory_emit_pkghistory() { pkginfo.rdepends = sortpkglist(squashspaces(getpkgvar(pkg, 'RDEPENDS') or "")) pkginfo.rrecommends = sortpkglist(squashspaces(getpkgvar(pkg, 'RRECOMMENDS') or "")) pkginfo.files = squashspaces(getpkgvar(pkg, 'FILES') or "") + for filevar in pkginfo.filevars: + pkginfo.filevars[filevar] = getpkgvar(pkg, filevar) # Gather information about packaged files pkgdestpkg = os.path.join(pkgdest, pkg)
@@ -191,16 +195,13 @@ def write_recipehistory(rcpinfo, d): pkghistdir = d.getVar('BUILDHISTORY_DIR_PACKAGE', True) infofile = os.path.join(pkghistdir, "latest") - f = open(infofile, "w") - try: + with open(infofile, "w") as f: if rcpinfo.pe != "0": f.write("PE = %s\n" % rcpinfo.pe) f.write("PV = %s\n" % rcpinfo.pv) f.write("PR = %s\n" % rcpinfo.pr) f.write("DEPENDS = %s\n" % rcpinfo.depends) f.write("PACKAGES = %s\n" % rcpinfo.packages) - finally: - f.close() def write_pkghistory(pkginfo, d):
@@ -213,8 +214,7 @@ def write_pkghistory(pkginfo, d): os.makedirs(pkgpath) infofile = os.path.join(pkgpath, "latest") - f = open(infofile, "w") - try: + with open(infofile, "w") as f: if pkginfo.pe != "0": f.write("PE = %s\n" % pkginfo.pe) f.write("PV = %s\n" % pkginfo.pv)
@@ -224,8 +224,16 @@ def write_pkghistory(pkginfo, d): f.write("PKGSIZE = %d\n" % pkginfo.size) f.write("FILES = %s\n" % pkginfo.files) f.write("FILELIST = %s\n" % pkginfo.filelist) - finally: - f.close() + + for filevar in pkginfo.filevars: + filevarpath = os.path.join(pkgpath, "latest.%s" % filevar) + val = pkginfo.filevars[filevar] + if val: + with open(filevarpath, "w") as f: + f.write(val) + else: + if os.path.exists(filevarpath): + os.unlink(filevarpath) buildhistory_get_image_installed() {
meta/lib/oe/buildhistory_analysis.py +49 -1
68 lines not on this page · raw message
Signature · 1 line
1.7.9.5
PATCH 5/9 classes/buildhistory: record PKG/PKGE/PKGV/PKGR
In reply to Paul EggletonSave PKG (the actual output package name, which is often different due to debian renaming), and PKGE/PKGV/PKGR (which may be manipulated in certain special cases e.g. gitpkgv.bbclass in meta-oe, the external-sourcery-toolchain recipe, etc.) Note that these are only written when they are different from the normal package name in the case of PKG, or PE/PV/PR for the other variables.
Also, use PKGE/PKGV/PKGR instead of PE/PV/PR when comparing package versions since these actually represent the version that the package manager sees.
Implements [YOCTO #2787].
--- meta/classes/buildhistory.bbclass | 66 +++++++++++++++++++++++++--------- meta/lib/oe/buildhistory_analysis.py | 22 +++++++++++- 2 files changed, 71 insertions(+), 17 deletions(-)
meta/classes/buildhistory.bbclass +50 -16
@@ -51,6 +51,11 @@ python buildhistory_emit_pkghistory() { self.pe = "0" self.pv = "0" self.pr = "r0" + # pkg/pkge/pkgv/pkgr should be empty because we want to be able to default them + self.pkg = "" + self.pkge = "" + self.pkgv = "" + self.pkgr = "" self.size = 0 self.depends = "" self.rdepends = ""
@@ -72,8 +77,7 @@ python buildhistory_emit_pkghistory() { def readPackageInfo(pkg, histfile): pkginfo = PackageInfo(pkg) - f = open(histfile, "r") - try: + with open(histfile, "r") as f: for line in f: lns = line.split('=') name = lns[0].strip()
@@ -84,6 +88,14 @@ python buildhistory_emit_pkghistory() { pkginfo.pv = value elif name == "PR": pkginfo.pr = value + elif name == "PKG": + pkginfo.pkg = value + elif name == "PKGE": + pkginfo.pkge = value + elif name == "PKGV": + pkginfo.pkgv = value + elif name == "PKGR": + pkginfo.pkgr = value elif name == "RDEPENDS": pkginfo.rdepends = value elif name == "RRECOMMENDS":
@@ -94,8 +106,15 @@ python buildhistory_emit_pkghistory() { pkginfo.files = value elif name == "FILELIST": pkginfo.filelist = value - finally: - f.close() + # Apply defaults + if not pkginfo.pkg: + pkginfo.pkg = pkginfo.name + if not pkginfo.pkge: + pkginfo.pkge = pkginfo.pe + if not pkginfo.pkgv: + pkginfo.pkgv = pkginfo.pv + if not pkginfo.pkgr: + pkginfo.pkgr = pkginfo.pr return pkginfo def getlastpkgversion(pkg):
@@ -143,29 +162,33 @@ python buildhistory_emit_pkghistory() { rcpinfo.packages = packages write_recipehistory(rcpinfo, d) - # Apparently the version can be different on a per-package basis (see Python) pkgdest = d.getVar('PKGDEST', True) for pkg in packagelist: - pe = getpkgvar(pkg, 'PE') or "0" - pv = getpkgvar(pkg, 'PV') - pr = getpkgvar(pkg, 'PR') + pkge = getpkgvar(pkg, 'PKGE') or "0" + pkgv = getpkgvar(pkg, 'PKGV') + pkgr = getpkgvar(pkg, 'PKGR') # # Find out what the last version was # Make sure the version did not decrease # lastversion = getlastpkgversion(pkg) if lastversion: - last_pe = lastversion.pe - last_pv = lastversion.pv - last_pr = lastversion.pr - r = bb.utils.vercmp((pe, pv, pr), (last_pe, last_pv, last_pr)) + last_pkge = lastversion.pkge + last_pkgv = lastversion.pkgv + last_pkgr = lastversion.pkgr + r = bb.utils.vercmp((pkge, pkgv, pkgr), (last_pkge, last_pkgv, last_pkgr)) if r < 0: - bb.error("Package version for package %s went backwards which would break package feeds from (%s:%s-%s to %s:%s-%s)" % (pkg, last_pe, last_pv, last_pr, pe, pv, pr)) + bb.error("Package version for package %s went backwards which would break package feeds from (%s:%s-%s to %s:%s-%s)" % (pkg, last_pkge, last_pkgv, last_pkgr, pkge, pkgv, pkgr)) pkginfo = PackageInfo(pkg) - pkginfo.pe = pe - pkginfo.pv = pv - pkginfo.pr = pr + # Apparently the version can be different on a per-package basis (see Python) + pkginfo.pe = getpkgvar(pkg, 'PE') or "0" + pkginfo.pv = getpkgvar(pkg, 'PV') + pkginfo.pr = getpkgvar(pkg, 'PR') + pkginfo.pkg = getpkgvar(pkg, 'PKG') or pkg + pkginfo.pkge = pkge + pkginfo.pkgv = pkgv + pkginfo.pkgr = pkgr pkginfo.rdepends = sortpkglist(squashspaces(getpkgvar(pkg, 'RDEPENDS') or "")) pkginfo.rrecommends = sortpkglist(squashspaces(getpkgvar(pkg, 'RRECOMMENDS') or "")) pkginfo.files = squashspaces(getpkgvar(pkg, 'FILES') or "")
@@ -219,6 +242,17 @@ def write_pkghistory(pkginfo, d): f.write("PE = %s\n" % pkginfo.pe) f.write("PV = %s\n" % pkginfo.pv) f.write("PR = %s\n" % pkginfo.pr) + + pkgvars = {} + pkgvars['PKG'] = pkginfo.pkg if pkginfo.pkg != pkginfo.name else '' + pkgvars['PKGE'] = pkginfo.pkge if pkginfo.pkge != pkginfo.pe else '' + pkgvars['PKGV'] = pkginfo.pkgv if pkginfo.pkgv != pkginfo.pv else '' + pkgvars['PKGR'] = pkginfo.pkgr if pkginfo.pkgr != pkginfo.pr else '' + for pkgvar in pkgvars: + val = pkgvars[pkgvar] + if val: + f.write("%s = %s\n" % (pkgvar, val)) + f.write("RDEPENDS = %s\n" % pkginfo.rdepends) f.write("RRECOMMENDS = %s\n" % pkginfo.rrecommends) f.write("PKGSIZE = %d\n" % pkginfo.size)
meta/lib/oe/buildhistory_analysis.py +21 -1
@@ -19,9 +19,10 @@ import bb.utils # How to display fields list_fields = ['DEPENDS', 'RDEPENDS', 'RRECOMMENDS', 'FILES', 'FILELIST', 'USER_CLASSES', 'IMAGE_CLASSES', 'IMAGE_FEATURES', 'IMAGE_LINGUAS', 'IMAGE_INSTALL', 'BAD_RECOMMENDATIONS'] list_order_fields = ['PACKAGES'] +defaultval_fields = ['PKG', 'PKGE', 'PKGV', 'PKGR'] numeric_fields = ['PKGSIZE', 'IMAGESIZE'] # Fields to monitor -monitor_fields = ['RDEPENDS', 'RRECOMMENDS', 'PACKAGES', 'FILELIST', 'PKGSIZE', 'IMAGESIZE'] +monitor_fields = ['RDEPENDS', 'RRECOMMENDS', 'PACKAGES', 'FILELIST', 'PKGSIZE', 'IMAGESIZE', 'PKG', 'PKGE', 'PKGV', 'PKGR'] # Percentage change to alert for numeric fields monitor_numeric_threshold = 20 # Image files to monitor (note that image-info.txt is handled separately)
@@ -90,6 +91,10 @@ class ChangeRecord: else: percentchg = 100 out = '%s changed from %s to %s (%s%d%%)' % (self.fieldname, self.oldvalue or "''", self.newvalue or "''", '+' if percentchg > 0 else '', percentchg) + elif self.fieldname in defaultval_fields: + out = '%s changed from %s to %s' % (self.fieldname, self.oldvalue, self.newvalue) + if self.fieldname == 'PKG' and '[default]' in self.newvalue: + out += ' - may indicate debian renaming failure' elif self.fieldname in ['pkg_preinst', 'pkg_postinst', 'pkg_prerm', 'pkg_postrm']: if self.oldvalue and self.newvalue: out = '%s changed:\n ' % self.fieldname
@@ -299,6 +304,14 @@ def compare_dict_blobs(path, ablob, bblob, report_all): adict = blob_to_dict(ablob) bdict = blob_to_dict(bblob) + defaultvals = {} + defaultvals['PKG'] = os.path.basename(path) + defaultvals['PKGE'] = adict.get('PE', '0') + defaultvals['PKGV'] = adict.get('PV', '') + defaultvals['PKGR'] = adict.get('PR', '') + for key in defaultvals: + defaultvals[key] = '%s [default]' % defaultvals[key] + changes = [] keys = list(set(adict.keys()) | set(bdict.keys())) for key in keys:
@@ -327,6 +340,13 @@ def compare_dict_blobs(path, ablob, bblob, report_all): blist.sort() if ' '.join(alist) == ' '.join(blist): continue + + if key in defaultval_fields: + if not astr: + astr = defaultvals[key] + elif not bstr: + bstr = defaultvals[key] + chg = ChangeRecord(path, key, astr, bstr, key in monitor_fields) changes.append(chg) return changes
Signature · 1 line
1.7.9.5
PATCH 6/9 scripts: add buildhistory-tag script
In reply to Paul EggletonThis trivial script determines BUILDHISTORY_DIR using bitbake and then passes the command line options to "git tag" on the buildhistory repository. This is useful for setting a tag before making some changes; even if the build takes a number of executions of bitbake (resulting in multiple commits to the buildhistory repository) you can still do the overall comparison easily.
Suggested by Ross Burton <[email protected]>.
--- scripts/buildhistory-tag | 42 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+) create mode 100755 scripts/buildhistory-tag
scripts/buildhistory-tag new file +42 -0
@@ -0,0 +1,42 @@ +#!/bin/sh + +# buildhistory-tag +# +# Copyright (C) 2012 Intel Corporation +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License version 2 as +# published by the Free Software Foundation. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# + +if [ -z $1 ] ; then + echo "Please specify at least the name of the tag to set" + exit 1 +fi + +BUILDHISTORY_DIR=`bitbake -e | grep '^BUILDHISTORY_DIR=' | sed -e 's/^[A-Z_]*=//' -e 's/"//g'` +if [ -z "$BUILDHISTORY_DIR" ]; then + echo "Unable to determine BUILDHISTORY_DIR" + exit 1 +fi + +if [ ! -d "$BUILDHISTORY_DIR" ]; then + echo "BUILDHISTORY_DIR $BUILDHISTORY_DIR does not exist" + exit 1 +fi + +if [ ! -d "$BUILDHISTORY_DIR/.git" ]; then + echo "BUILDHISTORY_DIR $BUILDHISTORY_DIR exists but is not a git repository" + exit 1 +fi + +(cd $BUILDHISTORY_DIR; git tag $@)
Signature · 1 line
1.7.9.5
Re: [PATCH 6/9] scripts: add buildhistory-tag script
Quoted 8 lines · Paul Eggleton
This trivial script determines
BUILDHISTORY_DIRusing bitbake and then passes the command line options to "git tag" on the buildhistory repository. This is useful for setting a tag before making some changes; even if the build takes a number of executions of bitbake (resulting in multiple commits to the buildhistory repository) you can still do the overall comparison easily.Suggested by Ross Burton <[email protected]>.
I've since changed my local script to be "buildhistory" that just runs git "$@", so "buildhistory log" etc work nicely.
Ross
Re: [PATCH 6/9] scripts: add buildhistory-tag script
Quoted 8 lines · Ross
Quoted 7 lines · Paul Eggleton
This trivial script determines
BUILDHISTORY_DIRusing bitbake and then passes the command line options to "git tag" on the buildhistory repository. This is useful for setting a tag before making some changes; even if the build takes a number of executions of bitbake (resulting in multiple commits to the buildhistory repository) you can still do the overall comparison easily.
Quoted 4 lines
Quoted 1 line
Suggested by Ross Burton <[email protected]>.
I've since changed my local script to be "buildhistory" that just runs git "$@", so "buildhistory log" etc work nicely.
Interesting - sounds like it would be useful, but I wonder how it would then fit in with buildhistory-diff as a separate script (particularly with diff as a valid option to "buildhistory")...
Cheers,
Paul
Signature · 2 lines
Paul Eggleton Intel Open Source Technology Centre
Re: [PATCH 6/9] scripts: add buildhistory-tag script
Quoted 3 lines · Paul Eggleton
Interesting - sounds like it would be useful, but I wonder how it would then fit in with buildhistory-diff as a separate script (particularly with diff as a valid option to "buildhistory")...
Rename it to -compare?
Ross
PATCH 7/9 buildhistory_analysis: tidy up duplicate split_version function
In reply to Paul EggletonThis function is now provided by bb.utils and since we have now bumped the minimum bitbake version, we can switch to that one instead.
--- meta/lib/oe/buildhistory_analysis.py | 20 +------------------- 1 file changed, 1 insertion(+), 19 deletions(-)
meta/lib/oe/buildhistory_analysis.py +1 -19
@@ -261,24 +261,6 @@ def compare_lists(alines, blines): return filechanges -def split_version(s): - """Split a version string into its constituent parts (PE, PV, PR) - FIXME: this is a duplicate of a new function in bitbake/lib/bb/utils - - we should switch to that once we can bump the minimum bitbake version - """ - s = s.strip(" <>=") - e = 0 - if s.count(':'): - e = int(s.split(":")[0]) - s = s.split(":")[1] - r = "" - if s.count('-'): - r = s.rsplit("-", 1)[1] - s = s.rsplit("-", 1)[0] - v = s - return (e, v, r) - - def compare_pkg_lists(astr, bstr): depvera = bb.utils.explode_dep_versions(astr) depverb = bb.utils.explode_dep_versions(bstr)
@@ -290,7 +272,7 @@ def compare_pkg_lists(astr, bstr): dva = depvera[k] dvb = depverb[k] if dva and dvb and dva != dvb: - if bb.utils.vercmp(split_version(dva), split_version(dvb)) < 0: + if bb.utils.vercmp(bb.utils.split_version(dva), bb.utils.split_version(dvb)) < 0: remove.append(k) for k in remove:
Signature · 1 line
1.7.9.5
PATCH 8/9 buildhistory_analysis: ignore removal of self-dependencies
In reply to Paul EggletonThe recent removal of self-dependencies for dbg/dev packages produced a fair amount of noise, so filter it out since we know this is a reasonable change.
--- meta/lib/oe/buildhistory_analysis.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
meta/lib/oe/buildhistory_analysis.py +5 -1
@@ -286,8 +286,9 @@ def compare_dict_blobs(path, ablob, bblob, report_all): adict = blob_to_dict(ablob) bdict = blob_to_dict(bblob) + pkgname = os.path.basename(path) defaultvals = {} - defaultvals['PKG'] = os.path.basename(path) + defaultvals['PKG'] = pkgname defaultvals['PKGE'] = adict.get('PE', '0') defaultvals['PKGV'] = adict.get('PV', '') defaultvals['PKGR'] = adict.get('PR', '')
@@ -320,6 +321,9 @@ def compare_dict_blobs(path, ablob, bblob, report_all): alist.sort() blist = bstr.split() blist.sort() + # We don't care about the removal of self-dependencies + if pkgname in alist and not pkgname in blist: + alist.remove(pkgname) if ' '.join(alist) == ' '.join(blist): continue
Signature · 1 line
1.7.9.5
PATCH 9/9 classes/buildhistory: save metadata revisions
In reply to Paul EggletonAlways write the metadata revisions for each layer into a machine- readable "metadata-revs" file so that you can potentially link changes in the output back to changes in the metadata. (Unlike the existing similar build-id file, this is not specific to image changes.)
--- meta/classes/buildhistory.bbclass | 23 +++++++++++++++++++---- 1 file changed, 19 insertions(+), 4 deletions(-)
meta/classes/buildhistory.bbclass +19 -4
@@ -354,6 +354,15 @@ def buildhistory_get_layers(d): layertext = "Configured metadata layers:\n%s\n" % '\n'.join(get_layers_branch_rev(d)) return layertext +def buildhistory_get_metadata_revs(d): + # We want an easily machine-readable format here, so get_layers_branch_rev isn't quite what we want + layers = (d.getVar("BBLAYERS", True) or "").split() + medadata_revs = ["%-17s = %s:%s" % (os.path.basename(i), \ + base_get_metadata_git_branch(i, None).strip(), \ + base_get_metadata_git_revision(i, None)) \ + for i in layers] + return '\n'.join(medadata_revs) + def squashspaces(string): import re
@@ -382,19 +391,25 @@ buildhistory_commit() { return fi + # Create a machine-readable list of metadata revisions for each layer + cat > ${BUILDHISTORY_DIR}/metadata-revs <<END +${@buildhistory_get_metadata_revs(d)} +END + ( cd ${BUILDHISTORY_DIR}/ # Initialise the repo if necessary if [ ! -d .git ] ; then git init -q fi - # Ensure there are new/changed files to commit - repostatus=`git status --porcelain` + # Check if there are new/changed files to commit (other than metadata-revs) + repostatus=`git status --porcelain | grep -v " metadata-revs$"` if [ "$repostatus" != "" ] ; then - git add ${BUILDHISTORY_DIR}/* + git add . HOSTNAME=`hostname 2>/dev/null || echo unknown` # porcelain output looks like "?? packages/foo/bar" + # Ensure we commit metadata-revs with the first commit for entry in `echo "$repostatus" | awk '{print $2}' | awk -F/ '{print $1}' | sort | uniq` ; do - git commit ${BUILDHISTORY_DIR}/$entry -m "$entry: Build ${BUILDNAME} of ${DISTRO} ${DISTRO_VERSION} for machine ${MACHINE} on $HOSTNAME" --author "${BUILDHISTORY_COMMIT_AUTHOR}" > /dev/null + git commit $entry metadata-revs -m "$entry: Build ${BUILDNAME} of ${DISTRO} ${DISTRO_VERSION} for machine ${MACHINE} on $HOSTNAME" --author "${BUILDHISTORY_COMMIT_AUTHOR}" > /dev/null done if [ "${BUILDHISTORY_PUSH_REPO}" != "" ] ; then git push -q ${BUILDHISTORY_PUSH_REPO}
Signature · 1 line
1.7.9.5
Quoted 3 lines · Paul Eggleton
A bunch of improvements for buildhistory, implementing some feature requests and todo items of my own, as well as tidying up some of the code.
I've been using this branch for an hour or so whilst reviewing changes, the new features are useful and it didn't break.
Ross
Martin Jansa
Acked-byQuoted 9 lines · Ross
Quoted 3 lines · Paul Eggleton
A bunch of improvements for buildhistory, implementing some feature requests and todo items of my own, as well as tidying up some of the code.
I've been using this branch for an hour or so whilst reviewing changes, the new features are useful and it didn't break.
Acked-by Ross Burton <[email protected]>
My tests also look good, some interesting (expected) changes recorded here
Thanks!
Cheers,
Signature · 1 line
Martin 'JaMa' Jansa jabber: [email protected]
Quoted 16 lines · Martin Jansa
Quoted 9 lines · Ross
Quoted 3 lines · Paul Eggleton
A bunch of improvements for buildhistory, implementing some feature requests and todo items of my own, as well as tidying up some of the code.
I've been using this branch for an hour or so whilst reviewing changes, the new features are useful and it didn't break.
Acked-by Ross Burton <[email protected]>My tests also look good, some interesting (expected) changes recorded here
Quoted 8 lines
Thanks!
Acked-by Martin Jansa <[email protected]>Cheers,
--
Martin 'JaMa' Jansa jabber: [email protected]
Although I have no time for testing at the moment: thanks for doing this
Andreas
Richard Purdie
AppliedQuoted 14 lines · Paul Eggleton
A bunch of improvements for buildhistory, implementing some feature requests and todo items of my own, as well as tidying up some of the code.
The following changes since commit
404d2d490fc347203e89d274530c17fb5f0aa20f:gcc-configure-target: Set native-system-header-dir for target gcc (2012-08-01 23:11:09 +0100)
are available in the git repository at:
git://git.openembedded.org/openembedded-core-contribpaule/buildhistory-fixes7
http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=paule/buildhistory-fixes7
Quoted 9 lines
Paul Eggleton (9):
classes/buildhistory: remove obsolete flat package history feature classes/buildhistory: ensure old package info is removed classes/buildhistory: remove unused functions classes/buildhistory: save preinst/postinst/prerm/postrm classes/buildhistory: record PKG/PKGE/PKGV/PKGR
buildhistory_analysis: tidy up duplicatesplit_versionfunction
buildhistory_analysis: ignore removal of self-dependencies classes/buildhistory: save metadata revisions
Merged to master apart from:
scripts: add buildhistory-tag script
which seems to need more discussion/work.
Cheers,
Richard