packagehistory improvement
cover packagehistory improvement
This is the initial implementation of the improvement for packagehistory.bbclass. This bbclass is intended to be used to help track changes to package output over time and hopefully catch unusual situations such as a package doubling in size. You can try it out by adding INHERIT += "packagehistory" to your local.conf, building something that performs do_package and then looking in TMPDIR/pkghistory to view the output.
I've tried to keep it fairly simple - i.e. no new storage backend, just plain files, although I anticipate most people using it will want to store the output in a git repository. There's potential for recording more per-recipe / per-package information, if there are suggestions I'd be keen to hear them.
Please review the following changes for suitability for inclusion. If you have any objections or suggestions for improvement, please respond to the patches. If you agree with the changes, please provide your Acked-by.
The following changes since commit 5f1dc1991f97037692193572298eb7f7a5640760:
staging.bbclass: Add BB_SETSCENE_VERIFY_FUNCTION function (2011-11-22 13:04:31 +0000)
are available in the git repository at:
git://git.openembedded.org/openembedded-core-contrib paule/packagehistory
http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=paule/packagehistory
Paul Eggleton (1):
classes/packagehistory: fix and extend
meta/classes/packagehistory.bbclass | 226 +++++++++++++++++++++++++++++------ 1 files changed, 189 insertions(+), 37 deletions(-)
Signature · 1 line
1.7.5.4
RFC PATCH 1/1 classes/packagehistory: fix and extend
* Replace use of BASEPKG_TARGET_SYS which is no longer available
* Change the file structure - use single files within PN/package subdirs rather than having a subdir level for each part of the version. There is a set of files for each recipe and for each package in directories underneath.
* Record more information - PACKAGES, DEPENDS, RDEPENDS, RRECOMMENDS, FILES, and the total size and a complete list of the packaged files.
* Record the values in simple text format. The "latest" file, rather than a symlink has been changed to a copy of the latest file so that if it is tracked in a VCS repository (e.g. git) you can compare it easily to the previous version.
Implements [YOCTO #1565].
--- meta/classes/packagehistory.bbclass | 226 +++++++++++++++++++++++++++++------ 1 files changed, 189 insertions(+), 37 deletions(-)
meta/classes/packagehistory.bbclass +189 -37
@@ -2,17 +2,39 @@ inherit package PACKAGEFUNCS += "emit_pkghistory" -PKGHIST_DIR = "${TMPDIR}/pkghistory/${BASEPKG_TARGET_SYS}/" - +PKGHIST_DIR = "${TMPDIR}/pkghistory/${MULTIMACH_TARGET_SYS}/" # # Called during do_package to write out metadata about this package # for comparision when writing future packages # python emit_pkghistory() { - packages = d.getVar('PACKAGES', True) - pkghistdir = d.getVar('PKGHIST_DIR', True) + import re + packages = d.getVar('PACKAGES', True) + pkghistdir = os.path.join(d.getVar('PKGHIST_DIR', True), d.getVar('PN', True)) + + class RecipeInfo: + def __init__(self, name): + self.name = name + self.pe = "0" + self.pv = "0" + self.pr = "r0" + self.depends = "" + self.packages = "" + + class PackageInfo: + def __init__(self, name): + self.name = name + self.pe = "0" + self.pv = "0" + self.pr = "r0" + self.size = 0 + self.depends = "" + self.rdepends = "" + self.rrecommends = "" + self.files = "" + self.filelist = "" # Should check PACKAGES here to see if anything removed
@@ -24,44 +46,136 @@ python emit_pkghistory() { return val - def getlastversion(pkg): + def readRecipeInfo(pn, histfile): + rcpinfo = RecipeInfo(pn) + f = open(histfile, "r") try: - pe = os.path.basename(os.readlink(os.path.join(pkghistdir, pkg, "latest"))) - pv = os.path.basename(os.readlink(os.path.join(pkghistdir, pkg, pe, "latest"))) - pr = os.path.basename(os.readlink(os.path.join(pkghistdir, pkg, pe, pv, "latest"))) - return (pe, pv, pr) - except OSError: - return (None, None, None) + for line in f: + lns = line.split('=') + name = lns[0].strip() + value = lns[1].strip(" \t\r\n").strip('"') + if name == "PE": + rcpinfo.pe = value + elif name == "PV": + rcpinfo.pv = value + elif name == "PR": + rcpinfo.pr = value + elif name == "DEPENDS": + rcpinfo.depends = value + elif name == "PACKAGES": + rcpinfo.packages = value + finally: + f.close() + return rcpinfo + + def readPackageInfo(pkg, histfile): + pkginfo = PackageInfo(pkg) + f = open(histfile, "r") + try: + for line in f: + lns = line.split('=') + name = lns[0].strip() + value = lns[1].strip(" \t\r\n").strip('"') + if name == "PE": + pkginfo.pe = value + elif name == "PV": + pkginfo.pv = value + elif name == "PR": + pkginfo.pr = value + elif name == "RDEPENDS": + pkginfo.rdepends = value + elif name == "RRECOMMENDS": + pkginfo.rrecommends = value + elif name == "PKGSIZE": + pkginfo.size = long(value) + elif name == "FILES": + pkginfo.files = value + elif name == "FILELIST": + pkginfo.filelist = value + finally: + f.close() + return pkginfo + + def getlastrecipeversion(pn): + try: + histfile = os.path.join(pkghistdir, "latest") + return readRecipeInfo(pn, histfile) + except EnvironmentError: + return None + def getlastpkgversion(pkg): + try: + histfile = os.path.join(pkghistdir, pkg, "latest") + return readPackageInfo(pkg, histfile) + except EnvironmentError: + return None + + def squashspaces(string): + return re.sub("\s+", " ", string) + + pn = d.getVar('PN', True) + pe = d.getVar('PE', True) or "0" + pv = d.getVar('PV', True) + pr = d.getVar('PR', True) + + rcpinfo = RecipeInfo(pn) + rcpinfo.pe = pe + rcpinfo.pv = pv + rcpinfo.pr = pr + rcpinfo.depends = squashspaces(d.getVar('DEPENDS', True) or "") + rcpinfo.packages = packages + write_recipehistory(rcpinfo, d) + write_latestlink(None, pe, pv, pr, d) + + # Apparently the version can be different on a per-package basis (see Python) + pkgdest = d.getVar('PKGDEST', True) for pkg in packages.split(): pe = getpkgvar(pkg, 'PE') or "0" pv = getpkgvar(pkg, 'PV') pr = getpkgvar(pkg, 'PR') - destdir = os.path.join(pkghistdir, pkg, pe, pv, pr) - # # Find out what the last version was # Make sure the version did not decrease # - lastversion = getlastversion(pkg) - (last_pe, last_pv, last_pr) = lastversion - - if last_pe is not None: - r = bb.utils.vercmp((pe, pv, pr), lastversion) + lastversion = getlastpkgversion(pkg) + if lastversion: + last_pe = lastversion.pe + last_pv = lastversion.pv + last_pr = lastversion.pr + r = bb.utils.vercmp((pe, pv, pr), (last_pe, last_pv, last_pr)) if r < 0: bb.fatal("Package version for package %s went backwards which would break package feeds from (%s:%s-%s to %s:%s-%s)" % (pkg, last_pe, last_pv, last_pr, pe, pv, pr)) - write_pkghistory(pkg, pe, pv, pr, d) - - if last_pe is not None: - check_pkghistory(pkg, pe, pv, pr, lastversion) - - write_latestlink(pkg, pe, pv, pr, d) + pkginfo = PackageInfo(pkg) + pkginfo.pe = pe + pkginfo.pv = pv + pkginfo.pr = pr + pkginfo.rdepends = squashspaces(getpkgvar(pkg, 'RDEPENDS') or "") + pkginfo.rrecommends = squashspaces(getpkgvar(pkg, 'RRECOMMENDS') or "") + pkginfo.files = squashspaces(getpkgvar(pkg, 'FILES') or "") + + # Gather information about packaged files + pkgdestpkg = os.path.join(pkgdest, pkg) + filelist = [] + pkginfo.size = 0 + for root, dirs, files in os.walk(pkgdestpkg): + relpth = os.path.relpath(root, pkgdestpkg) + for f in files: + fstat = os.lstat(os.path.join(root, f)) + pkginfo.size += fstat.st_size + filelist.append(os.sep + os.path.join(relpth, f)) + pkginfo.filelist = " ".join(filelist) + + write_pkghistory(pkginfo, d) + + if lastversion: + check_pkghistory(pkginfo, lastversion) + + write_latestlink(pkg, pe, pv, pr, d) } -def check_pkghistory(pkg, pe, pv, pr, lastversion): - (last_pe, last_pv, last_pr) = lastversion +def check_pkghistory(pkginfo, lastversion): bb.debug(2, "Checking package history") # RDEPENDS removed?
@@ -69,17 +183,56 @@ def check_pkghistory(pkg, pe, pv, pr, lastversion): # Each file list of each package for file removals? -def write_pkghistory(pkg, pe, pv, pr, d): +def write_recipehistory(rcpinfo, d): + bb.debug(2, "Writing recipe history") + + pkghistdir = os.path.join(d.getVar('PKGHIST_DIR', True), d.getVar('PN', True)) + + if not os.path.exists(pkghistdir): + os.makedirs(pkghistdir) + + verfile = os.path.join(pkghistdir, "%s:%s-%s" % (rcpinfo.pe, rcpinfo.pv, rcpinfo.pr)) + f = open(verfile, "w") + try: + if rcpinfo.pe != "0": + f.write("PE = %s\n" % rcpinfo.pe) + f.write("PV = %s\n" % rcpinfo.pv) + f.write("PR = %s\n" % rcpinfo.pr) + f.write("DEPENDS = %s\n" % rcpinfo.depends) + f.write("PACKAGES = %s\n" % rcpinfo.packages) + finally: + f.close() + + +def write_pkghistory(pkginfo, d): bb.debug(2, "Writing package history") - pkghistdir = d.getVar('PKGHIST_DIR', True) + pkghistdir = os.path.join(d.getVar('PKGHIST_DIR', True), d.getVar('PN', True)) - verpath = os.path.join(pkghistdir, pkg, pe, pv, pr) + verpath = os.path.join(pkghistdir, pkginfo.name) if not os.path.exists(verpath): os.makedirs(verpath) + verfile = os.path.join(verpath, "%s:%s-%s" % (pkginfo.pe, pkginfo.pv, pkginfo.pr)) + f = open(verfile, "w") + try: + if pkginfo.pe != "0": + f.write("PE = %s\n" % pkginfo.pe) + f.write("PV = %s\n" % pkginfo.pv) + f.write("PR = %s\n" % pkginfo.pr) + f.write("RDEPENDS = %s\n" % pkginfo.rdepends) + f.write("RRECOMMENDS = %s\n" % pkginfo.rrecommends) + f.write("PKGSIZE = %d\n" % pkginfo.size) + f.write("FILES = %s\n" % pkginfo.files) + f.write("FILELIST = %s\n" % pkginfo.filelist) + finally: + f.close() + + def write_latestlink(pkg, pe, pv, pr, d): - pkghistdir = d.getVar('PKGHIST_DIR', True) + import shutil + + pkghistdir = os.path.join(d.getVar('PKGHIST_DIR', True), d.getVar('PN', True)) def rm_link(path): try:
@@ -87,11 +240,10 @@ def write_latestlink(pkg, pe, pv, pr, d): except OSError: return - rm_link(os.path.join(pkghistdir, pkg, "latest")) - rm_link(os.path.join(pkghistdir, pkg, pe, "latest")) - rm_link(os.path.join(pkghistdir, pkg, pe, pv, "latest")) - - os.symlink(os.path.join(pkghistdir, pkg, pe), os.path.join(pkghistdir, pkg, "latest")) - os.symlink(os.path.join(pkghistdir, pkg, pe, pv), os.path.join(pkghistdir, pkg, pe, "latest")) - os.symlink(os.path.join(pkghistdir, pkg, pe, pv, pr), os.path.join(pkghistdir, pkg, pe, pv, "latest")) + if pkg: + filedir = os.path.join(pkghistdir, pkg) + else: + filedir = pkghistdir + rm_link(os.path.join(filedir, "latest")) + shutil.copy(os.path.join(filedir, "%s:%s-%s" % (pe, pv, pr)), os.path.join(filedir, "latest"))
Signature · 1 line
1.7.5.4
Op 22 nov. 2011, om 15:52 heeft Paul Eggleton het volgende geschreven:
Quoted 15 lines
This is the initial implementation of the improvement for packagehistory.bbclass. This bbclass is intended to be used to help track changes to package output over time and hopefully catch unusual situations such as a package doubling in size. You can try it out by adding INHERIT += "packagehistory" to your local.conf, building something that performs
do_packageand then looking in TMPDIR/pkghistory to view the output.I've tried to keep it fairly simple - i.e. no new storage backend, just plain files, although I anticipate most people using it will want to store the output in a git repository. There's potential for recording more per-recipe / per-package information, if there are suggestions I'd be keen to hear them.
Quoted 2 lines
Please review the following changes for suitability for inclusion. If you have any objections or suggestions for improvement, please respond to the patches.
I have a question instead:
Does it still suffer from the bug where it will break if you change machine and something goes backward in PV (e.g. gcc 4.6 is used on arm, but 4.5 on ppc)?
regards,
Koen
Quoted 3 lines · On Tuesday 22 November 2011 18:25:25 Koen Kooi wrote:
Does it still suffer from the bug where it will break if you change machine and something goes backward in PV (e.g. gcc 4.6 is used on arm, but 4.5 on ppc)?
If the package in question is not tied to the machine or the arch that was changed, yes it will still cause do_package to fail. I'd be happy to address this, but my question would be, how do we tell the difference between a legitimate version pinning and when something has gone wrong?
Cheers,
Paul
Signature · 2 lines
Paul Eggleton Intel Open Source Technology Centre
Quoted 3 lines · On Tuesday 22 November 2011 18:25:25 Koen Kooi wrote:
Does it still suffer from the bug where it will break if you change machine and something goes backward in PV (e.g. gcc 4.6 is used on arm, but 4.5 on ppc)?
So unless I'm mistaken the fix that you yourself applied to OE in 2009 (and which was included in Poky, and hence OE-core) already works around this specific situation, since the package history is always stored under a BASEPKG_TARGET_SYS directory (or MULTIMACH_TARGET_SYS now with my patch since the former is no longer available). Am I missing something?
Cheers,
Paul
Signature · 2 lines
Paul Eggleton Intel Open Source Technology Centre
Op 23 nov. 2011, om 14:24 heeft Paul Eggleton het volgende geschreven:
Quoted 10 lines · On Tuesday 22 November 2011 18:25:25 Koen Kooi wrote:
Quoted 3 lines
Does it still suffer from the bug where it will break if you change machine and something goes backward in PV (e.g. gcc 4.6 is used on arm, but 4.5 on ppc)?
So unless I'm mistaken the fix that you yourself applied to OE in 2009 (and which was included in Poky, and hence OE-core) already works around this specific situation, since the package history is always stored under a
BASEPKG_TARGET_SYSdirectory (orMULTIMACH_TARGET_SYSnow with my patch since the former is no longer available). Am I missing something?
I'm trying hard to remember why I disabled it last year, I was convinced it was the TARGET_SYS thing, but apparently not. Good to know that I fixed it already :) I'll ask my coworker when he gets back, but I vaguely remember something about it acting funny with nostamp making consecutive runs of 'bitbake some-image' really slow.
regards,
Koen