Skip to contentSkip to conversation

buildhistory improvements

PATCH Applied Paul Eggleton 5 participants 17 messages Aug 2, 2012 – Aug 6, 2012

lore
Share & tools
lore
#0 lore raw

cover buildhistory improvements

A bunch of improvements for buildhistory, implementing some feature requests and todo items of my own, as well as tidying up some of the code.

The following changes since commit 404d2d490fc347203e89d274530c17fb5f0aa20f:

gcc-configure-target: Set native-system-header-dir for target gcc (2012-08-01 23:11:09 +0100)

are available in the git repository at:

git://git.openembedded.org/openembedded-core-contrib paule/buildhistory-fixes7
http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=paule/buildhistory-fixes7

Paul Eggleton (9):
classes/buildhistory: remove obsolete flat package history feature classes/buildhistory: ensure old package info is removed classes/buildhistory: remove unused functions classes/buildhistory: save preinst/postinst/prerm/postrm classes/buildhistory: record PKG/PKGE/PKGV/PKGR
scripts: add buildhistory-tag script
buildhistory_analysis: tidy up duplicate split_version function
buildhistory_analysis: ignore removal of self-dependencies classes/buildhistory: save metadata revisions

 meta/classes/buildhistory.bbclass    |  197 ++++++++++++++++++----------------
 meta/lib/oe/buildhistory_analysis.py |   96 +++++++++++++----
 scripts/buildhistory-tag             |   42 ++++++++
 3 files changed, 221 insertions(+), 114 deletions(-)
 create mode 100755 scripts/buildhistory-tag
Signature · 1 line
1.7.9.5
#1 lore raw

PATCH 1/9 classes/buildhistory: remove obsolete flat package history feature

Remove the ability to set BUILDHISTORY_KEEP_VERSIONS = "1" to save the package history as flat versioned files rather than relying on git to keep previous versions of the package information. git has proven to work quite well in this capacity.

---
 meta/classes/buildhistory.bbclass |   43 +++++++------------------------------
 1 file changed, 8 insertions(+), 35 deletions(-)
meta/classes/buildhistory.bbclass +8 -35
index ddb76e8..f5494cd 100644
@@ -156,7 +156,6 @@ python buildhistory_emit_pkghistory() {
     rcpinfo.depends = sortlist(squashspaces(d.getVar('DEPENDS', True) or ""))
     rcpinfo.packages = packages
     write_recipehistory(rcpinfo, d)
-    write_latestlink(None, pe, pv, pr, d)
 
     # Apparently the version can be different on a per-package basis (see Python)
     pkgdest = d.getVar('PKGDEST', True)
@@ -199,8 +198,6 @@ python buildhistory_emit_pkghistory() {
         pkginfo.filelist = " ".join(filelist)
 
         write_pkghistory(pkginfo, d)
-
-        write_latestlink(pkg, pe, pv, pr, d)
 }
 
 
@@ -212,8 +209,8 @@ def write_recipehistory(rcpinfo, d):
     if not os.path.exists(pkghistdir):
         os.makedirs(pkghistdir)
 
-    verfile = os.path.join(pkghistdir, "%s:%s-%s" % (rcpinfo.pe, rcpinfo.pv, rcpinfo.pr))
-    f = open(verfile, "w")
+    infofile = os.path.join(pkghistdir, "latest")
+    f = open(infofile, "w")
     try:
         if rcpinfo.pe != "0":
             f.write("PE = %s\n" %  rcpinfo.pe)
@@ -226,16 +223,16 @@ def write_recipehistory(rcpinfo, d):
 
 
 def write_pkghistory(pkginfo, d):
-    bb.debug(2, "Writing package history")
+    bb.debug(2, "Writing package history for package %s" % pkginfo.name)
 
     pkghistdir = d.getVar('BUILDHISTORY_DIR_PACKAGE', True)
 
-    verpath = os.path.join(pkghistdir, pkginfo.name)
-    if not os.path.exists(verpath):
-        os.makedirs(verpath)
+    pkgpath = os.path.join(pkghistdir, pkginfo.name)
+    if not os.path.exists(pkgpath):
+        os.makedirs(pkgpath)
 
-    verfile = os.path.join(verpath, "%s:%s-%s" % (pkginfo.pe, pkginfo.pv, pkginfo.pr))
-    f = open(verfile, "w")
+    infofile = os.path.join(pkgpath, "latest")
+    f = open(infofile, "w")
     try:
         if pkginfo.pe != "0":
             f.write("PE = %s\n" %  pkginfo.pe)
@@ -250,30 +247,6 @@ def write_pkghistory(pkginfo, d):
         f.close()
 
 
-def write_latestlink(pkg, pe, pv, pr, d):
-    import shutil
-
-    pkghistdir = d.getVar('BUILDHISTORY_DIR_PACKAGE', True)
-
-    def rm_link(path):
-        try:
-            os.unlink(path)
-        except OSError:
-            return
-
-    if pkg:
-        filedir = os.path.join(pkghistdir, pkg)
-    else:
-        filedir = pkghistdir
-    latest_file = os.path.join(filedir, "latest")
-    ver_file = os.path.join(filedir, "%s:%s-%s" % (pe, pv, pr))
-    rm_link(latest_file)
-    if d.getVar('BUILDHISTORY_KEEP_VERSIONS', True) == '1':
-        shutil.copy(ver_file, latest_file)
-    else:
-        shutil.move(ver_file, latest_file)
-
-
 buildhistory_get_image_installed() {
 	# Anything requiring the use of the packaging system should be done in here
 	# in case the packaging files are going to be removed for this image
Signature · 1 line
1.7.9.5
#2 lore raw

PATCH 2/9 classes/buildhistory: ensure old package info is removed

In reply to Paul Eggleton

If a package is removed from PACKAGES, ensure that the package info file and directory are removed from buildhistory so that we don't have stale data lying around.

---
 meta/classes/buildhistory.bbclass |   18 ++++++++++++++----
 1 file changed, 14 insertions(+), 4 deletions(-)
meta/classes/buildhistory.bbclass +14 -4
index f5494cd..c3555da 100644
@@ -149,6 +149,19 @@ python buildhistory_emit_pkghistory() {
     pr = d.getVar('PR', True)
     packages = squashspaces(d.getVar('PACKAGES', True))
 
+    packagelist = packages.split()
+    if not os.path.exists(pkghistdir):
+        os.makedirs(pkghistdir)
+    else:
+        # Remove files for packages that no longer exist
+        for item in os.listdir(pkghistdir):
+            if item != "latest":
+                if item not in packagelist:
+                    subdir = os.path.join(pkghistdir, item)
+                    for subfile in os.listdir(subdir):
+                        os.unlink(os.path.join(subdir, subfile))
+                    os.rmdir(subdir)
+
     rcpinfo = RecipeInfo(pn)
     rcpinfo.pe = pe
     rcpinfo.pv = pv
@@ -159,7 +172,7 @@ python buildhistory_emit_pkghistory() {
 
     # Apparently the version can be different on a per-package basis (see Python)
     pkgdest = d.getVar('PKGDEST', True)
-    for pkg in packages.split():
+    for pkg in packagelist:
         pe = getpkgvar(pkg, 'PE') or "0"
         pv = getpkgvar(pkg, 'PV')
         pr = getpkgvar(pkg, 'PR')
@@ -206,9 +219,6 @@ def write_recipehistory(rcpinfo, d):
 
     pkghistdir = d.getVar('BUILDHISTORY_DIR_PACKAGE', True)
 
-    if not os.path.exists(pkghistdir):
-        os.makedirs(pkghistdir)
-
     infofile = os.path.join(pkghistdir, "latest")
     f = open(infofile, "w")
     try:
Signature · 1 line
1.7.9.5
#3 lore raw

PATCH 3/9 classes/buildhistory: remove unused functions

In reply to Paul Eggleton

getlastrecipeversion and readRecipeInfo weren't called anywhere. Remove them so we don't have to keep them up-to-date with future additions to the recipe info file.

---
 meta/classes/buildhistory.bbclass |   29 -----------------------------
 1 file changed, 29 deletions(-)
meta/classes/buildhistory.bbclass +0 -29
index c3555da..510a6df 100644
@@ -68,28 +68,6 @@ python buildhistory_emit_pkghistory() {
 
         return val
 
-    def readRecipeInfo(pn, histfile):
-        rcpinfo = RecipeInfo(pn)
-        f = open(histfile, "r")
-        try:
-            for line in f:
-                lns = line.split('=')
-                name = lns[0].strip()
-                value = lns[1].strip(" \t\r\n").strip('"')
-                if name == "PE":
-                    rcpinfo.pe = value
-                elif name == "PV":
-                    rcpinfo.pv = value
-                elif name == "PR":
-                    rcpinfo.pr = value
-                elif name == "DEPENDS":
-                    rcpinfo.depends = value
-                elif name == "PACKAGES":
-                    rcpinfo.packages = value
-        finally:
-            f.close()
-        return rcpinfo
-
     def readPackageInfo(pkg, histfile):
         pkginfo = PackageInfo(pkg)
         f = open(histfile, "r")
@@ -118,13 +96,6 @@ python buildhistory_emit_pkghistory() {
             f.close()
         return pkginfo
 
-    def getlastrecipeversion(pn):
-        try:
-            histfile = os.path.join(pkghistdir, "latest")
-            return readRecipeInfo(pn, histfile)
-        except EnvironmentError:
-            return None
-
     def getlastpkgversion(pkg):
         try:
             histfile = os.path.join(pkghistdir, pkg, "latest")
Signature · 1 line
1.7.9.5
#4 lore raw

PATCH 4/9 classes/buildhistory: save preinst/postinst/prerm/postrm

In reply to Paul Eggleton

Write the value of these package script variables into the packageinfo so that any changes to them can be tracked (in separate files since they are multi-line).

Inspired by an earlier patch from Andreas Müller <[email protected]>

---
 meta/classes/buildhistory.bbclass    |   24 ++++++++++------
 meta/lib/oe/buildhistory_analysis.py |   50 +++++++++++++++++++++++++++++++++-
 2 files changed, 65 insertions(+), 9 deletions(-)
meta/classes/buildhistory.bbclass +16 -8
index 510a6df..200d031 100644
@@ -57,6 +57,8 @@ python buildhistory_emit_pkghistory() {
             self.rrecommends = ""
             self.files = ""
             self.filelist = ""
+            # Variables that need to be written to their own separate file
+            self.filevars = dict.fromkeys(['pkg_preinst', 'pkg_postinst', 'pkg_prerm', 'pkg_postrm'])
 
     # Should check PACKAGES here to see if anything removed
 
@@ -167,6 +169,8 @@ python buildhistory_emit_pkghistory() {
         pkginfo.rdepends = sortpkglist(squashspaces(getpkgvar(pkg, 'RDEPENDS') or ""))
         pkginfo.rrecommends = sortpkglist(squashspaces(getpkgvar(pkg, 'RRECOMMENDS') or ""))
         pkginfo.files = squashspaces(getpkgvar(pkg, 'FILES') or "")
+        for filevar in pkginfo.filevars:
+            pkginfo.filevars[filevar] = getpkgvar(pkg, filevar)
 
         # Gather information about packaged files
         pkgdestpkg = os.path.join(pkgdest, pkg)
@@ -191,16 +195,13 @@ def write_recipehistory(rcpinfo, d):
     pkghistdir = d.getVar('BUILDHISTORY_DIR_PACKAGE', True)
 
     infofile = os.path.join(pkghistdir, "latest")
-    f = open(infofile, "w")
-    try:
+    with open(infofile, "w") as f:
         if rcpinfo.pe != "0":
             f.write("PE = %s\n" %  rcpinfo.pe)
         f.write("PV = %s\n" %  rcpinfo.pv)
         f.write("PR = %s\n" %  rcpinfo.pr)
         f.write("DEPENDS = %s\n" %  rcpinfo.depends)
         f.write("PACKAGES = %s\n" %  rcpinfo.packages)
-    finally:
-        f.close()
 
 
 def write_pkghistory(pkginfo, d):
@@ -213,8 +214,7 @@ def write_pkghistory(pkginfo, d):
         os.makedirs(pkgpath)
 
     infofile = os.path.join(pkgpath, "latest")
-    f = open(infofile, "w")
-    try:
+    with open(infofile, "w") as f:
         if pkginfo.pe != "0":
             f.write("PE = %s\n" %  pkginfo.pe)
         f.write("PV = %s\n" %  pkginfo.pv)
@@ -224,8 +224,16 @@ def write_pkghistory(pkginfo, d):
         f.write("PKGSIZE = %d\n" %  pkginfo.size)
         f.write("FILES = %s\n" %  pkginfo.files)
         f.write("FILELIST = %s\n" %  pkginfo.filelist)
-    finally:
-        f.close()
+
+    for filevar in pkginfo.filevars:
+        filevarpath = os.path.join(pkgpath, "latest.%s" % filevar)
+        val = pkginfo.filevars[filevar]
+        if val:
+            with open(filevarpath, "w") as f:
+                f.write(val)
+        else:
+            if os.path.exists(filevarpath):
+                os.unlink(filevarpath)
 
 
 buildhistory_get_image_installed() {
meta/lib/oe/buildhistory_analysis.py +49 -1

68 lines not on this page · raw message

Signature · 1 line
1.7.9.5
#5 lore raw

PATCH 5/9 classes/buildhistory: record PKG/PKGE/PKGV/PKGR

In reply to Paul Eggleton

Save PKG (the actual output package name, which is often different due to debian renaming), and PKGE/PKGV/PKGR (which may be manipulated in certain special cases e.g. gitpkgv.bbclass in meta-oe, the external-sourcery-toolchain recipe, etc.) Note that these are only written when they are different from the normal package name in the case of PKG, or PE/PV/PR for the other variables.

Also, use PKGE/PKGV/PKGR instead of PE/PV/PR when comparing package versions since these actually represent the version that the package manager sees.

Implements [YOCTO #2787].

---
 meta/classes/buildhistory.bbclass    |   66 +++++++++++++++++++++++++---------
 meta/lib/oe/buildhistory_analysis.py |   22 +++++++++++-
 2 files changed, 71 insertions(+), 17 deletions(-)
meta/classes/buildhistory.bbclass +50 -16
index 200d031..76648c9 100644
@@ -51,6 +51,11 @@ python buildhistory_emit_pkghistory() {
             self.pe = "0"
             self.pv = "0"
             self.pr = "r0"
+            # pkg/pkge/pkgv/pkgr should be empty because we want to be able to default them
+            self.pkg = ""
+            self.pkge = ""
+            self.pkgv = ""
+            self.pkgr = ""
             self.size = 0
             self.depends = ""
             self.rdepends = ""
@@ -72,8 +77,7 @@ python buildhistory_emit_pkghistory() {
 
     def readPackageInfo(pkg, histfile):
         pkginfo = PackageInfo(pkg)
-        f = open(histfile, "r")
-        try:
+        with open(histfile, "r") as f:
             for line in f:
                 lns = line.split('=')
                 name = lns[0].strip()
@@ -84,6 +88,14 @@ python buildhistory_emit_pkghistory() {
                     pkginfo.pv = value
                 elif name == "PR":
                     pkginfo.pr = value
+                elif name == "PKG":
+                    pkginfo.pkg = value
+                elif name == "PKGE":
+                    pkginfo.pkge = value
+                elif name == "PKGV":
+                    pkginfo.pkgv = value
+                elif name == "PKGR":
+                    pkginfo.pkgr = value
                 elif name == "RDEPENDS":
                     pkginfo.rdepends = value
                 elif name == "RRECOMMENDS":
@@ -94,8 +106,15 @@ python buildhistory_emit_pkghistory() {
                     pkginfo.files = value
                 elif name == "FILELIST":
                     pkginfo.filelist = value
-        finally:
-            f.close()
+        # Apply defaults
+        if not pkginfo.pkg:
+            pkginfo.pkg = pkginfo.name
+        if not pkginfo.pkge:
+            pkginfo.pkge = pkginfo.pe
+        if not pkginfo.pkgv:
+            pkginfo.pkgv = pkginfo.pv
+        if not pkginfo.pkgr:
+            pkginfo.pkgr = pkginfo.pr
         return pkginfo
 
     def getlastpkgversion(pkg):
@@ -143,29 +162,33 @@ python buildhistory_emit_pkghistory() {
     rcpinfo.packages = packages
     write_recipehistory(rcpinfo, d)
 
-    # Apparently the version can be different on a per-package basis (see Python)
     pkgdest = d.getVar('PKGDEST', True)
     for pkg in packagelist:
-        pe = getpkgvar(pkg, 'PE') or "0"
-        pv = getpkgvar(pkg, 'PV')
-        pr = getpkgvar(pkg, 'PR')
+        pkge = getpkgvar(pkg, 'PKGE') or "0"
+        pkgv = getpkgvar(pkg, 'PKGV')
+        pkgr = getpkgvar(pkg, 'PKGR')
         #
         # Find out what the last version was
         # Make sure the version did not decrease
         #
         lastversion = getlastpkgversion(pkg)
         if lastversion:
-            last_pe = lastversion.pe
-            last_pv = lastversion.pv
-            last_pr = lastversion.pr
-            r = bb.utils.vercmp((pe, pv, pr), (last_pe, last_pv, last_pr))
+            last_pkge = lastversion.pkge
+            last_pkgv = lastversion.pkgv
+            last_pkgr = lastversion.pkgr
+            r = bb.utils.vercmp((pkge, pkgv, pkgr), (last_pkge, last_pkgv, last_pkgr))
             if r < 0:
-                bb.error("Package version for package %s went backwards which would break package feeds from (%s:%s-%s to %s:%s-%s)" % (pkg, last_pe, last_pv, last_pr, pe, pv, pr))
+                bb.error("Package version for package %s went backwards which would break package feeds from (%s:%s-%s to %s:%s-%s)" % (pkg, last_pkge, last_pkgv, last_pkgr, pkge, pkgv, pkgr))
 
         pkginfo = PackageInfo(pkg)
-        pkginfo.pe = pe
-        pkginfo.pv = pv
-        pkginfo.pr = pr
+        # Apparently the version can be different on a per-package basis (see Python)
+        pkginfo.pe = getpkgvar(pkg, 'PE') or "0"
+        pkginfo.pv = getpkgvar(pkg, 'PV')
+        pkginfo.pr = getpkgvar(pkg, 'PR')
+        pkginfo.pkg = getpkgvar(pkg, 'PKG') or pkg
+        pkginfo.pkge = pkge
+        pkginfo.pkgv = pkgv
+        pkginfo.pkgr = pkgr
         pkginfo.rdepends = sortpkglist(squashspaces(getpkgvar(pkg, 'RDEPENDS') or ""))
         pkginfo.rrecommends = sortpkglist(squashspaces(getpkgvar(pkg, 'RRECOMMENDS') or ""))
         pkginfo.files = squashspaces(getpkgvar(pkg, 'FILES') or "")
@@ -219,6 +242,17 @@ def write_pkghistory(pkginfo, d):
             f.write("PE = %s\n" %  pkginfo.pe)
         f.write("PV = %s\n" %  pkginfo.pv)
         f.write("PR = %s\n" %  pkginfo.pr)
+
+        pkgvars = {}
+        pkgvars['PKG'] = pkginfo.pkg if pkginfo.pkg != pkginfo.name else ''
+        pkgvars['PKGE'] = pkginfo.pkge if pkginfo.pkge != pkginfo.pe else ''
+        pkgvars['PKGV'] = pkginfo.pkgv if pkginfo.pkgv != pkginfo.pv else ''
+        pkgvars['PKGR'] = pkginfo.pkgr if pkginfo.pkgr != pkginfo.pr else ''
+        for pkgvar in pkgvars:
+            val = pkgvars[pkgvar]
+            if val:
+                f.write("%s = %s\n" % (pkgvar, val))
+
         f.write("RDEPENDS = %s\n" %  pkginfo.rdepends)
         f.write("RRECOMMENDS = %s\n" %  pkginfo.rrecommends)
         f.write("PKGSIZE = %d\n" %  pkginfo.size)
meta/lib/oe/buildhistory_analysis.py +21 -1
index 6c6a085..55bd7b7 100644
@@ -19,9 +19,10 @@ import bb.utils
 # How to display fields
 list_fields = ['DEPENDS', 'RDEPENDS', 'RRECOMMENDS', 'FILES', 'FILELIST', 'USER_CLASSES', 'IMAGE_CLASSES', 'IMAGE_FEATURES', 'IMAGE_LINGUAS', 'IMAGE_INSTALL', 'BAD_RECOMMENDATIONS']
 list_order_fields = ['PACKAGES']
+defaultval_fields = ['PKG', 'PKGE', 'PKGV', 'PKGR']
 numeric_fields = ['PKGSIZE', 'IMAGESIZE']
 # Fields to monitor
-monitor_fields = ['RDEPENDS', 'RRECOMMENDS', 'PACKAGES', 'FILELIST', 'PKGSIZE', 'IMAGESIZE']
+monitor_fields = ['RDEPENDS', 'RRECOMMENDS', 'PACKAGES', 'FILELIST', 'PKGSIZE', 'IMAGESIZE', 'PKG', 'PKGE', 'PKGV', 'PKGR']
 # Percentage change to alert for numeric fields
 monitor_numeric_threshold = 20
 # Image files to monitor (note that image-info.txt is handled separately)
@@ -90,6 +91,10 @@ class ChangeRecord:
             else:
                 percentchg = 100
             out = '%s changed from %s to %s (%s%d%%)' % (self.fieldname, self.oldvalue or "''", self.newvalue or "''", '+' if percentchg > 0 else '', percentchg)
+        elif self.fieldname in defaultval_fields:
+            out = '%s changed from %s to %s' % (self.fieldname, self.oldvalue, self.newvalue)
+            if self.fieldname == 'PKG' and '[default]' in self.newvalue:
+                out += ' - may indicate debian renaming failure'
         elif self.fieldname in ['pkg_preinst', 'pkg_postinst', 'pkg_prerm', 'pkg_postrm']:
             if self.oldvalue and self.newvalue:
                 out = '%s changed:\n  ' % self.fieldname
@@ -299,6 +304,14 @@ def compare_dict_blobs(path, ablob, bblob, report_all):
     adict = blob_to_dict(ablob)
     bdict = blob_to_dict(bblob)
 
+    defaultvals = {}
+    defaultvals['PKG'] = os.path.basename(path)
+    defaultvals['PKGE'] = adict.get('PE', '0')
+    defaultvals['PKGV'] = adict.get('PV', '')
+    defaultvals['PKGR'] = adict.get('PR', '')
+    for key in defaultvals:
+        defaultvals[key] = '%s [default]' % defaultvals[key]
+
     changes = []
     keys = list(set(adict.keys()) | set(bdict.keys()))
     for key in keys:
@@ -327,6 +340,13 @@ def compare_dict_blobs(path, ablob, bblob, report_all):
                 blist.sort()
                 if ' '.join(alist) == ' '.join(blist):
                     continue
+
+            if key in defaultval_fields:
+                if not astr:
+                    astr = defaultvals[key]
+                elif not bstr:
+                    bstr = defaultvals[key]
+
             chg = ChangeRecord(path, key, astr, bstr, key in monitor_fields)
             changes.append(chg)
     return changes
Signature · 1 line
1.7.9.5
#6 lore raw

PATCH 6/9 scripts: add buildhistory-tag script

In reply to Paul Eggleton

This trivial script determines BUILDHISTORY_DIR using bitbake and then passes the command line options to "git tag" on the buildhistory repository. This is useful for setting a tag before making some changes; even if the build takes a number of executions of bitbake (resulting in multiple commits to the buildhistory repository) you can still do the overall comparison easily.

Suggested by Ross Burton <[email protected]>.

---
 scripts/buildhistory-tag |   42 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 42 insertions(+)
 create mode 100755 scripts/buildhistory-tag
scripts/buildhistory-tag new file +42 -0
new file mode 100755
index 0000000..da90bcf
@@ -0,0 +1,42 @@
+#!/bin/sh
+
+# buildhistory-tag
+#
+# Copyright (C) 2012 Intel Corporation
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+#
+
+if [ -z $1 ] ; then
+    echo "Please specify at least the name of the tag to set"
+    exit 1
+fi
+
+BUILDHISTORY_DIR=`bitbake -e | grep '^BUILDHISTORY_DIR=' | sed -e 's/^[A-Z_]*=//' -e 's/"//g'`
+if [ -z "$BUILDHISTORY_DIR" ]; then
+    echo "Unable to determine BUILDHISTORY_DIR"
+    exit 1
+fi
+
+if [ ! -d "$BUILDHISTORY_DIR" ]; then
+    echo "BUILDHISTORY_DIR $BUILDHISTORY_DIR does not exist"
+    exit 1
+fi
+
+if [ ! -d "$BUILDHISTORY_DIR/.git" ]; then
+    echo "BUILDHISTORY_DIR $BUILDHISTORY_DIR exists but is not a git repository"
+    exit 1
+fi
+
+(cd $BUILDHISTORY_DIR; git tag $@)
Signature · 1 line
1.7.9.5
#10 lore raw

PATCH 7/9 buildhistory_analysis: tidy up duplicate split_version function

In reply to Paul Eggleton

This function is now provided by bb.utils and since we have now bumped the minimum bitbake version, we can switch to that one instead.

---
 meta/lib/oe/buildhistory_analysis.py |   20 +-------------------
 1 file changed, 1 insertion(+), 19 deletions(-)
meta/lib/oe/buildhistory_analysis.py +1 -19
index 55bd7b7..a1a0408 100644
@@ -261,24 +261,6 @@ def compare_lists(alines, blines):
     return filechanges
 
 
-def split_version(s):
-    """Split a version string into its constituent parts (PE, PV, PR)
-    FIXME: this is a duplicate of a new function in bitbake/lib/bb/utils -
-    we should switch to that once we can bump the minimum bitbake version
-    """
-    s = s.strip(" <>=")
-    e = 0
-    if s.count(':'):
-        e = int(s.split(":")[0])
-        s = s.split(":")[1]
-    r = ""
-    if s.count('-'):
-        r = s.rsplit("-", 1)[1]
-        s = s.rsplit("-", 1)[0]
-    v = s
-    return (e, v, r)
-
-
 def compare_pkg_lists(astr, bstr):
     depvera = bb.utils.explode_dep_versions(astr)
     depverb = bb.utils.explode_dep_versions(bstr)
@@ -290,7 +272,7 @@ def compare_pkg_lists(astr, bstr):
             dva = depvera[k]
             dvb = depverb[k]
             if dva and dvb and dva != dvb:
-                if bb.utils.vercmp(split_version(dva), split_version(dvb)) < 0:
+                if bb.utils.vercmp(bb.utils.split_version(dva), bb.utils.split_version(dvb)) < 0:
                     remove.append(k)
 
     for k in remove:
Signature · 1 line
1.7.9.5
#11 lore raw

PATCH 8/9 buildhistory_analysis: ignore removal of self-dependencies

In reply to Paul Eggleton

The recent removal of self-dependencies for dbg/dev packages produced a fair amount of noise, so filter it out since we know this is a reasonable change.

---
 meta/lib/oe/buildhistory_analysis.py |    6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
meta/lib/oe/buildhistory_analysis.py +5 -1
index a1a0408..a5da7d9 100644
@@ -286,8 +286,9 @@ def compare_dict_blobs(path, ablob, bblob, report_all):
     adict = blob_to_dict(ablob)
     bdict = blob_to_dict(bblob)
 
+    pkgname = os.path.basename(path)
     defaultvals = {}
-    defaultvals['PKG'] = os.path.basename(path)
+    defaultvals['PKG'] = pkgname
     defaultvals['PKGE'] = adict.get('PE', '0')
     defaultvals['PKGV'] = adict.get('PV', '')
     defaultvals['PKGR'] = adict.get('PR', '')
@@ -320,6 +321,9 @@ def compare_dict_blobs(path, ablob, bblob, report_all):
                 alist.sort()
                 blist = bstr.split()
                 blist.sort()
+                # We don't care about the removal of self-dependencies
+                if pkgname in alist and not pkgname in blist:
+                    alist.remove(pkgname)
                 if ' '.join(alist) == ' '.join(blist):
                     continue
 
Signature · 1 line
1.7.9.5
#12 lore raw

PATCH 9/9 classes/buildhistory: save metadata revisions

In reply to Paul Eggleton

Always write the metadata revisions for each layer into a machine- readable "metadata-revs" file so that you can potentially link changes in the output back to changes in the metadata. (Unlike the existing similar build-id file, this is not specific to image changes.)

---
 meta/classes/buildhistory.bbclass |   23 +++++++++++++++++++----
 1 file changed, 19 insertions(+), 4 deletions(-)
meta/classes/buildhistory.bbclass +19 -4
index 76648c9..e121088 100644
@@ -354,6 +354,15 @@ def buildhistory_get_layers(d):
     layertext = "Configured metadata layers:\n%s\n" % '\n'.join(get_layers_branch_rev(d))
     return layertext
 
+def buildhistory_get_metadata_revs(d):
+    # We want an easily machine-readable format here, so get_layers_branch_rev isn't quite what we want
+    layers = (d.getVar("BBLAYERS", True) or "").split()
+    medadata_revs = ["%-17s = %s:%s" % (os.path.basename(i), \
+        base_get_metadata_git_branch(i, None).strip(), \
+        base_get_metadata_git_revision(i, None)) \
+            for i in layers]
+    return '\n'.join(medadata_revs)
+
 
 def squashspaces(string):
     import re
@@ -382,19 +391,25 @@ buildhistory_commit() {
 		return
 	fi
 
+	# Create a machine-readable list of metadata revisions for each layer
+	cat > ${BUILDHISTORY_DIR}/metadata-revs <<END
+${@buildhistory_get_metadata_revs(d)}
+END
+
 	( cd ${BUILDHISTORY_DIR}/
 		# Initialise the repo if necessary
 		if [ ! -d .git ] ; then
 			git init -q
 		fi
-		# Ensure there are new/changed files to commit
-		repostatus=`git status --porcelain`
+		# Check if there are new/changed files to commit (other than metadata-revs)
+		repostatus=`git status --porcelain | grep -v " metadata-revs$"`
 		if [ "$repostatus" != "" ] ; then
-			git add ${BUILDHISTORY_DIR}/*
+			git add .
 			HOSTNAME=`hostname 2>/dev/null || echo unknown`
 			# porcelain output looks like "?? packages/foo/bar"
+			# Ensure we commit metadata-revs with the first commit
 			for entry in `echo "$repostatus" | awk '{print $2}' | awk -F/ '{print $1}' | sort | uniq` ; do
-				git commit ${BUILDHISTORY_DIR}/$entry -m "$entry: Build ${BUILDNAME} of ${DISTRO} ${DISTRO_VERSION} for machine ${MACHINE} on $HOSTNAME" --author "${BUILDHISTORY_COMMIT_AUTHOR}" > /dev/null
+				git commit $entry metadata-revs -m "$entry: Build ${BUILDNAME} of ${DISTRO} ${DISTRO_VERSION} for machine ${MACHINE} on $HOSTNAME" --author "${BUILDHISTORY_COMMIT_AUTHOR}" > /dev/null
 			done
 			if [ "${BUILDHISTORY_PUSH_REPO}" != "" ] ; then
 				git push -q ${BUILDHISTORY_PUSH_REPO}
Signature · 1 line
1.7.9.5
#7 lore raw

Re: [PATCH 6/9] scripts: add buildhistory-tag script

In reply to Paul Eggleton
Quoted 8 lines · Paul Eggleton

This trivial script determines BUILDHISTORY_DIR using bitbake and then passes the command line options to "git tag" on the buildhistory repository. This is useful for setting a tag before making some changes; even if the build takes a number of executions of bitbake (resulting in multiple commits to the buildhistory repository) you can still do the overall comparison easily.

Suggested by Ross Burton <[email protected]>.

I've since changed my local script to be "buildhistory" that just runs git "$@", so "buildhistory log" etc work nicely.

Ross

#8 lore raw

Re: [PATCH 6/9] scripts: add buildhistory-tag script

Quoted 8 lines · Ross
Quoted 7 lines · Paul Eggleton

This trivial script determines BUILDHISTORY_DIR using bitbake and then passes the command line options to "git tag" on the buildhistory repository. This is useful for setting a tag before making some changes; even if the build takes a number of executions of bitbake (resulting in multiple commits to the buildhistory repository) you can still do the overall comparison easily.

Quoted 4 lines
Quoted 1 line

Suggested by Ross Burton <[email protected]>.

I've since changed my local script to be "buildhistory" that just runs git "$@", so "buildhistory log" etc work nicely.

Interesting - sounds like it would be useful, but I wonder how it would then fit in with buildhistory-diff as a separate script (particularly with diff as a valid option to "buildhistory")...

Cheers,
Paul

Signature · 2 lines
Paul Eggleton
Intel Open Source Technology Centre
#9 lore raw

Re: [PATCH 6/9] scripts: add buildhistory-tag script

Quoted 3 lines · Paul Eggleton

Interesting - sounds like it would be useful, but I wonder how it would then fit in with buildhistory-diff as a separate script (particularly with diff as a valid option to "buildhistory")...

Rename it to -compare?

Ross

Acked-by on the v1 series
#13 lore raw
In reply to Paul Eggleton
Quoted 3 lines · Paul Eggleton

A bunch of improvements for buildhistory, implementing some feature requests and todo items of my own, as well as tidying up some of the code.

I've been using this branch for an hour or so whilst reviewing changes, the new features are useful and it didn't break.

Ross

Acked-by
#14 lore raw
Quoted 9 lines · Ross
Quoted 3 lines · Paul Eggleton

A bunch of improvements for buildhistory, implementing some feature requests and todo items of my own, as well as tidying up some of the code.

I've been using this branch for an hour or so whilst reviewing changes, the new features are useful and it didn't break.

My tests also look good, some interesting (expected) changes recorded here

http://git.shr-project.org/git/?p=buildhistory.git;a=commitdiff;h=2fe33b15dcec9eb69f9075f138cea2405c061b00

Thanks!

Cheers,

Signature · 1 line
Martin 'JaMa' Jansa     jabber: [email protected]
#15 lore raw
Quoted 16 lines · Martin Jansa
Quoted 9 lines · Ross
Quoted 3 lines · Paul Eggleton

A bunch of improvements for buildhistory, implementing some feature requests and todo items of my own, as well as tidying up some of the code.

I've been using this branch for an hour or so whilst reviewing changes, the new features are useful and it didn't break.

My tests also look good, some interesting (expected) changes recorded here

http://git.shr-project.org/git/?p=buildhistory.git;a=commitdiff;h=2fe33b15dcec9eb69f9075f138cea2405c061b00

Quoted 8 lines

Thanks!

Cheers,

--
Martin 'JaMa' Jansa jabber: [email protected]

Although I have no time for testing at the moment: thanks for doing this

Andreas

#16 lore raw
In reply to Paul Eggleton
Quoted 14 lines · Paul Eggleton

A bunch of improvements for buildhistory, implementing some feature requests and todo items of my own, as well as tidying up some of the code.

The following changes since commit 404d2d490fc347203e89d274530c17fb5f0aa20f:

gcc-configure-target: Set native-system-header-dir for target gcc (2012-08-01 23:11:09 +0100)

are available in the git repository at:

git://git.openembedded.org/openembedded-core-contrib paule/buildhistory-fixes7
http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=paule/buildhistory-fixes7

Quoted 9 lines

Paul Eggleton (9):
classes/buildhistory: remove obsolete flat package history feature classes/buildhistory: ensure old package info is removed classes/buildhistory: remove unused functions classes/buildhistory: save preinst/postinst/prerm/postrm classes/buildhistory: record PKG/PKGE/PKGV/PKGR
buildhistory_analysis: tidy up duplicate split_version function
buildhistory_analysis: ignore removal of self-dependencies classes/buildhistory: save metadata revisions

Merged to master apart from:

scripts: add buildhistory-tag script

which seems to need more discussion/work.

Cheers,

Richard