buildhistory fixes

PATCH Applied Paul Eggleton 3 participants 12 messages Feb 14, 2012 – Feb 21, 2012

Share & tools

Paul Eggleton

author

2012-02-14 13:48 UTC #0 lore raw

cover buildhistory fixes

Analysis of the recent output from buildhistory on the Yocto Project and Angstrom autobuilders showed a few anomalies and annoyances; this set of patches addresses all of the ones I noticed.

The following changes since commit e5ad03093dfc4364d1407183f458df79f347c7a1:

guile: fix cross configure failure (2012-02-10 13:38:16 +0000)

are available in the git repository at:
git://git.openembedded.org/openembedded-core-contrib paule/buildhistory-fixes4
http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=paule/buildhistory-fixes4

Paul Eggleton (7):
classes/buildhistory: squash spaces out of image variables classes/buildhistory: sort and de-dupe dependency graphs classes/buildhistory: sort FILELIST in package info classes/buildhistory: use hostname instead of reading /etc/hostname classes/buildhistory: fix splitting on + in package list fields
buildhistory_analysis: avoid noise due to reordering
buildhistory_analysis: correctly split RDEPENDS/RRECOMMENDS

 meta/classes/buildhistory.bbclass    |   25 +++++++++++++++++--------
 meta/lib/oe/buildhistory_analysis.py |   31 ++++++++++++++++++++++++++-----
 2 files changed, 43 insertions(+), 13 deletions(-)

Signature · 1 line

1.7.5.4

Paul Eggleton

author

2012-02-14 13:48 UTC #1 lore raw

PATCH 1/7 classes/buildhistory: squash spaces out of image variables

Values of image variables that are lists (e.g. IMAGE_INSTALL) are easier to read if there are no extraneous spaces in them, so ensure that there is only one space between each item.

Signed-off-by Paul Eggleton <[email protected]>

---
 meta/classes/buildhistory.bbclass |   16 ++++++++++++----
 1 files changed, 12 insertions(+), 4 deletions(-)

meta/classes/buildhistory.bbclass +12 -4

index 1b6b249..d1a9670 100644
@@ -123,9 +123,6 @@ python buildhistory_emit_pkghistory() {
 		except EnvironmentError:
 			return None
 
-	def squashspaces(string):
-		return re.sub("\s+", " ", string)
-
 	def sortpkglist(string):
 		pkgiter = re.finditer(r'[a-zA-Z0-9.-]+( \([><=]+ [^ )]+\))?', string, 0)
 		pkglist = [p.group(0) for p in pkgiter]

@@ -349,12 +346,23 @@ def buildhistory_get_layers(d):
 	return layertext
 
 
+def squashspaces(string):
+	import re
+	return re.sub("\s+", " ", string).strip()
+
+
 def buildhistory_get_imagevars(d):
 	imagevars = "DISTRO DISTRO_VERSION USER_CLASSES IMAGE_CLASSES IMAGE_FEATURES IMAGE_LINGUAS IMAGE_INSTALL BAD_RECOMMENDATIONS ROOTFS_POSTPROCESS_COMMAND IMAGE_POSTPROCESS_COMMAND"
+	listvars = "USER_CLASSES IMAGE_CLASSES IMAGE_FEATURES IMAGE_LINGUAS IMAGE_INSTALL BAD_RECOMMENDATIONS"
 
+	imagevars = imagevars.split()
+	listvars = listvars.split()
 	ret = ""
-	for var in imagevars.split():
+	for var in imagevars:
 		value = d.getVar(var, True) or ""
+		if var in listvars:
+			# Squash out spaces
+			value = squashspaces(value)
 		ret += "%s = %s\n" % (var, value)
 	return ret.rstrip('\n')

Signature · 1 line

1.7.5.4

Paul Eggleton

author

2012-02-14 13:48 UTC #2 lore raw

PATCH 2/7 classes/buildhistory: sort and de-dupe dependency graphs

In reply to Paul Eggleton

Sort dependencies of each package which sometimes change order and cause noise in the buildhistory repo, and at the same time remove duplicates (which seem to be common especially for the RPM package query output).

Signed-off-by Paul Eggleton <[email protected]>

---
 meta/classes/buildhistory.bbclass |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

meta/classes/buildhistory.bbclass +2 -2

index d1a9670..0ee6a33 100644
@@ -286,12 +286,12 @@ buildhistory_get_image_installed() {
 			echo $pkgsize $pkg >> ${BUILDHISTORY_DIR_IMAGE}/installed-package-sizes.tmp
 		fi
 
-		deps=`list_package_depends $pkg`
+		deps=`list_package_depends $pkg | sort | uniq`
 		for dep in $deps ; do
 			echo "$pkg OPP $dep;" | sed -e 's:-:_:g' -e 's:\.:_:g' -e 's:+::g' | sed 's:OPP:->:g' >> ${BUILDHISTORY_DIR_IMAGE}/depends.dot
 		done
 
-		recs=`list_package_recommends $pkg`
+		recs=`list_package_recommends $pkg | sort | uniq`
 		for rec in $recs ; do
 			echo "$pkg OPP $rec [style=dotted];" | sed -e 's:-:_:g' -e 's:\.:_:g' -e 's:+::g' | sed 's:OPP:->:g' >> ${BUILDHISTORY_DIR_IMAGE}/depends.dot
 		done

Signature · 1 line

1.7.5.4

Paul Eggleton

author

2012-02-14 13:48 UTC #3 lore raw

PATCH 3/7 classes/buildhistory: sort FILELIST in package info

In reply to Paul Eggleton

The FILELIST order can vary depending on the order the files were written which may change between builds with no ill effect, so sort the list prior to writing it.

Signed-off-by Paul Eggleton <[email protected]>

---
 meta/classes/buildhistory.bbclass |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

meta/classes/buildhistory.bbclass +1 -0

index 0ee6a33..3fbe3a8 100644
@@ -186,6 +186,7 @@ python buildhistory_emit_pkghistory() {
 				fstat = os.lstat(os.path.join(root, f))
 				pkginfo.size += fstat.st_size
 				filelist.append(os.sep + os.path.join(relpth, f))
+		filelist.sort()
 		pkginfo.filelist = " ".join(filelist)
 
 		write_pkghistory(pkginfo, d)

Signature · 1 line

1.7.5.4

Paul Eggleton

author

2012-02-14 13:48 UTC #4 lore raw

PATCH 4/7 classes/buildhistory: use hostname instead of reading /etc/hostname

In reply to Paul Eggleton

For the purposes of querying the hostname to include it in the commit message, it seems "cat /etc/hostname" does not work on the Yocto Project autobuilder machines, and it's likely that the hostname command will be more generally reliable, so use that instead.

Signed-off-by Paul Eggleton <[email protected]>

---
 meta/classes/buildhistory.bbclass |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

meta/classes/buildhistory.bbclass +1 -1

index 3fbe3a8..dfc9b73 100644
@@ -383,7 +383,7 @@ buildhistory_commit() {
 		repostatus=`git status --porcelain`
 		if [ "$repostatus" != "" ] ; then
 			git add ${BUILDHISTORY_DIR}/*
-			HOSTNAME=`cat /etc/hostname 2>/dev/null || echo unknown`
+			HOSTNAME=`hostname 2>/dev/null || echo unknown`
 			git commit ${BUILDHISTORY_DIR}/ -m "Build ${BUILDNAME} of ${DISTRO} ${DISTRO_VERSION} for machine ${MACHINE} on $HOSTNAME" --author "${BUILDHISTORY_COMMIT_AUTHOR}" > /dev/null
 			if [ "${BUILDHISTORY_PUSH_REPO}" != "" ] ; then
 				git push -q ${BUILDHISTORY_PUSH_REPO}

Signature · 1 line

1.7.5.4

Paul Eggleton

author

2012-02-14 13:48 UTC #6 lore raw

PATCH 5/7 classes/buildhistory: fix splitting on + in package list fields

In reply to Paul Eggleton

Ensure we do not erroneously split on + in RDEPENDS/RRECOMMENDS e.g. libstdc++-dev was being split into libstdc and -dev.

Signed-off-by Paul Eggleton <[email protected]>

---
 meta/classes/buildhistory.bbclass |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

meta/classes/buildhistory.bbclass +1 -1

index dfc9b73..6c2d4e9 100644
@@ -124,7 +124,7 @@ python buildhistory_emit_pkghistory() {
 			return None
 
 	def sortpkglist(string):
-		pkgiter = re.finditer(r'[a-zA-Z0-9.-]+( \([><=]+ [^ )]+\))?', string, 0)
+		pkgiter = re.finditer(r'[a-zA-Z0-9.+-]+( \([><=]+ [^ )]+\))?', string, 0)
 		pkglist = [p.group(0) for p in pkgiter]
 		pkglist.sort()
 		return ' '.join(pkglist)

Signature · 1 line

1.7.5.4

Paul Eggleton

author

2012-02-14 13:48 UTC #7 lore raw

PATCH 6/7 buildhistory_analysis: avoid noise due to reordering

In reply to Paul Eggleton

Avoid noise in the output due to reordering of list variables (except for PACKAGES where we just report that the order changed). Recent changes to the buildhistory class itself will avoid this reordering from occurring but this allows us to examine the results before and after those changes.

Signed-off-by Paul Eggleton <[email protected]>

---
 meta/lib/oe/buildhistory_analysis.py |   17 ++++++++++++++---
 1 files changed, 14 insertions(+), 3 deletions(-)

meta/lib/oe/buildhistory_analysis.py +14 -3

index 103cfb4..bef6cd4 100644
@@ -15,7 +15,8 @@ import git
 
 
 # How to display fields
-list_fields = ['DEPENDS', 'RDEPENDS', 'RRECOMMENDS', 'PACKAGES', 'FILES', 'FILELIST', 'USER_CLASSES', 'IMAGE_CLASSES', 'IMAGE_FEATURES', 'IMAGE_LINGUAS', 'IMAGE_INSTALL', 'BAD_RECOMMENDATIONS']
+list_fields = ['DEPENDS', 'RDEPENDS', 'RRECOMMENDS', 'FILES', 'FILELIST', 'USER_CLASSES', 'IMAGE_CLASSES', 'IMAGE_FEATURES', 'IMAGE_LINGUAS', 'IMAGE_INSTALL', 'BAD_RECOMMENDATIONS']
+list_order_fields = ['PACKAGES']
 numeric_fields = ['PKGSIZE', 'IMAGESIZE']
 # Fields to monitor
 monitor_fields = ['RDEPENDS', 'RRECOMMENDS', 'PACKAGES', 'FILELIST', 'PKGSIZE', 'IMAGESIZE']

@@ -52,12 +53,15 @@ class ChangeRecord:
         else:
             prefix = ''
 
-        if self.fieldname in list_fields:
+        if self.fieldname in list_fields or self.fieldname in list_order_fields:
             aitems = self.oldvalue.split()
             bitems = self.newvalue.split()
             removed = list(set(aitems) - set(bitems))
             added = list(set(bitems) - set(aitems))
-            out = '%s:%s%s' % (self.fieldname, ' removed "%s"' % ' '.join(removed) if removed else '', ' added "%s"' % ' '.join(added) if added else '')
+            if removed or added:
+                out = '%s:%s%s' % (self.fieldname, ' removed "%s"' % ' '.join(removed) if removed else '', ' added "%s"' % ' '.join(added) if added else '')
+            else:
+                out = '%s changed order' % self.fieldname
         elif self.fieldname in numeric_fields:
             aval = int(self.oldvalue or 0)
             bval = int(self.newvalue or 0)

@@ -237,6 +241,13 @@ def compare_dict_blobs(path, ablob, bblob, report_all):
                     percentchg = 100
                 if percentchg < monitor_numeric_threshold:
                     continue
+            elif (not report_all) and key in list_fields:
+                alist = astr.split()
+                alist.sort()
+                blist = bstr.split()
+                blist.sort()
+                if ' '.join(alist) == ' '.join(blist):
+                    continue
             chg = ChangeRecord(path, key, astr, bstr, key in monitor_fields)
             changes.append(chg)
     return changes

Signature · 1 line

1.7.5.4

Paul Eggleton

author

2012-02-14 13:48 UTC #8 lore raw

PATCH 7/7 buildhistory_analysis: correctly split RDEPENDS/RRECOMMENDS

In reply to Paul Eggleton

Split RDEPENDS and RRECOMMENDS correctly (which may contain version number specifications after each item).

Signed-off-by Paul Eggleton <[email protected]>

---
 meta/lib/oe/buildhistory_analysis.py |   14 ++++++++++++--
 1 files changed, 12 insertions(+), 2 deletions(-)

meta/lib/oe/buildhistory_analysis.py +12 -2

index bef6cd4..4f3e635 100644
@@ -12,6 +12,7 @@ import sys
 import os.path
 import difflib
 import git
+import re
 
 
 # How to display fields

@@ -53,9 +54,18 @@ class ChangeRecord:
         else:
             prefix = ''
 
+        def pkglist_split(pkgs):
+            pkgit = re.finditer(r'[a-zA-Z0-9.+-]+( \([><=]+ [^ )]+\))?', pkgs, 0)
+            pkglist = [p.group(0) for p in pkgit]
+            return pkglist
+
         if self.fieldname in list_fields or self.fieldname in list_order_fields:
-            aitems = self.oldvalue.split()
-            bitems = self.newvalue.split()
+            if self.fieldname in ['RDEPENDS', 'RRECOMMENDS']:
+                aitems = pkglist_split(self.oldvalue)
+                bitems = pkglist_split(self.newvalue)
+            else:
+                aitems = self.oldvalue.split()
+                bitems = self.newvalue.split()
             removed = list(set(aitems) - set(bitems))
             added = list(set(bitems) - set(aitems))
             if removed or added:

Signature · 1 line

1.7.5.4

Koen Kooi

2012-02-14 14:43 UTC #5 lore raw

Re: [PATCH 4/7] classes/buildhistory: use hostname instead of reading /etc/hostname

In reply to Paul Eggleton

Op 14 feb. 2012, om 05:40 heeft Paul Eggleton het volgende geschreven:

Quoted 3 lines

For the purposes of querying the hostname to include it in the commit message, it seems "cat /etc/hostname" does not work on the Yocto Project autobuilder machines,

/etc/hostname is a debianism, but systemd is trying to make it a standard across all distros: http://0pointer.de/blog/projects/the-new-configuration-files.html

regards,

Koen

Quoted 29 lines

and it's likely that the hostname command will be
more generally reliable, so use that instead.

Signed-off-by Paul Eggleton <[email protected]>

---
meta/classes/buildhistory.bbclass | 2 +-

1 files changed, 1 insertions(+), 1 deletions(-)

meta/classes/buildhistory.bbclass +1 -1

index 3fbe3a8..dfc9b73 100644
@@ -383,7 +383,7 @@ buildhistory_commit() {
 		repostatus=`git status --porcelain`
 		if [ "$repostatus" != "" ] ; then
 			git add ${BUILDHISTORY_DIR}/*
-			HOSTNAME=`cat /etc/hostname 2>/dev/null || echo unknown`
+			HOSTNAME=`hostname 2>/dev/null || echo unknown`
 			git commit ${BUILDHISTORY_DIR}/ -m "Build ${BUILDNAME} of ${DISTRO} ${DISTRO_VERSION} for machine ${MACHINE} on $HOSTNAME" --author "${BUILDHISTORY_COMMIT_AUTHOR}" > /dev/null
 			if [ "${BUILDHISTORY_PUSH_REPO}" != "" ] ; then
 				git push -q ${BUILDHISTORY_PUSH_REPO}

Signature · 7 lines

1.7.5.4


_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

Paul Eggleton

author

2012-02-17 11:44 UTC #9 lore raw

In reply to Paul Eggleton

Quoted 3 lines · On Tuesday 14 February 2012 13:40:10 Paul Eggleton wrote:

Analysis of the recent output from buildhistory on the Yocto Project and Angstrom autobuilders showed a few anomalies and annoyances; this set of patches addresses all of the ones I noticed.

I've got one more patch to add to this, please ignore this set as v2 is on its way.

Cheers,
Paul

Signature · 2 lines

Paul Eggleton
Intel Open Source Technology Centre

Paul Eggleton

author

2012-02-21 15:05 UTC #10 lore raw

Quoted 2 lines · On Friday 17 February 2012 11:36:37 Paul Eggleton wrote:

I've got one more patch to add to this, please ignore this set as v2 is on its way.

Actually scratch that, as mentioned in another thread I got stuck on a pseudo issue; please merge this series.

Cheers,
Paul

Signature · 2 lines

Paul Eggleton
Intel Open Source Technology Centre

Saul Wold

Applied

2012-02-21 18:54 UTC #11 lore raw

In reply to Paul Eggleton

Quoted 21 lines · Paul Eggleton

Analysis of the recent output from buildhistory on the Yocto Project and Angstrom autobuilders showed a few anomalies and annoyances; this set of patches addresses all of the ones I noticed.
The following changes since commit e5ad03093dfc4364d1407183f458df79f347c7a1:
guile: fix cross configure failure (2012-02-10 13:38:16 +0000)
are available in the git repository at:
git://git.openembedded.org/openembedded-core-contrib paule/buildhistory-fixes4
http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=paule/buildhistory-fixes4
Paul Eggleton (7):
classes/buildhistory: squash spaces out of image variables classes/buildhistory: sort and de-dupe dependency graphs classes/buildhistory: sort FILELIST in package info classes/buildhistory: use hostname instead of reading /etc/hostname classes/buildhistory: fix splitting on + in package list fields
buildhistory_analysis: avoid noise due to reordering
buildhistory_analysis: correctly split RDEPENDS/RRECOMMENDS

Quoted 4 lines

  meta/classes/buildhistory.bbclass    |   25 +++++++++++++++++--------
  meta/lib/oe/buildhistory_analysis.py |   31 ++++++++++++++++++++++++++-----
  2 files changed, 43 insertions(+), 13 deletions(-)

Merged into OE-core

Thanks
Sau!