Skip to contentSkip to conversation

buildhistory improvements

PATCH New Paul Eggleton 1 participant 9 messages Aug 26, 2015

lore
Share & tools
lore
#0 lore raw

cover buildhistory improvements

A set of improvements for the buildhistory class:
* A fix for not pushing after making "No changes" commits
* Four fixes for the recently added package file listing functionality
* A patch to avoid errors if additional files are added at the recipe level in extension classes
* A couple of patches improving the commit messages

The following changes since commit f07045fcae859c902434062d1725f1348f42d1dd:

oeqa/oetest.py: add better package search for hasPackage() (2015-08-26 08:26:37 +0100)

are available in the git repository at:

git://git.openembedded.org/openembedded-core-contrib paule/buildhistory-fixes10
http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=paule/buildhistory-fixes10

Paul Eggleton (8):
classes/buildhistory: ensure we push when "no changes" commits are made classes/buildhistory: exclude . in file listings classes/buildhistory: indent recently added function consistently classes/buildhistory: tweak buildhistory_list_pkg_files classes/buildhistory: fix permissions in package file listing classes/buildhistory: handle additional files at recipe level classes/buildhistory: include metadata revisions in commit message classes/buildhistory: add build result to commit message

 meta/classes/buildhistory.bbclass | 95 ++++++++++++++++++++++++++++-----------
 1 file changed, 70 insertions(+), 25 deletions(-)
Signature · 1 line
2.1.0
#1 lore raw

PATCH 1/8 classes/buildhistory: ensure we push when "no changes" commits are made

If there aren't any changes, we still make a commit to the buildhistory repo, but this wasn't being pushed if BUILDHISTORY_PUSH_REPO is set. Move the push to the end to make it unconditional.

---
 meta/classes/buildhistory.bbclass | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
meta/classes/buildhistory.bbclass +3 -3
index cad5116..e60cdd7 100644
@@ -621,11 +621,11 @@ END
 				git commit $entry metadata-revs -m "$entry: Build ${BUILDNAME} of ${DISTRO} ${DISTRO_VERSION} for machine ${MACHINE} on $HOSTNAME" -m "cmd: $CMDLINE" --author "${BUILDHISTORY_COMMIT_AUTHOR}" > /dev/null
 			done
 			git gc --auto --quiet
-			if [ "${BUILDHISTORY_PUSH_REPO}" != "" ] ; then
-				git push -q ${BUILDHISTORY_PUSH_REPO}
-			fi
 		else
 			git commit ${BUILDHISTORY_DIR}/ --allow-empty -m "No changes: Build ${BUILDNAME} of ${DISTRO} ${DISTRO_VERSION} for machine ${MACHINE} on $HOSTNAME" -m "cmd: $CMDLINE" --author "${BUILDHISTORY_COMMIT_AUTHOR}" > /dev/null
+		fi
+		if [ "${BUILDHISTORY_PUSH_REPO}" != "" ] ; then
+			git push -q ${BUILDHISTORY_PUSH_REPO}
 		fi) || true
 }
 
Signature · 1 line
2.1.0
#2 lore raw

PATCH 2/8 classes/buildhistory: exclude . in file listings

In reply to Paul Eggleton

We don't care about the permissions on the top-level directory in which the files are contained, just everything under it; this also avoids lists with just this entry in it for empty packages. Affects file listings for both images and packages.

---
 meta/classes/buildhistory.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/classes/buildhistory.bbclass +1 -1
index e60cdd7..77a3c86 100644
@@ -435,7 +435,7 @@ buildhistory_get_sdk_installed_target() {
 buildhistory_list_files() {
 	# List the files in the specified directory, but exclude date/time etc.
 	# This awk script is somewhat messy, but handles where the size is not printed for device files under pseudo
-	( cd $1 && find . -printf "%M %-10u %-10g %10s %p -> %l\n" | sort -k5 | sed 's/ * -> $//' > $2 )
+	( cd $1 && find . ! -path . -printf "%M %-10u %-10g %10s %p -> %l\n" | sort -k5 | sed 's/ * -> $//' > $2 )
 }
 
 buildhistory_list_pkg_files() {
Signature · 1 line
2.1.0
#3 lore raw

PATCH 3/8 classes/buildhistory: indent recently added function consistently

In reply to Paul Eggleton

Shell functions use tabs in this file.

---
 meta/classes/buildhistory.bbclass | 28 ++++++++++++++--------------
 1 file changed, 14 insertions(+), 14 deletions(-)
meta/classes/buildhistory.bbclass +14 -14
index 77a3c86..054a81f 100644
@@ -439,20 +439,20 @@ buildhistory_list_files() {
 }
 
 buildhistory_list_pkg_files() {
-        file_prefix="files-in-"
-
-        # Create individual files-in-package for each recipe's package
-        for pkgdir in $(find ${PKGDEST}/* -maxdepth 0 -type d); do
-                pkgname=$(basename ${pkgdir})
-                outfolder="${BUILDHISTORY_DIR_PACKAGE}/${pkgname}"
-                outfile="${outfolder}/${file_prefix}${pkgname}.txt"
-                # Make sure the output folder, exist so we can create the files-in-$pkgname.txt file
-                if [ ! -d ${outfolder} ] ; then
-                        bbdebug 2 "Folder ${outfolder} does not exist, file ${outfile} not created"
-                        continue
-                fi
-                buildhistory_list_files ${pkgdir} ${outfile}
-        done
+	file_prefix="files-in-"
+
+	# Create individual files-in-package for each recipe's package
+	for pkgdir in $(find ${PKGDEST}/* -maxdepth 0 -type d); do
+		pkgname=$(basename ${pkgdir})
+		outfolder="${BUILDHISTORY_DIR_PACKAGE}/${pkgname}"
+		outfile="${outfolder}/${file_prefix}${pkgname}.txt"
+		# Make sure the output folder, exist so we can create the files-in-$pkgname.txt file
+		if [ ! -d ${outfolder} ] ; then
+			bbdebug 2 "Folder ${outfolder} does not exist, file ${outfile} not created"
+			continue
+		fi
+		buildhistory_list_files ${pkgdir} ${outfile}
+	done
 }
 
 buildhistory_get_imageinfo() {
Signature · 1 line
2.1.0
#4 lore raw

PATCH 4/8 classes/buildhistory: tweak buildhistory_list_pkg_files

In reply to Paul Eggleton

* Avoid using ${...} for shell variables (since they could be expanded as bitbake variables if present)
* Use files-in-package.txt rather than files-in-<packagename>.txt; the file is already in a subdirectory named with the package name and this naming is consistent with that of files-in-image.txt.

---
 meta/classes/buildhistory.bbclass | 14 ++++++--------
 1 file changed, 6 insertions(+), 8 deletions(-)
meta/classes/buildhistory.bbclass +6 -8
index 054a81f..3d459c7 100644
@@ -439,16 +439,14 @@ buildhistory_list_files() {
 }
 
 buildhistory_list_pkg_files() {
-	file_prefix="files-in-"
-
 	# Create individual files-in-package for each recipe's package
 	for pkgdir in $(find ${PKGDEST}/* -maxdepth 0 -type d); do
-		pkgname=$(basename ${pkgdir})
-		outfolder="${BUILDHISTORY_DIR_PACKAGE}/${pkgname}"
-		outfile="${outfolder}/${file_prefix}${pkgname}.txt"
-		# Make sure the output folder, exist so we can create the files-in-$pkgname.txt file
-		if [ ! -d ${outfolder} ] ; then
-			bbdebug 2 "Folder ${outfolder} does not exist, file ${outfile} not created"
+		pkgname=$(basename $pkgdir)
+		outfolder="${BUILDHISTORY_DIR_PACKAGE}/$pkgname"
+		outfile="$outfolder/files-in-package.txt"
+		# Make sure the output folder exists so we can create the file
+		if [ ! -d $outfolder ] ; then
+			bbdebug 2 "Folder $outfolder does not exist, file $outfile not created"
 			continue
 		fi
 		buildhistory_list_files ${pkgdir} ${outfile}
Signature · 1 line
2.1.0
#5 lore raw

PATCH 5/8 classes/buildhistory: fix permissions in package file listing

In reply to Paul Eggleton

If we want the correct file permissions to show up here as they would on the target, we need to run the command under pseudo. Normally we'd set the fakeroot varflag on the function and that would be enough, but it turns out that setting fakeroot on a non-task function that you execute using bb.build.exec_func() isn't working at the moment. Work around this by simply using FAKEROOTENV and FAKEROOTCMD. Unfortunately that means we have to duplicate the command for the two cases but I couldn't find a better means of doing that that actually works.

---
 meta/classes/buildhistory.bbclass | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)
meta/classes/buildhistory.bbclass +6 -2
index 3d459c7..190c38e 100644
@@ -435,7 +435,11 @@ buildhistory_get_sdk_installed_target() {
 buildhistory_list_files() {
 	# List the files in the specified directory, but exclude date/time etc.
 	# This awk script is somewhat messy, but handles where the size is not printed for device files under pseudo
-	( cd $1 && find . ! -path . -printf "%M %-10u %-10g %10s %p -> %l\n" | sort -k5 | sed 's/ * -> $//' > $2 )
+	if [ "$3" = "fakeroot" ] ; then
+		( cd $1 && ${FAKEROOTENV} ${FAKEROOTCMD} find . ! -path . -printf "%M %-10u %-10g %10s %p -> %l\n" | sort -k5 | sed 's/ * -> $//' > $2 )
+	else
+		( cd $1 && find . ! -path . -printf "%M %-10u %-10g %10s %p -> %l\n" | sort -k5 | sed 's/ * -> $//' > $2 )
+	fi
 }
 
 buildhistory_list_pkg_files() {
@@ -449,7 +453,7 @@ buildhistory_list_pkg_files() {
 			bbdebug 2 "Folder $outfolder does not exist, file $outfile not created"
 			continue
 		fi
-		buildhistory_list_files ${pkgdir} ${outfile}
+		buildhistory_list_files $pkgdir $outfile fakeroot
 	done
 }
 
Signature · 1 line
2.1.0
#6 lore raw

PATCH 6/8 classes/buildhistory: handle additional files at recipe level

In reply to Paul Eggleton

Avoid an error when attempting to remove previous data if it's not a subdirectory - we were assuming that anything that wasn't named "latest" or "latest_srcrev" had to be a directory. This makes it possible to have a buildhistory_emit_pkghistory_append which writes additional files at the recipe level.

---
 meta/classes/buildhistory.bbclass | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)
meta/classes/buildhistory.bbclass +7 -4
index 190c38e..57dc1e9 100644
@@ -172,10 +172,13 @@ python buildhistory_emit_pkghistory() {
         for item in os.listdir(pkghistdir):
             if item != "latest" and item != "latest_srcrev":
                 if item not in packagelist:
-                    subdir = os.path.join(pkghistdir, item)
-                    for subfile in os.listdir(subdir):
-                        os.unlink(os.path.join(subdir, subfile))
-                    os.rmdir(subdir)
+                    itempath = os.path.join(pkghistdir, item)
+                    if os.path.isdir(itempath):
+                        for subfile in os.listdir(itempath):
+                            os.unlink(os.path.join(itempath, subfile))
+                        os.rmdir(itempath)
+                    else:
+                        os.unlink(itempath)
 
     rcpinfo = RecipeInfo(pn)
     rcpinfo.pe = pe
Signature · 1 line
2.1.0
#7 lore raw

PATCH 7/8 classes/buildhistory: include metadata revisions in commit message

In reply to Paul Eggleton

We do already commit these into the repository itself, but have them in the commit message as well as a reference. As part of this, refactor out running "git commit" into a separate function so we don't have to duplicate the code in the two places we call it.

Implements [YOCTO #7966].

---
 meta/classes/buildhistory.bbclass | 25 +++++++++++++++++++++++--
 1 file changed, 23 insertions(+), 2 deletions(-)
meta/classes/buildhistory.bbclass +23 -2
index 57dc1e9..135d17a 100644
@@ -585,6 +585,27 @@ def buildhistory_get_cmdline(d):
     return '%s %s' % (bincmd, ' '.join(sys.argv[1:]))
 
 
+buildhistory_single_commit() {
+	if [ "$3" = "" ] ; then
+		commitopts="${BUILDHISTORY_DIR}/ --allow-empty"
+		item="No changes"
+	else
+		commitopts="$3 metadata-revs"
+		item="$3"
+	fi
+	commitmsgfile=`mktemp`
+	cat > $commitmsgfile << END
+$item: Build ${BUILDNAME} of ${DISTRO} ${DISTRO_VERSION} for machine ${MACHINE} on $2
+
+cmd: $1
+
+metadata revisions:
+END
+	cat ${BUILDHISTORY_DIR}/metadata-revs >> $commitmsgfile
+	git commit $commitopts -F $commitmsgfile --author "${BUILDHISTORY_COMMIT_AUTHOR}" > /dev/null
+	rm $commitmsgfile
+}
+
 buildhistory_commit() {
 	if [ ! -d ${BUILDHISTORY_DIR} ] ; then
 		# Code above that creates this dir never executed, so there can't be anything to commit
@@ -623,11 +644,11 @@ END
 			# porcelain output looks like "?? packages/foo/bar"
 			# Ensure we commit metadata-revs with the first commit
 			for entry in `echo "$repostatus" | awk '{print $2}' | awk -F/ '{print $1}' | sort | uniq` ; do
-				git commit $entry metadata-revs -m "$entry: Build ${BUILDNAME} of ${DISTRO} ${DISTRO_VERSION} for machine ${MACHINE} on $HOSTNAME" -m "cmd: $CMDLINE" --author "${BUILDHISTORY_COMMIT_AUTHOR}" > /dev/null
+				buildhistory_single_commit "$CMDLINE" "$HOSTNAME" "$entry"
 			done
 			git gc --auto --quiet
 		else
-			git commit ${BUILDHISTORY_DIR}/ --allow-empty -m "No changes: Build ${BUILDNAME} of ${DISTRO} ${DISTRO_VERSION} for machine ${MACHINE} on $HOSTNAME" -m "cmd: $CMDLINE" --author "${BUILDHISTORY_COMMIT_AUTHOR}" > /dev/null
+			buildhistory_single_commit "$CMDLINE" "$HOSTNAME"
 		fi
 		if [ "${BUILDHISTORY_PUSH_REPO}" != "" ] ; then
 			git push -q ${BUILDHISTORY_PUSH_REPO}
Signature · 1 line
2.1.0
#8 lore raw

PATCH 8/8 classes/buildhistory: add build result to commit message

In reply to Paul Eggleton

We have the command in the commit message, we might as well have the build result as well (succeeded/failed and whether or not it was interrupted by the user). The interrupted part relies upon a change to BitBake to extend the BuildCompleted event to include an attribute for that, but will not fail if the attribute is not present.

---
 meta/classes/buildhistory.bbclass | 21 ++++++++++++++++++++-
 1 file changed, 20 insertions(+), 1 deletion(-)
meta/classes/buildhistory.bbclass +20 -1
index 135d17a..4db0441 100644
@@ -593,12 +593,27 @@ buildhistory_single_commit() {
 		commitopts="$3 metadata-revs"
 		item="$3"
 	fi
+	if [ "${BUILDHISTORY_BUILD_FAILURES}" = "0" ] ; then
+		result="succeeded"
+	else
+		result="failed"
+	fi
+	case ${BUILDHISTORY_BUILD_INTERRUPTED} in
+		1)
+			result="$result (interrupted)"
+			;;
+		2)
+			result="$result (force interrupted)"
+			;;
+	esac
 	commitmsgfile=`mktemp`
 	cat > $commitmsgfile << END
 $item: Build ${BUILDNAME} of ${DISTRO} ${DISTRO_VERSION} for machine ${MACHINE} on $2
 
 cmd: $1
 
+result: $result
+
 metadata revisions:
 END
 	cat ${BUILDHISTORY_DIR}/metadata-revs >> $commitmsgfile
@@ -659,7 +674,11 @@ python buildhistory_eventhandler() {
     if e.data.getVar('BUILDHISTORY_FEATURES', True).strip():
         if e.data.getVar("BUILDHISTORY_COMMIT", True) == "1":
             bb.note("Writing buildhistory")
-            bb.build.exec_func("buildhistory_commit", e.data)
+            localdata = bb.data.createCopy(e.data)
+            localdata.setVar('BUILDHISTORY_BUILD_FAILURES', str(e._failures))
+            interrupted = getattr(e, '_interrupted', 0)
+            localdata.setVar('BUILDHISTORY_BUILD_INTERRUPTED', str(interrupted))
+            bb.build.exec_func("buildhistory_commit", localdata)
 }
 
 addhandler buildhistory_eventhandler
Signature · 1 line
2.1.0