buildhistory improvements
cover buildhistory improvements
A set of improvements for the buildhistory class:
* A fix for not pushing after making "No changes" commits
* Four fixes for the recently added package file listing functionality
* A patch to avoid errors if additional files are added at the recipe level in extension classes
* A couple of patches improving the commit messages
The following changes since commit f07045fcae859c902434062d1725f1348f42d1dd:
oeqa/oetest.py: add better package search for hasPackage() (2015-08-26 08:26:37 +0100)
are available in the git repository at:
git://git.openembedded.org/openembedded-core-contrib paule/buildhistory-fixes10
http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=paule/buildhistory-fixes10
Paul Eggleton (8):
classes/buildhistory: ensure we push when "no changes" commits are made classes/buildhistory: exclude . in file listings classes/buildhistory: indent recently added function consistently classes/buildhistory: tweak buildhistory_list_pkg_files classes/buildhistory: fix permissions in package file listing classes/buildhistory: handle additional files at recipe level classes/buildhistory: include metadata revisions in commit message classes/buildhistory: add build result to commit message
meta/classes/buildhistory.bbclass | 95 ++++++++++++++++++++++++++++----------- 1 file changed, 70 insertions(+), 25 deletions(-)
Signature · 1 line
2.1.0
PATCH 1/8 classes/buildhistory: ensure we push when "no changes" commits are made
If there aren't any changes, we still make a commit to the buildhistory repo, but this wasn't being pushed if BUILDHISTORY_PUSH_REPO is set. Move the push to the end to make it unconditional.
--- meta/classes/buildhistory.bbclass | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
meta/classes/buildhistory.bbclass +3 -3
@@ -621,11 +621,11 @@ END git commit $entry metadata-revs -m "$entry: Build ${BUILDNAME} of ${DISTRO} ${DISTRO_VERSION} for machine ${MACHINE} on $HOSTNAME" -m "cmd: $CMDLINE" --author "${BUILDHISTORY_COMMIT_AUTHOR}" > /dev/null done git gc --auto --quiet - if [ "${BUILDHISTORY_PUSH_REPO}" != "" ] ; then - git push -q ${BUILDHISTORY_PUSH_REPO} - fi else git commit ${BUILDHISTORY_DIR}/ --allow-empty -m "No changes: Build ${BUILDNAME} of ${DISTRO} ${DISTRO_VERSION} for machine ${MACHINE} on $HOSTNAME" -m "cmd: $CMDLINE" --author "${BUILDHISTORY_COMMIT_AUTHOR}" > /dev/null + fi + if [ "${BUILDHISTORY_PUSH_REPO}" != "" ] ; then + git push -q ${BUILDHISTORY_PUSH_REPO} fi) || true }
Signature · 1 line
2.1.0
PATCH 2/8 classes/buildhistory: exclude . in file listings
In reply to Paul EggletonWe don't care about the permissions on the top-level directory in which the files are contained, just everything under it; this also avoids lists with just this entry in it for empty packages. Affects file listings for both images and packages.
--- meta/classes/buildhistory.bbclass | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
meta/classes/buildhistory.bbclass +1 -1
@@ -435,7 +435,7 @@ buildhistory_get_sdk_installed_target() { buildhistory_list_files() { # List the files in the specified directory, but exclude date/time etc. # This awk script is somewhat messy, but handles where the size is not printed for device files under pseudo - ( cd $1 && find . -printf "%M %-10u %-10g %10s %p -> %l\n" | sort -k5 | sed 's/ * -> $//' > $2 ) + ( cd $1 && find . ! -path . -printf "%M %-10u %-10g %10s %p -> %l\n" | sort -k5 | sed 's/ * -> $//' > $2 ) } buildhistory_list_pkg_files() {
Signature · 1 line
2.1.0
PATCH 3/8 classes/buildhistory: indent recently added function consistently
In reply to Paul EggletonShell functions use tabs in this file.
--- meta/classes/buildhistory.bbclass | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-)
meta/classes/buildhistory.bbclass +14 -14
@@ -439,20 +439,20 @@ buildhistory_list_files() { } buildhistory_list_pkg_files() { - file_prefix="files-in-" - - # Create individual files-in-package for each recipe's package - for pkgdir in $(find ${PKGDEST}/* -maxdepth 0 -type d); do - pkgname=$(basename ${pkgdir}) - outfolder="${BUILDHISTORY_DIR_PACKAGE}/${pkgname}" - outfile="${outfolder}/${file_prefix}${pkgname}.txt" - # Make sure the output folder, exist so we can create the files-in-$pkgname.txt file - if [ ! -d ${outfolder} ] ; then - bbdebug 2 "Folder ${outfolder} does not exist, file ${outfile} not created" - continue - fi - buildhistory_list_files ${pkgdir} ${outfile} - done + file_prefix="files-in-" + + # Create individual files-in-package for each recipe's package + for pkgdir in $(find ${PKGDEST}/* -maxdepth 0 -type d); do + pkgname=$(basename ${pkgdir}) + outfolder="${BUILDHISTORY_DIR_PACKAGE}/${pkgname}" + outfile="${outfolder}/${file_prefix}${pkgname}.txt" + # Make sure the output folder, exist so we can create the files-in-$pkgname.txt file + if [ ! -d ${outfolder} ] ; then + bbdebug 2 "Folder ${outfolder} does not exist, file ${outfile} not created" + continue + fi + buildhistory_list_files ${pkgdir} ${outfile} + done } buildhistory_get_imageinfo() {
Signature · 1 line
2.1.0
PATCH 4/8 classes/buildhistory: tweak buildhistory_list_pkg_files
In reply to Paul Eggleton* Avoid using ${...} for shell variables (since they could be expanded as bitbake variables if present)
* Use files-in-package.txt rather than files-in-<packagename>.txt; the file is already in a subdirectory named with the package name and this naming is consistent with that of files-in-image.txt.
--- meta/classes/buildhistory.bbclass | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-)
meta/classes/buildhistory.bbclass +6 -8
@@ -439,16 +439,14 @@ buildhistory_list_files() { } buildhistory_list_pkg_files() { - file_prefix="files-in-" - # Create individual files-in-package for each recipe's package for pkgdir in $(find ${PKGDEST}/* -maxdepth 0 -type d); do - pkgname=$(basename ${pkgdir}) - outfolder="${BUILDHISTORY_DIR_PACKAGE}/${pkgname}" - outfile="${outfolder}/${file_prefix}${pkgname}.txt" - # Make sure the output folder, exist so we can create the files-in-$pkgname.txt file - if [ ! -d ${outfolder} ] ; then - bbdebug 2 "Folder ${outfolder} does not exist, file ${outfile} not created" + pkgname=$(basename $pkgdir) + outfolder="${BUILDHISTORY_DIR_PACKAGE}/$pkgname" + outfile="$outfolder/files-in-package.txt" + # Make sure the output folder exists so we can create the file + if [ ! -d $outfolder ] ; then + bbdebug 2 "Folder $outfolder does not exist, file $outfile not created" continue fi buildhistory_list_files ${pkgdir} ${outfile}
Signature · 1 line
2.1.0
PATCH 5/8 classes/buildhistory: fix permissions in package file listing
In reply to Paul EggletonIf we want the correct file permissions to show up here as they would on the target, we need to run the command under pseudo. Normally we'd set the fakeroot varflag on the function and that would be enough, but it turns out that setting fakeroot on a non-task function that you execute using bb.build.exec_func() isn't working at the moment. Work around this by simply using FAKEROOTENV and FAKEROOTCMD. Unfortunately that means we have to duplicate the command for the two cases but I couldn't find a better means of doing that that actually works.
--- meta/classes/buildhistory.bbclass | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)
meta/classes/buildhistory.bbclass +6 -2
@@ -435,7 +435,11 @@ buildhistory_get_sdk_installed_target() { buildhistory_list_files() { # List the files in the specified directory, but exclude date/time etc. # This awk script is somewhat messy, but handles where the size is not printed for device files under pseudo - ( cd $1 && find . ! -path . -printf "%M %-10u %-10g %10s %p -> %l\n" | sort -k5 | sed 's/ * -> $//' > $2 ) + if [ "$3" = "fakeroot" ] ; then + ( cd $1 && ${FAKEROOTENV} ${FAKEROOTCMD} find . ! -path . -printf "%M %-10u %-10g %10s %p -> %l\n" | sort -k5 | sed 's/ * -> $//' > $2 ) + else + ( cd $1 && find . ! -path . -printf "%M %-10u %-10g %10s %p -> %l\n" | sort -k5 | sed 's/ * -> $//' > $2 ) + fi } buildhistory_list_pkg_files() {
@@ -449,7 +453,7 @@ buildhistory_list_pkg_files() { bbdebug 2 "Folder $outfolder does not exist, file $outfile not created" continue fi - buildhistory_list_files ${pkgdir} ${outfile} + buildhistory_list_files $pkgdir $outfile fakeroot done }
Signature · 1 line
2.1.0
PATCH 6/8 classes/buildhistory: handle additional files at recipe level
In reply to Paul EggletonAvoid an error when attempting to remove previous data if it's not a subdirectory - we were assuming that anything that wasn't named "latest" or "latest_srcrev" had to be a directory. This makes it possible to have a buildhistory_emit_pkghistory_append which writes additional files at the recipe level.
--- meta/classes/buildhistory.bbclass | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-)
meta/classes/buildhistory.bbclass +7 -4
@@ -172,10 +172,13 @@ python buildhistory_emit_pkghistory() { for item in os.listdir(pkghistdir): if item != "latest" and item != "latest_srcrev": if item not in packagelist: - subdir = os.path.join(pkghistdir, item) - for subfile in os.listdir(subdir): - os.unlink(os.path.join(subdir, subfile)) - os.rmdir(subdir) + itempath = os.path.join(pkghistdir, item) + if os.path.isdir(itempath): + for subfile in os.listdir(itempath): + os.unlink(os.path.join(itempath, subfile)) + os.rmdir(itempath) + else: + os.unlink(itempath) rcpinfo = RecipeInfo(pn) rcpinfo.pe = pe
Signature · 1 line
2.1.0
PATCH 7/8 classes/buildhistory: include metadata revisions in commit message
In reply to Paul EggletonWe do already commit these into the repository itself, but have them in the commit message as well as a reference. As part of this, refactor out running "git commit" into a separate function so we don't have to duplicate the code in the two places we call it.
Implements [YOCTO #7966].
--- meta/classes/buildhistory.bbclass | 25 +++++++++++++++++++++++-- 1 file changed, 23 insertions(+), 2 deletions(-)
meta/classes/buildhistory.bbclass +23 -2
@@ -585,6 +585,27 @@ def buildhistory_get_cmdline(d): return '%s %s' % (bincmd, ' '.join(sys.argv[1:])) +buildhistory_single_commit() { + if [ "$3" = "" ] ; then + commitopts="${BUILDHISTORY_DIR}/ --allow-empty" + item="No changes" + else + commitopts="$3 metadata-revs" + item="$3" + fi + commitmsgfile=`mktemp` + cat > $commitmsgfile << END +$item: Build ${BUILDNAME} of ${DISTRO} ${DISTRO_VERSION} for machine ${MACHINE} on $2 + +cmd: $1 + +metadata revisions: +END + cat ${BUILDHISTORY_DIR}/metadata-revs >> $commitmsgfile + git commit $commitopts -F $commitmsgfile --author "${BUILDHISTORY_COMMIT_AUTHOR}" > /dev/null + rm $commitmsgfile +} + buildhistory_commit() { if [ ! -d ${BUILDHISTORY_DIR} ] ; then # Code above that creates this dir never executed, so there can't be anything to commit
@@ -623,11 +644,11 @@ END # porcelain output looks like "?? packages/foo/bar" # Ensure we commit metadata-revs with the first commit for entry in `echo "$repostatus" | awk '{print $2}' | awk -F/ '{print $1}' | sort | uniq` ; do - git commit $entry metadata-revs -m "$entry: Build ${BUILDNAME} of ${DISTRO} ${DISTRO_VERSION} for machine ${MACHINE} on $HOSTNAME" -m "cmd: $CMDLINE" --author "${BUILDHISTORY_COMMIT_AUTHOR}" > /dev/null + buildhistory_single_commit "$CMDLINE" "$HOSTNAME" "$entry" done git gc --auto --quiet else - git commit ${BUILDHISTORY_DIR}/ --allow-empty -m "No changes: Build ${BUILDNAME} of ${DISTRO} ${DISTRO_VERSION} for machine ${MACHINE} on $HOSTNAME" -m "cmd: $CMDLINE" --author "${BUILDHISTORY_COMMIT_AUTHOR}" > /dev/null + buildhistory_single_commit "$CMDLINE" "$HOSTNAME" fi if [ "${BUILDHISTORY_PUSH_REPO}" != "" ] ; then git push -q ${BUILDHISTORY_PUSH_REPO}
Signature · 1 line
2.1.0
PATCH 8/8 classes/buildhistory: add build result to commit message
In reply to Paul EggletonWe have the command in the commit message, we might as well have the build result as well (succeeded/failed and whether or not it was interrupted by the user). The interrupted part relies upon a change to BitBake to extend the BuildCompleted event to include an attribute for that, but will not fail if the attribute is not present.
--- meta/classes/buildhistory.bbclass | 21 ++++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-)
meta/classes/buildhistory.bbclass +20 -1
@@ -593,12 +593,27 @@ buildhistory_single_commit() { commitopts="$3 metadata-revs" item="$3" fi + if [ "${BUILDHISTORY_BUILD_FAILURES}" = "0" ] ; then + result="succeeded" + else + result="failed" + fi + case ${BUILDHISTORY_BUILD_INTERRUPTED} in + 1) + result="$result (interrupted)" + ;; + 2) + result="$result (force interrupted)" + ;; + esac commitmsgfile=`mktemp` cat > $commitmsgfile << END $item: Build ${BUILDNAME} of ${DISTRO} ${DISTRO_VERSION} for machine ${MACHINE} on $2 cmd: $1 +result: $result + metadata revisions: END cat ${BUILDHISTORY_DIR}/metadata-revs >> $commitmsgfile
@@ -659,7 +674,11 @@ python buildhistory_eventhandler() { if e.data.getVar('BUILDHISTORY_FEATURES', True).strip(): if e.data.getVar("BUILDHISTORY_COMMIT", True) == "1": bb.note("Writing buildhistory") - bb.build.exec_func("buildhistory_commit", e.data) + localdata = bb.data.createCopy(e.data) + localdata.setVar('BUILDHISTORY_BUILD_FAILURES', str(e._failures)) + interrupted = getattr(e, '_interrupted', 0) + localdata.setVar('BUILDHISTORY_BUILD_INTERRUPTED', str(interrupted)) + bb.build.exec_func("buildhistory_commit", localdata) } addhandler buildhistory_eventhandler
Signature · 1 line
2.1.0