extend buildhistory

The idea behind the implementation of Yocto #8138 was that an additional class can write additional files in the recipe directories, for example by hooking into the functions of buildhistory.bbclass or by implementing its own SSTATEPOSTINSTFUNCS function.

However, when these additional files get created before buildhistory_emit_pkghistory(), they get removed again by that function because it contains code which removes everything it does not know about. The reason for that is that these unknown items are probably obsolete.

This logic is the reason why the additional "kconfig" file from buildhistory-extra.bbclass never showed up in the final build history.

To fix this, the hard-coded list of known files in buildhistory_emit_pkghistory() must be turned into a variable which derived classes can extend.

Signed-off-by Patrick Ohly <[email protected]>

---
 meta/classes/buildhistory.bbclass | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)

(quoted diff) +0 -0

b/meta/classes/buildhistory.bbclass index 729dcd4..6023e5d 100644

meta/classes/buildhistory.bbclass +0 -0

@@ -23,6 +23,16 @@ sstate_install[vardepsexclude] +=

"buildhistory_emit_pkghistory" # then the value added to SSTATEPOSTINSTFUNCS:
SSTATEPOSTINSTFUNCS[vardepvalueexclude] .= "| buildhistory_emit_pkghistory"

meta/classes/buildhistory.bbclass +7 -0

+# All items excepts those listed here will be removed from a recipe's
+# build history directory by buildhistory_emit_pkghistory(). This is
+# necessary because some of these items (package directories, files that
+# we no longer emit) might be obsolete.
+#
+# When extending build history, derive your class from buildhistory.bbclass
+# and extend this list here with the additional files created by the

derived

meta/classes/buildhistory.bbclass +3 -0

+# class.
+BUILDHISTORY_PRESERVE = "latest latest_srcrev"
+
 #
 # Write out metadata about this package for comparison when writing future

packages #

meta/classes/buildhistory.bbclass +1 -0

@@ -165,12 +175,13 @@ python buildhistory_emit_pkghistory() {
             raise
 
     packagelist = packages.split()
+    preserve = d.getVar('BUILDHISTORY_PRESERVE', True).split()

buildhistory_emit_pkghistory() only runs for recipes with packages and when gathering information about packages is enabled. Information about source files is needed in all cases because one cannot assume that native compilation is based on exactly the same sources as compilation for the target and/or there might not be the corresponding information about the target compilation. This was not mentioned in the feature description of
https://bugzilla.yoctoproject.org/show_bug.cgi?id=8138 and thus not covered by the original implementation.

Therefore we need to ensure that the extra code runs independently from buildhistory_emit_pkghistory(). This is done in a similar way, except that the code runs earlier when populating the sysroot, which is done for native and target recipes.

Because the code now runs before buildhistory_emit_pkghistory(), we may have to create the recipe directory. This was already true for buildhistory_extra_emit_kernelconfig() which failed when starting with a build history that had no directory for the kernel. Even if it was written, that "kconfig" file then later got removed again by the directory cleaning code in buildhistory_emit_pkghistory(). Now the new BUILDHISTORY_PRESERVE mechanism is used to prevent that.

Handling the "latest" file is tricky: it mixes values that only exist for recipes with packages (PACKAGES, obviously) and values that always exist (like PV and PR), but the base class only writes it for recipes with packages. Instead of doing intrusive changes to the base class to fix that, the file now simply gets written by the extra code first and later gets overwritten with a more complete version by the base class. Another alternative would be to store PV and PR twice, but some redundant work during build time seems better than redundancy in the result.

Allowing a derived class to extend the file would cause even more complications (like having to write to the file three times, or introducing some kind of hook mechanism). While doable, it seems easier to just store extra variables in a separate "variables" file.

Signed-off-by Patrick Ohly <[email protected]>

---
 meta/classes/buildhistory-extra.bbclass | 49

+++++++++++++++++++++++++++++---- 1 file changed, 44 insertions(+), 5 deletions(-)

(quoted diff) +0 -0

b/meta/classes/buildhistory-extra.bbclass index 2dd5c5f..05ba4cf 100644

meta/classes/buildhistory-extra.bbclass +5 -0

@@ -6,15 +6,51 @@
 
 inherit buildhistory
 
+# Variables to record in the "variables" file of each recipe.
+#
+# WARNING: changing this does not cause a rewrite of that file unless
+# something causes do_populate_sysroot[_setscene] to be run again,
+# like removing the tempdir and rebuilding (with or without sstate).
 BUILDHISTORY_EXTRA_PKGVARS ?= "PACKAGECONFIG EXTRA_OEMAKE EXTRA_OECONF

EXTRA_OECMAKE EXTRA_OESCONS EXTRA_QMAKEVARS_PRE EXTRA_QMAKEVARS_POST OE_FEATURES SUMMARY DESCRIPTION HOMEPAGE LICENSE"

meta/classes/buildhistory-extra.bbclass +4 -1

-python buildhistory_emit_pkghistory_append() {
+BUILDHISTORY_PRESERVE += "sources metadata variables kconfig"
+
+SSTATEPOSTINSTFUNCS_append = " buildhistory_extra_emit_pkghistory"
+# We want to avoid influence the signatures of sstate tasks - first the

function itself: +sstate_install[vardepsexclude] += "buildhistory_extra_emit_pkghistory" +# then the value added to SSTATEPOSTINSTFUNCS:
+SSTATEPOSTINSTFUNCS[vardepvalueexclude] .= "| buildhistory_extra_emit_pkghistory" +

meta/classes/buildhistory-extra.bbclass +2 -0

+python buildhistory_extra_emit_pkghistory() {
+    bb.note('buildhistory_extra_emit_pkghistory %s' %

d.getVar('BB_CURRENTTASK', True)) + if not d.getVar('BB_CURRENTTASK',
True) in ['populate_sysroot', 'populate_sysroot_setscene']: + return
0

meta/classes/buildhistory-extra.bbclass +25 -4

+
     import codecs
 
     relpath = os.path.dirname(d.getVar('TOPDIR', True))
+    pkghistdir = d.getVar('BUILDHISTORY_DIR_PACKAGE', True)
+    if not os.path.exists(pkghistdir):
+        bb.utils.mkdirhier(pkghistdir)
+
+    # Record PV in the "latest" file. This duplicates work in
+    # buildhistory_emit_pkghistory(), but we do not know whether
+    # that will run (it gets skipped for recipes which never
+    # reach the packaging state or when "package" is not in
+    # BUILDHISTORY_FEATURES), so we write the file here
+    # and let buildhistory_emit_pkghistory() overwrite it again
+    # with more information later.
+    infofile = os.path.join(pkghistdir, "latest")
+    pe = d.getVar('PE', True) or "0"
+    pv = d.getVar('PV', True)
+    pr = d.getVar('PR', True)
+    with codecs.open(infofile, "w", encoding='utf8') as f:
+        if pe != "0":
+            f.write(u"PE = %s\n" % pe)
+        f.write(u"PV = %s\n" % pv)
+        f.write(u"PR = %s\n" % pr)
 
     # List sources
-    pkghistdir = d.getVar('BUILDHISTORY_DIR_PACKAGE', True)
     srcsfile = os.path.join(pkghistdir, "sources")
     with codecs.open(srcsfile, "w", encoding='utf8') as f:
         urls = (d.getVar('SRC_URI', True) or '').split()

@@ -39,15 +75,16 @@ python buildhistory_emit_pkghistory_append() {
                     path = os.path.relpath(path, relpath)
                 f.write('%s %s\n' % (path, sha256sum))
 
-    # List some extra var values
-    latestfile = os.path.join(pkghistdir, "latest")
     vars = (d.getVar('BUILDHISTORY_EXTRA_PKGVARS', True) or '').split()
+    varsfile = os.path.join(pkghistdir, "variables")
     if vars:
-        with codecs.open(latestfile, "a", encoding='utf8') as f:
+        with codecs.open(varsfile, "w", encoding='utf8') as f:
             for var in vars:
                 value = oe.utils.squashspaces(d.getVar(var, True) or '')
                 if value:
                     f.write('%s = %s\n' % (var, value))
+    elif os.path.exists(varsfile):
+        os.unlink(varsfile)
 }
 
 python() {

The initial implementation of YOCTO #8138 ("buildhistory-extra.bbclass: store more information about a build") had some issues ("kconfig" file not preserved) but more importantly, additional changes are needed to support also gathering information about native recipes.

I am proposing to include only the "buildhistory.bbclass: support extending the content of the build history" patch into OE-core.

The original buildhistory-extra.bbclass and the modifications made to it are included here merely to provide the context for that change and to publish the modifications.

Quoted 6 lines · On Wednesday 25 November 2015 10:30:28 Patrick Ohly wrote:

The initial implementation of YOCTO #8138 ("buildhistory-extra.bbclass: store more information about a build") had some issues ("kconfig" file not preserved) but more importantly, additional changes are needed to support also gathering information about native recipes.

Quoted 6 lines

I am proposing to include only the "buildhistory.bbclass: support extending the content of the build history" patch into OE-core.

The original buildhistory-extra.bbclass and the modifications made to it are included here merely to provide the context for that change and to publish the modifications.

For others' reference, buildhistory-extra.bbclass was previously attached to
https://bugzilla.yoctoproject.org/show_bug.cgi?id=8138 - I didn't and wouldn't propose introducing it into OE-Core in its current form because it's kind of turning buildhistory into something it wasn't designed for, though I don't disagree that it's useful. Happy to hear opinions from others on this though.

One aspect that will probably change in the future is how it picks up the kernel configuration - for
https://bugzilla.yoctoproject.org/show_bug.cgi?id=5574 (which I hope to get to for 2.1, but no promises) we will need to write the kernel configuration to the sysroot so that we have somewhere we can look for it; if that's the case there'll be no need to copy it specially in buildhistory-extra.bbclass.

Cheers,
Paul

Signature · 7 lines

Paul Eggleton
Intel Open Source Technology Centre
-- 
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

One aspect that will probably change in the future is how it picks up the kernel configuration - for
https://bugzilla.yoctoproject.org/show_bug.cgi?id=5574 (which I hope to get to for 2.1, but no promises) we will need to write the kernel configuration to the sysroot so that we have somewhere we can look for it; if that's the case there'll be no need to copy it specially in buildhistory-extra.bbclass.