Skip to contentSkip to conversation

buildhistory: record SRC_URI

RFC PATCH In review Paul Eggleton 4 participants 8 messages Oct 19, 2020 – Oct 29, 2020

lore
Share & tools
lore
#0 lore raw

cover buildhistory: record SRC_URI

Record SRC_URI for all built recipes in buildhistory. Since we also want this recorded for native recipes, enable writing out the "latest" file under packages/ for native recipes as well.

Please review the following changes for suitability for inclusion. If you have any objections or suggestions for improvement, please respond to the patches. If you agree with the changes, please provide your Acked-by.

The following changes since commit 1f26495884b8f567aecadc5936651846dfeed3f5:

siteinfo: Recognize bigendian sh3be and sh4be (2020-10-17 23:22:15 +0100)

are available in the git repository at:

git://git.openembedded.org/openembedded-core-contrib paule/buildhistory-src-uri
http://cgit.openembedded.org/openembedded-core-contrib/log/?h=paule/buildhistory-src-uri

Paul Eggleton (2):
classes/buildhistory: record SRC_URI classes/buildhistory: also save recipe info for native recipes

 meta/classes/buildhistory.bbclass | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)
Signature · 1 line
1.8.3.1
#1 lore raw

RFC PATCH 1/2 classes/buildhistory: record SRC_URI

From: Paul Eggleton <[email protected]>

It can be useful to record SRC_URI into buildhistory for the purposes of tracking exactly which sources got built (we already have SRCREV) as well as getting an indication when changes to the SRC_URI relate to changes in the output.

---
 meta/classes/buildhistory.bbclass | 3 +++
 1 file changed, 3 insertions(+)
meta/classes/buildhistory.bbclass +3 -0
index 0f26c3c..7d5e3eb 100644
@@ -116,6 +116,7 @@ python buildhistory_emit_pkghistory() {
             self.srcrev = ""
             self.layer = ""
             self.config = ""
+            self.src_uri = ""
 
 
     class PackageInfo:
@@ -258,6 +259,7 @@ python buildhistory_emit_pkghistory() {
     rcpinfo.packages = packages
     rcpinfo.layer = layer
     rcpinfo.config = sortlist(oe.utils.squashspaces(d.getVar('PACKAGECONFIG') or ""))
+    rcpinfo.src_uri = oe.utils.squashspaces(d.getVar('SRC_URI') or "")
     write_recipehistory(rcpinfo, d)
 
     bb.build.exec_func("read_subpackage_metadata", d)
@@ -368,6 +370,7 @@ def write_recipehistory(rcpinfo, d):
         f.write(u"PACKAGES = %s\n" %  rcpinfo.packages)
         f.write(u"LAYER = %s\n" %  rcpinfo.layer)
         f.write(u"CONFIG = %s\n" %  rcpinfo.config)
+        f.write(u"SRC_URI = %s\n" %  rcpinfo.src_uri)
 
     write_latest_srcrev(d, pkghistdir)
 
Signature · 1 line
1.8.3.1
#6 lore raw

RFC PATCH 2/2 classes/buildhistory: also save recipe info for native recipes

In reply to Paul Eggleton

From: Paul Eggleton <[email protected]>

If we want to also collect SRC_URI for native recipes we need to ensure that the code that writes out all of the recipe info is called - there isn't a do_packagedata for native recipes so we need to piggyback on do_populate_sysroot instead.

---
 meta/classes/buildhistory.bbclass | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
meta/classes/buildhistory.bbclass +4 -4
index 7d5e3eb..6d04d8c 100644
@@ -90,8 +90,7 @@ buildhistory_emit_sysroot() {
 python buildhistory_emit_pkghistory() {
     if d.getVar('BB_CURRENTTASK') in ['populate_sysroot', 'populate_sysroot_setscene']:
         bb.build.exec_func("buildhistory_emit_sysroot", d)
-
-    if not d.getVar('BB_CURRENTTASK') in ['packagedata', 'packagedata_setscene']:
+    elif not d.getVar('BB_CURRENTTASK') in ['packagedata', 'packagedata_setscene']:
         return 0
 
     if not "package" in (d.getVar('BUILDHISTORY_FEATURES') or "").split():
@@ -229,8 +228,9 @@ python buildhistory_emit_pkghistory() {
                     break
     except IOError as e:
         if e.errno == errno.ENOENT:
-            # Probably a -cross recipe, just ignore
-            return 0
+            if not bb.data.inherits_class('native', d):
+                # Probably a -cross recipe, just ignore
+                return 0
         else:
             raise
 
Signature · 1 line
1.8.3.1
#7 lore raw
In reply to Paul Eggleton
Quoted 3 lines · On Monday, 19 October 2020 17:03:55 NZDT Paul Eggleton wrote:

Please review the following changes for suitability for inclusion. If you have any objections or suggestions for improvement, please respond to the patches. If you agree with the changes, please provide your Acked-by.

Hmm, this should not have been RFC. Consider this a regular patch series.

Thanks
Paul

Acked-by on v1 1/2
#2 lore raw

Re: [OE-core] [RFC PATCH 1/2] classes/buildhistory: record SRC_URI

In reply to Paul Eggleton
Quoted 8 lines · Paul Eggleton

From: Paul Eggleton <[email protected]>

It can be useful to record SRC_URI into buildhistory for the purposes of tracking exactly which sources got built (we already have SRCREV) as well as getting an indication when changes to the SRC_URI relate to changes in the output.

I have similar patch in our poky trees. Also have patches to export LICENSE and CVE_PRODUCT to buildhistory. These are used by some post-build QA check scripts.

Cheers,

-Mikko

Quoted 39 lines
---
 meta/classes/buildhistory.bbclass | 3 +++
 1 file changed, 3 insertions(+)
meta/classes/buildhistory.bbclass +3 -0
index 0f26c3c..7d5e3eb 100644
@@ -116,6 +116,7 @@ python buildhistory_emit_pkghistory() {
             self.srcrev = ""
             self.layer = ""
             self.config = ""
+            self.src_uri = ""
 
 
     class PackageInfo:
@@ -258,6 +259,7 @@ python buildhistory_emit_pkghistory() {
     rcpinfo.packages = packages
     rcpinfo.layer = layer
     rcpinfo.config = sortlist(oe.utils.squashspaces(d.getVar('PACKAGECONFIG') or ""))
+    rcpinfo.src_uri = oe.utils.squashspaces(d.getVar('SRC_URI') or "")
     write_recipehistory(rcpinfo, d)
 
     bb.build.exec_func("read_subpackage_metadata", d)
@@ -368,6 +370,7 @@ def write_recipehistory(rcpinfo, d):
         f.write(u"PACKAGES = %s\n" %  rcpinfo.packages)
         f.write(u"LAYER = %s\n" %  rcpinfo.layer)
         f.write(u"CONFIG = %s\n" %  rcpinfo.config)
+        f.write(u"SRC_URI = %s\n" %  rcpinfo.src_uri)
 
     write_latest_srcrev(d, pkghistdir)
 
Signature · 1 line
1.8.3.1
#3 lore raw

Re: [OE-core] [RFC PATCH 1/2] classes/buildhistory: record SRC_URI

Quoted 10 lines · Mikko Rapeli
Quoted 9 lines · Paul Eggleton

From: Paul Eggleton <[email protected]>

It can be useful to record SRC_URI into buildhistory for the purposes of
tracking exactly which sources got built (we already have SRCREV) as
well as getting an indication when changes to the SRC_URI relate to changes in the output.

Quoted 7 lines
Quoted 1 line

I have similar patch in our poky trees. Also have patches to export LICENSE and CVE_PRODUCT to buildhistory. These are used by some post-build QA check scripts.

I'm actually fairly against some of these kinds of changes. Buildhistory is meant to be there to highlight changes in the output over time. Using it to create manifests and for license checking purposes is not what it was designed for.

I just feel if the original author of the class thinks its a good idea I need to give up :/.

Cheers,

Richard

#4 lore raw

Re: [OE-core] [RFC PATCH 1/2] classes/buildhistory: record SRC_URI

Quoted 19 lines · Richard Purdie
Quoted 10 lines · Mikko Rapeli
Quoted 9 lines · Paul Eggleton

From: Paul Eggleton <[email protected]>

It can be useful to record SRC_URI into buildhistory for the purposes of
tracking exactly which sources got built (we already have SRCREV) as
well as getting an indication when changes to the SRC_URI relate to changes in the output.

Quoted 7 lines
Quoted 1 line

I have similar patch in our poky trees. Also have patches to export LICENSE and CVE_PRODUCT to buildhistory. These are used by some post-build QA check scripts.

Quoted 7 lines

I'm actually fairly against some of these kinds of changes. Buildhistory is meant to be there to highlight changes in the output over time. Using it to create manifests and for license checking purposes is not what it was designed for.

I just feel if the original author of the class thinks its a good idea I need to give up :/.

Surely changes in LICENSE and SRC_URI over time are important for an overview of changes?

At least to me they are pretty much essential. I'm using buildhistory diffs to see what changes in major yocto updates for example once builds are passing.

Cheers,

-Mikko

Quoted 4 lines

Cheers,

Richard

#5 lore raw

Re: [OE-core] [RFC PATCH 1/2] classes/buildhistory: record SRC_URI

Quoted 28 lines · On Thu, 2020-10-29 at 09:18 +0000, [email protected] wrote:
Quoted 27 lines · Richard Purdie
Quoted 12 lines · Mikko Rapeli
Quoted 11 lines · Paul Eggleton

From: Paul Eggleton <[email protected]>

It can be useful to record SRC_URI into buildhistory for the purposes of
tracking exactly which sources got built (we already have SRCREV)
as
well as getting an indication when changes to the SRC_URI relate to
changes in the output.

Quoted 7 lines
Quoted 1 line

I have similar patch in our poky trees. Also have patches to export LICENSE and CVE_PRODUCT to buildhistory. These are used by some post-build QA check scripts.

I'm actually fairly against some of these kinds of changes. Buildhistory is meant to be there to highlight changes in the output
over time. Using it to create manifests and for license checking purposes is not what it was designed for.

Quoted 6 lines
Quoted 3 lines

I just feel if the original author of the class thinks its a good idea
I need to give up :/.

Surely changes in LICENSE and SRC_URI over time are important for an overview of changes?

LICENSE is important and I understand that one. SRC_URI is trickier. The recipe version is in some ways more relevant that SRC_URI. I guess with the latter, you can look at the change in the set of patches or perhaps source of the code but I'd suspect the version is the more useful thing people are looking for.

There is then the temptation for people to use the buildhistory data for license auditing which worries me as its not what its intended for.

Quoted 3 lines

At least to me they are pretty much essential. I'm using buildhistory diffs to see what changes in major yocto updates for example once builds are passing.

The question is whether the history is of the inputs, the outputs or tangential information like the license information. Traditionally it was focused very specifically on the output, we've then added more and more input data and people are using it for licensing (which there is supposed to be separate code for). It makes it hard to maintain and develop since the usage becomes so varied. I appreciate few people care about that though as it's someone else's problem though :(. I try and keep things true to their design and people end up unhappy. Where we deviate, I try and ensure its at least a conscious decision, I'm still not sure it is in this case.

Cheers,

Richard