Skip to contentSkip to conversation

bitbake.conf: Add COMMON_LICENSE_DIR to BB_HASHEXCLUDE_COMMON

PATCH In review Peter Kjellerstedt 3 participants 5 messages Feb 13, 2022 – Feb 14, 2022

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 bitbake.conf: Add COMMON_LICENSE_DIR to BB_HASHEXCLUDE_COMMON

Differences in COMMON_LICENSE_DIR should not affect the task hashes.

---
 meta/conf/bitbake.conf | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/conf/bitbake.conf +1 -1
index fba99e8f0c..47c8cb39f9 100644
@@ -922,7 +922,7 @@ BB_HASHEXCLUDE_COMMON ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH BBSERVER DL_DI
     BB_WORKERCONTEXT BB_LIMITEDDEPS BB_UNIHASH extend_recipe_sysroot DEPLOY_DIR \
     SSTATE_HASHEQUIV_METHOD SSTATE_HASHEQUIV_REPORT_TASKDATA \
     SSTATE_HASHEQUIV_OWNER CCACHE_TOP_DIR BB_HASHSERVE GIT_CEILING_DIRECTORIES \
-    OMP_NUM_THREADS BB_CURRENTTASK"
+    OMP_NUM_THREADS BB_CURRENTTASK COMMON_LICENSE_DIR"
 BB_HASHBASE_WHITELIST ?= "${BB_HASHEXCLUDE_COMMON} PSEUDO_IGNORE_PATHS BUILDHISTORY_DIR \
     SSTATE_DIR SOURCE_DATE_EPOCH"
 BB_HASHCONFIG_WHITELIST ?= "${BB_HASHEXCLUDE_COMMON} DATE TIME SSH_AGENT_PID \
#1 lore raw
Quoted 12 lines · Peter Kjellerstedt

Differences in COMMON_LICENSE_DIR should not affect the task hashes.

---
 meta/conf/bitbake.conf | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/conf/bitbake.conf +0 -0
index fba99e8f0c..47c8cb39f9 100644

meta/conf/bitbake.conf BB_HASHEXCLUDE_COMMON ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH BBSERVER DL_DI · line 928

Quoted 8 lines
meta/conf/bitbake.conf +1 -1
@@ -922,7 +922,7 @@ BB_HASHEXCLUDE_COMMON ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH BBSERVER DL_DI
     BB_WORKERCONTEXT BB_LIMITEDDEPS BB_UNIHASH extend_recipe_sysroot DEPLOY_DIR \
     SSTATE_HASHEQUIV_METHOD SSTATE_HASHEQUIV_REPORT_TASKDATA \
     SSTATE_HASHEQUIV_OWNER CCACHE_TOP_DIR BB_HASHSERVE GIT_CEILING_DIRECTORIES \
-    OMP_NUM_THREADS BB_CURRENTTASK"
+    OMP_NUM_THREADS BB_CURRENTTASK COMMON_LICENSE_DIR"
 BB_HASHBASE_WHITELIST ?= "${BB_HASHEXCLUDE_COMMON} PSEUDO_IGNORE_PATHS BUILDHISTORY_DIR \
     SSTATE_DIR SOURCE_DATE_EPOCH"
 BB_HASHCONFIG_WHITELIST ?= "${BB_HASHEXCLUDE_COMMON} DATE TIME SSH_AGENT_PID \

I think this has been discussed before and I'm very uneasy at the idea. Some users would expect that if they add "their" version of a license in a layer with higher priority, they'd expect the hashes to change.

Where is this causing an issue?

The whole idea of the "common" licenses probably needs revisiting anyway too :/

Cheers,

Richard

#2 lore raw
Quoted 23 lines

-----Original Message-----
From: Richard Purdie <[email protected]>
Sent: den 13 februari 2022 22:44
To: Peter Kjellerstedt <[email protected]>; [email protected]
Subject: Re: [OE-core] [PATCH] bitbake.conf: Add COMMON_LICENSE_DIR to BB_HASHEXCLUDE_COMMON

Quoted 16 lines · Peter Kjellerstedt

Differences in COMMON_LICENSE_DIR should not affect the task hashes.

---
 meta/conf/bitbake.conf | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/conf/bitbake.conf +0 -1
index fba99e8f0c..47c8cb39f9 100644
@@ -922,7 +922,7 @@ BB_HASHEXCLUDE_COMMON ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH BBSERVER DL_DI
     BB_WORKERCONTEXT BB_LIMITEDDEPS BB_UNIHASH extend_recipe_sysroot DEPLOY_DIR \
     SSTATE_HASHEQUIV_METHOD SSTATE_HASHEQUIV_REPORT_TASKDATA \
     SSTATE_HASHEQUIV_OWNER CCACHE_TOP_DIR BB_HASHSERVE GIT_CEILING_DIRECTORIES \
-    OMP_NUM_THREADS BB_CURRENTTASK"
Quoted 8 lines

meta/conf/bitbake.conf BB_HASHEXCLUDE_COMMON ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH BBSERVER DL_DI · line 928

Quoted 4 lines
meta/conf/bitbake.conf +1 -0
+    OMP_NUM_THREADS BB_CURRENTTASK COMMON_LICENSE_DIR"
 BB_HASHBASE_WHITELIST ?= "${BB_HASHEXCLUDE_COMMON} PSEUDO_IGNORE_PATHS BUILDHISTORY_DIR \
     SSTATE_DIR SOURCE_DATE_EPOCH"
 BB_HASHCONFIG_WHITELIST ?= "${BB_HASHEXCLUDE_COMMON} DATE TIME SSH_AGENT_PID \

I think this has been discussed before and I'm very uneasy at the idea. Some users would expect that if they add "their" version of a license in a layer with higher priority, they'd expect the hashes to change.

If the value was a relative path I could buy that, but not for an absolute path.

Quoted 1 line

Where is this causing an issue?

Due to the huge number of licenses in meta/files/common-licenses after all SPDX licenses were added, we cannot use that directory anymore as it triples the recipe parsing time for us (since we define INCOMPATIBLE LICENSE as AVAILABLE_LICENSES minus COMPATIBLE_LICENSES, and the license code doesn't really handle having many hundreds of licenses in INCOMPATIBLE LICENSE). Thus as a workaround I have had to create a common-licenses directory in one of our layers which only has symbolic links to the licenses in meta that we need. Then we set:

COMMON_LICENSE_DIR := "${LAYERDIR}/files/common-licenses"

in that layer's layer.conf, but since ${LAYERDIR} varies from build host to build host, our whole global sstate was now only useable on the build servers and not for local developer builds.

Quoted 7 lines

The whole idea of the "common" licenses probably needs revisiting anyway too :/

Cheers,

Richard

//Peter

#3 lore raw
Quoted 40 lines · Peter Kjellerstedt
Quoted 31 lines

-----Original Message-----
From: Richard Purdie <[email protected]>
Sent: den 13 februari 2022 22:44
To: Peter Kjellerstedt <[email protected]>; [email protected]
Subject: Re: [OE-core] [PATCH] bitbake.conf: Add COMMON_LICENSE_DIR to BB_HASHEXCLUDE_COMMON

Quoted 22 lines · Peter Kjellerstedt

Differences in COMMON_LICENSE_DIR should not affect the task hashes.

---
 meta/conf/bitbake.conf | 2 +-

 1 file changed, 1 insertion(+), 1 deletion(-)
meta/conf/bitbake.conf +0 -0
index fba99e8f0c..47c8cb39f9 100644
@@ -922,7 +922,7 @@ BB_HASHEXCLUDE_COMMON ?= "TMPDIR FILE PATH PWD

BB_TASKHASH BBPATH BBSERVER DL_DI
     BB_WORKERCONTEXT BB_LIMITEDDEPS BB_UNIHASH extend_recipe_sysroot DEPLOY_DIR \
     SSTATE_HASHEQUIV_METHOD SSTATE_HASHEQUIV_REPORT_TASKDATA \
     SSTATE_HASHEQUIV_OWNER CCACHE_TOP_DIR BB_HASHSERVE GIT_CEILING_DIRECTORIES \

meta/conf/bitbake.conf +1 -1
-    OMP_NUM_THREADS BB_CURRENTTASK"
+    OMP_NUM_THREADS BB_CURRENTTASK COMMON_LICENSE_DIR"

 BB_HASHBASE_WHITELIST ?= "${BB_HASHEXCLUDE_COMMON} PSEUDO_IGNORE_PATHS BUILDHISTORY_DIR \

Quoted 8 lines
Quoted 3 lines

     SSTATE_DIR SOURCE_DATE_EPOCH" BB_HASHCONFIG_WHITELIST ?= "${BB_HASHEXCLUDE_COMMON} DATE TIME SSH_AGENT_PID \

I think this has been discussed before and I'm very uneasy at the idea. Some users would expect that if they add "their" version of a license in a layer with
higher priority, they'd expect the hashes to change.

Quoted 2 lines

If the value was a relative path I could buy that, but not for an absolute path.

True, however my concern is more that if you're changing this you are changing the configuration and you'd expect the hashes to change as a result. Adding it to the exclusion list hides that.

Quoted 11 lines
Quoted 1 line

Where is this causing an issue?

Due to the huge number of licenses in meta/files/common-licenses after all SPDX licenses were added, we cannot use that directory anymore as it triples the recipe parsing time for us (since we define INCOMPATIBLE LICENSE as AVAILABLE_LICENSES minus COMPATIBLE_LICENSES, and the license code doesn't really handle having many hundreds of licenses in INCOMPATIBLE LICENSE). Thus as a workaround I have had to create a common-licenses directory in one of our layers which only has symbolic links to the licenses in meta that we need. Then we set:

Quoted 5 lines

COMMON_LICENSE_DIR := "${LAYERDIR}/files/common-licenses"

in that layer's layer.conf, but since ${LAYERDIR} varies from build host to build host, our whole global sstate was now only useable on the build servers and not for local developer builds.

So shouldn't you just add COMMON_LICENSE_DIR to BB_HASHEXCLUDE_COMMON in your config?

Cheers,

Richard

#4 lore raw
Quoted 53 lines

-----Original Message-----
From: Richard Purdie <[email protected]>
Sent: den 14 februari 2022 11:32
To: Peter Kjellerstedt <[email protected]>; openembedded- [email protected]
Subject: Re: [OE-core] [PATCH] bitbake.conf: Add COMMON_LICENSE_DIR to BB_HASHEXCLUDE_COMMON

Quoted 16 lines · Peter Kjellerstedt
Quoted 16 lines

-----Original Message-----
From: Richard Purdie <[email protected]>
Sent: den 13 februari 2022 22:44
To: Peter Kjellerstedt <[email protected]>; [email protected]
Subject: Re: [OE-core] [PATCH] bitbake.conf: Add COMMON_LICENSE_DIR to BB_HASHEXCLUDE_COMMON

Quoted 7 lines · Peter Kjellerstedt

Differences in COMMON_LICENSE_DIR should not affect the task hashes.

---
 meta/conf/bitbake.conf | 2 +-

 1 file changed, 1 insertion(+), 1 deletion(-)
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
meta/conf/bitbake.conf +0 -0
index fba99e8f0c..47c8cb39f9 100644
@@ -922,7 +922,7 @@ BB_HASHEXCLUDE_COMMON ?= "TMPDIR FILE PATH PWD

BB_TASKHASH BBPATH BBSERVER DL_DI
     BB_WORKERCONTEXT BB_LIMITEDDEPS BB_UNIHASH

extend_recipe_sysroot

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

DEPLOY_DIR \
     SSTATE_HASHEQUIV_METHOD SSTATE_HASHEQUIV_REPORT_TASKDATA \
     SSTATE_HASHEQUIV_OWNER CCACHE_TOP_DIR BB_HASHSERVE GIT_CEILING_DIRECTORIES \

meta/conf/bitbake.conf +1 -1
-    OMP_NUM_THREADS BB_CURRENTTASK"
+    OMP_NUM_THREADS BB_CURRENTTASK COMMON_LICENSE_DIR"

 BB_HASHBASE_WHITELIST ?= "${BB_HASHEXCLUDE_COMMON}

PSEUDO_IGNORE_PATHS

Quoted 6 lines
Quoted 6 lines
Quoted 4 lines

BUILDHISTORY_DIR \
     SSTATE_DIR SOURCE_DATE_EPOCH" BB_HASHCONFIG_WHITELIST ?= "${BB_HASHEXCLUDE_COMMON} DATE TIME SSH_AGENT_PID \

I think this has been discussed before and I'm very uneasy at the

idea. Some

Quoted 1 line
Quoted 1 line

users would expect that if they add "their" version of a license in a

layer

Quoted 3 lines
Quoted 2 lines

with
higher priority, they'd expect the hashes to change.

Quoted 6 lines
Quoted 2 lines

If the value was a relative path I could buy that, but not for an absolute path.

True, however my concern is more that if you're changing this you are changing the configuration and you'd expect the hashes to change as a result. Adding it to the exclusion list hides that.

On the other hand, most (all?) other variables that take absolute paths are also present in BB_HASHEXCLUDE_COMMON so that differences to where the build directory happens to be do not affect the sstate.

Quoted 11 lines
Quoted 11 lines
Quoted 1 line

Where is this causing an issue?

Due to the huge number of licenses in meta/files/common-licenses after all SPDX licenses were added, we cannot use that directory anymore as it triples the recipe parsing time for us (since we define INCOMPATIBLE LICENSE as AVAILABLE_LICENSES minus COMPATIBLE_LICENSES, and the license code doesn't really handle having many hundreds of licenses in INCOMPATIBLE LICENSE). Thus as a workaround I have had to create a common-licenses directory in one of our layers which only has symbolic links to the licenses in meta that we need. Then we set:

Quoted 8 lines
Quoted 5 lines

COMMON_LICENSE_DIR := "${LAYERDIR}/files/common-licenses"

in that layer's layer.conf, but since ${LAYERDIR} varies from build host to build host, our whole global sstate was now only useable on the build servers and not for local developer builds.

So shouldn't you just add COMMON_LICENSE_DIR to BB_HASHEXCLUDE_COMMON in your config?

Well, based on your reluctance to add it to bitbake.conf, I have now instead added COMMON_LICENSE_DIR[vardepvalue] = "" together with where we redefine COMMON_LICENSE_DIR.

However, the reason I suggested adding it to bitbake.conf is that if you change this variable, it is easy to miss that it can cause your sstate to be different for each build host. I know I did, and we have had it modified for half a year. It wasn't until yesterday, when I made a local build that I expected to fully build from our global sstate and it instead rebuilt a lot of recipes, that I found out there was something wrong. It took me a couple of hours to dig into the siginfo files to finally end up at the absolute path in COMMON_LICENSE_DIR as the source of the differing sstate.

Quoted 3 lines

Cheers,

Richard

//Peter