bitbake.conf: Add COMMON_LICENSE_DIR to BB_HASHEXCLUDE_COMMON
PATCH 1/1 bitbake.conf: Add COMMON_LICENSE_DIR to BB_HASHEXCLUDE_COMMON
Differences in COMMON_LICENSE_DIR should not affect the task hashes.
--- meta/conf/bitbake.conf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
meta/conf/bitbake.conf +1 -1
@@ -922,7 +922,7 @@ BB_HASHEXCLUDE_COMMON ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH BBSERVER DL_DI BB_WORKERCONTEXT BB_LIMITEDDEPS BB_UNIHASH extend_recipe_sysroot DEPLOY_DIR \ SSTATE_HASHEQUIV_METHOD SSTATE_HASHEQUIV_REPORT_TASKDATA \ SSTATE_HASHEQUIV_OWNER CCACHE_TOP_DIR BB_HASHSERVE GIT_CEILING_DIRECTORIES \ - OMP_NUM_THREADS BB_CURRENTTASK" + OMP_NUM_THREADS BB_CURRENTTASK COMMON_LICENSE_DIR" BB_HASHBASE_WHITELIST ?= "${BB_HASHEXCLUDE_COMMON} PSEUDO_IGNORE_PATHS BUILDHISTORY_DIR \ SSTATE_DIR SOURCE_DATE_EPOCH" BB_HASHCONFIG_WHITELIST ?= "${BB_HASHEXCLUDE_COMMON} DATE TIME SSH_AGENT_PID \
Quoted 12 lines · Peter Kjellerstedt
Differences in
COMMON_LICENSE_DIRshould not affect the task hashes.Signed-off-by Peter Kjellerstedt <[email protected]>--- meta/conf/bitbake.conf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)meta/conf/bitbake.conf +0
-0
meta/conf/bitbake.conf BB_HASHEXCLUDE_COMMON ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH BBSERVER DL_DI · line 928
Quoted 8 lines
meta/conf/bitbake.conf +1
-1@@ -922,7 +922,7 @@ BB_HASHEXCLUDE_COMMON ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH BBSERVER DL_DI BB_WORKERCONTEXT BB_LIMITEDDEPS BB_UNIHASH extend_recipe_sysroot DEPLOY_DIR \ SSTATE_HASHEQUIV_METHOD SSTATE_HASHEQUIV_REPORT_TASKDATA \ SSTATE_HASHEQUIV_OWNER CCACHE_TOP_DIR BB_HASHSERVE GIT_CEILING_DIRECTORIES \ - OMP_NUM_THREADS BB_CURRENTTASK" + OMP_NUM_THREADS BB_CURRENTTASK COMMON_LICENSE_DIR" BB_HASHBASE_WHITELIST ?= "${BB_HASHEXCLUDE_COMMON} PSEUDO_IGNORE_PATHS BUILDHISTORY_DIR \ SSTATE_DIR SOURCE_DATE_EPOCH" BB_HASHCONFIG_WHITELIST ?= "${BB_HASHEXCLUDE_COMMON} DATE TIME SSH_AGENT_PID \
I think this has been discussed before and I'm very uneasy at the idea. Some users would expect that if they add "their" version of a license in a layer with higher priority, they'd expect the hashes to change.
Where is this causing an issue?
The whole idea of the "common" licenses probably needs revisiting anyway too :/
Cheers,
Richard
Quoted 23 lines
-----Original Message-----
From: Richard Purdie <[email protected]>
Sent: den 13 februari 2022 22:44
To: Peter Kjellerstedt <[email protected]>; [email protected]
Subject: Re: [OE-core] [PATCH] bitbake.conf: AddCOMMON_LICENSE_DIRtoBB_HASHEXCLUDE_COMMONQuoted 16 lines · Peter Kjellerstedt
Differences in
COMMON_LICENSE_DIRshould not affect the task hashes.Signed-off-by Peter Kjellerstedt <[email protected]>--- meta/conf/bitbake.conf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)meta/conf/bitbake.conf +0
-1@@ -922,7 +922,7 @@ BB_HASHEXCLUDE_COMMON ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH BBSERVER DL_DI BB_WORKERCONTEXT BB_LIMITEDDEPS BB_UNIHASH extend_recipe_sysroot DEPLOY_DIR \ SSTATE_HASHEQUIV_METHOD SSTATE_HASHEQUIV_REPORT_TASKDATA \ SSTATE_HASHEQUIV_OWNER CCACHE_TOP_DIR BB_HASHSERVE GIT_CEILING_DIRECTORIES \ - OMP_NUM_THREADS BB_CURRENTTASK"
Quoted 8 lines
meta/conf/bitbake.conf BB_HASHEXCLUDE_COMMON ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH BBSERVER DL_DI · line 928
Quoted 4 lines
meta/conf/bitbake.conf +1
-0+ OMP_NUM_THREADS BB_CURRENTTASK COMMON_LICENSE_DIR" BB_HASHBASE_WHITELIST ?= "${BB_HASHEXCLUDE_COMMON} PSEUDO_IGNORE_PATHS BUILDHISTORY_DIR \ SSTATE_DIR SOURCE_DATE_EPOCH" BB_HASHCONFIG_WHITELIST ?= "${BB_HASHEXCLUDE_COMMON} DATE TIME SSH_AGENT_PID \I think this has been discussed before and I'm very uneasy at the idea. Some users would expect that if they add "their" version of a license in a layer with higher priority, they'd expect the hashes to change.
If the value was a relative path I could buy that, but not for an absolute path.
Quoted 1 line
Where is this causing an issue?
Due to the huge number of licenses in meta/files/common-licenses after all SPDX licenses were added, we cannot use that directory anymore as it triples the recipe parsing time for us (since we define INCOMPATIBLE LICENSE as AVAILABLE_LICENSES minus COMPATIBLE_LICENSES, and the license code doesn't really handle having many hundreds of licenses in INCOMPATIBLE LICENSE). Thus as a workaround I have had to create a common-licenses directory in one of our layers which only has symbolic links to the licenses in meta that we need. Then we set:
COMMON_LICENSE_DIR := "${LAYERDIR}/files/common-licenses"
in that layer's layer.conf, but since ${LAYERDIR} varies from build host to build host, our whole global sstate was now only useable on the build servers and not for local developer builds.
Quoted 7 lines
The whole idea of the "common" licenses probably needs revisiting anyway too :/
Cheers,
Richard
//Peter
Quoted 40 lines · Peter Kjellerstedt
Quoted 31 lines
-----Original Message-----
From: Richard Purdie <[email protected]>
Sent: den 13 februari 2022 22:44
To: Peter Kjellerstedt <[email protected]>; [email protected]
Subject: Re: [OE-core] [PATCH] bitbake.conf: AddCOMMON_LICENSE_DIRtoBB_HASHEXCLUDE_COMMONQuoted 22 lines · Peter Kjellerstedt
Differences in
COMMON_LICENSE_DIRshould not affect the task hashes.Signed-off-by Peter Kjellerstedt <[email protected]>---
meta/conf/bitbake.conf | 2 +-1 file changed, 1 insertion(+), 1 deletion(-)meta/conf/bitbake.conf +0
-0@@ -922,7 +922,7 @@ BB_HASHEXCLUDE_COMMON ?= "TMPDIR FILE PATH PWD
BB_TASKHASHBBPATH BBSERVERDL_DI
BB_WORKERCONTEXTBB_LIMITEDDEPSBB_UNIHASHextend_recipe_sysrootDEPLOY_DIR\
SSTATE_HASHEQUIV_METHODSSTATE_HASHEQUIV_REPORT_TASKDATA\
SSTATE_HASHEQUIV_OWNERCCACHE_TOP_DIRBB_HASHSERVEGIT_CEILING_DIRECTORIES\meta/conf/bitbake.conf +1
-1- OMP_NUM_THREADS BB_CURRENTTASK" + OMP_NUM_THREADS BB_CURRENTTASK COMMON_LICENSE_DIR"BB_HASHBASE_WHITELIST ?= "${BB_HASHEXCLUDE_COMMON}
PSEUDO_IGNORE_PATHSBUILDHISTORY_DIR\Quoted 8 lines
Quoted 3 lines
SSTATE_DIR
SOURCE_DATE_EPOCH"BB_HASHCONFIG_WHITELIST?= "${BB_HASHEXCLUDE_COMMON} DATE TIMESSH_AGENT_PID\I think this has been discussed before and I'm very uneasy at the idea. Some users would expect that if they add "their" version of a license in a layer with
higher priority, they'd expect the hashes to change.
Quoted 2 lines
If the value was a relative path I could buy that, but not for an absolute path.
True, however my concern is more that if you're changing this you are changing the configuration and you'd expect the hashes to change as a result. Adding it to the exclusion list hides that.
Quoted 11 lines
Quoted 1 line
Where is this causing an issue?
Due to the huge number of licenses in meta/files/common-licenses after all SPDX licenses were added, we cannot use that directory anymore as it triples the recipe parsing time for us (since we define INCOMPATIBLE LICENSE as
AVAILABLE_LICENSESminusCOMPATIBLE_LICENSES, and the license code doesn't really handle having many hundreds of licenses in INCOMPATIBLE LICENSE). Thus as a workaround I have had to create a common-licenses directory in one of our layers which only has symbolic links to the licenses in meta that we need. Then we set:
Quoted 5 lines
COMMON_LICENSE_DIR:= "${LAYERDIR}/files/common-licenses"in that layer's layer.conf, but since ${LAYERDIR} varies from build host to build host, our whole global sstate was now only useable on the build servers and not for local developer builds.
So shouldn't you just add COMMON_LICENSE_DIR to BB_HASHEXCLUDE_COMMON in your config?
Cheers,
Richard
Quoted 53 lines
-----Original Message-----
From: Richard Purdie <[email protected]>
Sent: den 14 februari 2022 11:32
To: Peter Kjellerstedt <[email protected]>; openembedded- [email protected]
Subject: Re: [OE-core] [PATCH] bitbake.conf: AddCOMMON_LICENSE_DIRtoBB_HASHEXCLUDE_COMMONQuoted 16 lines · Peter Kjellerstedt
Quoted 16 lines
-----Original Message-----
From: Richard Purdie <[email protected]>
Sent: den 13 februari 2022 22:44
To: Peter Kjellerstedt <[email protected]>; [email protected]
Subject: Re: [OE-core] [PATCH] bitbake.conf: AddCOMMON_LICENSE_DIRtoBB_HASHEXCLUDE_COMMONQuoted 7 lines · Peter Kjellerstedt
Differences in
COMMON_LICENSE_DIRshould not affect the task hashes.Signed-off-by Peter Kjellerstedt <[email protected]>---
meta/conf/bitbake.conf | 2 +-1 file changed, 1 insertion(+), 1 deletion(-)Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
meta/conf/bitbake.conf +0
-0@@ -922,7 +922,7 @@ BB_HASHEXCLUDE_COMMON ?= "TMPDIR FILE PATH PWD
BB_TASKHASHBBPATH BBSERVERDL_DI
BB_WORKERCONTEXTBB_LIMITEDDEPSBB_UNIHASH
extend_recipe_sysrootQuoted 7 lines
Quoted 7 lines
Quoted 7 lines
DEPLOY_DIR\
SSTATE_HASHEQUIV_METHODSSTATE_HASHEQUIV_REPORT_TASKDATA\
SSTATE_HASHEQUIV_OWNERCCACHE_TOP_DIRBB_HASHSERVEGIT_CEILING_DIRECTORIES\meta/conf/bitbake.conf +1
-1- OMP_NUM_THREADS BB_CURRENTTASK" + OMP_NUM_THREADS BB_CURRENTTASK COMMON_LICENSE_DIR"BB_HASHBASE_WHITELIST ?= "${BB_HASHEXCLUDE_COMMON}
PSEUDO_IGNORE_PATHSQuoted 6 lines
Quoted 6 lines
Quoted 4 lines
BUILDHISTORY_DIR\
SSTATE_DIRSOURCE_DATE_EPOCH"BB_HASHCONFIG_WHITELIST?= "${BB_HASHEXCLUDE_COMMON} DATE TIMESSH_AGENT_PID\I think this has been discussed before and I'm very uneasy at the
idea. Some
Quoted 1 line
Quoted 1 line
users would expect that if they add "their" version of a license in a
layer
Quoted 3 lines
Quoted 2 lines
with
higher priority, they'd expect the hashes to change.
Quoted 6 lines
Quoted 2 lines
If the value was a relative path I could buy that, but not for an absolute path.
True, however my concern is more that if you're changing this you are changing the configuration and you'd expect the hashes to change as a result. Adding it to the exclusion list hides that.
On the other hand, most (all?) other variables that take absolute paths are also present in BB_HASHEXCLUDE_COMMON so that differences to where the build directory happens to be do not affect the sstate.
Quoted 11 lines
Quoted 11 lines
Quoted 1 line
Where is this causing an issue?
Due to the huge number of licenses in meta/files/common-licenses after all SPDX licenses were added, we cannot use that directory anymore as it triples the recipe parsing time for us (since we define INCOMPATIBLE LICENSE as
AVAILABLE_LICENSESminusCOMPATIBLE_LICENSES, and the license code doesn't really handle having many hundreds of licenses in INCOMPATIBLE LICENSE). Thus as a workaround I have had to create a common-licenses directory in one of our layers which only has symbolic links to the licenses in meta that we need. Then we set:
Quoted 8 lines
Quoted 5 lines
COMMON_LICENSE_DIR:= "${LAYERDIR}/files/common-licenses"in that layer's layer.conf, but since ${LAYERDIR} varies from build host to build host, our whole global sstate was now only useable on the build servers and not for local developer builds.
So shouldn't you just add
COMMON_LICENSE_DIRtoBB_HASHEXCLUDE_COMMONin your config?
Well, based on your reluctance to add it to bitbake.conf, I have now instead added COMMON_LICENSE_DIR[vardepvalue] = "" together with where we redefine COMMON_LICENSE_DIR.
However, the reason I suggested adding it to bitbake.conf is that if you change this variable, it is easy to miss that it can cause your sstate to be different for each build host. I know I did, and we have had it modified for half a year. It wasn't until yesterday, when I made a local build that I expected to fully build from our global sstate and it instead rebuilt a lot of recipes, that I found out there was something wrong. It took me a couple of hours to dig into the siginfo files to finally end up at the absolute path in COMMON_LICENSE_DIR as the source of the differing sstate.
Quoted 3 lines
Cheers,
Richard
//Peter