bitbake.conf: Add resource vars to hash ignore list
PATCH 1/1 bitbake.conf: Add resource vars to hash ignore list
It should not matter what xz/zstd parallelism and resource usage is set to when determining hash signatures. This allows sharing more sstate across host machines with different CPU configurations.
--- meta/conf/bitbake.conf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
meta/conf/bitbake.conf +1 -1
@@ -944,7 +944,7 @@ BB_HASHEXCLUDE_COMMON ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH BBSERVER DL_DI BB_WORKERCONTEXT BB_LIMITEDDEPS BB_UNIHASH extend_recipe_sysroot DEPLOY_DIR \ SSTATE_HASHEQUIV_METHOD SSTATE_HASHEQUIV_REPORT_TASKDATA \ SSTATE_HASHEQUIV_OWNER CCACHE_TOP_DIR BB_HASHSERVE GIT_CEILING_DIRECTORIES \ - OMP_NUM_THREADS BB_CURRENTTASK" + XZ_MEMLIMIT XZ_THREADS XZ_DEFAULTS ZSTD_THREADS OMP_NUM_THREADS BB_CURRENTTASK" BB_BASEHASH_IGNORE_VARS ?= "${BB_HASHEXCLUDE_COMMON} PSEUDO_IGNORE_PATHS BUILDHISTORY_DIR \ SSTATE_DIR SOURCE_DATE_EPOCH RUST_BUILD_SYS RUST_HOST_SYS RUST_TARGET_SYS" BB_HASHCONFIG_IGNORE_VARS ?= "${BB_HASHEXCLUDE_COMMON} DATE TIME SSH_AGENT_PID \
Signature · 1 line
2.25.1
If you have a compressed initrd for example this could give you a surprise, so I'm not 100% sure about this.
/Regards
Den mån 22 aug. 2022 kl 07:00 skrev Wes Lindauer <[email protected]
Quoted 46 lines
:
It should not matter what xz/zstd parallelism and resource usage is set to when determining hash signatures. This allows sharing more sstate across host machines with different CPU configurations.
Signed-off-by Wes Lindauer <[email protected]>--- meta/conf/bitbake.conf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)meta/conf/bitbake.conf +0
-0@@ -944,7 +944,7 @@ BB_HASHEXCLUDE_COMMON ?= "TMPDIR FILE PATH PWD
BB_TASKHASHBBPATH BBSERVERDL_DI
BB_WORKERCONTEXTBB_LIMITEDDEPSBB_UNIHASHextend_recipe_sysrootDEPLOY_DIR\
SSTATE_HASHEQUIV_METHODSSTATE_HASHEQUIV_REPORT_TASKDATA\
SSTATE_HASHEQUIV_OWNERCCACHE_TOP_DIRBB_HASHSERVEGIT_CEILING_DIRECTORIES\meta/conf/bitbake.conf +1
-1- OMP_NUM_THREADS BB_CURRENTTASK" + XZ_MEMLIMIT XZ_THREADS XZ_DEFAULTS ZSTD_THREADS OMP_NUM_THREADS
BB_CURRENTTASK"
BB_BASEHASH_IGNORE_VARS?= "${BB_HASHEXCLUDE_COMMON}PSEUDO_IGNORE_PATHSBUILDHISTORY_DIR\
SSTATE_DIRSOURCE_DATE_EPOCHRUST_BUILD_SYSRUST_HOST_SYSRUST_TARGET_SYS"
BB_HASHCONFIG_IGNORE_VARS?= "${BB_HASHEXCLUDE_COMMON} DATE TIMESSH_AGENT_PID\
--
2.25.1-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#169658):
https://lists.openembedded.org/g/openembedded-core/message/169658
Mute This Topic: https://lists.openembedded.org/mt/93175519/4947266
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ [email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Hi Wes,
Quoted 4 lines · Wes Lindauer
It should not matter what xz/zstd parallelism and resource usage is set to when determining hash signatures. This allows sharing more sstate across host machines with different CPU configurations.
It seems like it does matter in some ways, c.f.:
https://git.openembedded.org/openembedded-core/commit/?id=0296dc71c01d1b7953d74ef37d56429e2f4fcfab
and
https://git.openembedded.org/openembedded-core/commit/?id=e72c0b94554a9bc293844ec2bddb0c04ea19791d
Cheers,
Quentin
Quoted 8 lines · Quentin Schulz
Hi Wes,
Quoted 4 lines · Wes Lindauer
It should not matter what xz/zstd parallelism and resource usage is set to when determining hash signatures. This allows sharing more sstate across host machines with different CPU configurations.
Quoted 4 lines
It seems like it does matter in some ways, c.f.:
https://git.openembedded.org/openembedded-core/commit/?id=0296dc71c01d1b7953d74ef37d56429e2f4fcfab
and
https://git.openembedded.org/openembedded-core/commit/?id=e72c0b94554a9bc293844ec2bddb0c04ea19791d
As Quentin mentions, the value does matter. In particular, those compiled with a single thread differ to those with multiple threads. As such the proposed patch can't be accepted.
Cheers,
Richard
Quoted 23 lines
-----Original Message-----
From: [email protected] <openembedded- [email protected]> On Behalf Of Richard Purdie
Sent: den 22 augusti 2022 13:19
To: Quentin Schulz <[email protected]>; Wes Lindauer <[email protected]>; [email protected]
Subject: Re: [OE-core][PATCH] bitbake.conf: Add resource vars to hash ignore listQuoted 8 lines · Quentin Schulz
Hi Wes,
Quoted 4 lines · Wes Lindauer
It should not matter what xz/zstd parallelism and resource usage is set to when determining hash signatures. This allows sharing more sstate across host machines with different CPU configurations.
Quoted 4 lines
It seems like it does matter in some ways, c.f.:
https://git.openembedded.org/openembedded-core/commit/?id=0296dc71c01d1b7953d74ef37d56429e2f4fcfab
and
https://git.openembedded.org/openembedded-core/commit/?id=e72c0b94554a9bc293844ec2bddb0c04ea19791d
Quoted 7 lines
As Quentin mentions, the value does matter. In particular, those compiled with a single thread differ to those with multiple threads. As such the proposed patch can't be accepted.
Cheers,
Richard
Isn't the problem of varying XZ_THREAD/ZSTD_THREAD vs taskhashes already taken care of by having:
XZ_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}" XZ_THREADS[vardepvalue] = "1"XZ_DEFAULTS ?= "--memlimit=${XZ_MEMLIMIT} --threads=${XZ_THREADS}" XZ_DEFAULTS[vardepsexclude] += "XZ_MEMLIMIT XZ_THREADS" ZSTD_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}" ZSTD_THREADS[vardepvalue] = "1"
in bitbake.conf.
//Peter
I think I understand. I am sorry for the FAD. I was experiencing this sstate mismatch on the dunfell branch that does not contain the 'XZ_THREADS[vardepvalue] = "1"' change. This is probably why I attempted my change in the hash exclude list and then tried to "port it forward to master". I appreciate everyone taking the time to explain. I am still a little confused why calculating every rpm task signature with a value of 1 is acceptable, but ignoring the variable in the signature is not. Are these not different implementations of the same thing?
Thanks,
Wes L
Quoted 50 lines · Peter Kjellerstedt
Quoted 9 lines
-----Original Message-----
From: [email protected] <openembedded- [email protected]> On Behalf Of Richard Purdie
Sent: den 22 augusti 2022 13:19
To: Quentin Schulz <[email protected]>; Wes Lindauer <[email protected]>; [email protected]
Subject: Re: [OE-core][PATCH] bitbake.conf: Add resource vars to hash ignore listQuoted 5 lines
Quoted 4 lines · Quentin Schulz
Hi Wes,
Quoted 1 line · Wes Lindauer
It should not matter what xz/zstd parallelism and resource usage is
set to
Quoted 1 line
Quoted 1 line
Quoted 1 line
when determining hash signatures. This allows sharing more sstate
across
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
host machines with different CPU configurations.
It seems like it does matter in some ways, c.f.:
https://git.openembedded.org/openembedded-core/commit/?id=0296dc71c01d1b7953d74ef37d56429e2f4fcfab
Quoted 2 lines
Quoted 2 lines
and
https://git.openembedded.org/openembedded-core/commit/?id=e72c0b94554a9bc293844ec2bddb0c04ea19791d
Quoted 8 lines
As Quentin mentions, the value does matter. In particular, those compiled with a single thread differ to those with multiple threads. As such the proposed patch can't be accepted.
Cheers,
Richard
Isn't the problem of varying XZ_THREAD/ZSTD_THREAD vs taskhashes already taken care of by having:
XZ_THREADS?= "${@oe.utils.cpu_count(at_least=2)}" XZ_THREADS[vardepvalue] = "1"XZ_DEFAULTS?= "--memlimit=${XZ_MEMLIMIT} --threads=${XZ_THREADS}" XZ_DEFAULTS[vardepsexclude] += "XZ_MEMLIMITXZ_THREADS"ZSTD_THREADS?= "${@oe.utils.cpu_count(at_least=2)}" ZSTD_THREADS[vardepvalue] = "1"in bitbake.conf.
//Peter
Quoted 12 lines · Wes Lindauer
I think I understand. I am sorry for the FAD. I was experiencing this sstate mismatch on the dunfell branch that does not contain the 'XZ_THREADS[vardepvalue] = "1"' change. This is probably why I attempted my change in the hash exclude list and then tried to "port it forward to master". I appreciate everyone taking the time to explain. I am still a little confused why calculating every rpm task signature with a value of 1 is acceptable, but ignoring the variable in the signature is not. Are these not different implementations of the same thing?
Quoted 2 lines
XZ_THREADS?= "${@oe.utils.cpu_count(at_least=2)}" XZ_THREADS[vardepvalue] = "1"
Note we do *two* things, we set it to at least a count of 2 and give it a known consistent value. It is the two things together than work out ok.
If the user changes XZ_THREADS, as long as it is set to a value or 2 or more it is still ok. If the user sets it to a value of 1, builds are not reproducible.
You're right that we could have just ignored it, but this at least gives a hint that a value of 2 or more should be used. Ideally we'd have put a warning around a value of 1 but there are a lot of things we'd do ideally and limited time...
Cheers,
Richard