Remove 'extend_recipe_sysroot' from 'BB_HASHEXCLUDE_COMMON'
cover Remove 'extend_recipe_sysroot' from 'BB_HASHEXCLUDE_COMMON'
It has lain there for a while, obscuring certain aspects of how task signatures are calculated.
This changeset does not change the current status quo, but rather changes the way in which it is implemented. It can be followed by changes which further refine what is excluded and how, from those tasks which refer to 'extend_recipe_sysroot'.
Related discussion:
https://lore.kernel.org/openembedded-core/aa893eb861bf4dd18e3e84e6bbebdee3b0367d1b.camel@linuxfoundation.org/
---
Adam Blank (7):
package_pkgdata: fix typo to stop calling undefined function
staging: add 'vardepsexclude' to 'staging_populate_sysroot_dir'
staging: add 'extend_recipe_sysroot' to 'vardepsexclude'
cross: add 'extend_recipe_sysroot' to 'vardepsexclude'
native: add 'extend_recipe_sysroot' to 'vardepsexclude'
wic-tool: add 'extend_recipe_sysroot' to 'vardepsexclude' bitbake.conf: remove 'extend_recipe_sysroot' from BB_HASHEXCLUDE_COMMON
meta/classes-global/package_pkgdata.bbclass | 4 +++- meta/classes-global/staging.bbclass | 5 ++++- meta/classes-recipe/cross.bbclass | 1 + meta/classes-recipe/native.bbclass | 1 + meta/conf/bitbake.conf | 2 +- meta/recipes-core/meta/wic-tools.bb | 1 + 6 files changed, 11 insertions(+), 3 deletions(-)
---
base-commit: d9b69d0a2d69e003b9432e1473830a89ff5a06c4
change-id: 20260418-extend_recipe_sysroot-d958ba9cfa9a
Best regards,
Signature · 1 line
Adam Blank <[email protected]>
PATCH 1/7 package_pkgdata: fix typo to stop calling undefined function
The function is named 'package_populate_pkgdata_dir' but the call was to 'staging_package_populate_pkgdata_dir'.
--- meta/classes-global/package_pkgdata.bbclass | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
meta/classes-global/package_pkgdata.bbclass +3 -1
@@ -31,6 +31,8 @@ def package_populate_pkgdata_dir(pkgdatadir, d): except FileExistsError: continue +package_populate_pkgdata_dir[vardepsexclude] += "MACHINE_ARCH PACKAGE_EXTRA_ARCHS" + python package_prepare_pkgdata() { import copy import glob
@@ -47,7 +49,7 @@ python package_prepare_pkgdata() { # Detect bitbake -b usage nodeps = d.getVar("BB_LIMITEDDEPS") or False if nodeps: - staging_package_populate_pkgdata_dir(pkgdatadir, d) + package_populate_pkgdata_dir(pkgdatadir, d) return start = None
Signature · 1 line
2.43.0
PATCH 2/7 staging: add 'vardepsexclude' to 'staging_populate_sysroot_dir'
In reply to Adam BlankIt will be needed when 'extend_recipe_sysroot' is no longer ignored when calculating signatures.
--- meta/classes-global/staging.bbclass | 2 ++ 1 file changed, 2 insertions(+)
meta/classes-global/staging.bbclass +2 -0
@@ -248,6 +248,8 @@ def staging_populate_sysroot_dir(targetsysroot, nativesysroot, native, d): for p in sorted(postinsts): bb.note("Running postinst {}, output:\n{}".format(p, subprocess.check_output(p, shell=True, stderr=subprocess.STDOUT))) +staging_populate_sysroot_dir[vardepsexclude] += "PACKAGE_EXTRA_ARCHS" + # # Manifests here are complicated. The main sysroot area has the unpacked sstate # which us unrelocated and tracked by the main sstate manifests. Each recipe
Signature · 1 line
2.43.0
PATCH 3/7 staging: add 'extend_recipe_sysroot' to 'vardepsexclude'
In reply to Adam Blank'do_prepare_recipe_sysroot' did not declare this explicitly, but made use of the fact, that 'extend_recipe_sysroot' is ignored on the base hash level.
--- meta/classes-global/staging.bbclass | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
meta/classes-global/staging.bbclass +2 -1
@@ -645,12 +645,13 @@ python extend_recipe_sysroot() { bb.utils.unlockfile(lock) } -extend_recipe_sysroot[vardepsexclude] += "MACHINE_ARCH PACKAGE_EXTRA_ARCHS SDK_ARCH BUILD_ARCH SDK_OS BB_TASKDEPDATA" +extend_recipe_sysroot[vardepsexclude] += "BB_TASKDEPDATA" do_prepare_recipe_sysroot[deptask] = "do_populate_sysroot" python do_prepare_recipe_sysroot () { bb.build.exec_func("extend_recipe_sysroot", d) } +do_prepare_recipe_sysroot[vardepsexclude] += "extend_recipe_sysroot" addtask do_prepare_recipe_sysroot before do_configure after do_fetch python staging_taskhandler() {
Signature · 1 line
2.43.0
PATCH 4/7 cross: add 'extend_recipe_sysroot' to 'vardepsexclude'
In reply to Adam Blank'do_addto_recipe_sysroot' did not declare this explicitly, but made use of the fact, that 'extend_recipe_sysroot' is ignored on the base hash level.
--- meta/classes-recipe/cross.bbclass | 1 + 1 file changed, 1 insertion(+)
meta/classes-recipe/cross.bbclass +1 -0
@@ -98,6 +98,7 @@ python do_addto_recipe_sysroot () { } addtask addto_recipe_sysroot after do_populate_sysroot do_addto_recipe_sysroot[deptask] = "do_populate_sysroot" +do_addto_recipe_sysroot[vardepsexclude] += "extend_recipe_sysroot" PATH:prepend = "${COREBASE}/scripts/cross-intercept:"
Signature · 1 line
2.43.0
PATCH 5/7 native: add 'extend_recipe_sysroot' to 'vardepsexclude'
In reply to Adam Blank'do_addto_recipe_sysroot' did not declare this explicitly, but made use of the fact, that 'extend_recipe_sysroot' is ignored on the base hash level.
--- meta/classes-recipe/native.bbclass | 1 + 1 file changed, 1 insertion(+)
meta/classes-recipe/native.bbclass +1 -0
@@ -176,6 +176,7 @@ python do_addto_recipe_sysroot () { } addtask addto_recipe_sysroot after do_populate_sysroot do_addto_recipe_sysroot[deptask] = "do_populate_sysroot" +do_addto_recipe_sysroot[vardepsexclude] += "extend_recipe_sysroot" inherit nopackages
Signature · 1 line
2.43.0
PATCH 6/7 wic-tool: add 'extend_recipe_sysroot' to 'vardepsexclude'
In reply to Adam Blank'do_build_sysroot' did not declare this explicitly,
but made use of the fact, that 'extend_recipe_sysroot' is
ignored on the base hash level.
--- meta/recipes-core/meta/wic-tools.bb | 1 + 1 file changed, 1 insertion(+)
meta/recipes-core/meta/wic-tools.bb +1 -0
@@ -27,3 +27,4 @@ python do_build_sysroot () { bb.build.exec_func("extend_recipe_sysroot", d) } addtask do_build_sysroot after do_prepare_recipe_sysroot before do_build +do_build_sysroot[vardepsexclude] += "extend_recipe_sysroot"
Signature · 1 line
2.43.0
PATCH 7/7 bitbake.conf: remove 'extend_recipe_sysroot' from BB_HASHEXCLUDE_COMMON
In reply to Adam Blank'extend_recipe_sysroot' can and should be handled just like any other function. Fine grained use of 'vardepsexclude' is capable of achieving the same effect as ingoring 'extend_recipe_sysroot' on the base hash level, but with a better visibility, and is a more uniform approach.
--- meta/conf/bitbake.conf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
meta/conf/bitbake.conf +1 -1
@@ -961,7 +961,7 @@ BB_HASHEXCLUDE_COMMON ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH BBSERVER DL_DI STAMPS_DIR PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE \ CCACHE_DIR EXTERNAL_TOOLCHAIN CCACHE CCACHE_NOHASHDIR LICENSE_PATH SDKPKGSUFFIX \ WARN_QA WORKDIR STAMPCLEAN PKGDATA_DIR BUILD_ARCH SSTATE_PKGARCH \ - BB_WORKERCONTEXT BB_LIMITEDDEPS BB_UNIHASH extend_recipe_sysroot DEPLOY_DIR \ + BB_WORKERCONTEXT BB_LIMITEDDEPS BB_UNIHASH DEPLOY_DIR \ SSTATE_HASHEQUIV_METHOD SSTATE_HASHEQUIV_REPORT_TASKDATA \ SSTATE_HASHEQUIV_OWNER CCACHE_TOP_DIR BB_HASHSERVE GIT_CEILING_DIRECTORIES \ OMP_NUM_THREADS BB_CURRENTTASK"
Signature · 1 line
2.43.0
Quoted 12 lines · Adam Blank via lists.openembedded.org
It has lain there for a while, obscuring certain aspects of how task signatures are calculated.
This changeset does not change the current status quo, but rather changes the way in which it is implemented. It can be followed by changes which further refine what is excluded and how, from those tasks which refer to '
extend_recipe_sysroot'.Related discussion:
https://lore.kernel.org/openembedded-core/aa893eb861bf4dd18e3e84e6bbebdee3b0367d1b.camel@linuxfoundation.org/Signed-off-by Adam Blank <[email protected]>
Quoted 10 lines
---
Adam Blank (7):
package_pkgdata: fix typo to stop calling undefined function
staging: add 'vardepsexclude' to 'staging_populate_sysroot_dir'
staging: add 'extend_recipe_sysroot' to 'vardepsexclude'
cross: add 'extend_recipe_sysroot' to 'vardepsexclude'
native: add 'extend_recipe_sysroot' to 'vardepsexclude'
wic-tool: add 'extend_recipe_sysroot' to 'vardepsexclude' bitbake.conf: remove 'extend_recipe_sysroot' fromBB_HASHEXCLUDE_COMMON
This change does have much wider impact than it would first appear. Even just looking at one of the first tasks in a build, quilt- native:do_fetch, generating sig data before and after these patches shows:
bitbake-diffsigs tmp/stamps/x86_64-linux/quilt-native/0.69.do_fetch.sigdata.*
NOTE: Starting bitbake server...basehash_ignore_vars changed from 'frozenset({'LOGNAME', 'BUILD_ARCH', 'PARALLEL_MAKE', 'USER', 'PRSERV_LOCKDOWN', 'BB_WORKERCONTEXT', 'RUST_BUILD_SYS', 'THISDIR', 'COREBASE', 'EXTERNAL_TOOLCHAIN', 'WARN_QA', 'FILESPATH', 'CCACHE_NOHASHDIR', 'BBPATH', 'SHELL', 'STAMPCLEAN', 'CCACHE', 'DEPLOY_DIR', 'LICENSE_PATH', 'OMP_NUM_THREADS', 'SSTATE_PKGARCH', 'STAGING_DIR_HOST', 'PRSERV_HOST', 'BUILDHISTORY_DIR', 'SOURCE_DATE_EPOCH', 'BBSERVER', 'BB_LIMITEDDEPS', 'HOME', 'BB_HASHSERVE', 'BB_TASKHASH', 'PATH', 'STAGING_DIR_TARGET', 'STAMPS_DIR', 'GIT_CEILING_DIRECTORIES', 'PRSERV_DUMPFILE', 'extend_recipe_sysroot', 'RUST_HOST_SYS', 'TMPDIR', 'CCACHE_DIR', 'SSTATE_HASHEQUIV_OWNER', 'DL_DIR', 'FILESEXTRAPATHS', 'SSTATE_DIR', 'FILE', 'PWD', 'SSTATE_HASHEQUIV_METHOD', 'FILE_DIRNAME', 'PSEUDO_INCLUDE_PATHS', 'WORKDIR', 'CCACHE_TOP_DIR', 'PRSERV_DUMPDIR', 'SDKPKGSUFFIX', 'SSTATE_HASHEQUIV_REPORT_TASKDATA', 'BB_CURRENTTASK', 'PKGDATA_DIR', 'RUST_TARGET_SYS', 'BB_UNIHASH'})' to 'frozenset({'LOGNAME', 'BUILD_ARCH', 'PARALLEL_MAKE', 'USER', 'PRSERV_LOCKDOWN', 'BB_WORKERCONTEXT', 'RUST_BUILD_SYS', 'THISDIR', 'COREBASE', 'EXTERNAL_TOOLCHAIN', 'WARN_QA', 'FILESPATH', 'CCACHE_NOHASHDIR', 'BBPATH', 'SHELL', 'STAMPCLEAN', 'CCACHE', 'DEPLOY_DIR', 'LICENSE_PATH', 'OMP_NUM_THREADS', 'SSTATE_PKGARCH', 'STAGING_DIR_HOST', 'PRSERV_HOST', 'BUILDHISTORY_DIR', 'SOURCE_DATE_EPOCH', 'BBSERVER', 'BB_LIMITEDDEPS', 'HOME', 'BB_HASHSERVE', 'BB_TASKHASH', 'PATH', 'STAGING_DIR_TARGET', 'STAMPS_DIR', 'GIT_CEILING_DIRECTORIES', 'PRSERV_DUMPFILE', 'RUST_HOST_SYS', 'TMPDIR', 'CCACHE_DIR', 'SSTATE_HASHEQUIV_OWNER', 'DL_DIR', 'FILESEXTRAPATHS', 'SSTATE_DIR', 'FILE', 'PWD', 'SSTATE_HASHEQUIV_METHOD', 'FILE_DIRNAME', 'PSEUDO_INCLUDE_PATHS', 'WORKDIR', 'CCACHE_TOP_DIR', 'PRSERV_DUMPDIR', 'SDKPKGSUFFIX', 'SSTATE_HASHEQUIV_REPORT_TASKDATA', 'BB_CURRENTTASK', 'PKGDATA_DIR', 'RUST_TARGET_SYS', 'BB_UNIHASH'})' changed items: frozenset({'extend_recipe_sysroot'})
Task dependencies changed from:
['OE_SHARED_UMASK', 'PV', 'SRC_URI', 'SRC_URI[sha256sum]', 'base_do_fetch', 'do_fetch[network]', 'do_fetch[umask]', 'fetcher_hashes_dummyfunc']
to:
['BB_RUNTASK', 'COMPONENTS_DIR', 'OE_SHARED_UMASK', 'PN', 'PV', 'RECIPE_SYSROOT', 'RECIPE_SYSROOT_MANIFEST_SUBDIR', 'RECIPE_SYSROOT_NATIVE', 'SRC_URI', 'SRC_URI[sha256sum]', 'SSTATETASKS', 'STAGING_DIR', 'base_do_fetch', 'do_fetch[network]', 'do_fetch[umask]', 'extend_recipe_sysroot', 'fetcher_hashes_dummyfunc', 'oe.path.remove', 'oe.sstatesig.find_sstate_manifest', 'oe.utils.get_multilib_datastore', 'setscene_depvalid', 'sstate_clean_manifest', 'staging_copydir', 'staging_copyfile', 'staging_populate_sysroot_dir', 'staging_processfixme']
basehash changed from fa1058e1378b6cf06cae11a9e251b7ada2d7f81a0de75644c1f693ac842576c0 to a11321c2775abee7697344dca7a79211c5de79a5de60f1e767ed9556ab434982
Dependency on variable BB_RUNTASK was added
Dependency on variable COMPONENTS_DIR was added
Dependency on variable PN was added
Dependency on variable RECIPE_SYSROOT was added
Dependency on variable RECIPE_SYSROOT_MANIFEST_SUBDIR was added
Dependency on variable RECIPE_SYSROOT_NATIVE was added
Dependency on variable SSTATETASKS was added
Dependency on variable STAGING_DIR was added
Dependency on variable extend_recipe_sysroot was added
Dependency on variable oe.path.remove was added
Dependency on variable oe.sstatesig.find_sstate_manifest was added
Dependency on variable oe.utils.get_multilib_datastore was added
Dependency on variable setscene_depvalid was added
Dependency on variable sstate_clean_manifest was added
Dependency on variable staging_copydir was added
Dependency on variable staging_copyfile was added
Dependency on variable staging_populate_sysroot_dir was added
Dependency on variable staging_processfixme was added
which is a lot of non-trivial change and a lot of extra data in the sigdata file.
Do we really want all of this data to be added into every signature computation?
Are we 100% sure that add the newly added variable dependencies are "safe" and won't trigger sstate cache reuse issues?
Right now, I'm very nervous about this.
Cheers,
Richard
Quoted 11 lines
which is a lot of non-trivial change and a lot of extra data in the sigdata file.
Do we really want all of this data to be added into every signature computation?
Are we 100% sure that add the newly added variable dependencies are "safe" and won't trigger sstate cache reuse issues?
Well, obviously such increase in the visibility of dependencies is not a comforting or welcome thing, but let's not forget, that those dependencies truly are there :-)
I'd ask the opposite question: given the extent and nature of the relations exposed by this change, do we want to keep wholesale hiding them with the current implementation?
Or in other words, are those particular dependencies (stemming from 'extend_recipe_sysroot') so irrelevant for the overall signature and sstate management, that it is justifiable to keep ignoring them in such a unique way and on such a fundamental level?
Unless it is truly a desirable situation and the whole subject can be dismissed, what would be the recommended way to tackle it?
Quoted 8 lines
Right now, I'm very nervous about this.
Cheers,
Richard
Quoted 23 lines · Adam Blank via lists.openembedded.org
Quoted 10 lines
which is a lot of non-trivial change and a lot of extra data in the sigdata file.
Do we really want all of this data to be added into every signature computation?
Are we 100% sure that add the newly added variable dependencies are "safe" and won't trigger sstate cache reuse issues?
Well, obviously such increase in the visibility of dependencies is not a comforting or welcome thing, but let's not forget, that those dependencies truly are there :-)
I'd ask the opposite question: given the extent and nature of the relations exposed by this change, do we want to keep wholesale hiding them with the current implementation?
Or in other words, are those particular dependencies (stemming from 'extend_recipe_sysroot') so irrelevant for the overall signature and sstate management, that it is justifiable to keep ignoring them in such a unique way and on such a fundamental level?
Quoted 2 lines
Unless it is truly a desirable situation and the whole subject can be dismissed, what would be the recommended way to tackle it?
I'm mentioning this as I wanted to explain the delay, and why I'm hesitant about this.
Not every piece of build information is put into the hashes and we previously made a conscious decision that extend_recipe_sysroot and anything hidden behind it were out of scope for the hashes. Looking at at the list of new dependencies, I can see why we did that.
Does having these dependencies help the average user? or does it waste space in the sigdata files and complicate the dependencies? Right now I'm torn and I'm not sure which way we should proceed.
I should also mention that in testing, we've had sstate mismatch issues too, e.g.
https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3747
We don't know which of the patches under test are at fault for that and the errors are non-specific, just that sstate wasn't matching. It could be something in this patchset, or it might be something else, we just can't tell and don't know.
Cheers,
Richard
Quoted 21 lines · Adam Blank via lists.openembedded.org
Quoted 9 lines
which is a lot of non-trivial change and a lot of extra data in the sigdata file.
Do we really want all of this data to be added into every signature computation?
Are we 100% sure that add the newly added variable dependencies are "safe" and won't trigger sstate cache reuse issues?
Well, obviously such increase in the visibility of dependencies is not a comforting or welcome thing, but let's not forget, that those dependencies truly are there :-)
I'd ask the opposite question: given the extent and nature of the relations exposed by this change, do we want to keep wholesale hiding them with the current implementation?
Or in other words, are those particular dependencies (stemming from 'extend_recipe_sysroot') so irrelevant for the overall signature and sstate management, that it is justifiable to keep ignoring them in such a unique way and on such a fundamental level?
Quoted 2 lines
Unless it is truly a desirable situation and the whole subject can be dismissed, what would be the recommended way to tackle it?
FWIW I was finally able to isolate one example of a change here which isn't desired.
For qemu-helper-native, we saw do_fetch change signatures when it shouldn't really have done. The change was:
basehash changed from ba2bcd325c42f37db9aa87567dbf0ccd217f4f12910558975a9e52e642dd426c to bd5120be022a43e6fd1273a14f0ceda3b72847308d4c4e65bfe91f9fea21683e
Variable SSTATETASKS value changed:
"do_create_package_spdx do_create_recipe_sbom do_create_recipe_spdx do_create_spdx do_deploy_source_date_epoch do_package do_package_qa [-do_package_write_deb-] do_package_write_ipk [-do_package_write_rpm-] do_packagedata do_populate_lic do_populate_sysroot do_recipe_qa"
which was probably a PACKAGE_CLASSES change in configuration triggering it.
Whether this is the only undesired change in dependencies, I'm not sure but we do need to track these kinds of issues down.
Cheers,
Richard
Quoted 41 lines · Richard Purdie
Quoted 21 lines · Adam Blank via lists.openembedded.org
Quoted 9 lines
which is a lot of non-trivial change and a lot of extra data in the sigdata file.
Do we really want all of this data to be added into every signature computation?
Are we 100% sure that add the newly added variable dependencies are "safe" and won't trigger sstate cache reuse issues?
Well, obviously such increase in the visibility of dependencies is not a comforting or welcome thing, but let's not forget, that those dependencies truly are there :-)
I'd ask the opposite question: given the extent and nature of the relations exposed by this change, do we want to keep wholesale hiding them with the current implementation?
Or in other words, are those particular dependencies (stemming from 'extend_recipe_sysroot') so irrelevant for the overall signature and sstate management, that it is justifiable to keep ignoring them in such a unique way and on such a fundamental level?Quoted 2 lines
Unless it is truly a desirable situation and the whole subject can be dismissed, what would be the recommended way to tackle it?
FWIW I was finally able to isolate one example of a change here which isn't desired.
For qemu-helper-native, we saw
do_fetchchange signatures when it shouldn't really have done. The change was:basehash changed from
ba2bcd325c42f37db9aa87567dbf0ccd217f4f12910558975a9e52e642dd426ctobd5120be022a43e6fd1273a14f0ceda3b72847308d4c4e65bfe91f9fea21683eVariable SSTATETASKS value changed:
"do_create_package_spdxdo_create_recipe_sbomdo_create_recipe_spdxdo_create_spdxdo_deploy_source_date_epochdo_packagedo_package_qa[-do_package_write_deb-]do_package_write_ipk[-do_package_write_rpm-]do_packagedatado_populate_licdo_populate_sysrootdo_recipe_qa"
Quoted 6 lines
which was probably a
PACKAGE_CLASSESchange in configuration triggering it.Whether this is the only undesired change in dependencies, I'm not sure but we do need to track these kinds of issues down.
I'd like to help somehow, but I don't have at my disposal a sufficiently powerful setup.
Please let me know if I can be of any assistance.
Thanks,
Adam
Quoted 5 lines
Cheers,
Richard
Quoted 22 lines · Adam Blank via lists.openembedded.org
It has lain there for a while, obscuring certain aspects of how task signatures are calculated.
This changeset does not change the current status quo, but rather changes the way in which it is implemented. It can be followed by changes which further refine what is excluded and how, from those tasks which refer to '
extend_recipe_sysroot'.Related discussion:
https://lore.kernel.org/openembedded-core/aa893eb861bf4dd18e3e84e6bbebdee3b0367d1b.camel@linuxfoundation.org/Signed-off-by Adam Blank <[email protected]>---
Adam Blank (7):
package_pkgdata: fix typo to stop calling undefined function
staging: add 'vardepsexclude' to 'staging_populate_sysroot_dir'
staging: add 'extend_recipe_sysroot' to 'vardepsexclude'
cross: add 'extend_recipe_sysroot' to 'vardepsexclude'
native: add 'extend_recipe_sysroot' to 'vardepsexclude'
wic-tool: add 'extend_recipe_sysroot' to 'vardepsexclude' bitbake.conf: remove 'extend_recipe_sysroot' fromBB_HASHEXCLUDE_COMMON
Quoted 7 lines
meta/classes-global/package_pkgdata.bbclass | 4 +++- meta/classes-global/staging.bbclass | 5 ++++- meta/classes-recipe/cross.bbclass | 1 + meta/classes-recipe/native.bbclass | 1 + meta/conf/bitbake.conf | 2 +- meta/recipes-core/meta/wic-tools.bb | 1 +
6 files changed, 11 insertions(+), 3 deletions(-)
I finally found a bit of time to look at this. I took some of it, I ended up reworking some of the patches as they needed rewording to justify them and explain the changes being made.
I didn't take the extend_recipe_sysroot removal from bitbake.conf as I'm still not convinced we have all the issues that might cause identified. The function name typo and the incorrect vardepsexclude are fixed though (or will be once a couple of patches pass testing/review and merge).
Cheers,
Richard
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 35 lines · Richard Purdie via lists.openembedded.org
Quoted 22 lines · Adam Blank via lists.openembedded.org
It has lain there for a while, obscuring certain aspects of how task signatures are calculated.
This changeset does not change the current status quo, but rather changes the way in which it is implemented. It can be followed by changes which further refine what is excluded and how, from those tasks which refer to '
extend_recipe_sysroot'.Related discussion:
https://lore.kernel.org/openembedded-core/aa893eb861bf4dd18e3e84e6bbebdee3b0367d1b.camel@linuxfoundation.org/Signed-off-by Adam Blank <[email protected]>---
Adam Blank (7):
package_pkgdata: fix typo to stop calling undefined function
staging: add 'vardepsexclude' to 'staging_populate_sysroot_dir'
staging: add 'extend_recipe_sysroot' to 'vardepsexclude'
cross: add 'extend_recipe_sysroot' to 'vardepsexclude'
native: add 'extend_recipe_sysroot' to 'vardepsexclude'
wic-tool: add 'extend_recipe_sysroot' to 'vardepsexclude' bitbake.conf: remove 'extend_recipe_sysroot' fromBB_HASHEXCLUDE_COMMONQuoted 7 lines
meta/classes-global/package_pkgdata.bbclass | 4 +++- meta/classes-global/staging.bbclass | 5 ++++- meta/classes-recipe/cross.bbclass | 1 + meta/classes-recipe/native.bbclass | 1 + meta/conf/bitbake.conf | 2 +- meta/recipes-core/meta/wic-tools.bb | 1 +
6 files changed, 11 insertions(+), 3 deletions(-)I finally found a bit of time to look at this. I took some of it, I ended up reworking some of the patches as they needed rewording to justify them and explain the changes being made.
Quoted 5 lines
I didn't take the
extend_recipe_sysrootremoval from bitbake.conf as I'm still not convinced we have all the issues that might cause identified. The function name typo and the incorrect vardepsexclude are fixed though (or will be once a couple of patches pass testing/review and merge).
I still wasn't 100% happy with this so I did go a little further and hacked up the attached script. It does have one hardcoded path but you can get the idea of what I'm trying to do.
You can start with a clean TMPDIR and run "bitbake world -S none". You can then edit bitbake.conf and remove extend_recipe_sysroot from BB_HASHEXCLUDE_COMMON and run that command a second time.
If you them run the attached script in tmp/stamps, after a bit of processing, you will see:
Added {'staging_processfixme[func]', 'staging_copydir', 'BB_RUNTASK', 'oe.utils.get_multilib_datastore', 'sstate_clean_manifest', 'extend_recipe_sysroot', 'DEFAULTTUNE', 'RECIPE_SYSROOT', 'oe.sstatesig.find_sstate_manifest', 'TUNE_PKGARCH:tune-x86-64-v3', 'STAGING_DIR', 'extend_recipe_sysroot[func]', 'RECIPE_SYSROOT_NATIVE', 'COMPONENTS_DIR', 'staging_copydir[func]', 'staging_copyfile[func]', 'sstate_clean_manifest[func]', 'staging_populate_sysroot_dir', 'setscene_depvalid', 'PN', 'oe.path.remove', 'TUNE_PKGARCH', 'RECIPE_SYSROOT_MANIFEST_SUBDIR', 'TARGET_ARCH', 'staging_populate_sysroot_dir[func]', 'staging_processfixme', 'setscene_depvalid[func]', 'staging_copyfile'} Removed set()
That means those variables were added as dependencies somewhere in the build for at least one task. This was basically my worry with the patch. We need to work out where/why those are being added and triage them in order to be able to remove extend_recipe_sysroot from the hash exclusion. Some of them may be acceptable, some of them may not, a lot depends on context.
I'm still not sure it is worth the work in changing that, but we do now at least have a way of checking the effect of such a change.
Cheers,
Richard
#!/usr/bin/env python3
import sys
import os
import re
sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(sys.argv[0])), 'lib')) sys.path.insert(0, '/media/build/poky/bitbake/lib')
from bb import __version__ as libbb_version
__version__ = "2.19.0"
if __version__ != libbb_version:
sys.exit("Bitbake core library version and program version mismatch!")
import bb
import bb.compress.zstd
import json
class SetEncoder(json.JSONEncoder):
def default(self, obj):
if isinstance(obj, set) or isinstance(obj, frozenset):
return dict(_set_object=list(sorted(obj)))
return json.JSONEncoder.default(self, obj)
def SetDecoder(dct):
if '_set_object' in dct:
return frozenset(dct['_set_object'])
return dct
def dict_diff(a, b, ignored_vars=set()):
sa = set(a.keys())
sb = set(b.keys())
common = sa & sb
changed = set()
for i in common:
if a[i] != b[i] and i not in ignored_vars:
changed.add(i)
added = sb - sa
removed = sa - sb
return changed, added, removed
def get_data(fn):
try:
with bb.compress.zstd.open(fn, "rt", encoding="utf-8", num_threads=1) as f:
a_data = json.load(f, object_hook=SetDecoder)
except (TypeError, OSError) as err:
bb.error("Failed to open sigdata file '%s': %s" % (fn, str(err))) raise err
return a_data
pattern = re.compile(r'^[\d.]+\.(do_[a-z_]+)\.sigdata\..*')
tasksplit = {}
for root, dirs, files in os.walk("."):
for f in files:
if ".sigdata." not in f:
continue
m = pattern.match(f)
if not m:
continue
task = m.group(1) + "-" + root
if task not in tasksplit:
tasksplit[task] = []
tasksplit[task].append(root + "/" + f)
adds = set()
removes = set()
for task in tasksplit:
files = tasksplit[task]
if len(files) == 1:
continue
if len(files) != 2:
print("Invalid number of sigdata files %s" % files) continue
files = sorted(files, key=lambda f: os.path.getmtime(f))
a_data = get_data(files[0])
b_data = get_data(files[1])
changed, added, removed = dict_diff(a_data['gendeps'], b_data['gendeps'], a_data['basehash_ignore_vars'] & b_data['basehash_ignore_vars']) adds.update(added)
removes.update(removed)
print("Added %s" % adds)print("Removed %s" % removes)
Quoted 77 lines · Richard Purdie
Quoted 1 line · Richard Purdie via lists.openembedded.org
On Sat, 2026-04-18 at 20:34 +0200, Adam Blank via lists.openembedded.org
Quoted 10 lines · wrote:
Quoted 10 lines
It has lain there for a while, obscuring certain aspects of how task signatures are calculated.
This changeset does not change the current status quo, but rather changes the way in which it is implemented. It can be followed by changes which further refine what is excluded and how, from those tasks which refer to '
extend_recipe_sysroot'.Related discussion:
Quoted 11 lines
Quoted 11 lines
Signed-off-by Adam Blank <[email protected]>---
Adam Blank (7):
package_pkgdata: fix typo to stop calling undefined function
staging: add 'vardepsexclude' to 'staging_populate_sysroot_dir'
staging: add 'extend_recipe_sysroot' to 'vardepsexclude'
cross: add 'extend_recipe_sysroot' to 'vardepsexclude'
native: add 'extend_recipe_sysroot' to 'vardepsexclude'
wic-tool: add 'extend_recipe_sysroot' to 'vardepsexclude' bitbake.conf: remove 'extend_recipe_sysroot' from
BB_HASHEXCLUDE_COMMONQuoted 13 lines
Quoted 8 lines
meta/classes-global/package_pkgdata.bbclass | 4 +++- meta/classes-global/staging.bbclass | 5 ++++- meta/classes-recipe/cross.bbclass | 1 + meta/classes-recipe/native.bbclass | 1 + meta/conf/bitbake.conf | 2 +- meta/recipes-core/meta/wic-tools.bb | 1 + 6 files changed, 11 insertions(+), 3 deletions(-)I finally found a bit of time to look at this. I took some of it, I ended up reworking some of the patches as they needed rewording to justify them and explain the changes being made.
Quoted 5 lines
I didn't take the
extend_recipe_sysrootremoval from bitbake.conf as I'm still not convinced we have all the issues that might cause identified. The function name typo and the incorrect vardepsexclude are fixed though (or will be once a couple of patches pass testing/review and merge).I still wasn't 100% happy with this so I did go a little further and hacked up the attached script. It does have one hardcoded path but you can get the idea of what I'm trying to do.
You can start with a clean TMPDIR and run "bitbake world -S none". You can then edit bitbake.conf and remove
extend_recipe_sysrootfromBB_HASHEXCLUDE_COMMONand run that command a second time.If you them run the attached script in tmp/stamps, after a bit of processing, you will see:
Added {'staging_processfixme[func]', '
staging_copydir', 'BB_RUNTASK', 'oe.utils.get_multilib_datastore', 'sstate_clean_manifest', 'extend_recipe_sysroot', 'DEFAULTTUNE', 'RECIPE_SYSROOT', 'oe.sstatesig.find_sstate_manifest', 'TUNE_PKGARCH:tune-x86-64-v3', 'STAGING_DIR', 'extend_recipe_sysroot[func]', 'RECIPE_SYSROOT_NATIVE', 'COMPONENTS_DIR', 'staging_copydir[func]', 'staging_copyfile[func]', 'sstate_clean_manifest[func]', 'staging_populate_sysroot_dir', 'setscene_depvalid', 'PN', 'oe.path.remove', 'TUNE_PKGARCH', 'RECIPE_SYSROOT_MANIFEST_SUBDIR', 'TARGET_ARCH', 'staging_populate_sysroot_dir[func]', 'staging_processfixme', 'setscene_depvalid[func]', 'staging_copyfile'}
Removedset()That means those variables were added as dependencies somewhere in the build for at least one task. This was basically my worry with the patch. We need to work out where/why those are being added and triage them in order to be able to remove
extend_recipe_sysrootfrom the hash exclusion. Some of them may be acceptable, some of them may not, a lot depends on context.
Quoted 3 lines
I'm still not sure it is worth the work in changing that, but we do now at least have a way of checking the effect of such a change.
I've allowed myself to have an AI agent run this and one other analysis of this subject (hopefully this is not prohibited or frowned upon) and I think there might be an attractive compromise.
What could be done is the following set of changes, which would assure, that:
- the 'extend_recipe_sysroot' exclusion from the signature calculation would remain active
- this exclusion would be closer to the code which it logically complements (staging.bbclass) - this would improve the comprehensibility of this design choice
- other cases (native.bbclass, cross.bbclass, et al) would be more transparent about their relation with 'extend_recipe_sysroot'
--- meta/classes-global/staging.bbclass | 2 ++ meta/classes-recipe/cross.bbclass | 1 + meta/classes-recipe/native.bbclass | 1 + meta/conf/bitbake.conf | 2 +- meta/recipes-core/meta/wic-tools.bb | 1 + meta/recipes-devtools/gcc/gcc-runtime.inc | 1 + 6 files changed, 7 insertions(+), 1 deletion(-)
(quoted diff) +0 -0
b/meta/classes-global/staging.bbclass
index f0b3c6eedc..5547980ee4 100644
meta/classes-global/staging.bbclass +2 -0
@@ -647,6 +647,7 @@ python extend_recipe_sysroot() { extend_recipe_sysroot[vardepsexclude] += "BB_TASKDEPDATA SSTATETASKS" do_prepare_recipe_sysroot[deptask] = "do_populate_sysroot" +do_prepare_recipe_sysroot[vardepsexclude] += "extend_recipe_sysroot" python do_prepare_recipe_sysroot () { bb.build.exec_func("extend_recipe_sysroot", d) }
@@ -665,6 +666,7 @@ python staging_taskhandler() { deps = d.getVarFlag(task, "depends") if task == "do_configure" or (deps and "populate_sysroot" in deps): d.prependVarFlag(task, "prefuncs", "extend_recipe_sysroot ") + d.appendVarFlag(task, "vardepsexclude", "
extend_recipe_sysroot")
}
staging_taskhandler[eventmask] = "bb.event.RecipeTaskPreProcess" addhandler staging_taskhandler
(quoted diff) +0 -0
b/meta/classes-recipe/cross.bbclass
index 82a49dc8f5..bceee5d6e6 100644
meta/classes-recipe/cross.bbclass +1 -0
@@ -98,6 +98,7 @@ python do_addto_recipe_sysroot () { } addtask addto_recipe_sysroot after do_populate_sysroot do_addto_recipe_sysroot[deptask] = "do_populate_sysroot" +do_addto_recipe_sysroot[vardepsexclude] += "extend_recipe_sysroot" PATH:prepend = "${COREBASE}/scripts/cross-intercept:"
(quoted diff) +0 -0
b/meta/classes-recipe/native.bbclass
index 8aa86f9cb6..b357b2da7b 100644
meta/classes-recipe/native.bbclass +1 -0
@@ -170,6 +170,7 @@ python do_addto_recipe_sysroot () { } addtask addto_recipe_sysroot after do_populate_sysroot do_addto_recipe_sysroot[deptask] = "do_populate_sysroot" +do_addto_recipe_sysroot[vardepsexclude] += "extend_recipe_sysroot" inherit nopackages
meta/conf/bitbake.conf +0 -0
@@ -964,7 +964,7 @@ BB_HASHEXCLUDE_COMMON ?= "TMPDIR FILE PATH PWD
BB_TASKHASH BBPATH BBSERVER DL_DI
STAMPS_DIR PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWNPARALLEL_MAKE \
CCACHE_DIR EXTERNAL_TOOLCHAIN CCACHE CCACHE_NOHASHDIR LICENSE_PATH
SDKPKGSUFFIX \
WARN_QA WORKDIR STAMPCLEAN PKGDATA_DIR BUILD_ARCH SSTATE_PKGARCH \- BB_WORKERCONTEXT BB_LIMITEDDEPS BB_UNIHASH extend_recipe_sysrootDEPLOY_DIR \
+ BB_WORKERCONTEXT BB_LIMITEDDEPS BB_UNIHASH DEPLOY_DIR \
SSTATE_HASHEQUIV_METHOD SSTATE_HASHEQUIV_REPORT_TASKDATA \
SSTATE_HASHEQUIV_OWNER CCACHE_TOP_DIR BB_HASHSERVEGIT_CEILING_DIRECTORIES \
OMP_NUM_THREADS BB_CURRENTTASK"
meta/recipes-core/meta/ renamed from meta/recipes-core/meta/wic-tools.bb +0 -0
wic-tools.bb
index 823dbe6db6..9a3cf8f371 100644
meta/recipes-core/meta/wic-tools.bb +1 -0
@@ -28,4 +28,5 @@ RM_WORK_EXCLUDE += "${PN}" python do_build_sysroot () { bb.build.exec_func("extend_recipe_sysroot", d) } +do_build_sysroot[vardepsexclude] += "extend_recipe_sysroot" addtask do_build_sysroot after do_prepare_recipe_sysroot before do_build
(quoted diff) +0 -0
b/meta/recipes-devtools/gcc/gcc-runtime.inc
index e305180c64..15ae9c337f 100644
meta/recipes-devtools/gcc/gcc-runtime.inc +0 -0
@@ -309,6 +309,7 @@ do_check[depends] += "virtual/libc:do_populate_sysroot" do_check[depends] += "${@'qemu-native:do_populate_sysroot' if "user" in
d.getVar('TOOLCHAIN_TEST_TARGET') else ''}"
# extend the recipe sysroot to include the built libraries (for qemu
usermode)
do_check[prefuncs] += "extend_recipe_sysroot"
+do_check[vardepsexclude] += "extend_recipe_sysroot"
do_check[prefuncs] += "check_prepare"
do_check[dirs] = "${WORKDIR}/dejagnu ${B}"
do_check[nostamp] = "1"
--
2.34.1Thanks,
Adam
Quoted 5 lines
Cheers,
Richard