classes/native: Propagate dependencies to outhash
RFC 1/1 classes/native: Propagate dependencies to outhash
Native task outputs are directly run on the target (host) system after being built. Even if the output of a native recipe doesn't change, a change in one of its dependencies may cause a change in the output it generates (e.g. rpm output depends on the output of its dependent zstd library).
This can cause poor interactions with hash equivalence, since this recipes output-changing dependency is "hidden" and downstream task only see that this recipe has the same outhash and therefore is equivalent. This can result in different output in different cases.
To resolve this, unhide the output-changing dependency by adding it's unihash to this tasks outhash calculation. Unfortunately, don't know specifically know which dependencies are output-changing, so we have to add all of them.
[YOCTO #14685]
--- meta/classes/native.bbclass | 31 +++++++++++++++++++++++++++++++ meta/lib/oe/sstatesig.py | 10 +++++++--- 2 files changed, 38 insertions(+), 3 deletions(-)
meta/classes/native.bbclass +31 -0
@@ -195,3 +195,34 @@ USE_NLS = "no" RECIPERDEPTASK = "do_populate_sysroot" do_populate_sysroot[rdeptask] = "${RECIPERDEPTASK}" + +# +# Native task outputs are directly run on the target (host) system after being +# built. Even if the output of this recipe doesn't change, a change in one of +# its dependencies may cause a change in the output it generates (e.g. rpm +# output depends on the output of its dependent zstd library). +# +# This can cause poor interactions with hash equivalence, since this recipes +# output-changing dependency is "hidden" and downstream task only see that this +# recipe has the same outhash and therefore is equivalent. This can result in +# different output in different cases. +# +# To resolve this, unhide the output-changing dependency by adding its unihash +# to this tasks outhash calculation. Unfortunately, don't know specifically +# know which dependencies are output-changing, so we have to add all of them. +# +python native_add_do_populate_sysroot_deps () { + current_task = "do_" + d.getVar("BB_CURRENTTASK") + if current_task != "do_populate_sysroot": + return + + taskdepdata = d.getVar("BB_TASKDEPDATA", False) + pn = d.getVar("PN") + deps = { + dep[0]:dep[6] for dep in taskdepdata.values() if + dep[1] == current_task and dep[0] != pn + } + + d.setVar("HASHEQUIV_EXTRA_SIGDATA", "\n".join("%s: %s" % (k, deps[k]) for k in sorted(deps.keys()))) +} +SSTATECREATEFUNCS += "native_add_do_populate_sysroot_deps"
meta/lib/oe/sstatesig.py +7 -3
@@ -491,7 +491,8 @@ def OEOuthashBasic(path, sigfile, task, d): if task == "package": include_timestamps = True include_root = False - extra_content = d.getVar('HASHEQUIV_HASH_VERSION') + hash_version = d.getVar('HASHEQUIV_HASH_VERSION') + extra_sigdata = d.getVar("HASHEQUIV_EXTRA_SIGDATA") filemaps = {} for m in (d.getVar('SSTATE_HASHEQUIV_FILEMAP') or '').split():
@@ -506,8 +507,11 @@ def OEOuthashBasic(path, sigfile, task, d): basepath = os.path.normpath(path) update_hash("OEOuthashBasic\n") - if extra_content: - update_hash(extra_content + "\n") + if hash_version: + update_hash(hash_version + "\n") + + if extra_sigdata: + update_hash(extra_sigdata + "\n") # It is only currently useful to get equivalent hashes for things that # can be restored from sstate. Since the sstate object is named using
Signature · 1 line
2.33.0
Quoted 1 line · Joshua Watt
Native task outputs are directly run on the target (host) system after
"target" or "host" ? the latter i suppose
Quoted 10 lines
being built. Even if the output of a native recipe doesn't change, a change in one of its dependencies may cause a change in the output it generates (e.g. rpm output depends on the output of its dependent zstd library).
This can cause poor interactions with hash equivalence, since this recipes output-changing dependency is "hidden" and downstream task only see that this recipe has the same outhash and therefore is equivalent. This can result in different output in different cases.
Quoted 5 lines
To resolve this, unhide the output-changing dependency by adding it's unihash to this tasks outhash calculation. Unfortunately, don't know specifically know which dependencies are output-changing, so we have to add all of them.
"don't know specifically know which.."
Quoted 13 lines
[YOCTO #14685]
Signed-off-by Joshua Watt <[email protected]>--- meta/classes/native.bbclass | 31 +++++++++++++++++++++++++++++++ meta/lib/oe/sstatesig.py | 10 +++++++--- 2 files changed, 38 insertions(+), 3 deletions(-)meta/classes/native.bbclass +0
-0
meta/classes/native.bbclass USE_NLS = "no" · line 200
Quoted 6 lines
meta/classes/native.bbclass +3
-0@@ -195,3 +195,34 @@ USE_NLS = "no" RECIPERDEPTASK = "do_populate_sysroot" do_populate_sysroot[rdeptask] = "${RECIPERDEPTASK}" + +# +# Native task outputs are directly run on the target (host) system after being
see above
meta/classes/native.bbclass USE_NLS = "no" · line 213
Quoted 13 lines
meta/classes/native.bbclass +13
-0+# built. Even if the output of this recipe doesn't change, a change in one of +# its dependencies may cause a change in the output it generates (e.g. rpm +# output depends on the output of its dependent zstd library). +# +# This can cause poor interactions with hash equivalence, since this recipes +# output-changing dependency is "hidden" and downstream task only see that this +# recipe has the same outhash and therefore is equivalent. This can result in +# different output in different cases. +# +# To resolve this, unhide the output-changing dependency by adding its unihash +# to this tasks outhash calculation. Unfortunately, don't know specifically +# know which dependencies are output-changing, so we have to add all of them. +#
see above
Quoted 47 lines
meta/classes/native.bbclass +15
-0+python native_add_do_populate_sysroot_deps () { + current_task = "do_" + d.getVar("BB_CURRENTTASK") + if current_task != "do_populate_sysroot": + return + + taskdepdata = d.getVar("BB_TASKDEPDATA", False) + pn = d.getVar("PN") + deps = { + dep[0]:dep[6] for dep in taskdepdata.values() if + dep[1] == current_task and dep[0] != pn + } + + d.setVar("HASHEQUIV_EXTRA_SIGDATA", "\n".join("%s: %s" % (k, deps[k]) for k in sorted(deps.keys()))) +} +SSTATECREATEFUNCS += "native_add_do_populate_sysroot_deps"meta/lib/oe/sstatesig.py +7
-4@@ -491,7 +491,8 @@ def OEOuthashBasic(path, sigfile, task, d): if task == "package": include_timestamps = True include_root = False - extra_content = d.getVar('HASHEQUIV_HASH_VERSION') + hash_version = d.getVar('HASHEQUIV_HASH_VERSION') + extra_sigdata = d.getVar("HASHEQUIV_EXTRA_SIGDATA") filemaps = {} for m in (d.getVar('SSTATE_HASHEQUIV_FILEMAP') or '').split():@@ -506,8 +507,11 @@ def OEOuthashBasic(path, sigfile, task, d): basepath = os.path.normpath(path) update_hash("OEOuthashBasic\n") - if extra_content: - update_hash(extra_content + "\n") + if hash_version: + update_hash(hash_version + "\n") + + if extra_sigdata: + update_hash(extra_sigdata + "\n") # It is only currently useful to get equivalent hashes for things that # can be restored from sstate. Since the sstate object is named using -=-=-=-=-=-=-=-=-=-=-=-
Quoted 7 lines
Links: You receive all messages sent to this group.
View/Reply Online (#160571): https://lists.openembedded.org/g/openembedded-core/message/160571
Mute This Topic: https://lists.openembedded.org/mt/88425608/4454410
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Sounds to me like something we should do.
Jacob
While we're here, recipes inheriting qemu class should do this too, but for the target dependencies.
Alex
Quoted 137 lines · Jacob Kroon
Quoted 1 line · Joshua Watt
Native task outputs are directly run on the target (host) system after
"target" or "host" ? the latter i suppose
Quoted 10 lines
being built. Even if the output of a native recipe doesn't change, a change in one of its dependencies may cause a change in the output it generates (e.g. rpm output depends on the output of its dependent zstd library).
This can cause poor interactions with hash equivalence, since this recipes output-changing dependency is "hidden" and downstream task only see that this recipe has the same outhash and therefore is equivalent. This can result in different output in different cases.
Quoted 5 lines
To resolve this, unhide the output-changing dependency by adding it's unihash to this tasks outhash calculation. Unfortunately, don't know specifically know which dependencies are output-changing, so we have to add all of them.
"don't know specifically know which.."
Quoted 13 lines
[YOCTO #14685]
Signed-off-by Joshua Watt <[email protected]>--- meta/classes/native.bbclass | 31 +++++++++++++++++++++++++++++++ meta/lib/oe/sstatesig.py | 10 +++++++--- 2 files changed, 38 insertions(+), 3 deletions(-)meta/classes/native.bbclass +0
-0meta/classes/native.bbclass USE_NLS = "no" · line 200
Quoted 6 lines
meta/classes/native.bbclass +3
-0@@ -195,3 +195,34 @@ USE_NLS = "no" RECIPERDEPTASK = "do_populate_sysroot" do_populate_sysroot[rdeptask] = "${RECIPERDEPTASK}" + +# +# Native task outputs are directly run on the target (host) systemafter being
see above
meta/classes/native.bbclass USE_NLS = "no" · line 201
Quoted 1 line
meta/classes/native.bbclass +1
-0+# built. Even if the output of this recipe doesn't change, a change inone of
meta/classes/native.bbclass USE_NLS = "no" · line 202
Quoted 1 line
meta/classes/native.bbclass +1
-0+# its dependencies may cause a change in the output it generates (e.g.rpm
meta/classes/native.bbclass USE_NLS = "no"
Quoted 3 lines
meta/classes/native.bbclass +3
-0+# output depends on the output of its dependent zstd library). +# +# This can cause poor interactions with hash equivalence, since thisrecipes
meta/classes/native.bbclass USE_NLS = "no" · line 206
Quoted 1 line
meta/classes/native.bbclass +1
-0+# output-changing dependency is "hidden" and downstream task only seethat this
meta/classes/native.bbclass USE_NLS = "no" · line 207
Quoted 1 line
meta/classes/native.bbclass +1
-0+# recipe has the same outhash and therefore is equivalent. This canresult in
meta/classes/native.bbclass USE_NLS = "no"
Quoted 3 lines
meta/classes/native.bbclass +3
-0+# different output in different cases. +# +# To resolve this, unhide the output-changing dependency by adding itsunihash
meta/classes/native.bbclass USE_NLS = "no" · line 211
Quoted 1 line
meta/classes/native.bbclass +1
-0+# to this tasks outhash calculation. Unfortunately, don't knowspecifically
meta/classes/native.bbclass USE_NLS = "no" · line 212
Quoted 1 line
meta/classes/native.bbclass +1
-0+# know which dependencies are output-changing, so we have to add all ofthem.
meta/classes/native.bbclass USE_NLS = "no" · line 213
Quoted 1 line
meta/classes/native.bbclass +1
-0+#see above
meta/classes/native.bbclass USE_NLS = "no"
Quoted 13 lines
meta/classes/native.bbclass +13
-0+python native_add_do_populate_sysroot_deps () { + current_task = "do_" + d.getVar("BB_CURRENTTASK") + if current_task != "do_populate_sysroot": + return + + taskdepdata = d.getVar("BB_TASKDEPDATA", False) + pn = d.getVar("PN") + deps = { + dep[0]:dep[6] for dep in taskdepdata.values() if + dep[1] == current_task and dep[0] != pn + } + + d.setVar("HASHEQUIV_EXTRA_SIGDATA", "\n".join("%s: %s" % (k,deps[k]) for k in sorted(deps.keys())))
Quoted 19 lines
meta/classes/native.bbclass +2
-0+} +SSTATECREATEFUNCS += "native_add_do_populate_sysroot_deps"meta/lib/oe/sstatesig.py +2
-1@@ -491,7 +491,8 @@ def OEOuthashBasic(path, sigfile, task, d): if task == "package": include_timestamps = True include_root = False - extra_content = d.getVar('HASHEQUIV_HASH_VERSION') + hash_version = d.getVar('HASHEQUIV_HASH_VERSION') + extra_sigdata = d.getVar("HASHEQUIV_EXTRA_SIGDATA") filemaps = {} for m in (d.getVar('SSTATE_HASHEQUIV_FILEMAP') or '').split():@@ -506,8 +507,11 @@ def OEOuthashBasic(path, sigfile, task, d): basepath = os.path.normpath(path)meta/lib/oe/sstatesig.py def OEOuthashBasic(path, sigfile, task, d): · line 516
Quoted 10 lines
meta/lib/oe/sstatesig.py +5
-2update_hash("OEOuthashBasic\n") - if extra_content: - update_hash(extra_content + "\n") + if hash_version: + update_hash(hash_version + "\n") + + if extra_sigdata: + update_hash(extra_sigdata + "\n") # It is only currently useful to get equivalent hashes forthings that
meta/lib/oe/sstatesig.py def OEOuthashBasic(path, sigfile, task, d): · line 517
Quoted 1 line
meta/lib/oe/sstatesig.py +0
-0# can be restored from sstate. Since the sstate object is namedusing
Quoted 5 lines
Sounds to me like something we should do.
Jacob
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#160573):
https://lists.openembedded.org/g/openembedded-core/message/160573
Mute This Topic: https://lists.openembedded.org/mt/88425608/1686489
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ [email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Quoted 16 lines · Joshua Watt
Native task outputs are directly run on the target (host) system after being built. Even if the output of a native recipe doesn't change, a change in one of its dependencies may cause a change in the output it generates (e.g. rpm output depends on the output of its dependent zstd library).
This can cause poor interactions with hash equivalence, since this recipes output-changing dependency is "hidden" and downstream task only see that this recipe has the same outhash and therefore is equivalent. This can result in different output in different cases.
To resolve this, unhide the output-changing dependency by adding it's unihash to this tasks outhash calculation. Unfortunately, don't know specifically know which dependencies are output-changing, so we have to add all of them.
Quoted 7 lines
[YOCTO #14685]
Signed-off-by Joshua Watt <[email protected]>--- meta/classes/native.bbclass | 31 +++++++++++++++++++++++++++++++ meta/lib/oe/sstatesig.py | 10 +++++++--- 2 files changed, 38 insertions(+), 3 deletions(-)
Thanks for this. I know it was an RFC but after testing over the weekend and doing some checks of my own, I've merged it. I think it is the correct solution and we don't have any other good option to fix the issue which is breaking builds on the autobuilder.
Cheers,
Richard