Skip to contentSkip to conversation

classes/native: Propagate dependencies to outhash

RFC In review Joshua Watt 4 participants 4 messages Jan 14, 2022 – Jan 17, 2022

lore
Share & tools
lore
#0 lore raw

RFC 1/1 classes/native: Propagate dependencies to outhash

Native task outputs are directly run on the target (host) system after being built. Even if the output of a native recipe doesn't change, a change in one of its dependencies may cause a change in the output it generates (e.g. rpm output depends on the output of its dependent zstd library).

This can cause poor interactions with hash equivalence, since this recipes output-changing dependency is "hidden" and downstream task only see that this recipe has the same outhash and therefore is equivalent. This can result in different output in different cases.

To resolve this, unhide the output-changing dependency by adding it's unihash to this tasks outhash calculation. Unfortunately, don't know specifically know which dependencies are output-changing, so we have to add all of them.

[YOCTO #14685]

---
 meta/classes/native.bbclass | 31 +++++++++++++++++++++++++++++++
 meta/lib/oe/sstatesig.py    | 10 +++++++---
 2 files changed, 38 insertions(+), 3 deletions(-)
meta/classes/native.bbclass +31 -0
index 76a599bc15..fc7422c5d7 100644
@@ -195,3 +195,34 @@ USE_NLS = "no"
 
 RECIPERDEPTASK = "do_populate_sysroot"
 do_populate_sysroot[rdeptask] = "${RECIPERDEPTASK}"
+
+#
+# Native task outputs are directly run on the target (host) system after being
+# built. Even if the output of this recipe doesn't change, a change in one of
+# its dependencies may cause a change in the output it generates (e.g. rpm
+# output depends on the output of its dependent zstd library).
+#
+# This can cause poor interactions with hash equivalence, since this recipes
+# output-changing dependency is "hidden" and downstream task only see that this
+# recipe has the same outhash and therefore is equivalent. This can result in
+# different output in different cases.
+#
+# To resolve this, unhide the output-changing dependency by adding its unihash
+# to this tasks outhash calculation. Unfortunately, don't know specifically
+# know which dependencies are output-changing, so we have to add all of them.
+#
+python native_add_do_populate_sysroot_deps () {
+    current_task = "do_" + d.getVar("BB_CURRENTTASK")
+    if current_task != "do_populate_sysroot":
+        return
+
+    taskdepdata = d.getVar("BB_TASKDEPDATA", False)
+    pn = d.getVar("PN")
+    deps = {
+        dep[0]:dep[6] for dep in taskdepdata.values() if
+            dep[1] == current_task and dep[0] != pn
+    }
+
+    d.setVar("HASHEQUIV_EXTRA_SIGDATA", "\n".join("%s: %s" % (k, deps[k]) for k in sorted(deps.keys())))
+}
+SSTATECREATEFUNCS += "native_add_do_populate_sysroot_deps"
meta/lib/oe/sstatesig.py +7 -3
index 038404e377..abcd96231e 100644
@@ -491,7 +491,8 @@ def OEOuthashBasic(path, sigfile, task, d):
     if task == "package":
         include_timestamps = True
         include_root = False
-    extra_content = d.getVar('HASHEQUIV_HASH_VERSION')
+    hash_version = d.getVar('HASHEQUIV_HASH_VERSION')
+    extra_sigdata = d.getVar("HASHEQUIV_EXTRA_SIGDATA")
 
     filemaps = {}
     for m in (d.getVar('SSTATE_HASHEQUIV_FILEMAP') or '').split():
@@ -506,8 +507,11 @@ def OEOuthashBasic(path, sigfile, task, d):
         basepath = os.path.normpath(path)
 
         update_hash("OEOuthashBasic\n")
-        if extra_content:
-            update_hash(extra_content + "\n")
+        if hash_version:
+            update_hash(hash_version + "\n")
+
+        if extra_sigdata:
+            update_hash(extra_sigdata + "\n")
 
         # It is only currently useful to get equivalent hashes for things that
         # can be restored from sstate. Since the sstate object is named using
Signature · 1 line
2.33.0
#1 lore raw
Quoted 1 line · Joshua Watt

Native task outputs are directly run on the target (host) system after

"target" or "host" ? the latter i suppose

Quoted 10 lines

being built. Even if the output of a native recipe doesn't change, a change in one of its dependencies may cause a change in the output it generates (e.g. rpm output depends on the output of its dependent zstd library).

This can cause poor interactions with hash equivalence, since this recipes output-changing dependency is "hidden" and downstream task only see that this recipe has the same outhash and therefore is equivalent. This can result in different output in different cases.

Quoted 5 lines

To resolve this, unhide the output-changing dependency by adding it's unihash to this tasks outhash calculation. Unfortunately, don't know specifically know which dependencies are output-changing, so we have to add all of them.

"don't know specifically know which.."

Quoted 13 lines

[YOCTO #14685]

---
 meta/classes/native.bbclass | 31 +++++++++++++++++++++++++++++++
 meta/lib/oe/sstatesig.py    | 10 +++++++---
 2 files changed, 38 insertions(+), 3 deletions(-)
meta/classes/native.bbclass +0 -0
index 76a599bc15..fc7422c5d7 100644

meta/classes/native.bbclass USE_NLS = "no" · line 200

Quoted 6 lines
meta/classes/native.bbclass +3 -0
@@ -195,3 +195,34 @@ USE_NLS = "no"
 
 RECIPERDEPTASK = "do_populate_sysroot"
 do_populate_sysroot[rdeptask] = "${RECIPERDEPTASK}"
+
+#
+# Native task outputs are directly run on the target (host) system after being

see above

meta/classes/native.bbclass USE_NLS = "no" · line 213

Quoted 13 lines
meta/classes/native.bbclass +13 -0
+# built. Even if the output of this recipe doesn't change, a change in one of
+# its dependencies may cause a change in the output it generates (e.g. rpm
+# output depends on the output of its dependent zstd library).
+#
+# This can cause poor interactions with hash equivalence, since this recipes
+# output-changing dependency is "hidden" and downstream task only see that this
+# recipe has the same outhash and therefore is equivalent. This can result in
+# different output in different cases.
+#
+# To resolve this, unhide the output-changing dependency by adding its unihash
+# to this tasks outhash calculation. Unfortunately, don't know specifically
+# know which dependencies are output-changing, so we have to add all of them.
+#

see above

Quoted 47 lines
meta/classes/native.bbclass +15 -0
+python native_add_do_populate_sysroot_deps () {
+    current_task = "do_" + d.getVar("BB_CURRENTTASK")
+    if current_task != "do_populate_sysroot":
+        return
+
+    taskdepdata = d.getVar("BB_TASKDEPDATA", False)
+    pn = d.getVar("PN")
+    deps = {
+        dep[0]:dep[6] for dep in taskdepdata.values() if
+            dep[1] == current_task and dep[0] != pn
+    }
+
+    d.setVar("HASHEQUIV_EXTRA_SIGDATA", "\n".join("%s: %s" % (k, deps[k]) for k in sorted(deps.keys())))
+}
+SSTATECREATEFUNCS += "native_add_do_populate_sysroot_deps"
meta/lib/oe/sstatesig.py +7 -4
index 038404e377..abcd96231e 100644
@@ -491,7 +491,8 @@ def OEOuthashBasic(path, sigfile, task, d):
     if task == "package":
         include_timestamps = True
         include_root = False
-    extra_content = d.getVar('HASHEQUIV_HASH_VERSION')
+    hash_version = d.getVar('HASHEQUIV_HASH_VERSION')
+    extra_sigdata = d.getVar("HASHEQUIV_EXTRA_SIGDATA")
 
     filemaps = {}
     for m in (d.getVar('SSTATE_HASHEQUIV_FILEMAP') or '').split():
@@ -506,8 +507,11 @@ def OEOuthashBasic(path, sigfile, task, d):
         basepath = os.path.normpath(path)
 
         update_hash("OEOuthashBasic\n")
-        if extra_content:
-            update_hash(extra_content + "\n")
+        if hash_version:
+            update_hash(hash_version + "\n")
+
+        if extra_sigdata:
+            update_hash(extra_sigdata + "\n")
 
         # It is only currently useful to get equivalent hashes for things that
         # can be restored from sstate. Since the sstate object is named using
 
 
 
-=-=-=-=-=-=-=-=-=-=-=-
Quoted 7 lines

Links: You receive all messages sent to this group.
View/Reply Online (#160571): https://lists.openembedded.org/g/openembedded-core/message/160571
Mute This Topic: https://lists.openembedded.org/mt/88425608/4454410
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

Sounds to me like something we should do.

Jacob

#2 lore raw

While we're here, recipes inheriting qemu class should do this too, but for the target dependencies.

Alex

Quoted 137 lines · Jacob Kroon
Quoted 1 line · Joshua Watt

Native task outputs are directly run on the target (host) system after

"target" or "host" ? the latter i suppose

Quoted 10 lines

being built. Even if the output of a native recipe doesn't change, a change in one of its dependencies may cause a change in the output it generates (e.g. rpm output depends on the output of its dependent zstd library).

This can cause poor interactions with hash equivalence, since this recipes output-changing dependency is "hidden" and downstream task only see that this recipe has the same outhash and therefore is equivalent. This can result in different output in different cases.

Quoted 5 lines

To resolve this, unhide the output-changing dependency by adding it's unihash to this tasks outhash calculation. Unfortunately, don't know specifically know which dependencies are output-changing, so we have to add all of them.

"don't know specifically know which.."

Quoted 13 lines

[YOCTO #14685]

---
 meta/classes/native.bbclass | 31 +++++++++++++++++++++++++++++++
 meta/lib/oe/sstatesig.py    | 10 +++++++---
 2 files changed, 38 insertions(+), 3 deletions(-)
meta/classes/native.bbclass +0 -0
index 76a599bc15..fc7422c5d7 100644

meta/classes/native.bbclass USE_NLS = "no" · line 200

Quoted 6 lines
meta/classes/native.bbclass +3 -0
@@ -195,3 +195,34 @@ USE_NLS = "no"
 
 RECIPERDEPTASK = "do_populate_sysroot"
 do_populate_sysroot[rdeptask] = "${RECIPERDEPTASK}"
+
+#
+# Native task outputs are directly run on the target (host) system

after being

see above

meta/classes/native.bbclass USE_NLS = "no" · line 201

Quoted 1 line
meta/classes/native.bbclass +1 -0
+# built. Even if the output of this recipe doesn't change, a change in

one of

meta/classes/native.bbclass USE_NLS = "no" · line 202

Quoted 1 line
meta/classes/native.bbclass +1 -0
+# its dependencies may cause a change in the output it generates (e.g.

rpm

meta/classes/native.bbclass USE_NLS = "no"

Quoted 3 lines
meta/classes/native.bbclass +3 -0
+# output depends on the output of its dependent zstd library).
+#
+# This can cause poor interactions with hash equivalence, since this

recipes

meta/classes/native.bbclass USE_NLS = "no" · line 206

Quoted 1 line
meta/classes/native.bbclass +1 -0
+# output-changing dependency is "hidden" and downstream task only see

that this

meta/classes/native.bbclass USE_NLS = "no" · line 207

Quoted 1 line
meta/classes/native.bbclass +1 -0
+# recipe has the same outhash and therefore is equivalent. This can

result in

meta/classes/native.bbclass USE_NLS = "no"

Quoted 3 lines
meta/classes/native.bbclass +3 -0
+# different output in different cases.
+#
+# To resolve this, unhide the output-changing dependency by adding its

unihash

meta/classes/native.bbclass USE_NLS = "no" · line 211

Quoted 1 line
meta/classes/native.bbclass +1 -0
+# to this tasks outhash calculation. Unfortunately, don't know

specifically

meta/classes/native.bbclass USE_NLS = "no" · line 212

Quoted 1 line
meta/classes/native.bbclass +1 -0
+# know which dependencies are output-changing, so we have to add all of

them.

meta/classes/native.bbclass USE_NLS = "no" · line 213

Quoted 1 line
meta/classes/native.bbclass +1 -0
+#

see above

meta/classes/native.bbclass USE_NLS = "no"

Quoted 13 lines
meta/classes/native.bbclass +13 -0
+python native_add_do_populate_sysroot_deps () {
+    current_task = "do_" + d.getVar("BB_CURRENTTASK")
+    if current_task != "do_populate_sysroot":
+        return
+
+    taskdepdata = d.getVar("BB_TASKDEPDATA", False)
+    pn = d.getVar("PN")
+    deps = {
+        dep[0]:dep[6] for dep in taskdepdata.values() if
+            dep[1] == current_task and dep[0] != pn
+    }
+
+    d.setVar("HASHEQUIV_EXTRA_SIGDATA", "\n".join("%s: %s" % (k,

deps[k]) for k in sorted(deps.keys())))

Quoted 19 lines
meta/classes/native.bbclass +2 -0
+}
+SSTATECREATEFUNCS += "native_add_do_populate_sysroot_deps"
meta/lib/oe/sstatesig.py +2 -1
index 038404e377..abcd96231e 100644
@@ -491,7 +491,8 @@ def OEOuthashBasic(path, sigfile, task, d):
     if task == "package":
         include_timestamps = True
         include_root = False
-    extra_content = d.getVar('HASHEQUIV_HASH_VERSION')
+    hash_version = d.getVar('HASHEQUIV_HASH_VERSION')
+    extra_sigdata = d.getVar("HASHEQUIV_EXTRA_SIGDATA")
 
     filemaps = {}
     for m in (d.getVar('SSTATE_HASHEQUIV_FILEMAP') or '').split():
@@ -506,8 +507,11 @@ def OEOuthashBasic(path, sigfile, task, d):
         basepath = os.path.normpath(path)
 

meta/lib/oe/sstatesig.py def OEOuthashBasic(path, sigfile, task, d): · line 516

Quoted 10 lines
meta/lib/oe/sstatesig.py +5 -2
         update_hash("OEOuthashBasic\n")
-        if extra_content:
-            update_hash(extra_content + "\n")
+        if hash_version:
+            update_hash(hash_version + "\n")
+
+        if extra_sigdata:
+            update_hash(extra_sigdata + "\n")
 
         # It is only currently useful to get equivalent hashes for

things that

meta/lib/oe/sstatesig.py def OEOuthashBasic(path, sigfile, task, d): · line 517

Quoted 1 line
meta/lib/oe/sstatesig.py +0 -0
         # can be restored from sstate. Since the sstate object is named

using

Quoted 5 lines

Sounds to me like something we should do.

Jacob

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#160573):
https://lists.openembedded.org/g/openembedded-core/message/160573
Mute This Topic: https://lists.openembedded.org/mt/88425608/1686489
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ [email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#3 lore raw
In reply to Joshua Watt
Quoted 16 lines · Joshua Watt

Native task outputs are directly run on the target (host) system after being built. Even if the output of a native recipe doesn't change, a change in one of its dependencies may cause a change in the output it generates (e.g. rpm output depends on the output of its dependent zstd library).

This can cause poor interactions with hash equivalence, since this recipes output-changing dependency is "hidden" and downstream task only see that this recipe has the same outhash and therefore is equivalent. This can result in different output in different cases.

To resolve this, unhide the output-changing dependency by adding it's unihash to this tasks outhash calculation. Unfortunately, don't know specifically know which dependencies are output-changing, so we have to add all of them.

Quoted 7 lines

[YOCTO #14685]

---
 meta/classes/native.bbclass | 31 +++++++++++++++++++++++++++++++
 meta/lib/oe/sstatesig.py    | 10 +++++++---
 2 files changed, 38 insertions(+), 3 deletions(-)

Thanks for this. I know it was an RFC but after testing over the weekend and doing some checks of my own, I've merged it. I think it is the correct solution and we don't have any other good option to fix the issue which is breaking builds on the autobuilder.

Cheers,

Richard