Skip to contentSkip to conversation

eSDK: fix locked signature handling for multiconfig builds

PATCH In review Mehmet Fide 2 participants 2 messages Mar 15

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 eSDK: fix locked signature handling for multiconfig builds

Building an extensible SDK with populate_sdk_ext fails for machines that use BBMULTICONFIG. Three separate issues contribute to the failure:

1. generate_locked_sigs() in copy_buildsystem.py builds the task filter from BB_TASKDEPDATA which does not include multiconfig tasks. The resulting locked-sigs.inc lacks entries for mc builds, so the eSDK internal build cannot find sstate for those tasks and fails with setscene enforcement errors.

Fix by also collecting mc task IDs from siggen.runtaskdeps.

2. sstate_lockedsigs() in sstatesig.py stores a single hash per pn:task pair. When the same PN appears in multiple tune-specific type sections with different hashes (as happens with multiconfig), the last type overwrites earlier entries. get_taskhash() then returns the wrong locked hash for one of the multiconfigs.

Fix by storing a list of entries per pn:task and selecting the entry whose hash matches the computed hash in get_taskhash().

3. translate_virtualfns() in oe-check-sstate crashes with a KeyError when multiconfig tasks reference recipe files not present in the recipe cache for that mc context.

Fix by skipping unresolvable tasks gracefully.

---
 meta/lib/oe/copy_buildsystem.py | 10 +++++++++-
 meta/lib/oe/sstatesig.py        | 17 ++++++++++++++---
 scripts/oe-check-sstate         |  4 ++++
 3 files changed, 27 insertions(+), 4 deletions(-)
meta/lib/oe/copy_buildsystem.py +9 -1
index 81abfbf9e2..c62fbd3fbb 100644
@@ -172,7 +172,15 @@ class BuildSystem(object):
 def generate_locked_sigs(sigfile, d):
     bb.utils.mkdirhier(os.path.dirname(sigfile))
     depd = d.getVar('BB_TASKDEPDATA', False)
-    tasks = ['%s:%s' % (v[2], v[1]) for v in depd.values()]
+    tasks = set(['%s:%s' % (v[2], v[1]) for v in depd.values()])
+    # BB_TASKDEPDATA does not include multiconfig tasks, so the eSDK
+    # locked-sigs.inc would lack entries for mc builds causing setscene
+    # enforcement failures.  Add any mc tasks tracked by siggen.
+    for tid in bb.parse.siggen.runtaskdeps:
+        if isinstance(tid, tuple):
+            tid = tid[1]
+        if tid.startswith('mc:'):
+            tasks.add(tid)
     bb.parse.siggen.dump_lockedsigs(sigfile, tasks)
 
 def prune_lockedsigs(excluded_tasks, excluded_targets, lockedsigs, onlynative, pruned_output):
meta/lib/oe/sstatesig.py +14 -3
index d818fce8f1..9736e0b4e6 100644
@@ -90,7 +90,9 @@ def sstate_lockedsigs(d):
             pn, task, h = ls.split(":", 2)
             if pn not in sigs:
                 sigs[pn] = {}
-            sigs[pn][task] = [h, siggen_lockedsigs_var]
+            if task not in sigs[pn]:
+                sigs[pn][task] = []
+            sigs[pn][task].append([h, siggen_lockedsigs_var])
     return sigs
 
 class SignatureGeneratorOEBasicHashMixIn(object):
@@ -182,8 +184,17 @@ class SignatureGeneratorOEBasicHashMixIn(object):
 
         if not unlocked and recipename in self.lockedsigs:
             if task in self.lockedsigs[recipename]:
-                h_locked = self.lockedsigs[recipename][task][0]
-                var = self.lockedsigs[recipename][task][1]
+                entries = self.lockedsigs[recipename][task]
+                # When multiple locked hashes exist for the same pn:task
+                # (e.g. from multiconfig builds), pick the one matching
+                # the computed hash.  Fall back to the first entry.
+                h_locked = entries[0][0]
+                var = entries[0][1]
+                for entry in entries:
+                    if entry[0] == h:
+                        h_locked = entry[0]
+                        var = entry[1]
+                        break
                 self.lockedhashes[tid] = h_locked
                 self._internal = True
                 unihash = self.get_unihash(tid)
scripts/oe-check-sstate +4 -0
index 0d171c4463..239aff0a97 100755
@@ -35,6 +35,10 @@ def translate_virtualfns(tasks):
             (mc, fn, taskname) = bb.runqueue.split_tid(task)
             if taskname.endswith('_setscene'):
                 taskname = taskname[:-9]
+            if mc not in recipecaches or fn not in recipecaches[mc].pkg_fn:
+                # Multiconfig tasks may reference fns not in the recipe
+                # cache for this mc context.  Skip them quietly.
+                continue
             outtasks.append('%s:%s' % (recipecaches[mc].pkg_fn[fn], taskname))
     finally:
         tinfoil.shutdown()
Signature · 1 line
2.43.0
#1 lore raw
Quoted 45 lines · Mehmet Fide

Building an extensible SDK with populate_sdk_ext fails for machines that use BBMULTICONFIG.  Three separate issues contribute to the failure:

1. generate_locked_sigs() in copy_buildsystem.py builds the task filter from BB_TASKDEPDATA which does not include multiconfig tasks.  The resulting locked-sigs.inc lacks entries for mc builds, so the eSDK internal build cannot find sstate for those tasks and fails with setscene enforcement errors.

   Fix by also collecting mc task IDs from siggen.runtaskdeps.

2. sstate_lockedsigs() in sstatesig.py stores a single hash per pn:task pair.  When the same PN appears in multiple tune-specific type sections with different hashes (as happens with multiconfig), the last type overwrites earlier entries.  get_taskhash() then returns the wrong locked hash for one of the multiconfigs.

   Fix by storing a list of entries per pn:task and selecting the entry whose hash matches the computed hash in get_taskhash().

3. translate_virtualfns() in oe-check-sstate crashes with a KeyError when multiconfig tasks reference recipe files not present in the recipe cache for that mc context.

   Fix by skipping unresolvable tasks gracefully.

---
 meta/lib/oe/copy_buildsystem.py | 10 +++++++++- meta/lib/oe/sstatesig.py        | 17 ++++++++++++++--- scripts/oe-check-sstate         |  4 ++++

 3 files changed, 27 insertions(+), 4 deletions(-)
meta/lib/oe/copy_buildsystem.py +0 -0
index 81abfbf9e2..c62fbd3fbb 100644
@@ -172,7 +172,15 @@ class BuildSystem(object):

 def generate_locked_sigs(sigfile, d):
     bb.utils.mkdirhier(os.path.dirname(sigfile))
     depd = d.getVar('BB_TASKDEPDATA', False)
-    tasks = ['%s:%s' % (v[2], v[1]) for v in depd.values()] +    tasks = set(['%s:%s' % (v[2], v[1]) for v in depd.values()]) +    # BB_TASKDEPDATA does not include multiconfig tasks, so the eSDK

Quoted 8 lines

+    # locked-sigs.inc would lack entries for mc builds causing setscene +    # enforcement failures.  Add any mc tasks tracked by siggen. +    for tid in bb.parse.siggen.runtaskdeps:
+        if isinstance(tid, tuple):
+            tid = tid[1]
+        if tid.startswith('mc:'):
+            tasks.add(tid) bb.parse.siggen.dump_lockedsigs(sigfile, tasks)

BB_TASKDEPDATA only contains the tasks that the current task depends upon. This is by design so that you can't "see" outside your own current scope, which would be a dependency issue.

The way you've changed the code, it will add any task with a multiconfig set, regardless of whether you depend on it or not.

It is also possible you're running this code within a multiconfig, in which case you'd want to non-mc tasks, so it isn't correct from that angle either.

The complexities of this are why we've so far not chosen to try and handle it.

Quoted 28 lines

 def prune_lockedsigs(excluded_tasks, excluded_targets, lockedsigs, onlynative, pruned_output):

meta/lib/oe/sstatesig.py +0 -0
index d818fce8f1..9736e0b4e6 100644
@@ -90,7 +90,9 @@ def sstate_lockedsigs(d):
              pn, task, h = ls.split(":", 2)

             if pn not in sigs:
                 sigs[pn] = {}
-            sigs[pn][task] = [h, siggen_lockedsigs_var] +            if task not in sigs[pn]:
+                sigs[pn][task] = []
+            sigs[pn][task].append([h, siggen_lockedsigs_var]) return sigs

 class SignatureGeneratorOEBasicHashMixIn(object):

meta/lib/oe/sstatesig.py +0 -0
@@ -182,8 +184,17 @@ class SignatureGeneratorOEBasicHashMixIn(object):


         if not unlocked and recipename in self.lockedsigs:
             if task in self.lockedsigs[recipename]:
-                h_locked = self.lockedsigs[recipename][task][0] -                var = self.lockedsigs[recipename][task][1]
+                entries = self.lockedsigs[recipename][task] +                # When multiple locked hashes exist for the same pn:task +                # (e.g. from multiconfig builds), pick the one matching +                # the computed hash.  Fall back to the first entry. +                h_locked = entries[0][0]
+                var = entries[0][1]

Quoted 8 lines

+                for entry in entries:
+                    if entry[0] == h:
+                        h_locked = entry[0] +                        var = entry[1]
+                        break self.lockedhashes[tid] = h_locked self._internal = True unihash = self.get_unihash(tid)

Multiconfigs are deliberately meant to behave like separate builds, which don't need to be multiconfig aware. The way you're changing this, changes that. Instead for example, the locked sigs inc file could include the MC in the filename, then you'd just need each config to write it out.

Cheers,

Richard