eSDK: fix locked signature handling for multiconfig builds
PATCH 1/1 eSDK: fix locked signature handling for multiconfig builds
Building an extensible SDK with populate_sdk_ext fails for machines that use BBMULTICONFIG. Three separate issues contribute to the failure:
1. generate_locked_sigs() in copy_buildsystem.py builds the task filter from BB_TASKDEPDATA which does not include multiconfig tasks. The resulting locked-sigs.inc lacks entries for mc builds, so the eSDK internal build cannot find sstate for those tasks and fails with setscene enforcement errors.
Fix by also collecting mc task IDs from siggen.runtaskdeps.
2. sstate_lockedsigs() in sstatesig.py stores a single hash per pn:task pair. When the same PN appears in multiple tune-specific type sections with different hashes (as happens with multiconfig), the last type overwrites earlier entries. get_taskhash() then returns the wrong locked hash for one of the multiconfigs.
Fix by storing a list of entries per pn:task and selecting the entry whose hash matches the computed hash in get_taskhash().
3. translate_virtualfns() in oe-check-sstate crashes with a KeyError when multiconfig tasks reference recipe files not present in the recipe cache for that mc context.
Fix by skipping unresolvable tasks gracefully.
--- meta/lib/oe/copy_buildsystem.py | 10 +++++++++- meta/lib/oe/sstatesig.py | 17 ++++++++++++++--- scripts/oe-check-sstate | 4 ++++ 3 files changed, 27 insertions(+), 4 deletions(-)
meta/lib/oe/copy_buildsystem.py +9 -1
@@ -172,7 +172,15 @@ class BuildSystem(object): def generate_locked_sigs(sigfile, d): bb.utils.mkdirhier(os.path.dirname(sigfile)) depd = d.getVar('BB_TASKDEPDATA', False) - tasks = ['%s:%s' % (v[2], v[1]) for v in depd.values()] + tasks = set(['%s:%s' % (v[2], v[1]) for v in depd.values()]) + # BB_TASKDEPDATA does not include multiconfig tasks, so the eSDK + # locked-sigs.inc would lack entries for mc builds causing setscene + # enforcement failures. Add any mc tasks tracked by siggen. + for tid in bb.parse.siggen.runtaskdeps: + if isinstance(tid, tuple): + tid = tid[1] + if tid.startswith('mc:'): + tasks.add(tid) bb.parse.siggen.dump_lockedsigs(sigfile, tasks) def prune_lockedsigs(excluded_tasks, excluded_targets, lockedsigs, onlynative, pruned_output):
meta/lib/oe/sstatesig.py +14 -3
@@ -90,7 +90,9 @@ def sstate_lockedsigs(d): pn, task, h = ls.split(":", 2) if pn not in sigs: sigs[pn] = {} - sigs[pn][task] = [h, siggen_lockedsigs_var] + if task not in sigs[pn]: + sigs[pn][task] = [] + sigs[pn][task].append([h, siggen_lockedsigs_var]) return sigs class SignatureGeneratorOEBasicHashMixIn(object):
@@ -182,8 +184,17 @@ class SignatureGeneratorOEBasicHashMixIn(object): if not unlocked and recipename in self.lockedsigs: if task in self.lockedsigs[recipename]: - h_locked = self.lockedsigs[recipename][task][0] - var = self.lockedsigs[recipename][task][1] + entries = self.lockedsigs[recipename][task] + # When multiple locked hashes exist for the same pn:task + # (e.g. from multiconfig builds), pick the one matching + # the computed hash. Fall back to the first entry. + h_locked = entries[0][0] + var = entries[0][1] + for entry in entries: + if entry[0] == h: + h_locked = entry[0] + var = entry[1] + break self.lockedhashes[tid] = h_locked self._internal = True unihash = self.get_unihash(tid)
scripts/oe-check-sstate +4 -0
@@ -35,6 +35,10 @@ def translate_virtualfns(tasks): (mc, fn, taskname) = bb.runqueue.split_tid(task) if taskname.endswith('_setscene'): taskname = taskname[:-9] + if mc not in recipecaches or fn not in recipecaches[mc].pkg_fn: + # Multiconfig tasks may reference fns not in the recipe + # cache for this mc context. Skip them quietly. + continue outtasks.append('%s:%s' % (recipecaches[mc].pkg_fn[fn], taskname)) finally: tinfoil.shutdown()
Signature · 1 line
2.43.0
Quoted 45 lines · Mehmet Fide
Building an extensible SDK with
populate_sdk_extfails for machines that use BBMULTICONFIG. Three separate issues contribute to the failure:1.
generate_locked_sigs()incopy_buildsystem.pybuilds the task filter fromBB_TASKDEPDATAwhich does not include multiconfig tasks. The resulting locked-sigs.inc lacks entries for mc builds, so the eSDK internal build cannot find sstate for those tasks and fails with setscene enforcement errors.Fix by also collecting mc task IDs from siggen.runtaskdeps.
2.
sstate_lockedsigs()insstatesig.pystores a single hash per pn:task pair. When the same PN appears in multiple tune-specific type sections with different hashes (as happens with multiconfig), the last type overwrites earlier entries.get_taskhash()then returns the wrong locked hash for one of the multiconfigs.Fix by storing a list of entries per pn:task and selecting the entry whose hash matches the computed hash in
get_taskhash().3.
translate_virtualfns()in oe-check-sstate crashes with a KeyError when multiconfig tasks reference recipe files not present in the recipe cache for that mc context.Fix by skipping unresolvable tasks gracefully.
Signed-off-by Mehmet Fide <[email protected]>---
meta/lib/oe/copy_buildsystem.py | 10 +++++++++- meta/lib/oe/sstatesig.py | 17 ++++++++++++++--- scripts/oe-check-sstate | 4 ++++3 files changed, 27 insertions(+), 4 deletions(-)meta/lib/oe/copy_buildsystem.py +0
-0@@ -172,7 +172,15 @@ class BuildSystem(object):def
generate_locked_sigs(sigfile, d):
bb.utils.mkdirhier(os.path.dirname(sigfile))
depd =d.getVar('BB_TASKDEPDATA', False)
- tasks = ['%s:%s' % (v[2], v[1]) for v indepd.values()] + tasks =set(['%s:%s' % (v[2], v[1]) for v in depd.values()])+ #BB_TASKDEPDATAdoes not include multiconfig tasks, so the eSDK
Quoted 8 lines
+ # locked-sigs.inc would lack entries for mc builds causing setscene + # enforcement failures. Add any mc tasks tracked by siggen. + for tid in bb.parse.siggen.runtaskdeps:
+ ifisinstance(tid, tuple):
+ tid = tid[1]
+ iftid.startswith('mc:'):
+tasks.add(tid)bb.parse.siggen.dump_lockedsigs(sigfile, tasks)
BB_TASKDEPDATA only contains the tasks that the current task depends upon. This is by design so that you can't "see" outside your own current scope, which would be a dependency issue.
The way you've changed the code, it will add any task with a multiconfig set, regardless of whether you depend on it or not.
It is also possible you're running this code within a multiconfig, in which case you'd want to non-mc tasks, so it isn't correct from that angle either.
The complexities of this are why we've so far not chosen to try and handle it.
Quoted 28 lines
def
prune_lockedsigs(excluded_tasks, excluded_targets, lockedsigs, onlynative, pruned_output):meta/lib/oe/sstatesig.py +0
-0@@ -90,7 +90,9 @@ def sstate_lockedsigs(d): pn, task, h = ls.split(":", 2)if pn not in sigs:
sigs[pn] = {}
- sigs[pn][task] = [h,siggen_lockedsigs_var] + if task not in sigs[pn]:
+ sigs[pn][task] = []
+ sigs[pn][task].append([h,siggen_lockedsigs_var]) return sigs
class SignatureGeneratorOEBasicHashMixIn(object):meta/lib/oe/sstatesig.py +0
-0@@ -182,8 +184,17 @@ class SignatureGeneratorOEBasicHashMixIn(object):
if not unlocked and recipename in self.lockedsigs:
if task in self.lockedsigs[recipename]:
-h_locked= self.lockedsigs[recipename][task][0] - var = self.lockedsigs[recipename][task][1]
+ entries = self.lockedsigs[recipename][task] + # When multiple locked hashes exist for the same pn:task + # (e.g. from multiconfig builds), pick the one matching + # the computed hash. Fall back to the first entry. +h_locked= entries[0][0]
+ var = entries[0][1]
Quoted 8 lines
+ for entry in entries:
+ if entry[0] == h:
+h_locked= entry[0] + var = entry[1]
+ break self.lockedhashes[tid] =h_lockedself._internal = True unihash =self.get_unihash(tid)
Multiconfigs are deliberately meant to behave like separate builds, which don't need to be multiconfig aware. The way you're changing this, changes that. Instead for example, the locked sigs inc file could include the MC in the filename, then you'd just need each config to write it out.
Cheers,
Richard