Add support for locked down sstate
cover Add support for locked down sstate
Patches 4/5, 3/5 are mine, based on Richard's 1/5, 2/5, 3/5.
Test Steps:
1. Prepare a locked sstat-cache by dumping lockedsigs file
1) vim local.conf
...SIGGEN_DUMP_LOCKEDSIGS = "1" SIGGEN_LOCKEDSIGS_CONFIG = "/path/to/locked-sigs.inc" SSTATE_DIR ?= "/path/to/sstate-cache"
...
2) Build db-native from a clean environment
$ bitbake db-native
3) Back up lockedsigs file
$ mv /path/to/locked-sigs.inc /path/to/locked-sigs.inc-back
4) Incrementally build ldconfig-native
$ vim local.conf
...
require ${SIGGEN_LOCKEDSIGS_CONFIG}
...
$ bitbake ldconfig-native5) Check incrementally dumping lockedsiged file also
$ vimdiff /path/to/locked-sigs.inc-back /path/to/locked-sigs.inc
2. Create a new build which uses previous locked sstat-cache
1) vim local.conf
...SSTATE_DIR ?= "/path/to/sstate-cache" SIGGEN_LOCKEDSIGS_CONFIG = "/path/to/locked-sigs.inc" require ${SIGGEN_LOCKEDSIGS_CONFIG} SIGGEN_LOCKEDSIGS_CHECK_LEVEL = "error"
...
2) Build db-native
$ bitbake db-native
3) Tweak db's do_configure task by adding comments
meta/recipes-support/db/db_6.0.30.bb +1 -0
@@ -82,6 +82,7 @@ do_configure() { gnu-configize --force ${S} export STRIP="true" oe_runconf + echo "hello" }
4) Build db-native and abort
$ bitbake db-native ... ERROR: The db-native:do_configure sig (00546ab0e73b3afd50efc03f26e5f040) changed, use locked sig 9d8f2210506446e76d012d1ebd49a021 to instead The db-native:do_compile sig (4442f979d97bdbed71a8fecbbbe41ed3) changed, use locked sig d29a7c568d10859176d733773433fe92 to instead The db-native:do_install sig (562cb98d8eb25a7c79ded6a5359c56c1) changed, use locked sig 1e917d8d619c58987a1b2fe286e9288c to instead The db-native:do_populate_sysroot sig (91627e36da9ac8fe489f36b67a87ed2e) changed, use locked sig f780d45dbf680e40a871820327c4c56c to instead The db-native:do_build sig (ba289cba4a3bfda0c07aed9868fdcd5d) changed, use locked sig d9b84e5fb30011fd31bc0e7d7652fead to instead ...
5) Remove db-native sstate-cache
$ bitbake db-native -ccleansstate
6) Build db-native and abort
$ bitbake db-native ... Locked sig is set for db-native:do_populate_sysroot (f780d45dbf680e40a871820327c4c56c) yet not in sstate cache? ...
7) Do not break the build
vim local.conf
...SIGGEN_LOCKEDSIGS_CHECK_LEVEL = "warn"
...
8) Build db-native
$ bitbake db-native ... WARNING Locked sig is set for db-native:do_populate_sysroot (f780d45dbf680e40a871820327c4c56c) yet not in sstate cache? ...
//Hongxu
The following changes since commit bd1a3ab8a63d6b2c109d595c67fcca99817e1daa:
nspr: Upgrade to 4.10.7 (2014-09-10 15:38:57 +0100)
are available in the git repository at:
git://git.pokylinux.org/poky-contrib hongxu/locked-sstate
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=hongxu/locked-sstate
Hongxu Jia (2):
sstate/sstatesig: optimize the support for locked down sstate cache
usage
sstatesig: incremental dump lockedsigs
Richard Purdie (1):
siggen/runqueue/bitbake-worker: Improve siggen data transfer interface
[email protected] (2):
Richard Purdie : sstatesig/sstate: Add support for locked down sstate
cache usage
Richard Purdie : sstatesig: Improve to handle locking of multiple
machines
bitbake/bin/bitbake-worker | 2 +- bitbake/lib/bb/runqueue.py | 4 +- bitbake/lib/bb/siggen.py | 9 ++-- meta/classes/sstate.bbclass | 17 ++++++ meta/lib/oe/sstatesig.py | 123 ++++++++++++++++++++++++++++++++++++++++++++ scripts/gen-lockedsig-cache | 40 ++++++++++++++ 6 files changed, 187 insertions(+), 8 deletions(-) create mode 100755 scripts/gen-lockedsig-cache
Signature · 1 line
1.9.1
PATCH 1/5 Richard Purdie : sstatesig/sstate: Add support for locked down sstate cache usage
From: "[email protected]" <[email protected]>
Module: openembedded-core.git Branch: master-next Commit: a12e33a584a77df4bdd9ad6a5d1a58f4dde10317 URL: http://git.openembedded.org/?p=openembedded-core.git&a=commit;h=a12e33a584a77df4bdd9ad6a5d1a58f4dde10317
Author: Richard Purdie <[email protected]> Date: Fri Sep 5 10:40:02 2014 +0100
sstatesig/sstate: Add support for locked down sstate cache usage
I've been giving things some thought, specifically why sstate doesn't get used more and why we have people requesting external toolchains. I'm guessing the issue is that people don't like how often sstate can change and the lack of an easy way to lock it down.
Locking it down is actually quite easy so patch implements some basics of how you can do this (for example to a specific toolchain). With an addition like this to local.conf (or wherever):
SIGGEN_LOCKEDSIGS = "\
gcc-cross:do_populate_sysroot:a8d91b35b98e1494957a2ddaf4598956 \
eglibc:do_populate_sysroot:13e8c68553dc61f9d67564f13b9b2d67 \
eglibc:do_packagedata:bfca0db1782c719d373f8636282596ee \
gcc-cross:do_packagedata:4b601ff4f67601395ee49c46701122f6 \
"
the code at the end of the email will force the hashes to those values for the recipes mentioned. The system would then find and use those specific objects from the sstate cache instead of trying to build anything.
Obviously this is a little simplistic, you might need to put an override against this to only apply those revisions for a specific architecture for example. You'd also probably want to put code in the sstate hash validation code to ensure it really did install these from sstate since if it didn't you'd want to abort the build.
This patch also implements support to add to bitbake -S which dumps the locked sstate checksums for each task into a ready prepared include file locked-sigs.inc (currently placed into cwd). There is a function, bb.parse.siggen.dump_lockedsigs() which can be called to trigger the same functionality from task space.
A warning is added to sstate.bbclass through a call back into the siggen class to warn if objects are not used from the locked cache. The SIGGEN_ENFORCE_LOCKEDSIGS variable controls whether this is just a warning or a fatal error.
A script is provided to generate sstate directory from a locked-sigs file.
--- meta/classes/sstate.bbclass | 3 ++ meta/lib/oe/sstatesig.py | 74 +++++++++++++++++++++++++++++++++++++++++++++ scripts/gen-lockedsig-cache | 40 ++++++++++++++++++++++++ 3 files changed, 117 insertions(+) create mode 100755 scripts/gen-lockedsig-cache
meta/classes/sstate.bbclass +3 -0
@@ -712,6 +712,9 @@ def sstate_checkhashes(sq_fn, sq_task, sq_hash, sq_hashfn, d): evdata['found'].append( (sq_fn[task], sq_task[task], sq_hash[task], sstatefile ) ) bb.event.fire(bb.event.MetadataEvent("MissedSstate", evdata), d) + if hasattr(bb.parse.siggen, "checkhashes"): + bb.parse.siggen.checkhashes(missed, ret, sq_fn, sq_task, sq_hash, sq_hashfn, d) + return ret BB_SETSCENE_DEPVALID = "setscene_depvalid"
meta/lib/oe/sstatesig.py +74 -0
90 lines not on this page · raw message
scripts/gen-lockedsig-cache new file +40 -0
@@ -0,0 +1,40 @@ +#!/usr/bin/env python +# +# gen-lockedsig-cache <locked-sigs.inc> <input-cachedir> <output-cachedir> +# + +import os +import sys +import glob +import shutil +import errno + +def mkdir(d): + try: + os.makedirs(d) + except OSError as e: + if e.errno != errno.EEXIST: + raise e + +if len(sys.argv) < 3: + print("Incorrect number of arguments specified") + sys.exit(1) + +sigs = [] +with open(sys.argv[1]) as f: + for l in f.readlines(): + if ":" in l: + sigs.append(l.split(":")[2].split()[0]) + +files = set() +for s in sigs: + p = sys.argv[2] + "/" + s[:2] + "/*" + s + "*" + files |= set(glob.glob(p)) + p = sys.argv[2] + "/*/" + s[:2] + "/*" + s + "*" + files |= set(glob.glob(p)) + +for f in files: + dst = f.replace(sys.argv[2], sys.argv[3]) + mkdir(os.path.dirname(dst)) + os.link(f, dst) +
Signature · 1 line
1.9.1
PATCH 2/5 Richard Purdie : sstatesig: Improve to handle locking of multiple machines
In reply to Hongxu JiaFrom: "[email protected]" <[email protected]>
Module: openembedded-core.git Branch: master-next Commit: a702eb50674ba7017c63829272082d4005237f13 URL: http://git.openembedded.org/?p=openembedded-core.git&a=commit;h=a702eb50674ba7017c63829272082d4005237f13
Author: Richard Purdie <[email protected]> Date: Fri Sep 5 11:55:18 2014 +0100
sstatesig: Improve to handle locking of multiple machines
Instead of a single monolithic SIGGEN_LOCKEDSIGS, split this into separate variables, one per sstate package architecture. Add in a new SIGGEN_LOCKEDSIGS_TYPES variable which lists the package architectures to load in.
SIGGEN_LOCKEDSIGS_TYPES is made machine specific using overrides.
Also sort the hashes in the lists by PN to make diffing them easier.
--- meta/lib/oe/sstatesig.py | 41 ++++++++++++++++++++++++++++------------- 1 file changed, 28 insertions(+), 13 deletions(-)
meta/lib/oe/sstatesig.py +28 -13
@@ -63,12 +63,14 @@ def sstate_rundepfilter(siggen, fn, recipename, task, dep, depname, dataCache): def sstate_lockedsigs(d): sigs = {} - lockedsigs = (d.getVar("SIGGEN_LOCKEDSIGS", True) or "").split() - for ls in lockedsigs: - pn, task, h = ls.split(":", 2) - if pn not in sigs: - sigs[pn] = {} - sigs[pn][task] = h + types = (d.getVar("SIGGEN_LOCKEDSIGS_TYPES", True) or "").split() + for t in types: + lockedsigs = (d.getVar("SIGGEN_LOCKEDSIGS_%s" % t, True) or "").split() + for ls in lockedsigs: + pn, task, h = ls.split(":", 2) + if pn not in sigs: + sigs[pn] = {} + sigs[pn][task] = h return sigs class SignatureGeneratorOEBasic(bb.siggen.SignatureGeneratorBasic):
@@ -88,16 +90,18 @@ class SignatureGeneratorOEBasicHash(bb.siggen.SignatureGeneratorBasicHash): self.lockedsigs = sstate_lockedsigs(data) self.lockedhashes = {} self.lockedpnmap = {} + self.lockedhashfn = {} + self.machine = data.getVar("MACHINE", True) pass def rundep_check(self, fn, recipename, task, dep, depname, dataCache = None): return sstate_rundepfilter(self, fn, recipename, task, dep, depname, dataCache) def get_taskdata(self): data = super(bb.siggen.SignatureGeneratorBasicHash, self).get_taskdata() - return (data, self.lockedpnmap) + return (data, self.lockedpnmap, self.lockedhashfn) def set_taskdata(self, data): - coredata, self.lockedpnmap = data + coredata, self.lockedpnmap, self.lockedhashfn = data super(bb.siggen.SignatureGeneratorBasicHash, self).set_taskdata(coredata) def dump_sigs(self, dataCache, options):
@@ -107,6 +111,7 @@ class SignatureGeneratorOEBasicHash(bb.siggen.SignatureGeneratorBasicHash): def get_taskhash(self, fn, task, deps, dataCache): recipename = dataCache.pkg_fn[fn] self.lockedpnmap[fn] = recipename + self.lockedhashfn[fn] = dataCache.hashfn[fn] if recipename in self.lockedsigs: if task in self.lockedsigs[recipename]: k = fn + "." + task
@@ -127,17 +132,27 @@ class SignatureGeneratorOEBasicHash(bb.siggen.SignatureGeneratorBasicHash): def dump_lockedsigs(self): bb.plain("Writing locked sigs to " + os.getcwd() + "/locked-sigs.inc") + types = {} + for k in self.runtaskdeps: + fn = k.rsplit(".",1)[0] + t = self.lockedhashfn[fn].split(" ")[1].split(":")[5] + if t not in types: + types[t] = [] + types[t].append(k) + with open("locked-sigs.inc", "w") as f: - f.write('SIGGEN_LOCKEDSIGS = "\\\n') - #for fn in self.taskdeps: - for k in self.runtaskdeps: - #k = fn + "." + task + for t in types: + f.write('SIGGEN_LOCKEDSIGS_%s = "\\\n' % t) + types[t].sort() + sortedk = sorted(types[t], key=lambda k: self.lockedpnmap[k.rsplit(".",1)[0]]) + for k in sortedk: fn = k.rsplit(".",1)[0] task = k.rsplit(".",1)[1] if k not in self.taskhash: continue f.write(" " + self.lockedpnmap[fn] + ":" + task + ":" + self.taskhash[k] + " \\\n") - f.write(' "\n') + f.write(' "\n') + f.write('SIGGEN_LOCKEDSIGS_TYPES_%s = "%s"' % (self.machine, " ".join(types.keys()))) def checkhashes(self, missed, ret, sq_fn, sq_task, sq_hash, sq_hashfn, d): enforce = (d.getVar("SIGGEN_ENFORCE_LOCKEDSIGS", True) or "1") == "1"
Signature · 1 line
1.9.1
PATCH 3/5 Richard Purdie : siggen/runqueue/bitbake-worker: Improve siggen data transfer interface
In reply to Hongxu JiaFrom: Richard Purdie <[email protected]>
We need to transfer some of the siggen data from the core/cooker into the worker instances. There was a partial API created for this but its ugly and its not possible to extend it from the siggen class.
This patch completes the interface/abstraction for the data and means the class can extend/customise it in any siggen class.
--- bitbake/bin/bitbake-worker | 2 +- bitbake/lib/bb/runqueue.py | 4 +--- bitbake/lib/bb/siggen.py | 9 +++++---- 3 files changed, 7 insertions(+), 8 deletions(-)
bitbake/bin/bitbake-worker +1 -1
@@ -168,7 +168,7 @@ def fork_off_task(cfg, data, workerdata, fn, task, taskname, appends, taskdepdat data.setVar("BUILDNAME", workerdata["buildname"]) data.setVar("DATE", workerdata["date"]) data.setVar("TIME", workerdata["time"]) - bb.parse.siggen.set_taskdata(workerdata["hashes"], workerdata["hash_deps"], workerdata["sigchecksums"]) + bb.parse.siggen.set_taskdata(workerdata["sigdata"]) ret = 0 try: the_data = bb.cache.Cache.loadDataFull(fn, appends, data)
bitbake/lib/bb/runqueue.py +1 -3
@@ -879,9 +879,7 @@ class RunQueue: "fakerootenv" : self.rqdata.dataCache.fakerootenv, "fakerootdirs" : self.rqdata.dataCache.fakerootdirs, "fakerootnoenv" : self.rqdata.dataCache.fakerootnoenv, - "hashes" : bb.parse.siggen.taskhash, - "hash_deps" : bb.parse.siggen.runtaskdeps, - "sigchecksums" : bb.parse.siggen.file_checksum_values, + "sigdata" : bb.parse.siggen.get_taskdata(), "runq_hash" : self.rqdata.runq_hash, "logdefaultdebug" : bb.msg.loggerDefaultDebugLevel, "logdefaultverbose" : bb.msg.loggerDefaultVerbose,
bitbake/lib/bb/siggen.py +5 -4
@@ -197,10 +197,11 @@ class SignatureGeneratorBasic(SignatureGenerator): #d.setVar("BB_TASKHASH_task-%s" % task, taskhash[task]) return h - def set_taskdata(self, hashes, deps, checksums): - self.runtaskdeps = deps - self.taskhash = hashes - self.file_checksum_values = checksums + def get_taskdata(self): + return (self.runtaskdeps, self.taskhash, self.file_checksum_values) + + def set_taskdata(self, data): + self.runtaskdeps, self.taskhash, self.file_checksum_values = data def dump_sigtask(self, fn, task, stampbase, runtime): k = fn + "." + task
Signature · 1 line
1.9.1
PATCH 4/5 sstate/sstatesig: optimize the support for locked down sstate cache usage
In reply to Hongxu JiaThis fix is based on Richard Purdie's 'sstatesig: Improve to handle locking of multiple machines' which located in master-next.
Add code in the sstate hash validation code to ensure it really did install these from sstate since if it didn't should to warn/abort the build. The judgment condition is:
1) If a build is replaced by locked sstate-cache, it will triger a warn/error;
2) If objects are not used from the locked cache, it will triger a warn/error;
3) Use SIGGEN_LOCKEDSIGS_CHECK_LEVEL variable controls whether this is just a warning or a fatal error or nothing to report.
Use SIGGEN_DUMP_LOCKEDSIGS variable controls whether to dump lockedsigs. Add a event handler at 'bb.event.BuildCompleted', so while the locked sstate-cache was created, it will dump the lockedsigs files rather than manually invoking 'bitbake -S **' again.
Use SIGGEN_LOCKEDSIGS_CONFIG variable controls where to dump the lockedsigs file, the default is placed into ${SSTATE_DIR}/locked-sigs.inc.
[YOCTO #6639]
--- meta/classes/sstate.bbclass | 14 ++++++++++++++ meta/lib/oe/sstatesig.py | 43 +++++++++++++++++++++++++++---------------- 2 files changed, 41 insertions(+), 16 deletions(-)
meta/classes/sstate.bbclass +14 -0
@@ -802,3 +802,17 @@ python sstate_eventhandler() { bb.siggen.dump_this_task(sstatepkg + '_' + taskname + ".tgz" ".siginfo", d) } +SIGGEN_LOCKEDSIGS_CONFIG ?= "${SSTATE_DIR}/locked-sigs.inc" +addhandler sstate_dump_lockedsig +sstate_dump_lockedsig[eventmask] = "bb.event.BuildCompleted" +python sstate_dump_lockedsig() { + d = e.data + if d.getVar('SIGGEN_DUMP_LOCKEDSIGS', True) == '1': + if e.getFailures(): + return + + if hasattr(bb.parse.siggen, "dump_lockedsigs"): + lockedsigs = d.getVar('SIGGEN_LOCKEDSIGS_CONFIG', True) + bb.parse.siggen.dump_lockedsigs(lockedsigs) +} +
meta/lib/oe/sstatesig.py +27 -16
87 lines not on this page · raw message
Signature · 1 line
1.9.1
Re: [PATCH 4/5] sstate/sstatesig: optimize the support for locked down sstate cache usage
Quoted 40 lines · Hongxu Jia
This fix is based on Richard Purdie's 'sstatesig: Improve to handle locking of multiple machines' which located in master-next.
Add code in the sstate hash validation code to ensure it really did install these from sstate since if it didn't should to warn/abort the build. The judgment condition is:
1) If a build is replaced by locked sstate-cache, it will triger a warn/error;
2) If objects are not used from the locked cache, it will triger a warn/error;
3) UseSIGGEN_LOCKEDSIGS_CHECK_LEVELvariable controls whether this is just a warning or a fatal error or nothing to report.Use
SIGGEN_DUMP_LOCKEDSIGSvariable controls whether to dump lockedsigs. Add a event handler at 'bb.event.BuildCompleted', so while the locked sstate-cache was created, it will dump the lockedsigs files rather than manually invoking 'bitbake -S **' again.Use
SIGGEN_LOCKEDSIGS_CONFIGvariable controls where to dump the lockedsigs file, the default is placed into ${SSTATE_DIR}/locked-sigs.inc.[YOCTO #6639]
Signed-off-by Hongxu Jia <[email protected]>--- meta/classes/sstate.bbclass | 14 ++++++++++++++ meta/lib/oe/sstatesig.py | 43 +++++++++++++++++++++++++++---------------- 2 files changed, 41 insertions(+), 16 deletions(-)meta/classes/sstate.bbclass +3
-0@@ -802,3 +802,17 @@ python sstate_eventhandler() { bb.siggen.dump_this_task(sstatepkg + '_' + taskname + ".tgz" ".siginfo", d) } +SIGGEN_LOCKEDSIGS_CONFIG ?= "${SSTATE_DIR}/locked-sigs.inc" +addhandler sstate_dump_lockedsig +sstate_dump_lockedsig[eventmask] = "bb.event.BuildCompleted"
meta/classes/sstate.bbclass python sstate_eventhandler() { · line 817
Quoted 10 lines
meta/classes/sstate.bbclass +10
-0+python sstate_dump_lockedsig() { + d = e.data + if d.getVar('SIGGEN_DUMP_LOCKEDSIGS', True) == '1': + if e.getFailures(): + return + + if hasattr(bb.parse.siggen, "dump_lockedsigs"): + lockedsigs = d.getVar('SIGGEN_LOCKEDSIGS_CONFIG', True) + bb.parse.siggen.dump_lockedsigs(lockedsigs) +}
Firstly, I think the change to sstate.bbclass should be as a separate patch and I'm not sure I like that patch as it stands. Should this perhaps be as a separate class?
meta/lib/oe/sstatesig.py class SignatureGeneratorOEBasicHash(bb.siggen.SignatureGeneratorBasicHash): · line 95
Quoted 9 lines
meta/lib/oe/sstatesig.py +1
-0@@ -92,6 +92,7 @@ class SignatureGeneratorOEBasicHash(bb.siggen.SignatureGeneratorBasicHash): self.lockedpnmap = {} self.lockedhashfn = {} self.machine = data.getVar("MACHINE", True) + self.checkmsgs = []
I think this should be called "mismatch_msgs" instead.
Quoted 34 lines
meta/lib/oe/sstatesig.py +11
-5pass def rundep_check(self, fn, recipename, task, dep, depname, dataCache = None): return sstate_rundepfilter(self, fn, recipename, task, dep, depname, dataCache)@@ -109,18 +110,24 @@ class SignatureGeneratorOEBasicHash(bb.siggen.SignatureGeneratorBasicHash): return super(bb.siggen.SignatureGeneratorBasicHash, self).dump_sigs(dataCache, options) def get_taskhash(self, fn, task, deps, dataCache): + h = super(bb.siggen.SignatureGeneratorBasicHash, self).get_taskhash(fn, task, deps, dataCache) + recipename = dataCache.pkg_fn[fn] self.lockedpnmap[fn] = recipename self.lockedhashfn[fn] = dataCache.hashfn[fn] if recipename in self.lockedsigs: if task in self.lockedsigs[recipename]: k = fn + "." + task - h = self.lockedsigs[recipename][task] - self.lockedhashes[k] = h - self.taskhash[k] = h + h_locked = self.lockedsigs[recipename][task] + self.lockedhashes[k] = h_locked + self.taskhash[k] = h_locked #bb.warn("Using %s %s %s" % (recipename, task, h)) - return h - h = super(bb.siggen.SignatureGeneratorBasicHash, self).get_taskhash(fn, task, deps, dataCache) + + if h != h_locked: + self.checkmsgs.append('The %s:%s sig (%s) changed, use locked sig %s to instead' + % (recipename, task, h, h_locked)) + + return h_locked #bb.warn("%s %s %s" % (recipename, task, h)) return h
meta/lib/oe/sstatesig.py class SignatureGeneratorOEBasicHash(bb.siggen.SignatureGeneratorBasicHash): · line 140
Quoted 6 lines
meta/lib/oe/sstatesig.py +1
-2@@ -130,8 +137,11 @@ class SignatureGeneratorOEBasicHash(bb.siggen.SignatureGeneratorBasicHash): return super(bb.siggen.SignatureGeneratorBasicHash, self).dump_sigtask(fn, task, stampbase, runtime) - def dump_lockedsigs(self): - bb.plain("Writing locked sigs to " + os.getcwd() + "/locked-sigs.inc") + def dump_lockedsigs(self, where_to_dump=None):
where_to_dump -> sigfile
Quoted 22 lines
meta/lib/oe/sstatesig.py +6
-2+ if not where_to_dump: + where_to_dump = os.getcwd() + "/locked-sigs.inc" + + bb.plain("Writing locked sigs to %s" % where_to_dump) types = {} for k in self.runtaskdeps: fn = k.rsplit(".",1)[0]@@ -140,11 +150,11 @@ class SignatureGeneratorOEBasicHash(bb.siggen.SignatureGeneratorBasicHash): types[t] = [] types[t].append(k) - with open("locked-sigs.inc", "w") as f: + with open(where_to_dump, "w") as f: for t in types: f.write('SIGGEN_LOCKEDSIGS_%s = "\\\n' % t) types[t].sort() - sortedk = sorted(types[t], key=lambda k: self.lockedpnmap[k.rsplit(".",1)[0]]) + sortedk = sorted(types[t], key=lambda k: self.lockedpnmap[k.rsplit(".",1)[0]]) for k in sortedk: fn = k.rsplit(".",1)[0] task = k.rsplit(".",1)[1]
meta/lib/oe/sstatesig.py class SignatureGeneratorOEBasicHash(bb.siggen.SignatureGeneratorBasicHash): · line 168
Quoted 6 lines
meta/lib/oe/sstatesig.py +1
-2@@ -155,17 +165,18 @@ class SignatureGeneratorOEBasicHash(bb.siggen.SignatureGeneratorBasicHash): f.write('SIGGEN_LOCKEDSIGS_TYPES_%s = "%s"' % (self.machine, " ".join(types.keys()))) def checkhashes(self, missed, ret, sq_fn, sq_task, sq_hash, sq_hashfn, d): - enforce = (d.getVar("SIGGEN_ENFORCE_LOCKEDSIGS", True) or "1") == "1" - msgs = [] + checklevel = d.getVar("SIGGEN_LOCKEDSIGS_CHECK_LEVEL", True)
Perhaps this should default to error?
Quoted 9 lines
meta/lib/oe/sstatesig.py +0
-5for task in range(len(sq_fn)): if task not in ret: for pn in self.lockedsigs: if sq_hash[task] in self.lockedsigs[pn].itervalues(): - msgs.append("Locked sig is set for %s:%s (%s) yet not in sstate cache?" % (pn, sq_task[task], sq_hash[task])) - if msgs and enforce: - bb.fatal("\n".join(msgs)) - elif msgs: - bb.warn("\n".join(msgs))
meta/lib/oe/sstatesig.py class SignatureGeneratorOEBasicHash(bb.siggen.SignatureGeneratorBasicHash): · line 182
Quoted 10 lines
meta/lib/oe/sstatesig.py +7
-0+ self.checkmsgs.append("Locked sig is set for %s:%s (%s) yet not in sstate cache?" + % (pn, sq_task[task], sq_hash[task])) + + if self.checkmsgs and checklevel == 'warn': + bb.warn("\n".join(self.checkmsgs)) + elif self.checkmsgs and checklevel == 'error': + bb.fatal("\n".join(self.checkmsgs)) # Insert these classes into siggen's namespace so it can see and select them
I have an updated version of this patch at
http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t222&id=17c255cd72810616139e36ae8ad30cc4daefe9d1
if you're ok with the changes although I do need to update the commit message.
Cheers,
Richard
Re: [PATCH 4/5] sstate/sstatesig: optimize the support for locked down sstate cache usage
Quoted 47 lines · Richard Purdie
Quoted 46 lines · Hongxu Jia
This fix is based on Richard Purdie's 'sstatesig: Improve to handle locking of multiple machines' which located in master-next.
Add code in the sstate hash validation code to ensure it really did install these from sstate since if it didn't should to warn/abort the build. The judgment condition is:
1) If a build is replaced by locked sstate-cache, it will triger awarn/error; 2) If objects are not used from the locked cache, it will triger a warn/error;3) Use
SIGGEN_LOCKEDSIGS_CHECK_LEVELvariable controls whether this is just a warning or a fatal error or nothing to report.Use
SIGGEN_DUMP_LOCKEDSIGSvariable controls whether to dump lockedsigs. Add a event handler at 'bb.event.BuildCompleted', so while the locked sstate-cache was created, it will dump the lockedsigs files rather than manually invoking 'bitbake -S **' again.Use
SIGGEN_LOCKEDSIGS_CONFIGvariable controls where to dump the lockedsigs file, the default is placed into ${SSTATE_DIR}/locked-sigs.inc.[YOCTO #6639]
Signed-off-by Hongxu Jia <[email protected]>--- meta/classes/sstate.bbclass | 14 ++++++++++++++ meta/lib/oe/sstatesig.py | 43 +++++++++++++++++++++++++++---------------- 2 files changed, 41 insertions(+), 16 deletions(-)meta/classes/sstate.bbclass +9
-0@@ -802,3 +802,17 @@ python sstate_eventhandler() { bb.siggen.dump_this_task(sstatepkg + '_' + taskname + ".tgz" ".siginfo", d) } +SIGGEN_LOCKEDSIGS_CONFIG ?= "${SSTATE_DIR}/locked-sigs.inc" +addhandler sstate_dump_lockedsig +sstate_dump_lockedsig[eventmask] = "bb.event.BuildCompleted" +python sstate_dump_lockedsig() { + d = e.data + if d.getVar('SIGGEN_DUMP_LOCKEDSIGS', True) == '1': + if e.getFailures(): + return +
Quoted 8 lines
meta/classes/sstate.bbclass python sstate_eventhandler() { · line 817
Quoted 4 lines
meta/classes/sstate.bbclass +4
-0+ if hasattr(bb.parse.siggen, "dump_lockedsigs"): + lockedsigs = d.getVar('SIGGEN_LOCKEDSIGS_CONFIG', True) + bb.parse.siggen.dump_lockedsigs(lockedsigs) +}Firstly, I think the change to sstate.bbclass should be as a separate patch and I'm not sure I like that patch as it stands. Should this perhaps be as a separate class?
It provides a method to dump lockedsigs at recipe building time, so no need to invoke 'bitbake -S **' again.
And I will name a new class 'sstate_lockedsig' and move the event handler to it.
Quoted 10 lines
meta/lib/oe/sstatesig.py class SignatureGeneratorOEBasicHash(bb.siggen.SignatureGeneratorBasicHash): · line 95
Quoted 9 lines
meta/lib/oe/sstatesig.py +1
-0@@ -92,6 +92,7 @@ class SignatureGeneratorOEBasicHash(bb.siggen.SignatureGeneratorBasicHash): self.lockedpnmap = {} self.lockedhashfn = {} self.machine = data.getVar("MACHINE", True) + self.checkmsgs = []I think this should be called "
mismatch_msgs" instead.
Agree
Quoted 33 lines
Quoted 33 lines
meta/lib/oe/sstatesig.py +11
-5pass def rundep_check(self, fn, recipename, task, dep, depname, dataCache = None): return sstate_rundepfilter(self, fn, recipename, task, dep, depname, dataCache)@@ -109,18 +110,24 @@ class SignatureGeneratorOEBasicHash(bb.siggen.SignatureGeneratorBasicHash): return super(bb.siggen.SignatureGeneratorBasicHash, self).dump_sigs(dataCache, options) def get_taskhash(self, fn, task, deps, dataCache): + h = super(bb.siggen.SignatureGeneratorBasicHash, self).get_taskhash(fn, task, deps, dataCache) + recipename = dataCache.pkg_fn[fn] self.lockedpnmap[fn] = recipename self.lockedhashfn[fn] = dataCache.hashfn[fn] if recipename in self.lockedsigs: if task in self.lockedsigs[recipename]: k = fn + "." + task - h = self.lockedsigs[recipename][task] - self.lockedhashes[k] = h - self.taskhash[k] = h + h_locked = self.lockedsigs[recipename][task] + self.lockedhashes[k] = h_locked + self.taskhash[k] = h_locked #bb.warn("Using %s %s %s" % (recipename, task, h)) - return h - h = super(bb.siggen.SignatureGeneratorBasicHash, self).get_taskhash(fn, task, deps, dataCache) + + if h != h_locked: + self.checkmsgs.append('The %s:%s sig (%s) changed, use locked sig %s to instead' + % (recipename, task, h, h_locked)) + + return h_locked #bb.warn("%s %s %s" % (recipename, task, h)) return h
Quoted 8 lines
meta/lib/oe/sstatesig.py class SignatureGeneratorOEBasicHash(bb.siggen.SignatureGeneratorBasicHash): · line 140
Quoted 7 lines
meta/lib/oe/sstatesig.py +1
-2@@ -130,8 +137,11 @@ class SignatureGeneratorOEBasicHash(bb.siggen.SignatureGeneratorBasicHash): return super(bb.siggen.SignatureGeneratorBasicHash, self).dump_sigtask(fn, task, stampbase, runtime) - def dump_lockedsigs(self): - bb.plain("Writing locked sigs to " + os.getcwd() + "/locked-sigs.inc") + def dump_lockedsigs(self, where_to_dump=None):
where_to_dump-> sigfile
Agree
Quoted 24 lines
Quoted 24 lines
meta/lib/oe/sstatesig.py +6
-2+ if not where_to_dump: + where_to_dump = os.getcwd() + "/locked-sigs.inc" + + bb.plain("Writing locked sigs to %s" % where_to_dump) types = {} for k in self.runtaskdeps: fn = k.rsplit(".",1)[0]@@ -140,11 +150,11 @@ class SignatureGeneratorOEBasicHash(bb.siggen.SignatureGeneratorBasicHash): types[t] = [] types[t].append(k) - with open("locked-sigs.inc", "w") as f: + with open(where_to_dump, "w") as f: for t in types: f.write('SIGGEN_LOCKEDSIGS_%s = "\\\n' % t) types[t].sort() - sortedk = sorted(types[t], key=lambda k: self.lockedpnmap[k.rsplit(".",1)[0]]) + sortedk = sorted(types[t], key=lambda k: self.lockedpnmap[k.rsplit(".",1)[0]]) for k in sortedk: fn = k.rsplit(".",1)[0] task = k.rsplit(".",1)[1]@@ -155,17 +165,18 @@ class SignatureGeneratorOEBasicHash(bb.siggen.SignatureGeneratorBasicHash): f.write('SIGGEN_LOCKEDSIGS_TYPES_%s = "%s"' % (self.machine, " ".join(types.keys())))
Quoted 5 lines
meta/lib/oe/sstatesig.py class SignatureGeneratorOEBasicHash(bb.siggen.SignatureGeneratorBasicHash): · line 168
Quoted 4 lines
meta/lib/oe/sstatesig.py +1
-2def checkhashes(self, missed, ret, sq_fn, sq_task, sq_hash, sq_hashfn, d): - enforce = (d.getVar("SIGGEN_ENFORCE_LOCKEDSIGS", True) or "1") == "1" - msgs = [] + checklevel = d.getVar("SIGGEN_LOCKEDSIGS_CHECK_LEVEL", True)Perhaps this should default to error?
Agree, I will initial it with 'error' in class 'sstate'
Quoted 18 lines
Quoted 18 lines
meta/lib/oe/sstatesig.py +7
-5for task in range(len(sq_fn)): if task not in ret: for pn in self.lockedsigs: if sq_hash[task] in self.lockedsigs[pn].itervalues(): - msgs.append("Locked sig is set for %s:%s (%s) yet not in sstate cache?" % (pn, sq_task[task], sq_hash[task])) - if msgs and enforce: - bb.fatal("\n".join(msgs)) - elif msgs: - bb.warn("\n".join(msgs)) + self.checkmsgs.append("Locked sig is set for %s:%s (%s) yet not in sstate cache?" + % (pn, sq_task[task], sq_hash[task])) + + if self.checkmsgs and checklevel == 'warn': + bb.warn("\n".join(self.checkmsgs)) + elif self.checkmsgs and checklevel == 'error': + bb.fatal("\n".join(self.checkmsgs))
Quoted 6 lines
meta/lib/oe/sstatesig.py class SignatureGeneratorOEBasicHash(bb.siggen.SignatureGeneratorBasicHash): · line 182
Quoted 1 line
meta/lib/oe/sstatesig.py +0
-0# Insert these classes into siggen's namespace so it can see and select themI have an updated version of this patch at
http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t222&id=17c255cd72810616139e36ae8ad30cc4daefe9d1
if you're ok with the changes although I do need to update the commit message.
Got it, I will rebase it according the latest changes
//Hongxu
Quoted 5 lines
Cheers,
Richard
PATCH 5/5 sstatesig: incremental dump lockedsigs
In reply to Hongxu JiaWhile including an existed lockedsigs file to build from locked sstate-cache, and we want to incremental build the locked sstate-cache. Config with:
...SIGGEN_DUMP_LOCKEDSIGS = '1' SIGGEN_LOCKEDSIGS_CONFIG = "${TOPDIR}/locked-sigs.inc" require ${SIGGEN_LOCKEDSIGS_CONFIG}
...
So we support to dump lockedsigs file incrementally
Since we improve to handle locking of multiple machines and add a new SIGGEN_LOCKEDSIGS_TYPES variable which lists the package architectures to load in. We should add package architectures as type to self.lockedsigs to reflect that. Such as:
{recipename:{task:{hash1}}} --> {type:{recipename:{task:{hash1}}}}
--- meta/lib/oe/sstatesig.py | 67 ++++++++++++++++++++++++++++++++---------------- 1 file changed, 45 insertions(+), 22 deletions(-)
meta/lib/oe/sstatesig.py +45 -22
@@ -65,12 +65,15 @@ def sstate_lockedsigs(d): sigs = {} types = (d.getVar("SIGGEN_LOCKEDSIGS_TYPES", True) or "").split() for t in types: + if t not in sigs: + sigs[t] = {} + lockedsigs = (d.getVar("SIGGEN_LOCKEDSIGS_%s" % t, True) or "").split() for ls in lockedsigs: pn, task, h = ls.split(":", 2) - if pn not in sigs: - sigs[pn] = {} - sigs[pn][task] = h + if pn not in sigs[t]: + sigs[t][pn] = {} + sigs[t][pn][task] = h return sigs class SignatureGeneratorOEBasic(bb.siggen.SignatureGeneratorBasic):
@@ -115,10 +118,14 @@ class SignatureGeneratorOEBasicHash(bb.siggen.SignatureGeneratorBasicHash): recipename = dataCache.pkg_fn[fn] self.lockedpnmap[fn] = recipename self.lockedhashfn[fn] = dataCache.hashfn[fn] - if recipename in self.lockedsigs: - if task in self.lockedsigs[recipename]: + + t = self.lockedhashfn[fn].split(" ")[1].split(":")[5] + if t not in self.lockedsigs: + return h + if recipename in self.lockedsigs[t]: + if task in self.lockedsigs[t][recipename]: k = fn + "." + task - h_locked = self.lockedsigs[recipename][task] + h_locked = self.lockedsigs[t][recipename][task] self.lockedhashes[k] = h_locked self.taskhash[k] = h_locked #bb.warn("Using %s %s %s" % (recipename, task, h))
@@ -137,6 +144,19 @@ class SignatureGeneratorOEBasicHash(bb.siggen.SignatureGeneratorBasicHash): return super(bb.siggen.SignatureGeneratorBasicHash, self).dump_sigtask(fn, task, stampbase, runtime) + def update_lockedsigs(self, type, recipename, task, hash): + if type not in self.lockedsigs: + self.lockedsigs[type] = {} + + if recipename not in self.lockedsigs[type]: + self.lockedsigs[type][recipename] = {} + + if task not in self.lockedsigs[type][recipename]: + self.lockedsigs[type][recipename][task] = hash + elif hash != self.lockedsigs[type][recipename][task]: + bb.warn('Conflict %s %s %s (new %s, old %s)' % + (type, recipename, task, hash, self.lockedsigs[type][recipename][task])) + def dump_lockedsigs(self, where_to_dump=None): if not where_to_dump: where_to_dump = os.getcwd() + "/locked-sigs.inc"
@@ -146,30 +166,33 @@ class SignatureGeneratorOEBasicHash(bb.siggen.SignatureGeneratorBasicHash): for k in self.runtaskdeps: fn = k.rsplit(".",1)[0] t = self.lockedhashfn[fn].split(" ")[1].split(":")[5] - if t not in types: - types[t] = [] - types[t].append(k) + recipename = self.lockedpnmap[fn] + task = k.rsplit(".",1)[1] + hash = self.taskhash[k] + self.update_lockedsigs(t, recipename, task, hash) with open(where_to_dump, "w") as f: - for t in types: - f.write('SIGGEN_LOCKEDSIGS_%s = "\\\n' % t) - types[t].sort() - sortedk = sorted(types[t], key=lambda k: self.lockedpnmap[k.rsplit(".",1)[0]]) - for k in sortedk: - fn = k.rsplit(".",1)[0] - task = k.rsplit(".",1)[1] - if k not in self.taskhash: - continue - f.write(" " + self.lockedpnmap[fn] + ":" + task + ":" + self.taskhash[k] + " \\\n") + for type in sorted(self.lockedsigs): + f.write('SIGGEN_LOCKEDSIGS_%s = "\\\n' % type) + + for recipename in sorted(self.lockedsigs[type]): + for task in sorted(self.lockedsigs[type][recipename]): + hash = self.lockedsigs[type][recipename][task] + f.write(" " + recipename + ":" + task + ":" + hash + " \\\n") f.write(' "\n') - f.write('SIGGEN_LOCKEDSIGS_TYPES_%s = "%s"' % (self.machine, " ".join(types.keys()))) + + f.write('SIGGEN_LOCKEDSIGS_TYPES_%s = "%s"' % (self.machine, " ".join(self.lockedsigs.keys()))) def checkhashes(self, missed, ret, sq_fn, sq_task, sq_hash, sq_hashfn, d): checklevel = d.getVar("SIGGEN_LOCKEDSIGS_CHECK_LEVEL", True) for task in range(len(sq_fn)): if task not in ret: - for pn in self.lockedsigs: - if sq_hash[task] in self.lockedsigs[pn].itervalues(): + t = sq_hashfn[task].split(" ")[1].split(":")[5] + if t not in self.lockedsigs: + continue + + for pn in self.lockedsigs[t]: + if sq_hash[task] in self.lockedsigs[t][pn].itervalues(): self.checkmsgs.append("Locked sig is set for %s:%s (%s) yet not in sstate cache?" % (pn, sq_task[task], sq_hash[task]))
Signature · 1 line
1.9.1
Re: [PATCH 5/5] sstatesig: incremental dump lockedsigs
Quoted 17 lines · Hongxu Jia
While including an existed lockedsigs file to build from locked sstate-cache, and we want to incremental build the locked sstate-cache. Config with:
...SIGGEN_DUMP_LOCKEDSIGS= '1'SIGGEN_LOCKEDSIGS_CONFIG= "${TOPDIR}/locked-sigs.inc" require ${SIGGEN_LOCKEDSIGS_CONFIG}
...
So we support to dump lockedsigs file incrementallySince we improve to handle locking of multiple machines and add a new
SIGGEN_LOCKEDSIGS_TYPESvariable which lists the package architectures to load in. We should add package architectures as type to self.lockedsigs to reflect that. Such as:
{recipename:{task:{hash1}}} --> {type:{recipename:{task:{hash1}}}}
Quoted 4 lines
Signed-off-by Hongxu Jia <[email protected]>--- meta/lib/oe/sstatesig.py | 67 ++++++++++++++++++++++++++++++++---------------- 1 file changed, 45 insertions(+), 22 deletions(-)
I've stared long and hard at this patch and I simply don't understand why we need to complicate the structure of the lockedsigs variable.
I understand wanting to generate an incremental sig file, that much is fine, I just don't like the other seemingly unrelated change. Regardless, they should be split into separate patches.
Can you explain more about what you're trying to do?
Cheers,
Richard
Quoted 111 lines
meta/lib/oe/sstatesig.py +45
-22@@ -65,12 +65,15 @@ def sstate_lockedsigs(d): sigs = {} types = (d.getVar("SIGGEN_LOCKEDSIGS_TYPES", True) or "").split() for t in types: + if t not in sigs: + sigs[t] = {} + lockedsigs = (d.getVar("SIGGEN_LOCKEDSIGS_%s" % t, True) or "").split() for ls in lockedsigs: pn, task, h = ls.split(":", 2) - if pn not in sigs: - sigs[pn] = {} - sigs[pn][task] = h + if pn not in sigs[t]: + sigs[t][pn] = {} + sigs[t][pn][task] = h return sigs class SignatureGeneratorOEBasic(bb.siggen.SignatureGeneratorBasic):@@ -115,10 +118,14 @@ class SignatureGeneratorOEBasicHash(bb.siggen.SignatureGeneratorBasicHash): recipename = dataCache.pkg_fn[fn] self.lockedpnmap[fn] = recipename self.lockedhashfn[fn] = dataCache.hashfn[fn] - if recipename in self.lockedsigs: - if task in self.lockedsigs[recipename]: + + t = self.lockedhashfn[fn].split(" ")[1].split(":")[5] + if t not in self.lockedsigs: + return h + if recipename in self.lockedsigs[t]: + if task in self.lockedsigs[t][recipename]: k = fn + "." + task - h_locked = self.lockedsigs[recipename][task] + h_locked = self.lockedsigs[t][recipename][task] self.lockedhashes[k] = h_locked self.taskhash[k] = h_locked #bb.warn("Using %s %s %s" % (recipename, task, h))@@ -137,6 +144,19 @@ class SignatureGeneratorOEBasicHash(bb.siggen.SignatureGeneratorBasicHash): return super(bb.siggen.SignatureGeneratorBasicHash, self).dump_sigtask(fn, task, stampbase, runtime) + def update_lockedsigs(self, type, recipename, task, hash): + if type not in self.lockedsigs: + self.lockedsigs[type] = {} + + if recipename not in self.lockedsigs[type]: + self.lockedsigs[type][recipename] = {} + + if task not in self.lockedsigs[type][recipename]: + self.lockedsigs[type][recipename][task] = hash + elif hash != self.lockedsigs[type][recipename][task]: + bb.warn('Conflict %s %s %s (new %s, old %s)' % + (type, recipename, task, hash, self.lockedsigs[type][recipename][task])) + def dump_lockedsigs(self, where_to_dump=None): if not where_to_dump: where_to_dump = os.getcwd() + "/locked-sigs.inc"@@ -146,30 +166,33 @@ class SignatureGeneratorOEBasicHash(bb.siggen.SignatureGeneratorBasicHash): for k in self.runtaskdeps: fn = k.rsplit(".",1)[0] t = self.lockedhashfn[fn].split(" ")[1].split(":")[5] - if t not in types: - types[t] = [] - types[t].append(k) + recipename = self.lockedpnmap[fn] + task = k.rsplit(".",1)[1] + hash = self.taskhash[k] + self.update_lockedsigs(t, recipename, task, hash) with open(where_to_dump, "w") as f: - for t in types: - f.write('SIGGEN_LOCKEDSIGS_%s = "\\\n' % t) - types[t].sort() - sortedk = sorted(types[t], key=lambda k: self.lockedpnmap[k.rsplit(".",1)[0]]) - for k in sortedk: - fn = k.rsplit(".",1)[0] - task = k.rsplit(".",1)[1] - if k not in self.taskhash: - continue - f.write(" " + self.lockedpnmap[fn] + ":" + task + ":" + self.taskhash[k] + " \\\n") + for type in sorted(self.lockedsigs): + f.write('SIGGEN_LOCKEDSIGS_%s = "\\\n' % type) + + for recipename in sorted(self.lockedsigs[type]): + for task in sorted(self.lockedsigs[type][recipename]): + hash = self.lockedsigs[type][recipename][task] + f.write(" " + recipename + ":" + task + ":" + hash + " \\\n") f.write(' "\n') - f.write('SIGGEN_LOCKEDSIGS_TYPES_%s = "%s"' % (self.machine, " ".join(types.keys()))) + + f.write('SIGGEN_LOCKEDSIGS_TYPES_%s = "%s"' % (self.machine, " ".join(self.lockedsigs.keys()))) def checkhashes(self, missed, ret, sq_fn, sq_task, sq_hash, sq_hashfn, d): checklevel = d.getVar("SIGGEN_LOCKEDSIGS_CHECK_LEVEL", True) for task in range(len(sq_fn)): if task not in ret: - for pn in self.lockedsigs: - if sq_hash[task] in self.lockedsigs[pn].itervalues(): + t = sq_hashfn[task].split(" ")[1].split(":")[5] + if t not in self.lockedsigs: + continue + + for pn in self.lockedsigs[t]: + if sq_hash[task] in self.lockedsigs[t][pn].itervalues(): self.checkmsgs.append("Locked sig is set for %s:%s (%s) yet not in sstate cache?" % (pn, sq_task[task], sq_hash[task]))
Re: [PATCH 5/5] sstatesig: incremental dump lockedsigs
Quoted 18 lines · Richard Purdie
Quoted 17 lines · Hongxu Jia
While including an existed lockedsigs file to build from locked sstate-cache, and we want to incremental build the locked sstate-cache. Config with:
...SIGGEN_DUMP_LOCKEDSIGS= '1'SIGGEN_LOCKEDSIGS_CONFIG= "${TOPDIR}/locked-sigs.inc" require ${SIGGEN_LOCKEDSIGS_CONFIG}
...
So we support to dump lockedsigs file incrementallySince we improve to handle locking of multiple machines and add a new
SIGGEN_LOCKEDSIGS_TYPESvariable which lists the package architectures to load in. We should add package architectures as type to self.lockedsigs to reflect that. Such as:
{recipename:{task:{hash1}}} --> {type:{recipename:{task:{hash1}}}}
Quoted 6 lines
Quoted 4 lines
Signed-off-by Hongxu Jia <[email protected]>--- meta/lib/oe/sstatesig.py | 67 ++++++++++++++++++++++++++++++++---------------- 1 file changed, 45 insertions(+), 22 deletions(-)I've stared long and hard at this patch and I simply don't understand why we need to complicate the structure of the lockedsigs variable.
Ok, I will try to simplify and not complicate the structure of the lockedsigs variable.
Quoted 5 lines
I understand wanting to generate an incremental sig file, that much is fine, I just don't like the other seemingly unrelated change. Regardless, they should be split into separate patches.
Can you explain more about what you're trying to do?
I want to save self.lockedsigs to the new sig file.
The idea of incremental sig is:
New sig file = Old sig file (if available) + New sig items in current build.
The self.lockedsigs contains the old locked-sigs.inc but missing 'type' (machine), that's why I complicate it.
I will limit the modification within the dump_lockedsigs in V2, and I have to add two variables 'self.lockedsigs_types' and 'self.lockedsigs_raw' keep old sig file.
...
meta/lib/oe/sstatesig.py +4 -0
@@ -93,6 +93,10 @@ class SignatureGeneratorOEBasicHash(bb.siggen.SignatureGeneratorBasicHash): self.lockedhashfn = {} self.machine = data.getVar("MACHINE", True) self.mismatch_msgs = [] + self.lockedsigs_types = (data.getVar("SIGGEN_LOCKEDSIGS_TYPES", True) or "").split() + self.lockedsigs_raw = {} + for t in self.lockedsigs_types: + self.lockedsigs_raw[t] = (data.getVar("SIGGEN_LOCKEDSIGS_%s" % t, True) or "").split()
...
V2 incoming
//Hongxu
Quoted 117 lines
Cheers,
Richard
Quoted 111 lines
meta/lib/oe/sstatesig.py +45
-22@@ -65,12 +65,15 @@ def sstate_lockedsigs(d): sigs = {} types = (d.getVar("SIGGEN_LOCKEDSIGS_TYPES", True) or "").split() for t in types: + if t not in sigs: + sigs[t] = {} + lockedsigs = (d.getVar("SIGGEN_LOCKEDSIGS_%s" % t, True) or "").split() for ls in lockedsigs: pn, task, h = ls.split(":", 2) - if pn not in sigs: - sigs[pn] = {} - sigs[pn][task] = h + if pn not in sigs[t]: + sigs[t][pn] = {} + sigs[t][pn][task] = h return sigs class SignatureGeneratorOEBasic(bb.siggen.SignatureGeneratorBasic):@@ -115,10 +118,14 @@ class SignatureGeneratorOEBasicHash(bb.siggen.SignatureGeneratorBasicHash): recipename = dataCache.pkg_fn[fn] self.lockedpnmap[fn] = recipename self.lockedhashfn[fn] = dataCache.hashfn[fn] - if recipename in self.lockedsigs: - if task in self.lockedsigs[recipename]: + + t = self.lockedhashfn[fn].split(" ")[1].split(":")[5] + if t not in self.lockedsigs: + return h + if recipename in self.lockedsigs[t]: + if task in self.lockedsigs[t][recipename]: k = fn + "." + task - h_locked = self.lockedsigs[recipename][task] + h_locked = self.lockedsigs[t][recipename][task] self.lockedhashes[k] = h_locked self.taskhash[k] = h_locked #bb.warn("Using %s %s %s" % (recipename, task, h))@@ -137,6 +144,19 @@ class SignatureGeneratorOEBasicHash(bb.siggen.SignatureGeneratorBasicHash): return super(bb.siggen.SignatureGeneratorBasicHash, self).dump_sigtask(fn, task, stampbase, runtime) + def update_lockedsigs(self, type, recipename, task, hash): + if type not in self.lockedsigs: + self.lockedsigs[type] = {} + + if recipename not in self.lockedsigs[type]: + self.lockedsigs[type][recipename] = {} + + if task not in self.lockedsigs[type][recipename]: + self.lockedsigs[type][recipename][task] = hash + elif hash != self.lockedsigs[type][recipename][task]: + bb.warn('Conflict %s %s %s (new %s, old %s)' % + (type, recipename, task, hash, self.lockedsigs[type][recipename][task])) + def dump_lockedsigs(self, where_to_dump=None): if not where_to_dump: where_to_dump = os.getcwd() + "/locked-sigs.inc"@@ -146,30 +166,33 @@ class SignatureGeneratorOEBasicHash(bb.siggen.SignatureGeneratorBasicHash): for k in self.runtaskdeps: fn = k.rsplit(".",1)[0] t = self.lockedhashfn[fn].split(" ")[1].split(":")[5] - if t not in types: - types[t] = [] - types[t].append(k) + recipename = self.lockedpnmap[fn] + task = k.rsplit(".",1)[1] + hash = self.taskhash[k] + self.update_lockedsigs(t, recipename, task, hash) with open(where_to_dump, "w") as f: - for t in types: - f.write('SIGGEN_LOCKEDSIGS_%s = "\\\n' % t) - types[t].sort() - sortedk = sorted(types[t], key=lambda k: self.lockedpnmap[k.rsplit(".",1)[0]]) - for k in sortedk: - fn = k.rsplit(".",1)[0] - task = k.rsplit(".",1)[1] - if k not in self.taskhash: - continue - f.write(" " + self.lockedpnmap[fn] + ":" + task + ":" + self.taskhash[k] + " \\\n") + for type in sorted(self.lockedsigs): + f.write('SIGGEN_LOCKEDSIGS_%s = "\\\n' % type) + + for recipename in sorted(self.lockedsigs[type]): + for task in sorted(self.lockedsigs[type][recipename]): + hash = self.lockedsigs[type][recipename][task] + f.write(" " + recipename + ":" + task + ":" + hash + " \\\n") f.write(' "\n') - f.write('SIGGEN_LOCKEDSIGS_TYPES_%s = "%s"' % (self.machine, " ".join(types.keys()))) + + f.write('SIGGEN_LOCKEDSIGS_TYPES_%s = "%s"' % (self.machine, " ".join(self.lockedsigs.keys()))) def checkhashes(self, missed, ret, sq_fn, sq_task, sq_hash, sq_hashfn, d): checklevel = d.getVar("SIGGEN_LOCKEDSIGS_CHECK_LEVEL", True) for task in range(len(sq_fn)): if task not in ret: - for pn in self.lockedsigs: - if sq_hash[task] in self.lockedsigs[pn].itervalues(): + t = sq_hashfn[task].split(" ")[1].split(":")[5] + if t not in self.lockedsigs: + continue + + for pn in self.lockedsigs[t]: + if sq_hash[task] in self.lockedsigs[t][pn].itervalues(): self.checkmsgs.append("Locked sig is set for %s:%s (%s) yet not in sstate cache?" % (pn, sq_task[task], sq_hash[task]))