improve locked down sstate
v2 cover improve locked down sstate
Changed in V2:
- Set SIGGEN_LOCKEDSIGS_CHECK_LEVEL default to error
- Move dump sig file event handler to a separate class
- Fix typo
- Fix overrides behaviour to remove SIGGEN_LOCKEDSIGS_i586
- Support incremental dump lockedsigs without complicate the structure of the lockedsigs variable.
Test Steps:
1. Prepare a locked sstat-cache by dumping lockedsigs file
1) vim local.conf
...USER_CLASSES += "sstate_lockedsig" SIGGEN_LOCKEDSIGS_CONFIG = "/path/to/locked-sigs.inc" SSTATE_DIR ?= "/path/to/sstate-cache"
...
2) Build db-native from a clean environment
$ bitbake db-native
3) Back up lockedsigs file
$ mv /path/to/locked-sigs.inc /path/to/locked-sigs.inc-back
4) Incremental build ldconfig-native
$ vim local.conf
...
require ${SIGGEN_LOCKEDSIGS_CONFIG}
...
$ bitbake ldconfig-native5) Compare the two sig files, and check whether it is incremental dump or not
$ vimdiff /path/to/locked-sigs.inc-back /path/to/locked-sigs.inc
2. Create a new build which uses previous locked sstat-cache
1) vim local.conf
...SSTATE_DIR ?= "/path/to/sstate-cache" SIGGEN_LOCKEDSIGS_CONFIG = "/path/to/locked-sigs.inc" require ${SIGGEN_LOCKEDSIGS_CONFIG}
...
2) Build db-native
$ bitbake db-native
3) Tweak db's do_configure task by adding comments
meta/recipes-support/db/db_6.0.30.bb +1 -0
@@ -82,6 +82,7 @@ do_configure() { gnu-configize --force ${S} export STRIP="true" oe_runconf + echo "hello" }
4) Build db-native and abort
$ bitbake db-native ... ERROR: The db-native:do_configure sig (00546ab0e73b3afd50efc03f26e5f040) changed, use locked sig 9d8f2210506446e76d012d1ebd49a021 to instead The db-native:do_compile sig (4442f979d97bdbed71a8fecbbbe41ed3) changed, use locked sig d29a7c568d10859176d733773433fe92 to instead The db-native:do_install sig (562cb98d8eb25a7c79ded6a5359c56c1) changed, use locked sig 1e917d8d619c58987a1b2fe286e9288c to instead The db-native:do_populate_sysroot sig (91627e36da9ac8fe489f36b67a87ed2e) changed, use locked sig f780d45dbf680e40a871820327c4c56c to instead The db-native:do_build sig (ba289cba4a3bfda0c07aed9868fdcd5d) changed, use locked sig d9b84e5fb30011fd31bc0e7d7652fead to instead ...
5) Remove db-native sstate-cache
$ bitbake db-native -ccleansstate
6) Build db-native and abort
$ bitbake db-native ... Locked sig is set for db-native:do_populate_sysroot (f780d45dbf680e40a871820327c4c56c) yet not in sstate cache? ...
7) Do not break the build
vim local.conf
...SIGGEN_LOCKEDSIGS_CHECK_LEVEL = "warn"
...
8) Build db-native
$ bitbake db-native ... WARNING Locked sig is set for db-native:do_populate_sysroot (f780d45dbf680e40a871820327c4c56c) yet not in sstate cache? ...
//Hongxu
The following changes since commit 7d80f8e9468253496a7097685aac8f468940a9c5:
bitbake: data_smart: Clarify what 'computed' means in the data store history context (2014-09-16 22:15:24 +0100)
are available in the git repository at:
git://git.pokylinux.org/poky-contrib hongxu/locked-sstate
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=hongxu/locked-sstate
Hongxu Jia (5):
sstate: set SIGGEN_LOCKEDSIGS_CHECK_LEVEL default to error sstate_lockedsig.bbclass: add event handler to dump lockedsigs at
recipe building time
sstatesig.py: fix typo of locke sstate
sstatesig: fix overrides behaviour to remove SIGGEN_LOCKEDSIGS_i586
sstatesig: incremental dump lockedsigs
[email protected] (4):
Richard Purdie : sstatesig/sstate: Add support for locked down sstate
cache usage
Richard Purdie : sstatesig: Improve to handle locking of multiple
machines
Randy Witt : sstatesig.py: Replace '_'s with '-' s in
SSTATE_LOCKEDSIGS* names.
Hongxu Jia : sstatesig: Improve the support for locked down sstate
cache usage
meta/classes/sstate.bbclass | 5 ++ meta/classes/sstate_lockedsig.bbclass | 12 ++++ meta/lib/oe/sstatesig.py | 106 ++++++++++++++++++++++++++++++++++ scripts/gen-lockedsig-cache | 40 +++++++++++++ 4 files changed, 163 insertions(+) create mode 100644 meta/classes/sstate_lockedsig.bbclass create mode 100755 scripts/gen-lockedsig-cache
Signature · 1 line
1.9.1
PATCH 5/9 sstate: set SIGGEN_LOCKEDSIGS_CHECK_LEVEL default to error
--- meta/classes/sstate.bbclass | 2 ++ 1 file changed, 2 insertions(+)
meta/classes/sstate.bbclass +2 -0
@@ -39,6 +39,8 @@ SSTATEPOSTUNPACKFUNCS = "sstate_hardcode_path_unpack" SSTATEPOSTINSTFUNCS = "" EXTRA_STAGING_FIXMES ?= "" +SIGGEN_LOCKEDSIGS_CHECK_LEVEL ?= 'error' + # Specify dirs in which the shell function is executed and don't use ${B} # as default dirs to avoid possible race about ${B} with other task. sstate_create_package[dirs] = "${SSTATE_BUILDDIR}"
Signature · 1 line
1.9.1
PATCH 6/9 sstate_lockedsig.bbclass: add event handler to dump lockedsigs at recipe building time
In reply to Hongxu JiaWhile the class in use, it adds a event handler at 'bb.event.BuildCompleted', so after the build completed, it will dump the lockedsigs files.
Use SIGGEN_LOCKEDSIGS_CONFIG variable controls where to dump the lockedsigs file, the default is placed into ${SSTATE_DIR}/locked-sigs.inc.
--- meta/classes/sstate_lockedsig.bbclass | 12 ++++++++++++ 1 file changed, 12 insertions(+) create mode 100644 meta/classes/sstate_lockedsig.bbclass
meta/classes/sstate_lockedsig.bbclass new file +12 -0
@@ -0,0 +1,12 @@ +SIGGEN_LOCKEDSIGS_CONFIG ?= "${SSTATE_DIR}/locked-sigs.inc" +addhandler sstate_dump_lockedsig +sstate_dump_lockedsig[eventmask] = "bb.event.BuildCompleted" +python sstate_dump_lockedsig() { + d = e.data + if e.getFailures(): + return + + if hasattr(bb.parse.siggen, "dump_lockedsigs"): + lockedsigs = d.getVar('SIGGEN_LOCKEDSIGS_CONFIG', True) + bb.parse.siggen.dump_lockedsigs(lockedsigs) +}
Signature · 1 line
1.9.1
Re: [PATCH 6/9] sstate_lockedsig.bbclass: add event handler to dump lockedsigs at recipe building time
Hi Richard,
Is this patch not acceptable, it is not merged to master or master-next.
//Hongxu
Quoted 30 lines · Hongxu Jia
While the class in use, it adds a event handler at 'bb.event.BuildCompleted', so after the build completed, it will dump the lockedsigs files.
Use
SIGGEN_LOCKEDSIGS_CONFIGvariable controls where to dump the lockedsigs file, the default is placed into ${SSTATE_DIR}/locked-sigs.inc.Signed-off-by Hongxu Jia <[email protected]>--- meta/classes/sstate_lockedsig.bbclass | 12 ++++++++++++ 1 file changed, 12 insertions(+)create mode 100644 meta/classes/sstate_lockedsig.bbclass
meta/classes/sstate_lockedsig.bbclass new file +12
-0@@ -0,0 +1,12 @@ +SIGGEN_LOCKEDSIGS_CONFIG ?= "${SSTATE_DIR}/locked-sigs.inc" +addhandler sstate_dump_lockedsig +sstate_dump_lockedsig[eventmask] = "bb.event.BuildCompleted" +python sstate_dump_lockedsig() { + d = e.data + if e.getFailures(): + return + + if hasattr(bb.parse.siggen, "dump_lockedsigs"): + lockedsigs = d.getVar('SIGGEN_LOCKEDSIGS_CONFIG', True) + bb.parse.siggen.dump_lockedsigs(lockedsigs) +}
PATCH 7/9 sstatesig.py: fix typo of locke sstate
In reply to Hongxu JiaWe used sigfile to instead of where_to_dump. Fix the missing one.
--- meta/lib/oe/sstatesig.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
meta/lib/oe/sstatesig.py +1 -1
@@ -138,7 +138,7 @@ class SignatureGeneratorOEBasicHash(bb.siggen.SignatureGeneratorBasicHash): super(bb.siggen.SignatureGeneratorBasicHash, self).dump_sigtask(fn, task, stampbase, runtime) def dump_lockedsigs(self, sigfile=None): - if not where_to_dump: + if not sigfile: sigfile = os.getcwd() + "/locked-sigs.inc" bb.plain("Writing locked sigs to %s" % sigfile)
Signature · 1 line
1.9.1
PATCH 8/9 sstatesig: fix overrides behaviour to remove SIGGEN_LOCKEDSIGS_i586
In reply to Hongxu JiaRequire a sig file which SIGGEN_LOCKEDSIGS_i586 is not null, but the actual SIGGEN_LOCKEDSIGS_i586 is null.
Invoking 'bitbake -e' and we got:
...
3935 # $SIGGEN_LOCKEDSIGS_i586 [2 operations]
3936 # set /path/to/locked-sigs.inc:8576
3938 # del data_smart.py:406 [finalize]
3939 # ""
...
It was caused by the following commit:
...
(Bitbake rev: 899d45b90061eb3cf3e71029072eee42cd80930c)
Author: Richard Purdie <[email protected]> Date: Tue May 31 23:52:50 2011 +0100
bitbake/data_smart: Change overrides behaviour to remove expanded
variables from the datastore
...
We add prefix 't-' to type to workaround the overrides behaviour.
--- meta/lib/oe/sstatesig.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
meta/lib/oe/sstatesig.py +1 -1
@@ -146,7 +146,7 @@ class SignatureGeneratorOEBasicHash(bb.siggen.SignatureGeneratorBasicHash): for k in self.runtaskdeps: fn = k.rsplit(".",1)[0] t = self.lockedhashfn[fn].split(" ")[1].split(":")[5] - t = t.replace('_', '-') + t = 't-' + t.replace('_', '-') if t not in types: types[t] = [] types[t].append(k)
Signature · 1 line
1.9.1
PATCH 9/9 sstatesig: incremental dump lockedsigs
In reply to Hongxu JiaThe idea of incremental sig is:
New sig file = Old sig file (if available) + New sig items in current build.
Limit the modification within the dump_lockedsigs, and add two variables 'self.lockedsigs_types' and 'self.lockedsigs_raw' keep old sig file.
How to config for incremental dump:
...USER_CLASSES += "sstate_lockedsig" SIGGEN_LOCKEDSIGS_CONFIG = "${TOPDIR}/locked-sigs.inc" require ${SIGGEN_LOCKEDSIGS_CONFIG}
...
--- meta/lib/oe/sstatesig.py | 33 +++++++++++++++++++-------------- 1 file changed, 19 insertions(+), 14 deletions(-)
meta/lib/oe/sstatesig.py +19 -14
@@ -93,6 +93,10 @@ class SignatureGeneratorOEBasicHash(bb.siggen.SignatureGeneratorBasicHash): self.lockedhashfn = {} self.machine = data.getVar("MACHINE", True) self.mismatch_msgs = [] + self.lockedsigs_types = (data.getVar("SIGGEN_LOCKEDSIGS_TYPES", True) or "").split() + self.lockedsigs_raw = {} + for t in self.lockedsigs_types: + self.lockedsigs_raw[t] = (data.getVar("SIGGEN_LOCKEDSIGS_%s" % t, True) or "").split() pass def rundep_check(self, fn, recipename, task, dep, depname, dataCache = None): return sstate_rundepfilter(self, fn, recipename, task, dep, depname, dataCache)
@@ -142,28 +146,29 @@ class SignatureGeneratorOEBasicHash(bb.siggen.SignatureGeneratorBasicHash): sigfile = os.getcwd() + "/locked-sigs.inc" bb.plain("Writing locked sigs to %s" % sigfile) - types = {} for k in self.runtaskdeps: fn = k.rsplit(".",1)[0] t = self.lockedhashfn[fn].split(" ")[1].split(":")[5] t = 't-' + t.replace('_', '-') - if t not in types: - types[t] = [] - types[t].append(k) + if t not in self.lockedsigs_types: + self.lockedsigs_types.append(t) + self.lockedsigs_raw[t] = [] + + pn = self.lockedpnmap[fn] + task = k.rsplit(".",1)[1] + hash = self.taskhash[k] + lockedsig = "%s:%s:%s" % (pn, task, hash) + if lockedsig not in self.lockedsigs_raw[t]: + self.lockedsigs_raw[t].append(lockedsig) with open(sigfile, "w") as f: - for t in types: + for t in sorted(self.lockedsigs_types): f.write('SIGGEN_LOCKEDSIGS_%s = "\\\n' % t) - types[t].sort() - sortedk = sorted(types[t], key=lambda k: self.lockedpnmap[k.rsplit(".",1)[0]]) - for k in sortedk: - fn = k.rsplit(".",1)[0] - task = k.rsplit(".",1)[1] - if k not in self.taskhash: - continue - f.write(" " + self.lockedpnmap[fn] + ":" + task + ":" + self.taskhash[k] + " \\\n") + for ls in sorted(self.lockedsigs_raw[t]): + f.write(" " + ls + " \\\n") f.write(' "\n') - f.write('SIGGEN_LOCKEDSIGS_TYPES_%s = "%s"' % (self.machine, " ".join(types.keys()))) + f.write('SIGGEN_LOCKEDSIGS_TYPES_%s = "%s"' % + (self.machine, " ".join(sorted(self.lockedsigs_types)))) def checkhashes(self, missed, ret, sq_fn, sq_task, sq_hash, sq_hashfn, d): checklevel = d.getVar("SIGGEN_LOCKEDSIGS_CHECK_LEVEL", True)
Signature · 1 line
1.9.1
Re: [PATCH 9/9] sstatesig: incremental dump lockedsigs
Quoted 7 lines · Hongxu Jia
The idea of incremental sig is:
New sig file = Old sig file (if available) + New sig items in current build.
Limit the modification within the
dump_lockedsigs, and add two variables 'self.lockedsigs_types' and 'self.lockedsigs_raw' keep old sig file.
Quoted 8 lines
How to config for incremental dump:
...USER_CLASSES+= "sstate_lockedsig"SIGGEN_LOCKEDSIGS_CONFIG= "${TOPDIR}/locked-sigs.inc" require ${SIGGEN_LOCKEDSIGS_CONFIG}
...Signed-off-by Hongxu Jia <[email protected]>
I'm fine with the idea in principle. Why can't we do something like:
meta/lib/oe/sstatesig.py +5 -1
@@ -158,10 +158,14 @@ class SignatureGeneratorOEBasicHash(bb.siggen.SignatureGeneratorBasicHash): sortedk = sorted(types[t], key=lambda k: self.lockedpnmap[k.rsplit(".",1)[0]]) for k in sortedk: fn = k.rsplit(".",1)[0] + pn = self.lockedpnmap[fn] task = k.rsplit(".",1)[1] if k not in self.taskhash: continue - f.write(" " + self.lockedpnmap[fn] + ":" + task + ":" + self.taskhash[k] + " \\\n") + if pn in self.lockedsigs and task in self.lockedsigs[pn] and self.hashtask[k] == self.lockedsigs[pn][task]: + continue + sigentry = pn + ":" + task + ":" + self.taskhash[k] + f.write(" " + sigentry + " \\\n") f.write(' "\n') f.write('SIGGEN_LOCKEDSIGS_TYPES_%s = "%s"' % (self.machine, " ".join(types.keys())))
which is substantially simpler though?
Cheers,
Richard
Re: [PATCH 9/9] sstatesig: incremental dump lockedsigs
Quoted 18 lines · Richard Purdie
Quoted 7 lines · Hongxu Jia
The idea of incremental sig is:
New sig file = Old sig file (if available) + New sig items in current build.
Limit the modification within the
dump_lockedsigs, and add two variables 'self.lockedsigs_types' and 'self.lockedsigs_raw' keep old sig file.Quoted 8 lines
How to config for incremental dump:
...USER_CLASSES+= "sstate_lockedsig"SIGGEN_LOCKEDSIGS_CONFIG= "${TOPDIR}/locked-sigs.inc" require ${SIGGEN_LOCKEDSIGS_CONFIG}
...Signed-off-by Hongxu Jia <[email protected]>I'm fine with the idea in principle. Why can't we do something like:
...
Quoted 1 line
which is substantially simpler though?
Wouldn't it keep accumulating locked hashes for same recipes?
Signature · 3 lines
Otavio Salvador O.S. Systems http://www.ossystems.com.br http://code.ossystems.com.br Mobile: +55 (53) 9981-7854 Mobile: +1 (347) 903-9750
Re: [PATCH 9/9] sstatesig: incremental dump lockedsigs
Quoted 17 lines · Otavio Salvador
Quoted 15 lines · Richard Purdie
Quoted 14 lines · Hongxu Jia
The idea of incremental sig is:
New sig file = Old sig file (if available) + New sig items in current build.
Limit the modification within the
dump_lockedsigs, and add two variables 'self.lockedsigs_types' and 'self.lockedsigs_raw' keep old sig file.How to config for incremental dump:
...USER_CLASSES+= "sstate_lockedsig"SIGGEN_LOCKEDSIGS_CONFIG= "${TOPDIR}/locked-sigs.inc" require ${SIGGEN_LOCKEDSIGS_CONFIG}
...
Quoted 5 lines
Quoted 2 lines
Quoted 1 line
Signed-off-by Hongxu Jia <[email protected]>I'm fine with the idea in principle. Why can't we do something like:
...
Quoted 1 line
which is substantially simpler though?
Wouldn't it keep accumulating locked hashes for same recipes?
Of course not, checking added before write to file
...
+ if pn in self.lockedsigs and task in self.lockedsigs[pn] and self.hashtask[k] == self.lockedsigs[pn][task]:
+ continue
...
But there are still accumulating empty SIGGEN_LOCKEDSIGS added, such as:
...
SIGGEN_LOCKEDSIGS_t-x86-64 += "\
"SIGGEN_LOCKEDSIGS_TYPES_qemux86 += "t-x86-64"
...
I will fix it in V3
//Hongxu
Quoted 2 lines
Re: [PATCH 9/9] sstatesig: incremental dump lockedsigs
In reply to Richard PurdieQuoted 39 lines · Richard Purdie
Quoted 7 lines · Hongxu Jia
The idea of incremental sig is:
New sig file = Old sig file (if available) + New sig items in current build.
Limit the modification within the
dump_lockedsigs, and add two variables 'self.lockedsigs_types' and 'self.lockedsigs_raw' keep old sig file.Quoted 8 lines
How to config for incremental dump:
...USER_CLASSES+= "sstate_lockedsig"SIGGEN_LOCKEDSIGS_CONFIG= "${TOPDIR}/locked-sigs.inc" require ${SIGGEN_LOCKEDSIGS_CONFIG}
...Signed-off-by Hongxu Jia <[email protected]>I'm fine with the idea in principle. Why can't we do something like:
meta/lib/oe/sstatesig.py +5
-1@@ -158,10 +158,14 @@ class SignatureGeneratorOEBasicHash(bb.siggen.SignatureGeneratorBasicHash): sortedk = sorted(types[t], key=lambda k: self.lockedpnmap[k.rsplit(".",1)[0]]) for k in sortedk: fn = k.rsplit(".",1)[0] + pn = self.lockedpnmap[fn] task = k.rsplit(".",1)[1] if k not in self.taskhash: continue - f.write(" " + self.lockedpnmap[fn] + ":" + task + ":" + self.taskhash[k] + " \\\n") + if pn in self.lockedsigs and task in self.lockedsigs[pn] and self.hashtask[k] == self.lockedsigs[pn][task]: + continue + sigentry = pn + ":" + task + ":" + self.taskhash[k] + f.write(" " + sigentry + " \\\n") f.write(' "\n') f.write('SIGGEN_LOCKEDSIGS_TYPES_%s = "%s"' % (self.machine, " ".join(types.keys())))
Quoted 1 line
which is substantially simpler though?
Agree, thanks for the improvement, I will send V3 with some trivial fixes.
//Hongxu
Quoted 6 lines
Cheers,
Richard