bbclass/sstate: only allowed sstate-cache objects are allowed in a build (read-only sstate-cache)
cover bbclass/sstate: only allowed sstate-cache objects are allowed in a build (read-only sstate-cache)
Test steps:
1) Create a sstate cache for testing
$ bitbake db
2) Add gzip to SSTATECACHE_WHITELIST, it enabled read-only sstate-cache, vim local.conf
...SSTATECACHE_WHITELIST = 'gzip'
...
3) Remove tmp dir and build db from sstate-cache succeed
$ mv tmp tmp-back && bitbake db
4) tweak db's do_configure task by adding comments
meta/recipes-support/db/db_6.0.30.bb +1 -0
@@ -27,6 +27,8 @@ LIC_FILES_CHKSUM = "file://../LICENSE;md5=1ec8b0b17cc31513fe35ab10716f8490"
@@ -82,6 +84,7 @@ do_configure() { gnu-configize --force ${S} export STRIP="true" oe_runconf + echo "hello" }
5) build db and there is a build failure
$ bitbake db
6) clean db and there is a clean failure
bitbake db -ccleansstate
7) Append the missing recipe's PN to SSTATECACHE_WHITELIST
vim local.conf
...SSTATECACHE_WHITELIST = 'gzip db rpm-native gcc-runtime eglibc linux-libc-headers libgcc'
...
8) Build db and clean db succeed
$ bitbake db && bitbake db -ccleansstate
//Hongxu
The following changes since commit 34436672f3ef4915e7526770a0fa8dcff328f93d:
bitbake: runqueue.py: Fix typoes/grammar in comments. (2014-08-19 20:41:26 +0100)
are available in the git repository at:
git://git.pokylinux.org/poky-contrib hongxu/readonly-sstatecache
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=hongxu/readonly-sstatecache
Hongxu Jia (1):
bbclass/sstate: only allowed sstate-cache objects are allowed in a
build (read-only sstate-cache)
meta/classes/sstate.bbclass | 40 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+)
Signature · 1 line
1.9.1
PATCH 1/1 bbclass/sstate: only allowed sstate-cache objects are allowed in a build (read-only sstate-cache)
The requirement is the developer who demand only the "new" software they write is allowed to be compiled from source, they only want to reuse binaries from an existed sstate-cache, if the developer makes a change that triggers a rebuild, it should be an instant error.
The purpose of this is for the sstate-cache to check if the item exists or not. If it doesn't the item needs to be in a whitelist or we need to fail.
In the sstate-cache code, add a checking in the return path of sstate_checkhashes. If read-only sstate-cache enable, and the recipe's ${PN} not in the ${SSTATECACHE_WHITELIST}, it trigered an instant error.
...
$ bitbake db ERROR: Read-only sstate-cache is enabled, the build of "db rpm-native gcc-runtime eglibc linux-libc-headers libgcc" did not come from sstate-cache. Only the recipe listed in SSTATECACHE_WHITELIST is allowed to build from source
Summary: There was 1 ERROR message shown, returning a non-zero exit code.
...
[YOCTO #6639]
--- meta/classes/sstate.bbclass | 40 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+)
meta/classes/sstate.bbclass +40 -0
@@ -41,6 +41,14 @@ EXTRA_STAGING_FIXMES ?= "" sstate_create_package[dirs] = "${SSTATE_BUILDDIR}" sstate_unpack_package[dirs] = "${SSTATE_INSTDIR}" +# 1) If ${SSTATECACHE_WHITELIST} is "", it means read-only sstate-cache +# disabled; +# +# 2) If read-only sstate-cache enabled and the recipe's ${PN} not listed +# in ${SSTATECACHE_WHITELIST}, the build from source will triger an +# instant error; +SSTATECACHE_WHITELIST ?= "" + python () { if bb.data.inherits_class('native', d): d.setVar('SSTATE_PKGARCH', d.getVar('BUILD_ARCH'))
@@ -382,6 +390,15 @@ sstate_clean[vardepsexclude] = "SSTATE_MANFILEPREFIX" CLEANFUNCS += "sstate_cleanall" python sstate_cleanall() { + whitelist = d.getVar('SSTATECACHE_WHITELIST', True) + if whitelist: + pn = d.getVar('PN', True) + if pn not in whitelist.split(): + msg = 'Read-only sstate-cache is enabled, the clean of \n' + msg += '%s is not allowed. Only the recipe listed in\n' % pn + msg += 'SSTATECACHE_WHITELIST is allowed to clean sstate-cache' + bb.fatal(msg) + bb.note("Removing shared state for package %s" % d.getVar('PN', True)) manifest_dir = d.getVar('SSTATE_MANIFESTS', True)
@@ -704,6 +721,29 @@ def sstate_checkhashes(sq_fn, sq_task, sq_hash, sq_hashfn, d): evdata['found'].append( (sq_fn[task], sq_task[task], sq_hash[task], sstatefile ) ) bb.event.fire(bb.event.MetadataEvent("MissedSstate", evdata), d) + whitelist = d.getVar('SSTATECACHE_WHITELIST', True) + if whitelist: + missed_pn = [] + for task in missed: + fn = sq_fn[task] + data = bb.cache.Cache.loadDataFull(fn, '', d) + pn = data.getVar('PN', True) or "" + if pn and pn not in missed_pn: + missed_pn.append(pn) + + if missed_pn: + blacklist = [pn for pn in missed_pn if pn not in whitelist.split()] + if blacklist: + # We should manually unlock the bitbake lock, because the fatal + # msg will exit the build immediately. + lockfile = d.expand("${TOPDIR}/bitbake.lock") + os.unlink(lockfile) + msg = 'Read-only sstate-cache is enabled, the build of \n' + msg += '"' + ' '.join(blacklist) + '"\n' + msg += 'did not come from sstate-cache. Only the recipe listed in\n' + msg += 'SSTATECACHE_WHITELIST is allowed to build from source' + bb.msg.fatal('sstate', msg) + return ret BB_SETSCENE_DEPVALID = "setscene_depvalid"
Signature · 1 line
1.9.1
Hi Hongxu,
Quoted 14 lines · Hongxu Jia
The requirement is the developer who demand only the "new" software they write is allowed to be compiled from source, they only want to reuse binaries from an existed sstate-cache, if the developer makes a change that triggers a rebuild, it should be an instant error.
The purpose of this is for the sstate-cache to check if the item exists or not. If it doesn't the item needs to be in a whitelist or we need to fail.
In the sstate-cache code, add a checking in the return path of
sstate_checkhashes. If read-only sstate-cache enable, and the recipe's ${PN} not in the ${SSTATECACHE_WHITELIST}, it trigered an instant error.
Quoted 8 lines
...
$ bitbake db ERROR: Read-only sstate-cache is enabled, the build of "db rpm-native gcc-runtime eglibc linux-libc-headers libgcc" did not come from sstate-cache. Only the recipe listed in SSTATECACHE_WHITELIST is allowed to build from sourceSummary: There was 1 ERROR message shown, returning a non-zero exit code.
Sorry for the delay in getting to this, with the other patches for the release, time has been difficult.
I have been able to take a look at this and the other locked sstate patches I've had pending for some time.
Having thought quite a bit about this, I think we really want to make this functionality part of the siggen class code. Where we need to add hooks, we should do so with callbacks into the siggen code.
I've just sent out a patch which my locked code in it. Could you take a look and see if we can make that approach work with your code too?
In particular, I don't really want to have multiple whitelist type variables. Could we use the SIGGEN_LOCKEDSIGS variable as the definitive way to control which recipes can float and which should not?
Cheers,
Richard
Quoted 36 lines · Richard Purdie
Hi Hongxu,
Quoted 14 lines · Hongxu Jia
The requirement is the developer who demand only the "new" software they write is allowed to be compiled from source, they only want to reuse binaries from an existed sstate-cache, if the developer makes a change that triggers a rebuild, it should be an instant error.
The purpose of this is for the sstate-cache to check if the item exists or not. If it doesn't the item needs to be in a whitelist or we need to fail.
In the sstate-cache code, add a checking in the return path of
sstate_checkhashes. If read-only sstate-cache enable, and the recipe's ${PN} not in the ${SSTATECACHE_WHITELIST}, it trigered an instant error.Quoted 8 lines
...
$ bitbake db ERROR: Read-only sstate-cache is enabled, the build of "db rpm-native gcc-runtime eglibc linux-libc-headers libgcc" did not come from sstate-cache. Only the recipe listed in SSTATECACHE_WHITELIST is allowed to build from sourceSummary: There was 1 ERROR message shown, returning a non-zero exit code.
Sorry for the delay in getting to this, with the other patches for the release, time has been difficult.
I have been able to take a look at this and the other locked sstate patches I've had pending for some time.
Having thought quite a bit about this, I think we really want to make this functionality part of the siggen class code. Where we need to add hooks, we should do so with callbacks into the siggen code.
Quoted 6 lines
I've just sent out a patch which my locked code in it. Could you take a look and see if we can make that approach work with your code too?
In particular, I don't really want to have multiple whitelist type variables. Could we use the
SIGGEN_LOCKEDSIGSvariable as the definitive way to control which recipes can float and which should not?
I'm all for combining this.. however it appears the patches have two slightly separate goals.
The locked sstate patch tells the system to use a specific hash value and ONLY that hash value. So if a user modified a system behavior in a way that would change the behavior of the component there is no warning or error.
The readonly sstate-cache on the other hand says, to reuse the sstate cache items (there of course may be more then one per recipe name with a different hash based on different configurations), but will warn/error when if the system configuration changes in a way the hash would have changed.
So there needs to be an interface that says which items are allowed to be built from source, and which items require either the locked or readonly sstate behavior.
Something like:
SIGGEN_LOCKEDSIGS = "\
gcc-cross:do_populate_sysroot:ro \
eglibc:do_populate_sysroot:ro \
eglibc:do_packagedata:ro \
gcc-cross:do_packagedata:ro \
"
could work to extend it, but I have a concern with this. It only affects individual recipes. The request we've had from our customers is that once they have built their environment(s), and distributed the sstate-cache to their developers -- only the items they distributed are allowed to be used, unless an exception is granted.
So having to maintain a SIGGEN_LOCKSIGS = variable with every possible package in the system seems complicated at best (for the RO case). They really want a switch that sets everything, except exceptions as read-only.
SIGGEN_LOCKEDSIGS = "\
*:*:ro \
bash:*:rw \
"
but I don't know if the glob/wildcarding would add more processing overhead then is acceptable.
--Mark
Quoted 4 lines
Cheers,
Richard
Quoted 78 lines · Mark Hatle
Quoted 36 lines · Richard Purdie
Hi Hongxu,
Quoted 14 lines · Hongxu Jia
The requirement is the developer who demand only the "new" software they write is allowed to be compiled from source, they only want to reuse binaries from an existed sstate-cache, if the developer makes a change that triggers a rebuild, it should be an instant error.
The purpose of this is for the sstate-cache to check if the item exists or not. If it doesn't the item needs to be in a whitelist or we need to fail.
In the sstate-cache code, add a checking in the return path of
sstate_checkhashes. If read-only sstate-cache enable, and the recipe's ${PN} not in the ${SSTATECACHE_WHITELIST}, it trigered an instant error.Quoted 8 lines
...
$ bitbake db ERROR: Read-only sstate-cache is enabled, the build of "db rpm-native gcc-runtime eglibc linux-libc-headers libgcc" did not come from sstate-cache. Only the recipe listed in SSTATECACHE_WHITELIST is allowed to build from sourceSummary: There was 1 ERROR message shown, returning a non-zero exit code.
Sorry for the delay in getting to this, with the other patches for the release, time has been difficult.
I have been able to take a look at this and the other locked sstate patches I've had pending for some time.
Having thought quite a bit about this, I think we really want to make this functionality part of the siggen class code. Where we need to add hooks, we should do so with callbacks into the siggen code.
Quoted 6 lines
I've just sent out a patch which my locked code in it. Could you take a look and see if we can make that approach work with your code too?
In particular, I don't really want to have multiple whitelist type variables. Could we use the
SIGGEN_LOCKEDSIGSvariable as the definitive way to control which recipes can float and which should not?I'm all for combining this.. however it appears the patches have two slightly separate goals.
The locked sstate patch tells the system to use a specific hash value and ONLY
that hash value. So if a user modified a system behavior in a way that would
change the behavior of the component there is no warning or error.The readonly sstate-cache on the other hand says, to reuse the sstate cache items (there of course may be more then one per recipe name with a different hash based on different configurations), but will warn/error when if the system configuration changes in a way the hash would have changed.
So there needs to be an interface that says which items are allowed to be built from source, and which items require either the locked or readonly sstate behavior.
Something like:
SIGGEN_LOCKEDSIGS= "\
gcc-cross:do_populate_sysroot:ro \
eglibc:do_populate_sysroot:ro \
eglibc:do_packagedata:ro \
gcc-cross:do_packagedata:ro \
"could work to extend it, but I have a concern with this. It only affects individual recipes. The request we've had from our customers is that once they have built their environment(s), and distributed the sstate-cache to their developers -- only the items they distributed are allowed to be used, unless an exception is granted.
So having to maintain a
SIGGEN_LOCKSIGS= variable with every possible package in the system seems complicated at best (for the RO case). They really want a switch that sets everything, except exceptions as read-only.
Quoted 7 lines
SIGGEN_LOCKEDSIGS= "\
*:*:ro \
bash:*:rw \
"but I don't know if the glob/wildcarding would add more processing overhead then is acceptable.
I guess my main point right now is more about the controlling interface of this which I believe should be the "siggen" class. What I'd like to try and do is standardise on that as being the interface which handles this policy, whatever it may be. If it needs hooks adding, we can do that like the one I added from sstate.
Whether we end up with one siggen class or we have different ones (perhaps subclassing each other) and we switch between them for different behaviours I'm less sure about right now.
I'll give the locked sstate interface a bit more thought based on the above, thanks. I suspect it may be a case of adding a new subclass with some different functionality (e.g. readonly cache), then if that is what the user requires, we select that class using BB_SIGNATURE_HANDLER?
Richard
Quoted 93 lines · Richard Purdie
Quoted 78 lines · Mark Hatle
Quoted 36 lines · Richard Purdie
Hi Hongxu,
Quoted 14 lines · Hongxu Jia
The requirement is the developer who demand only the "new" software they write is allowed to be compiled from source, they only want to reuse binaries from an existed sstate-cache, if the developer makes a change that triggers a rebuild, it should be an instant error.
The purpose of this is for the sstate-cache to check if the item exists or not. If it doesn't the item needs to be in a whitelist or we need to fail.
In the sstate-cache code, add a checking in the return path of
sstate_checkhashes. If read-only sstate-cache enable, and the recipe's ${PN} not in the ${SSTATECACHE_WHITELIST}, it trigered an instant error.Quoted 8 lines
...
$ bitbake db ERROR: Read-only sstate-cache is enabled, the build of "db rpm-native gcc-runtime eglibc linux-libc-headers libgcc" did not come from sstate-cache. Only the recipe listed in SSTATECACHE_WHITELIST is allowed to build from sourceSummary: There was 1 ERROR message shown, returning a non-zero exit code.
Sorry for the delay in getting to this, with the other patches for the release, time has been difficult.
I have been able to take a look at this and the other locked sstate patches I've had pending for some time.
Having thought quite a bit about this, I think we really want to make this functionality part of the siggen class code. Where we need to add hooks, we should do so with callbacks into the siggen code.
Quoted 6 lines
I've just sent out a patch which my locked code in it. Could you take a look and see if we can make that approach work with your code too?
In particular, I don't really want to have multiple whitelist type variables. Could we use the
SIGGEN_LOCKEDSIGSvariable as the definitive way to control which recipes can float and which should not?I'm all for combining this.. however it appears the patches have two slightly separate goals.
The locked sstate patch tells the system to use a specific hash value and ONLY
that hash value. So if a user modified a system behavior in a way that would
change the behavior of the component there is no warning or error.The readonly sstate-cache on the other hand says, to reuse the sstate cache items (there of course may be more then one per recipe name with a different hash based on different configurations), but will warn/error when if the system configuration changes in a way the hash would have changed.
So there needs to be an interface that says which items are allowed to be built from source, and which items require either the locked or readonly sstate behavior.
Something like:
SIGGEN_LOCKEDSIGS= "\
gcc-cross:do_populate_sysroot:ro \
eglibc:do_populate_sysroot:ro \
eglibc:do_packagedata:ro \
gcc-cross:do_packagedata:ro \
"could work to extend it, but I have a concern with this. It only affects individual recipes. The request we've had from our customers is that once they have built their environment(s), and distributed the sstate-cache to their developers -- only the items they distributed are allowed to be used, unless an exception is granted.
So having to maintain a
SIGGEN_LOCKSIGS= variable with every possible package in the system seems complicated at best (for the RO case). They really want a switch that sets everything, except exceptions as read-only.Quoted 7 lines
SIGGEN_LOCKEDSIGS= "\
*:*:ro \
bash:*:rw \
"but I don't know if the glob/wildcarding would add more processing overhead then is acceptable.
I guess my main point right now is more about the controlling interface of this which I believe should be the "siggen" class. What I'd like to try and do is standardise on that as being the interface which handles this policy, whatever it may be. If it needs hooks adding, we can do that like the one I added from sstate.
Quoted 8 lines
Whether we end up with one siggen class or we have different ones (perhaps subclassing each other) and we switch between them for different behaviours I'm less sure about right now.
I'll give the locked sstate interface a bit more thought based on the above, thanks. I suspect it may be a case of adding a new subclass with some different functionality (e.g. readonly cache), then if that is what the user requires, we select that class using
BB_SIGNATURE_HANDLER?
Works for me. I don't mind the behavior of subclassing at all. The key thing is we've gotten requests for odd behaviors (which like I've said before, I don't think the community is or should be interested in -- but something that we have to do for our own customer..)
Nothing like a change (pgp signatures) the cripples the speed of the sstate-cache. :P
--Mark
Quoted 4 lines
Richard