only allowed sstate-cache objects are allowed in a build (read-only sstate-cache?)
cover only allowed sstate-cache objects are allowed in a build (read-only sstate-cache?)
Issue description:
The issue is that the developer who demand only the "new" software they write is allowed to be compiled from source, they only want to reuse binaries from an existed sstate-cache, if the developer makes a change that triggers a rebuild, it should be an instant error.
The purpose of this is for the sstate-cache to check if the item exists or not. If it doesn't the item needs to be in a whitelist or we need to fail.
I dig into three approaches to implement, and send them to oe-core list to see if the community has a preference as to the approach, and any additional comments.
In these approaches, I'm not sure we should protect the 'clean' or not. Since it's ok to clean the sstate-cache, as long as a mirror fetch will pull it back down. Any suggestion is welcomed.
If you have any better ideas, please don't hesitate to share with us.
Test steps:
1) For approach 1
INHERIT += 'sstate_readonly'
For approach 2
INHERIT += 'sstate_readonly_2'
2) Create a sstate cache for testing
$ bitbake db
3) Add gzip to SSTATECACHE_WHITELIST, it enabled read-only sstate-cache, vim local.conf
...SSTATECACHE_WHITELIST = 'gzip'
...
4) Remove tmp dir and build db from sstate-cache succeed
$ mv tmp tmp-back && bitbake db
5) tweak db's do_configure task by adding comments
meta/recipes-support/db/db_6.0.30.bb +1 -0
@@ -27,6 +27,8 @@ LIC_FILES_CHKSUM = "file://../LICENSE;md5=1ec8b0b17cc31513fe35ab10716f8490"
@@ -82,6 +84,7 @@ do_configure() { gnu-configize --force ${S} export STRIP="true" oe_runconf + echo "hello" }
6) build db and there is a build failure
$ bitbake db
7) clean db and there is a clean failure
bitbake db -ccleansstate
8) Append the missing recipe's PN to SSTATECACHE_WHITELIST
vim local.conf
...SSTATECACHE_WHITELIST = 'gzip db rpm-native gcc-runtime eglibc linux-libc-headers libgcc'
...
9) Build db and clean db succeed
$ bitbake db && bitbake db -ccleansstate
//Hongxu
The following changes since commit 870bb8d35547b8313b3a487d7e8b914ab9470e64:
local.conf.sample.extended: fix example for EXTRA_USERS_PARAMS (2014-08-04 17:38:24 +0100)
are available in the git repository at:
git://git.pokylinux.org/poky-contrib hongxu/readonly-sstatecache
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=hongxu/readonly-sstatecache
Hongxu Jia (3):
bbclass/sstate_readonly approach 1: add prefuncs to SSTATETASKS bbclass/sstate_readonly_2 approach 2: add event handler at TaskStarted
time
bbclass/sstate approach 3: add checking in the return path of
sstate_checkhashes
meta/classes/sstate.bbclass | 40 ++++++++++++++++++++++++++++++++++ meta/classes/sstate_readonly.bbclass | 38 ++++++++++++++++++++++++++++++++ meta/classes/sstate_readonly_2.bbclass | 25 +++++++++++++++++++++ 3 files changed, 103 insertions(+) create mode 100644 meta/classes/sstate_readonly.bbclass create mode 100644 meta/classes/sstate_readonly_2.bbclass
Signature · 1 line
1.9.1
PATCH 1/3 bbclass/sstate_readonly approach 1: add prefuncs to SSTATETASKS
Add prefuncs to SSTATETASKS to handle the sstate-cache read-only cheking. If read-only sstate-cache enable, and the building recipe not in the ${SSTATECACHE_WHITELIST}, it trigered an instant error.
Flaws:
1. For each SSTATETASKS task, there is a read-only sstate-cache checking, it increased the build loads.
2. The instant error breaks the building randomly while there are multible recipes not in the ${SSTATECACHE_WHITELIST}, and also could not list all the missing recipes which sould be added to ${SSTATECACHE_WHITELIST}.
3. The checking time is postponed to SSTATETASKS, take db-native for example:
if you tweak db-native's do_configure, the checking will occur at do_populate_sysroot time. The tasks 'do_compile', 'do_install' has been invoked.
...
$ bitbake db ERROR: Read-only sstate-cache is enabled, the build of gettext-minimal-native did not come from sstate-cache. Only the recipe listed in SSTATECACHE_WHITELIST is allowed to build from source ERROR: Function failed: sstate_readonly_check ERROR: Logfile of failure stored in: tmp/work/x86_64-linux/gettext-minimal-native/0.18.3.2-r0/temp/log.do_populate_sysroot.23801 ERROR: Task 155 (poky/meta/recipes-core/gettext/gettext-minimal-native_0.18.3.2.bb, do_populate_sysroot) failed with exit code '1' NOTE: Tasks Summary: Attempted 84 tasks of which 49 didn't need to be rerun and 1 failed. ...
--- meta/classes/sstate_readonly.bbclass | 38 ++++++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+) create mode 100644 meta/classes/sstate_readonly.bbclass
meta/classes/sstate_readonly.bbclass new file +38 -0
@@ -0,0 +1,38 @@ +# 1) If ${SSTATECACHE_WHITELIST} is "", it means read-only sstate-cache +# disabled; +# +# 2) If read-only sstate-cache enabled and the recipe's ${PN} not listed +# in ${SSTATECACHE_WHITELIST}, the build from source will triger an +# instant error; +SSTATECACHE_WHITELIST ?= "" + +python () { + unique_tasks = set((d.getVar('SSTATETASKS', True) or "").split()) + d.setVar('SSTATETASKS', " ".join(unique_tasks)) + for task in unique_tasks: + d.prependVarFlag(task, 'prefuncs', "sstate_readonly_check") +} + +_sstate_readonly_check[vardepsexclude] += "SSTATECACHE_WHITELIST" +def _sstate_readonly_check(d): + whitelist = d.getVar('SSTATECACHE_WHITELIST', True) or "" + if whitelist != "": + pn = d.getVar('PN', True) + if pn not in whitelist.split(): + msg = 'Read-only sstate-cache is enabled, the build of %s\n' % pn + msg += 'did not come from sstate-cache. Only the recipe listed in\n' + msg += 'SSTATECACHE_WHITELIST is allowed to build from source' + bb.fatal(msg) + +python sstate_readonly_check(){ + _sstate_readonly_check(d) +} + +python do_cleansstate_prepend() { + _sstate_readonly_check(d) +} + +python do_cleanall_prepend() { + _sstate_readonly_check(d) +} +
Signature · 1 line
1.9.1
PATCH 2/3 bbclass/sstate_readonly_2 approach 2: add event handler at TaskStarted time
In reply to Hongxu JiaAdd event handler at TaskStarted time to handle the sstate-cache read-only cheking. If read-only sstate-cache enable, and the building recipe not in the ${SSTATECACHE_WHITELIST}, it trigered an instant error.
Flaws:
1. For every task, there is a read-only sstate-cache checking, it increased the build loads.
2. The instant error breaks the building randomly while there are multible recipes not in the ${SSTATECACHE_WHITELIST}, and also could not list all the missing recipes which sould be added to ${SSTATECACHE_WHITELIST}.
...
$ bitbake db ERROR: Read-only sstate-cache is enabled, the build of ncurses-native did not come from sstate-cache. Only the recipe listed in SSTATECACHE_WHITELIST is allowed to build from source ERROR: Execution of event handler 'sstate_readonly_eventhandler' failed ERROR: Task 350 (virtual:native:poky/meta/recipes-core/ncurses/ncurses_5.9.bb, do_fetch) failed with exit code '1' ERROR: Read-only sstate-cache is enabled, the build of gcc-cross-initial-i586 did not come from sstate-cache. Only the recipe listed in SSTATECACHE_WHITELIST is allowed to build from source ERROR: Execution of event handler 'sstate_readonly_eventhandler' failed ERROR: Task 290 (poky/meta/recipes-devtools/gcc/gcc-cross-initial_4.9.bb, do_fetch) failed with exit code '1' ...
--- meta/classes/sstate_readonly_2.bbclass | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) create mode 100644 meta/classes/sstate_readonly_2.bbclass
meta/classes/sstate_readonly_2.bbclass new file +25 -0
@@ -0,0 +1,25 @@ +# 1) If ${SSTATECACHE_WHITELIST} is "", it means read-only sstate-cache +# disabled; +# +# 2) If read-only sstate-cache enabled and the recipe's ${PN} not listed +# in ${SSTATECACHE_WHITELIST}, the build from source will triger an +# instant error; +SSTATECACHE_WHITELIST ?= "" + +addhandler sstate_readonly_eventhandler +sstate_readonly_eventhandler[eventmask] = "bb.build.TaskStarted" +python sstate_readonly_eventhandler() { + d = e.data + whitelist = d.getVar('SSTATECACHE_WHITELIST', True) or "" + if whitelist != "": + taskname = d.getVar('BB_CURRENTTASK', True) + if not taskname.startswith('do_'): + taskname = 'do_%s' % taskname + if not taskname.endswith("_setscene") and taskname != "do_setscene": + pn = d.getVar('PN', True) + if pn not in whitelist.split(): + msg = '\nRead-only sstate-cache is enabled, the build of %s\n' % pn + msg += 'did not come from sstate-cache. Only the recipe listed in\n' + msg += 'SSTATECACHE_WHITELIST is allowed to build from source' + bb.msg.fatal('sstate', msg) +}
Signature · 1 line
1.9.1
PATCH 3/3 bbclass/sstate approach 3: add checking in the return path of sstate_checkhashes
In reply to Hongxu JiaIn the sstate-cache code, add a checking in the return path of sstate_checkhashes. If read-only sstate-cache enable, and the recipe's ${PN} not in the ${SSTATECACHE_WHITELIST}, it trigered an instant error.
Flaws:
1. We should manually unlock the bitbake lock which the instant error msg will exit the build immediately.
...
$ bitbake db ERROR: Read-only sstate-cache is enabled, the build of "db rpm-native gcc-runtime eglibc linux-libc-headers libgcc" did not come from sstate-cache. Only the recipe listed in SSTATECACHE_WHITELIST is allowed to build from source
Summary: There was 1 ERROR message shown, returning a non-zero exit code.
...
--- meta/classes/sstate.bbclass | 40 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+)
meta/classes/sstate.bbclass +40 -0
@@ -41,6 +41,14 @@ EXTRA_STAGING_FIXMES ?= "" sstate_create_package[dirs] = "${SSTATE_BUILDDIR}" sstate_unpack_package[dirs] = "${SSTATE_INSTDIR}" +# 1) If ${SSTATECACHE_WHITELIST} is "", it means read-only sstate-cache +# disabled; +# +# 2) If read-only sstate-cache enabled and the recipe's ${PN} not listed +# in ${SSTATECACHE_WHITELIST}, the build from source will triger an +# instant error; +SSTATECACHE_WHITELIST ?= "" + python () { if bb.data.inherits_class('native', d): d.setVar('SSTATE_PKGARCH', d.getVar('BUILD_ARCH'))
@@ -382,6 +390,15 @@ sstate_clean[vardepsexclude] = "SSTATE_MANFILEPREFIX" CLEANFUNCS += "sstate_cleanall" python sstate_cleanall() { + whitelist = d.getVar('SSTATECACHE_WHITELIST', True) or "" + if whitelist: + pn = d.getVar('PN', True) + if pn not in whitelist.split(): + msg = 'Read-only sstate-cache is enabled, the clean of \n' + msg += '%s is not allowed. Only the recipe listed in\n' % pn + msg += 'SSTATECACHE_WHITELIST is allowed to clean sstate-cache' + bb.fatal(msg) + bb.note("Removing shared state for package %s" % d.getVar('PN', True)) manifest_dir = d.getVar('SSTATE_MANIFESTS', True)
@@ -704,6 +721,29 @@ def sstate_checkhashes(sq_fn, sq_task, sq_hash, sq_hashfn, d): evdata['found'].append( (sq_fn[task], sq_task[task], sq_hash[task], sstatefile ) ) bb.event.fire(bb.event.MetadataEvent("MissedSstate", evdata), d) + whitelist = d.getVar('SSTATECACHE_WHITELIST', True) or "" + if whitelist: + missed_pn = [] + for task in missed: + fn = sq_fn[task] + data = bb.cache.Cache.loadDataFull(fn, '', d) + pn = data.getVar('PN', True) or "" + if pn and pn not in missed_pn: + missed_pn.append(pn) + + if missed_pn: + blacklist = [pn for pn in missed_pn if pn not in whitelist.split()] + if blacklist: + # We should manually unlock the bitbake lock, because the fatal + # msg will exit the build immediately. + lockfile = d.expand("${TOPDIR}/bitbake.lock") + os.unlink(lockfile) + msg = 'Read-only sstate-cache is enabled, the build of \n' + msg += '"' + ' '.join(blacklist) + '"\n' + msg += 'did not come from sstate-cache. Only the recipe listed in\n' + msg += 'SSTATECACHE_WHITELIST is allowed to build from source' + bb.msg.fatal('sstate', msg) + return ret BB_SETSCENE_DEPVALID = "setscene_depvalid"
Signature · 1 line
1.9.1
Ping
//Hongxu
Quoted 92 lines · Hongxu Jia
Issue description:
The issue is that the developer who demand only the "new" software they write is allowed to be compiled from source, they only want to reuse binaries from an existed sstate-cache, if the developer makes a change that triggers a rebuild, it should be an instant error.
The purpose of this is for the sstate-cache to check if the item exists or not. If it doesn't the item needs to be in a whitelist or we need to fail.
I dig into three approaches to implement, and send them to oe-core list to see if the community has a preference as to the approach, and any additional comments.
In these approaches, I'm not sure we should protect the 'clean' or not. Since it's ok to clean the sstate-cache, as long as a mirror fetch will pull it back down. Any suggestion is welcomed.
If you have any better ideas, please don't hesitate to share with us.
Test steps:
1) For approach 1
INHERIT += 'sstate_readonly' For approach 2 INHERIT += 'sstate_readonly_2'2) Create a sstate cache for testing
$ bitbake db3) Add gzip to
SSTATECACHE_WHITELIST, it enabled read-only sstate-cache, vim local.conf
...SSTATECACHE_WHITELIST= 'gzip'
...4) Remove tmp dir and build db from sstate-cache succeed
$ mv tmp tmp-back && bitbake db5) tweak db's
do_configuretask by adding commentsmeta/recipes-support/db/db_6.0.30.bb +1
-0@@ -27,6 +27,8 @@ LIC_FILES_CHKSUM = "file://../LICENSE;md5=1ec8b0b17cc31513fe35ab10716f8490"@@ -82,6 +84,7 @@ do_configure() { gnu-configize --force ${S} export STRIP="true" oe_runconf + echo "hello" }6) build db and there is a build failure
$ bitbake db7) clean db and there is a clean failure
bitbake db -ccleansstate8) Append the missing recipe's PN to
SSTATECACHE_WHITELIST
vim local.conf
...SSTATECACHE_WHITELIST= 'gzip db rpm-native gcc-runtime eglibc linux-libc-headers libgcc'
...9) Build db and clean db succeed
$ bitbake db && bitbake db -ccleansstate//Hongxu
The following changes since commit
870bb8d35547b8313b3a487d7e8b914ab9470e64:local.conf.sample.extended: fix example for EXTRA_USERS_PARAMS (2014-08-04 17:38:24 +0100)are available in the git repository at:
git://git.pokylinux.org/poky-contribhongxu/readonly-sstatecache
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=hongxu/readonly-sstatecacheHongxu Jia (3):
bbclass/sstate_readonly approach 1: add prefuncs to SSTATETASKS bbclass/sstate_readonly_2 approach 2: add event handler at TaskStarted
time
bbclass/sstate approach 3: add checking in the return path of
sstate_checkhashesmeta/classes/sstate.bbclass | 40 ++++++++++++++++++++++++++++++++++ meta/classes/sstate_readonly.bbclass | 38 ++++++++++++++++++++++++++++++++ meta/classes/sstate_readonly_2.bbclass | 25 +++++++++++++++++++++ 3 files changed, 103 insertions(+)create mode 100644 meta/classes/sstate_readonly.bbclass create mode 100644 meta/classes/sstate_readonly_2.bbclass