sstate_readonly: only allowed sstate-cache objects to be built (read-only sstate-cache)

Changed in V4:
- Enable read-only sstate-cache by inheriting the bbclass rather than by the SSTATECACHE_WHITELIST value is not null. There will be cases where customers don't want to build -anything- from source. In V3, So they would need to enter a dummy value to get that behavior, and that would not be intuitive.

Changed in V3:
- remove bitbake patch, which was sent to bitbake-devel
- rebase 'bbclass/sstate' patch, which was conflicted with 'sstate: Add extra intercept functions' patch in master-next

Changed in V2:

- add hook SSTATE_CHECK_FUNCTIONS for sstate checking, and add a new bbclass/sstate_readonly, rather than directly modify bbclass/sstate

- terminate build safely while RunQueueExecuteScenequeue init failed, the previous bb.msg.fatal is too hack.

Test Steps:

1) Prepare an existed sstate_cache with cache directory "Sstate_Cache_Dir" It contained db-native sstate_cache.

2) Enable read-only sstate-cache by inheriting, and assign SSTATE_DIR with Sstate_Cache_Dir, vim local.conf
...
INHERIT += 'sstate_readonly'
SSTATE_DIR = "Sstate_Cache_Dir"
...

3) bitbake db-native

4) tweak db's do_configure task by adding comments

@@ -82,6 +82,7 @@ do_configure() {
        gnu-configize --force ${S}
        export STRIP="true"
        oe_runconf
+       echo "hello"
 }
 
 

5) build db-native and there is a build failure

$ bitbake db-native
...
ERROR: Read-only sstate-cache is enabled, the build of 
"db-native"
did not come from sstate-cache. Only the recipe listed in
SSTATECACHE_WHITELIST is allowed to build from source
ERROR: Hash validation failed in RunQueueExecuteScenequeue Function failed: sstate_readonly_check
...

6) clean db-native failed

$ bitbake db-native -ccleansstate
...
ERROR: Read-only sstate-cache is enabled, the clean of 
db-native is not allowed. Only the recipe listed in
SSTATECACHE_WHITELIST is allowed to clean sstate-cache
...

$ bitbake db-native -ccleanall
...
ERROR: Read-only sstate-cache is enabled, the clean of 
db-native is not allowed. Only the recipe listed in
SSTATECACHE_WHITELIST is allowed to clean sstate-cache
...

7) Add db-native to SSTATECACHE_WHITELIST
vim local.conf
...
SSTATECACHE_WHITELIST = 'db-native'
...

8) build/clean db-native succee

$ bitbake db-native
$ bitbake db-native -ccleansstate
$ bitbake db-native -ccleanall

//Hongxu

The following changes since commit 669c07d6022174d01fe5a95b7b0faa9ef86da1e2:

bitbake: build/data: Write out more complete python run files (2014-08-28 15:12:45 +0100)

are available in the git repository at:

git://git.pokylinux.org/poky-contrib hongxu/readonly-sstatecache
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=hongxu/readonly-sstatecache

Hongxu Jia (2):
bbclass/sstate: add hook SSTATE_CHECK_FUNCTIONS for sstate checking bbclass/sstate_readonly: only allowed sstate-cache objects to be built
(read-only sstate-cache)

 meta/classes/sstate.bbclass          | 26 ++++++++++++++++++
 meta/classes/sstate_readonly.bbclass | 53 ++++++++++++++++++++++++++++++++++++
 2 files changed, 79 insertions(+)
 create mode 100644 meta/classes/sstate_readonly.bbclass

1.9.1

Provide hook for sstate checking, the checking failure will break the building immediately. Pass input parameters by bb.data (d.setVar/d.getVar).

Hook : SSTATE_CHECK_FUNCTIONS
Input : missed - list of missing sstate file index in sq

      : found     - list of existing sstate file index in sq
      : sq_fn     - list of filenames in the sq
      : sq_task   - list of task names in the sq
      : sq_hash   - list of task hash in the sq

: sq_hashfn - list of hashfn in the sq

[YOCTO #6639]

---
 meta/classes/sstate.bbclass | 26 ++++++++++++++++++++++++++
 1 file changed, 26 insertions(+)

index ead829e..e7569be 100644
@@ -622,6 +622,18 @@ sstate_unpack_package () {
 
 BB_HASHCHECK_FUNCTION = "sstate_checkhashes"
 
+# Provide hook for sstate checking, the checking failure will
+# break the building immediately. Pass input parameters by
+# bb.data (d.setVar/d.getVar).
+# Input : missed    - list of missing sstate file index in sq
+#       : found     - list of existing sstate file index in sq
+#       : sq_fn     - list of filenames in the sq
+#       : sq_task   - list of task names in the sq
+#       : sq_hash   - list of task hash in the sq
+#       : sq_hashfn - list of hashfn in the sq
+#
+SSTATE_CHECK_FUNCTIONS ?= ""
+
 def sstate_checkhashes(sq_fn, sq_task, sq_hash, sq_hashfn, d):
 
     ret = []

@@ -710,6 +722,20 @@ def sstate_checkhashes(sq_fn, sq_task, sq_hash, sq_hashfn, d):
             evdata['found'].append( (sq_fn[task], sq_task[task], sq_hash[task], sstatefile ) )
         bb.event.fire(bb.event.MetadataEvent("MissedSstate", evdata), d)
 
+    localdata = bb.data.createCopy(d)
+    bb.data.update_data(localdata)
+    for f in (localdata.getVar('SSTATE_CHECK_FUNCTIONS', True) or '').split():
+        localdata = bb.data.createCopy(d)
+        bb.data.update_data(localdata)
+        localdata.setVar('missed', missed)
+        localdata.setVar('found', ret)
+        localdata.setVar('sq_fn', sq_fn)
+        localdata.setVar('sq_task', sq_task)
+        localdata.setVar('sq_hash', sq_hash)
+        localdata.setVar('sq_hashfn', sq_hashfn)
+
+        bb.build.exec_func(f, localdata)
+
     return ret
 
 BB_SETSCENE_DEPVALID = "setscene_depvalid"

1.9.1

The requirement is the developer who demand only the "new" software they write is allowed to be compiled from source, they only want to reuse binaries from an existed sstate-cache, if the developer makes a change that triggers a rebuild, it should be an instant error.

When the readonly sstate-cache is enabled, an error will be generated if a recipe is not available within the sstate-cache. Adding recipes to the whitelist will allow only select recipes to be allowed to build from source.

[YOCTO #6639]

---
 meta/classes/sstate_readonly.bbclass | 53 ++++++++++++++++++++++++++++++++++++
 1 file changed, 53 insertions(+)
 create mode 100644 meta/classes/sstate_readonly.bbclass

new file mode 100644
index 0000000..488353d
@@ -0,0 +1,53 @@
+SSTATE_CHECK_FUNCTIONS_append = " sstate_readonly_check"
+# 1) The read-only sstate-cache will always be enabled if this bbclass
+#    inherited
+#
+# 2) If ${SSTATECACHE_WHITELIST} is "", it means always blacklist
+#    everything
+#
+# 3) Adding recipes to ${SSTATECACHE_WHITELIST} will allow only select
+#    recipes to be allowed to build
+#
+# 4) While recipes not in ${SSTATECACHE_WHITELIST}, an error will be
+#    generated if a recipe is not available within sstate-cache.
+SSTATECACHE_WHITELIST ?= ""
+
+python sstate_readonly_check(){
+    whitelist = (d.getVar('SSTATECACHE_WHITELIST', True) or '').split()
+    sq_fn = d.getVar('sq_fn', True) or []
+    missed = d.getVar('missed', True) or []
+    missed_pn = []
+    for task in missed:
+        fn = sq_fn[task]
+        data = bb.cache.Cache.loadDataFull(fn, '', d)
+        pn = data.getVar('PN', True)
+        if pn and pn not in missed_pn:
+            missed_pn.append(pn)
+
+    if missed_pn:
+        blacklist = [pn for pn in missed_pn if pn not in whitelist]
+        if blacklist:
+            msg =  'Read-only sstate-cache is enabled, the build of \n'
+            msg += '"' + ' '.join(blacklist) + '"\n'
+            msg += 'did not come from sstate-cache. Only the recipe listed in\n'
+            msg += 'SSTATECACHE_WHITELIST is allowed to build from source'
+            bb.fatal(msg)
+}
+
+def _sstate_readonly_clean_check(d):
+    whitelist = (d.getVar('SSTATECACHE_WHITELIST', True) or '').split()
+    pn = d.getVar('PN', True)
+    if pn not in whitelist:
+        msg =  'Read-only sstate-cache is enabled, the clean of \n'
+        msg += '%s is not allowed. Only the recipe listed in\n' % pn
+        msg += 'SSTATECACHE_WHITELIST is allowed to clean sstate-cache'
+        bb.fatal(msg)
+
+python do_cleansstate_prepend() {
+        _sstate_readonly_clean_check(d)
+}
+
+python do_cleanall_prepend() {
+    _sstate_readonly_clean_check(d)
+}
+

1.9.1