bbclass/sstate: only allowed sstate-cache objects are allowed in a build (read-only sstate-cache)

Changed in V3:
- remove bitbake patch, which was sent to bitbake-devel
- rebase 'bbclass/sstate' patch, which was conflicted with 'sstate: Add extra intercept functions' patch in master-next

Changed in V2:

- add hook SSTATE_CHECK_FUNCTIONS for sstate checking, and add a new bbclass/sstate_readonly, rather than directly modify bbclass/sstate

- terminate build safely while RunQueueExecuteScenequeue init failed, the previous bb.msg.fatal is too hack.

Test steps:

1) vim local.conf
INHERIT += 'sstate_readonly'

2) Create a sstate cache for testing

$ bitbake db-native

3) Remove tmp dir and build db-native from sstate-cache

$ mv tmp tmp-back && bitbake db-native

4) Add gzip to SSTATECACHE_WHITELIST, it enabled read-only sstate-cache, vim local.conf
...
SSTATECACHE_WHITELIST = 'gzip'
...

5) Remove tmp dir and build db-native from sstate-cache

$ mv tmp tmp-back && bitbake db-native

6) tweak db's do_configure task by adding comments

@@ -82,6 +82,7 @@ do_configure() {
        gnu-configize --force ${S}
        export STRIP="true"
        oe_runconf
+       echo "hello"
 }
 
 

7) build db-native and there is a build failure

$ bitbake db-native
...
NOTE: Preparing runqueue
ERROR: Read-only sstate-cache is enabled, the build of 
"db-native"
did not come from sstate-cache. Only the recipe listed in
SSTATECACHE_WHITELIST is allowed to build from source
...

8) clean db-native failed

$ bitbake db-native -ccleansstate
...
ERROR: Read-only sstate-cache is enabled, the clean of 
db-native is not allowed. Only the recipe listed in
SSTATECACHE_WHITELIST is allowed to clean sstate-cache
...

$ bitbake db-native -ccleanall
...
ERROR: Read-only sstate-cache is enabled, the clean of 
db-native is not allowed. Only the recipe listed in
SSTATECACHE_WHITELIST is allowed to clean sstate-cache
...

9) Add db-native to SSTATECACHE_WHITELIST
vim local.conf
...
SSTATECACHE_WHITELIST = 'gzip db-native'
...

10) build/clean db-native succee

$ bitbake db-native
$ bitbake db-native -ccleansstate
$ bitbake db-native -ccleanall

//Hongxu

The following changes since commit c16795477ea7e2159b5d3808268912b572887e2d:

qemu: add PACKAGECONFIG for numa (2014-08-27 12:13:38 +0100)

are available in the git repository at:

git://git.pokylinux.org/poky-contrib hongxu/readonly-sstatecache
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=hongxu/readonly-sstatecache

Hongxu Jia (2):
bbclass/sstate: add hook SSTATE_CHECK_FUNCTIONS for sstate checking bbclass/sstate_readonly: only allowed sstate-cache objects are allowed
in a build (read-only sstate-cache)

 meta/classes/sstate.bbclass          | 26 ++++++++++++++++++
 meta/classes/sstate_readonly.bbclass | 51 ++++++++++++++++++++++++++++++++++++
 2 files changed, 77 insertions(+)
 create mode 100644 meta/classes/sstate_readonly.bbclass

1.9.1

Provide hook for sstate checking, the checking failure will break the building immediately. Pass input parameters by bb.data (d.setVar/d.getVar).

Hook : SSTATE_CHECK_FUNCTIONS
Input : missed - list of missing sstate file index in sq

      : found     - list of existing sstate file index in sq
      : sq_fn     - list of filenames in the sq
      : sq_task   - list of task names in the sq
      : sq_hash   - list of task hash in the sq

: sq_hashfn - list of hashfn in the sq

[YOCTO #6639]

---
 meta/classes/sstate.bbclass | 26 ++++++++++++++++++++++++++
 1 file changed, 26 insertions(+)

index 4eec6bd..9edd7c4 100644
@@ -616,6 +616,18 @@ sstate_unpack_package () {
 
 BB_HASHCHECK_FUNCTION = "sstate_checkhashes"
 
+# Provide hook for sstate checking, the checking failure will
+# break the building immediately. Pass input parameters by
+# bb.data (d.setVar/d.getVar).
+# Input : missed    - list of missing sstate file index in sq
+#       : found     - list of existing sstate file index in sq
+#       : sq_fn     - list of filenames in the sq
+#       : sq_task   - list of task names in the sq
+#       : sq_hash   - list of task hash in the sq
+#       : sq_hashfn - list of hashfn in the sq
+#
+SSTATE_CHECK_FUNCTIONS ?= ""
+
 def sstate_checkhashes(sq_fn, sq_task, sq_hash, sq_hashfn, d):
 
     ret = []

@@ -704,6 +716,20 @@ def sstate_checkhashes(sq_fn, sq_task, sq_hash, sq_hashfn, d):
             evdata['found'].append( (sq_fn[task], sq_task[task], sq_hash[task], sstatefile ) )
         bb.event.fire(bb.event.MetadataEvent("MissedSstate", evdata), d)
 
+    localdata = bb.data.createCopy(d)
+    bb.data.update_data(localdata)
+    for f in (localdata.getVar('SSTATE_CHECK_FUNCTIONS', True) or '').split():
+        localdata = bb.data.createCopy(d)
+        bb.data.update_data(localdata)
+        localdata.setVar('missed', missed)
+        localdata.setVar('found', ret)
+        localdata.setVar('sq_fn', sq_fn)
+        localdata.setVar('sq_task', sq_task)
+        localdata.setVar('sq_hash', sq_hash)
+        localdata.setVar('sq_hashfn', sq_hashfn)
+
+        bb.build.exec_func(f, localdata)
+
     return ret
 
 BB_SETSCENE_DEPVALID = "setscene_depvalid"

1.9.1

The requirement is the developer who demand only the "new" software they write is allowed to be compiled from source, they only want to reuse binaries from an existed sstate-cache, if the developer makes a change that triggers a rebuild, it should be an instant error.

The purpose of this is for the sstate-cache to check if the item exists or not. If it doesn't the item needs to be in a whitelist or we need to fail.

If read-only sstate-cache enable, and the recipe's ${PN} not in the ${SSTATECACHE_WHITELIST}, it trigered an instant error.

[YOCTO #6639]

---
 meta/classes/sstate_readonly.bbclass | 51 ++++++++++++++++++++++++++++++++++++
 1 file changed, 51 insertions(+)
 create mode 100644 meta/classes/sstate_readonly.bbclass

new file mode 100644
index 0000000..9eabf81
@@ -0,0 +1,51 @@
+SSTATE_CHECK_FUNCTIONS_append = " sstate_readonly_check"
+
+# 1) If ${SSTATECACHE_WHITELIST} is "", it means read-only sstate-cache
+#    disabled;
+#
+# 2) If read-only sstate-cache enabled and the recipe's ${PN} not listed
+#    in ${SSTATECACHE_WHITELIST}, the build from source will triger an
+#    instant error;
+SSTATECACHE_WHITELIST ?= ""
+
+python sstate_readonly_check(){
+    whitelist = (d.getVar('SSTATECACHE_WHITELIST', True) or '').split()
+    if whitelist:
+        sq_fn = d.getVar('sq_fn', True) or []
+        missed = d.getVar('missed', True) or []
+        missed_pn = []
+        for task in missed:
+            fn = sq_fn[task]
+            data = bb.cache.Cache.loadDataFull(fn, '', d)
+            pn = data.getVar('PN', True)
+            if pn and pn not in missed_pn:
+                missed_pn.append(pn)
+
+        if missed_pn:
+            blacklist = [pn for pn in missed_pn if pn not in whitelist]
+            if blacklist:
+                msg =  'Read-only sstate-cache is enabled, the build of \n'
+                msg += '"' + ' '.join(blacklist) + '"\n'
+                msg += 'did not come from sstate-cache. Only the recipe listed in\n'
+                msg += 'SSTATECACHE_WHITELIST is allowed to build from source'
+                bb.fatal(msg)
+}
+
+def _sstate_readonly_clean_check(d):
+    whitelist = (d.getVar('SSTATECACHE_WHITELIST', True) or '').split()
+    if whitelist:
+        pn = d.getVar('PN', True)
+        if pn not in whitelist:
+            msg =  'Read-only sstate-cache is enabled, the clean of \n'
+            msg += '%s is not allowed. Only the recipe listed in\n' % pn
+            msg += 'SSTATECACHE_WHITELIST is allowed to clean sstate-cache'
+            bb.fatal(msg)
+
+python do_cleansstate_prepend() {
+        _sstate_readonly_clean_check(d)
+}
+
+python do_cleanall_prepend() {
+    _sstate_readonly_clean_check(d)
+}
+

1.9.1