Skip to contentSkip to conversation

improve locked down sstate

PATCH New Hongxu Jia 3 participants 11 messages Sep 17, 2014 – Sep 18, 2014

lore
Share & tools
lore

Hongxu Jia

author
#0 lore raw

v2 cover improve locked down sstate

Changed in V2:
- Set SIGGEN_LOCKEDSIGS_CHECK_LEVEL default to error

- Move dump sig file event handler to a separate class

- Fix typo

- Fix overrides behaviour to remove SIGGEN_LOCKEDSIGS_i586

- Support incremental dump lockedsigs without complicate the structure of the lockedsigs variable.

Test Steps:

1. Prepare a locked sstat-cache by dumping lockedsigs file
1) vim local.conf
...
USER_CLASSES += "sstate_lockedsig" SIGGEN_LOCKEDSIGS_CONFIG = "/path/to/locked-sigs.inc" SSTATE_DIR ?= "/path/to/sstate-cache"
...

2) Build db-native from a clean environment

$ bitbake db-native

3) Back up lockedsigs file

$ mv /path/to/locked-sigs.inc  /path/to/locked-sigs.inc-back

4) Incremental build ldconfig-native

$ vim local.conf
...
require ${SIGGEN_LOCKEDSIGS_CONFIG}
...
$ bitbake ldconfig-native

5) Compare the two sig files, and check whether it is incremental dump or not

$ vimdiff /path/to/locked-sigs.inc-back /path/to/locked-sigs.inc

2. Create a new build which uses previous locked sstat-cache
1) vim local.conf
...
SSTATE_DIR ?= "/path/to/sstate-cache" SIGGEN_LOCKEDSIGS_CONFIG = "/path/to/locked-sigs.inc" require ${SIGGEN_LOCKEDSIGS_CONFIG}
...

2) Build db-native

$ bitbake db-native

3) Tweak db's do_configure task by adding comments

meta/recipes-support/db/db_6.0.30.bb +1 -0
@@ -82,6 +82,7 @@ do_configure() {
        gnu-configize --force ${S}
        export STRIP="true"
        oe_runconf
+       echo "hello"
 }
 

4) Build db-native and abort

$ bitbake db-native
...
ERROR: The db-native:do_configure sig (00546ab0e73b3afd50efc03f26e5f040) changed, use locked sig 9d8f2210506446e76d012d1ebd49a021 to instead
The db-native:do_compile sig (4442f979d97bdbed71a8fecbbbe41ed3) changed, use locked sig d29a7c568d10859176d733773433fe92 to instead
The db-native:do_install sig (562cb98d8eb25a7c79ded6a5359c56c1) changed, use locked sig 1e917d8d619c58987a1b2fe286e9288c to instead
The db-native:do_populate_sysroot sig (91627e36da9ac8fe489f36b67a87ed2e) changed, use locked sig f780d45dbf680e40a871820327c4c56c to instead
The db-native:do_build sig (ba289cba4a3bfda0c07aed9868fdcd5d) changed, use locked sig d9b84e5fb30011fd31bc0e7d7652fead to instead
...

5) Remove db-native sstate-cache

$ bitbake db-native -ccleansstate

6) Build db-native and abort

$ bitbake db-native
...
Locked sig is set for db-native:do_populate_sysroot (f780d45dbf680e40a871820327c4c56c) yet not in sstate cache?
...

7) Do not break the build
vim local.conf
...
SIGGEN_LOCKEDSIGS_CHECK_LEVEL = "warn"
...

8) Build db-native

$ bitbake db-native
...
WARNING Locked sig is set for db-native:do_populate_sysroot (f780d45dbf680e40a871820327c4c56c) yet not in sstate cache?
...

//Hongxu

The following changes since commit 7d80f8e9468253496a7097685aac8f468940a9c5:

bitbake: data_smart: Clarify what 'computed' means in the data store history context (2014-09-16 22:15:24 +0100)

are available in the git repository at:

git://git.pokylinux.org/poky-contrib hongxu/locked-sstate
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=hongxu/locked-sstate

Hongxu Jia (5):
sstate: set SIGGEN_LOCKEDSIGS_CHECK_LEVEL default to error sstate_lockedsig.bbclass: add event handler to dump lockedsigs at
recipe building time
sstatesig.py: fix typo of locke sstate
sstatesig: fix overrides behaviour to remove SIGGEN_LOCKEDSIGS_i586
sstatesig: incremental dump lockedsigs

[email protected] (4):
Richard Purdie : sstatesig/sstate: Add support for locked down sstate
cache usage
Richard Purdie : sstatesig: Improve to handle locking of multiple
machines
Randy Witt : sstatesig.py: Replace '_'s with '-' s in
SSTATE_LOCKEDSIGS* names.
Hongxu Jia : sstatesig: Improve the support for locked down sstate
cache usage

 meta/classes/sstate.bbclass           |   5 ++
 meta/classes/sstate_lockedsig.bbclass |  12 ++++
 meta/lib/oe/sstatesig.py              | 106 ++++++++++++++++++++++++++++++++++
 scripts/gen-lockedsig-cache           |  40 +++++++++++++
 4 files changed, 163 insertions(+)
 create mode 100644 meta/classes/sstate_lockedsig.bbclass
 create mode 100755 scripts/gen-lockedsig-cache
Signature · 1 line
1.9.1
author
#1 lore raw

PATCH 5/9 sstate: set SIGGEN_LOCKEDSIGS_CHECK_LEVEL default to error

---
 meta/classes/sstate.bbclass | 2 ++
 1 file changed, 2 insertions(+)
meta/classes/sstate.bbclass +2 -0
index d3e251c..4057c8c 100644
@@ -39,6 +39,8 @@ SSTATEPOSTUNPACKFUNCS = "sstate_hardcode_path_unpack"
 SSTATEPOSTINSTFUNCS = ""
 EXTRA_STAGING_FIXMES ?= ""
 
+SIGGEN_LOCKEDSIGS_CHECK_LEVEL ?= 'error'
+
 # Specify dirs in which the shell function is executed and don't use ${B}
 # as default dirs to avoid possible race about ${B} with other task.
 sstate_create_package[dirs] = "${SSTATE_BUILDDIR}"
Signature · 1 line
1.9.1
author
#2 lore raw

PATCH 6/9 sstate_lockedsig.bbclass: add event handler to dump lockedsigs at recipe building time

In reply to Hongxu Jia

While the class in use, it adds a event handler at 'bb.event.BuildCompleted', so after the build completed, it will dump the lockedsigs files.

Use SIGGEN_LOCKEDSIGS_CONFIG variable controls where to dump the lockedsigs file, the default is placed into ${SSTATE_DIR}/locked-sigs.inc.

---
 meta/classes/sstate_lockedsig.bbclass | 12 ++++++++++++
 1 file changed, 12 insertions(+)
 create mode 100644 meta/classes/sstate_lockedsig.bbclass
meta/classes/sstate_lockedsig.bbclass new file +12 -0
new file mode 100644
index 0000000..ea88260
@@ -0,0 +1,12 @@
+SIGGEN_LOCKEDSIGS_CONFIG ?= "${SSTATE_DIR}/locked-sigs.inc"
+addhandler sstate_dump_lockedsig
+sstate_dump_lockedsig[eventmask] = "bb.event.BuildCompleted"
+python sstate_dump_lockedsig() {
+    d = e.data
+    if e.getFailures():
+        return
+
+    if hasattr(bb.parse.siggen, "dump_lockedsigs"):
+        lockedsigs = d.getVar('SIGGEN_LOCKEDSIGS_CONFIG', True)
+        bb.parse.siggen.dump_lockedsigs(lockedsigs)
+}
Signature · 1 line
1.9.1
author
#3 lore raw

Re: [PATCH 6/9] sstate_lockedsig.bbclass: add event handler to dump lockedsigs at recipe building time

Hi Richard,

Is this patch not acceptable, it is not merged to master or master-next.

//Hongxu

Quoted 30 lines · Hongxu Jia

While the class in use, it adds a event handler at 'bb.event.BuildCompleted', so after the build completed, it will dump the lockedsigs files.

Use SIGGEN_LOCKEDSIGS_CONFIG variable controls where to dump the lockedsigs file, the default is placed into ${SSTATE_DIR}/locked-sigs.inc.

---
  meta/classes/sstate_lockedsig.bbclass | 12 ++++++++++++
  1 file changed, 12 insertions(+)

create mode 100644 meta/classes/sstate_lockedsig.bbclass

meta/classes/sstate_lockedsig.bbclass new file +12 -0
new file mode 100644
index 0000000..ea88260
@@ -0,0 +1,12 @@
+SIGGEN_LOCKEDSIGS_CONFIG ?= "${SSTATE_DIR}/locked-sigs.inc"
+addhandler sstate_dump_lockedsig
+sstate_dump_lockedsig[eventmask] = "bb.event.BuildCompleted"
+python sstate_dump_lockedsig() {
+    d = e.data
+    if e.getFailures():
+        return
+
+    if hasattr(bb.parse.siggen, "dump_lockedsigs"):
+        lockedsigs = d.getVar('SIGGEN_LOCKEDSIGS_CONFIG', True)
+        bb.parse.siggen.dump_lockedsigs(lockedsigs)
+}
author
#4 lore raw

PATCH 7/9 sstatesig.py: fix typo of locke sstate

In reply to Hongxu Jia

We used sigfile to instead of where_to_dump. Fix the missing one.

---
 meta/lib/oe/sstatesig.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/lib/oe/sstatesig.py +1 -1
index df325d0..65dbac1 100644
@@ -138,7 +138,7 @@ class SignatureGeneratorOEBasicHash(bb.siggen.SignatureGeneratorBasicHash):
         super(bb.siggen.SignatureGeneratorBasicHash, self).dump_sigtask(fn, task, stampbase, runtime)
 
     def dump_lockedsigs(self, sigfile=None):
-        if not where_to_dump:
+        if not sigfile:
             sigfile = os.getcwd() + "/locked-sigs.inc"
 
         bb.plain("Writing locked sigs to %s" % sigfile)
Signature · 1 line
1.9.1
author
#5 lore raw

PATCH 8/9 sstatesig: fix overrides behaviour to remove SIGGEN_LOCKEDSIGS_i586

In reply to Hongxu Jia

Require a sig file which SIGGEN_LOCKEDSIGS_i586 is not null, but the actual SIGGEN_LOCKEDSIGS_i586 is null.
Invoking 'bitbake -e' and we got:
...
3935 # $SIGGEN_LOCKEDSIGS_i586 [2 operations]
3936 # set /path/to/locked-sigs.inc:8576
3938 # del data_smart.py:406 [finalize]
3939 # ""
...

It was caused by the following commit:
...
(Bitbake rev: 899d45b90061eb3cf3e71029072eee42cd80930c)

Author: Richard Purdie <[email protected]>
Date:   Tue May 31 23:52:50 2011 +0100

bitbake/data_smart: Change overrides behaviour to remove expanded
variables from the datastore
...

We add prefix 't-' to type to workaround the overrides behaviour.

---
 meta/lib/oe/sstatesig.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/lib/oe/sstatesig.py +1 -1
index 65dbac1..af7617e 100644
@@ -146,7 +146,7 @@ class SignatureGeneratorOEBasicHash(bb.siggen.SignatureGeneratorBasicHash):
         for k in self.runtaskdeps:
             fn = k.rsplit(".",1)[0]
             t = self.lockedhashfn[fn].split(" ")[1].split(":")[5]
-            t = t.replace('_', '-')
+            t = 't-' + t.replace('_', '-')
             if t not in types:
                 types[t] = []
             types[t].append(k)
Signature · 1 line
1.9.1
author
#6 lore raw

PATCH 9/9 sstatesig: incremental dump lockedsigs

In reply to Hongxu Jia

The idea of incremental sig is:

New sig file = Old sig file (if available) + New sig items in current build.

Limit the modification within the dump_lockedsigs, and add two variables 'self.lockedsigs_types' and 'self.lockedsigs_raw' keep old sig file.

How to config for incremental dump:
...
USER_CLASSES += "sstate_lockedsig" SIGGEN_LOCKEDSIGS_CONFIG = "${TOPDIR}/locked-sigs.inc" require ${SIGGEN_LOCKEDSIGS_CONFIG}
...

---
 meta/lib/oe/sstatesig.py | 33 +++++++++++++++++++--------------
 1 file changed, 19 insertions(+), 14 deletions(-)
meta/lib/oe/sstatesig.py +19 -14
index af7617e..d8cd8e6 100644
@@ -93,6 +93,10 @@ class SignatureGeneratorOEBasicHash(bb.siggen.SignatureGeneratorBasicHash):
         self.lockedhashfn = {}
         self.machine = data.getVar("MACHINE", True)
         self.mismatch_msgs = []
+        self.lockedsigs_types = (data.getVar("SIGGEN_LOCKEDSIGS_TYPES", True) or "").split()
+        self.lockedsigs_raw = {}
+        for t in self.lockedsigs_types:
+            self.lockedsigs_raw[t] = (data.getVar("SIGGEN_LOCKEDSIGS_%s" % t, True) or "").split()
         pass
     def rundep_check(self, fn, recipename, task, dep, depname, dataCache = None):
         return sstate_rundepfilter(self, fn, recipename, task, dep, depname, dataCache)
@@ -142,28 +146,29 @@ class SignatureGeneratorOEBasicHash(bb.siggen.SignatureGeneratorBasicHash):
             sigfile = os.getcwd() + "/locked-sigs.inc"
 
         bb.plain("Writing locked sigs to %s" % sigfile)
-        types = {}
         for k in self.runtaskdeps:
             fn = k.rsplit(".",1)[0]
             t = self.lockedhashfn[fn].split(" ")[1].split(":")[5]
             t = 't-' + t.replace('_', '-')
-            if t not in types:
-                types[t] = []
-            types[t].append(k)
+            if t not in self.lockedsigs_types:
+                self.lockedsigs_types.append(t)
+                self.lockedsigs_raw[t] = []
+
+            pn = self.lockedpnmap[fn]
+            task = k.rsplit(".",1)[1]
+            hash = self.taskhash[k]
+            lockedsig = "%s:%s:%s" % (pn, task, hash)
+            if lockedsig not in self.lockedsigs_raw[t]:
+                self.lockedsigs_raw[t].append(lockedsig)
 
         with open(sigfile, "w") as f:
-            for t in types:
+            for t in sorted(self.lockedsigs_types):
                 f.write('SIGGEN_LOCKEDSIGS_%s = "\\\n' % t)
-                types[t].sort()
-                sortedk = sorted(types[t], key=lambda k: self.lockedpnmap[k.rsplit(".",1)[0]])
-                for k in sortedk:
-                    fn = k.rsplit(".",1)[0]
-                    task = k.rsplit(".",1)[1]
-                    if k not in self.taskhash:
-                        continue
-                    f.write("    " + self.lockedpnmap[fn] + ":" + task + ":" + self.taskhash[k] + " \\\n")
+                for ls in sorted(self.lockedsigs_raw[t]):
+                    f.write("    " + ls + " \\\n")
                 f.write('    "\n')
-            f.write('SIGGEN_LOCKEDSIGS_TYPES_%s = "%s"' % (self.machine, " ".join(types.keys())))
+            f.write('SIGGEN_LOCKEDSIGS_TYPES_%s = "%s"' %
+                    (self.machine, " ".join(sorted(self.lockedsigs_types))))
 
     def checkhashes(self, missed, ret, sq_fn, sq_task, sq_hash, sq_hashfn, d):
         checklevel = d.getVar("SIGGEN_LOCKEDSIGS_CHECK_LEVEL", True)
Signature · 1 line
1.9.1
#7 lore raw

Re: [PATCH 9/9] sstatesig: incremental dump lockedsigs

Quoted 7 lines · Hongxu Jia

The idea of incremental sig is:

New sig file = Old sig file (if available) + New sig items in current build.

Limit the modification within the dump_lockedsigs, and add two variables 'self.lockedsigs_types' and 'self.lockedsigs_raw' keep old sig file.

Quoted 8 lines

How to config for incremental dump:
...
USER_CLASSES += "sstate_lockedsig" SIGGEN_LOCKEDSIGS_CONFIG = "${TOPDIR}/locked-sigs.inc" require ${SIGGEN_LOCKEDSIGS_CONFIG}
...

I'm fine with the idea in principle. Why can't we do something like:

meta/lib/oe/sstatesig.py +5 -1
index af7617e..1da282e 100644
@@ -158,10 +158,14 @@ class SignatureGeneratorOEBasicHash(bb.siggen.SignatureGeneratorBasicHash):
                 sortedk = sorted(types[t], key=lambda k: self.lockedpnmap[k.rsplit(".",1)[0]])
                 for k in sortedk:
                     fn = k.rsplit(".",1)[0]
+                    pn = self.lockedpnmap[fn]
                     task = k.rsplit(".",1)[1]
                     if k not in self.taskhash:
                         continue
-                    f.write("    " + self.lockedpnmap[fn] + ":" + task + ":" + self.taskhash[k] + " \\\n")
+                    if pn in self.lockedsigs and task in self.lockedsigs[pn] and self.hashtask[k] == self.lockedsigs[pn][task]:
+                        continue
+                    sigentry = pn + ":" + task + ":" + self.taskhash[k]
+                    f.write("    " + sigentry + " \\\n")
                 f.write('    "\n')
             f.write('SIGGEN_LOCKEDSIGS_TYPES_%s = "%s"' % (self.machine, " ".join(types.keys())))
 

which is substantially simpler though?

Cheers,

Richard

#8 lore raw

Re: [PATCH 9/9] sstatesig: incremental dump lockedsigs

Quoted 18 lines · Richard Purdie
Quoted 7 lines · Hongxu Jia

The idea of incremental sig is:

New sig file = Old sig file (if available) + New sig items in current build.

Limit the modification within the dump_lockedsigs, and add two variables 'self.lockedsigs_types' and 'self.lockedsigs_raw' keep old sig file.

Quoted 8 lines

How to config for incremental dump:
...
USER_CLASSES += "sstate_lockedsig" SIGGEN_LOCKEDSIGS_CONFIG = "${TOPDIR}/locked-sigs.inc" require ${SIGGEN_LOCKEDSIGS_CONFIG}
...

I'm fine with the idea in principle. Why can't we do something like:

...

Quoted 1 line

which is substantially simpler though?

Wouldn't it keep accumulating locked hashes for same recipes?

Signature · 3 lines
Otavio Salvador                             O.S. Systems
http://www.ossystems.com.br        http://code.ossystems.com.br
Mobile: +55 (53) 9981-7854            Mobile: +1 (347) 903-9750
author
#9 lore raw

Re: [PATCH 9/9] sstatesig: incremental dump lockedsigs

Quoted 17 lines · Otavio Salvador
Quoted 15 lines · Richard Purdie
Quoted 14 lines · Hongxu Jia

The idea of incremental sig is:

New sig file = Old sig file (if available) + New sig items in current build.

Limit the modification within the dump_lockedsigs, and add two variables 'self.lockedsigs_types' and 'self.lockedsigs_raw' keep old sig file.

How to config for incremental dump:
...
USER_CLASSES += "sstate_lockedsig" SIGGEN_LOCKEDSIGS_CONFIG = "${TOPDIR}/locked-sigs.inc" require ${SIGGEN_LOCKEDSIGS_CONFIG}
...

Quoted 5 lines
Quoted 2 lines
Quoted 1 line

I'm fine with the idea in principle. Why can't we do something like:

...

Quoted 1 line

which is substantially simpler though?

Wouldn't it keep accumulating locked hashes for same recipes?

Of course not, checking added before write to file
...

+ if pn in self.lockedsigs and task in self.lockedsigs[pn] and self.hashtask[k] == self.lockedsigs[pn][task]:
+ continue

...

But there are still accumulating empty SIGGEN_LOCKEDSIGS added, such as:
...

SIGGEN_LOCKEDSIGS_t-x86-64 += "\
    "

SIGGEN_LOCKEDSIGS_TYPES_qemux86 += "t-x86-64"
...

I will fix it in V3

//Hongxu

Quoted 2 lines
author
#10 lore raw

Re: [PATCH 9/9] sstatesig: incremental dump lockedsigs

In reply to Richard Purdie
Quoted 39 lines · Richard Purdie
Quoted 7 lines · Hongxu Jia

The idea of incremental sig is:

New sig file = Old sig file (if available) + New sig items in current build.

Limit the modification within the dump_lockedsigs, and add two variables 'self.lockedsigs_types' and 'self.lockedsigs_raw' keep old sig file.

Quoted 8 lines

How to config for incremental dump:
...
USER_CLASSES += "sstate_lockedsig" SIGGEN_LOCKEDSIGS_CONFIG = "${TOPDIR}/locked-sigs.inc" require ${SIGGEN_LOCKEDSIGS_CONFIG}
...

I'm fine with the idea in principle. Why can't we do something like:

meta/lib/oe/sstatesig.py +5 -1
index af7617e..1da282e 100644
@@ -158,10 +158,14 @@ class SignatureGeneratorOEBasicHash(bb.siggen.SignatureGeneratorBasicHash):
                  sortedk = sorted(types[t], key=lambda k: self.lockedpnmap[k.rsplit(".",1)[0]])
                  for k in sortedk:
                      fn = k.rsplit(".",1)[0]
+                    pn = self.lockedpnmap[fn]
                      task = k.rsplit(".",1)[1]
                      if k not in self.taskhash:
                          continue
-                    f.write("    " + self.lockedpnmap[fn] + ":" + task + ":" + self.taskhash[k] + " \\\n")
+                    if pn in self.lockedsigs and task in self.lockedsigs[pn] and self.hashtask[k] == self.lockedsigs[pn][task]:
+                        continue
+                    sigentry = pn + ":" + task + ":" + self.taskhash[k]
+                    f.write("    " + sigentry + " \\\n")
                  f.write('    "\n')
              f.write('SIGGEN_LOCKEDSIGS_TYPES_%s = "%s"' % (self.machine, " ".join(types.keys())))
 
Quoted 1 line

which is substantially simpler though?

Agree, thanks for the improvement, I will send V3 with some trivial fixes.

//Hongxu

Quoted 6 lines

Cheers,

Richard