Skip to contentSkip to conversation

Fix for two issues related to sstate-cache

PATCH New Mark Hatle 1 participant 3 messages May 13, 2020

lore
Share & tools
lore

Mark Hatle

author
#0 lore raw

cover Fix for two issues related to sstate-cache

Patch 1 is a backport from master (it's already in Dunfell). If you have a locked-sigs file, and the sstate for that item isn't available the .siginfo and sigdata files were not being written. Patch 1/2 resolved that with the fragment in "dump_sigtask".

Patch 2 was sent for master as well, and this is just to get it applied to Zeus. Due to problem #1, we were generating sstate-cache that did not have siginfo files. Due to this we were getting failures later on using that sstate-cache in a subsequent build. This patch ensures if this happens again (of if someone intentionally strips those) it won't fail.

Mark Hatle (1):
sstate.bbclass: When siginfo or sig files are missing, stop fetcher
errors

Richard Purdie (1):
sstatesig: Optimise get_taskhash for hashequiv

 meta/classes/sstate.bbclass |  6 +++++-
 meta/lib/oe/sstatesig.py    | 13 +++++++++++--
 2 files changed, 16 insertions(+), 3 deletions(-)
Signature · 1 line
2.17.1
author
#1 lore raw

PATCH 1/2 sstatesig: Optimise get_taskhash for hashequiv

From: Richard Purdie <[email protected]>

With hashequiv the get_taskhash function is called much more regularly and contains expensive operations. This these don't change based upon hash in a given build, improve the caching within the function to reduce overhead.

(From OE-Core rev: de98cfe3cde4b8d5f4b163b5fba3f129651ef06a)

---
 meta/lib/oe/sstatesig.py | 13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)
meta/lib/oe/sstatesig.py +11 -2
index b2316b12b8..f1abff0c45 100644
@@ -151,6 +151,13 @@ class SignatureGeneratorOEBasicHashMixIn(object):
 
     def get_taskhash(self, tid, deps, dataCache):
         h = super(bb.siggen.SignatureGeneratorBasicHash, self).get_taskhash(tid, deps, dataCache)
+        if tid in self.lockedhashes:
+            if self.lockedhashes[tid]:
+                return self.lockedhashes[tid]
+            else:
+                return h
+
+        h = super(bb.siggen.SignatureGeneratorBasicHash, self).get_taskhash(tid, deps, dataCache)
 
         (mc, _, task, fn) = bb.runqueue.split_tid_mcfn(tid)
 
@@ -187,17 +194,19 @@ class SignatureGeneratorOEBasicHashMixIn(object):
                                           % (recipename, task, h, h_locked, var))
 
                 return h_locked
+
+        self.lockedhashes[tid] = False
         #bb.warn("%s %s %s" % (recipename, task, h))
         return h
 
     def get_unihash(self, tid):
-        if tid in self.lockedhashes:
+        if tid in self.lockedhashes and self.lockedhashes[tid]:
             return self.lockedhashes[tid]
         return super().get_unihash(tid)
 
     def dump_sigtask(self, fn, task, stampbase, runtime):
         tid = fn + ":" + task
-        if tid in self.lockedhashes:
+        if tid in self.lockedhashes and self.lockedhashes[tid]:
             return
         super(bb.siggen.SignatureGeneratorBasicHash, self).dump_sigtask(fn, task, stampbase, runtime)
 
Signature · 1 line
2.17.1
author
#2 lore raw

PATCH 2/2 sstate.bbclass: When siginfo or sig files are missing, stop fetcher errors

In reply to Mark Hatle

From: Mark Hatle <[email protected]>

Prior to fetching, the system checks if the sstate file is present either locally or on the mirror. If it is, then it goes to the fetch stage. Up to three files can be fetched, sstate, sstate.siginfo and sstate.sig (if signature validation is enabled).

The previous pstaging_fetch function would iterate over these, and if a download error occurred would spew forth a great amount of fetcher failure messages as well as stop fetching the next item in the set.

This was resolved by adding a fetcher.checkstatus() call prior to the download. If the file isn't present, then the exception will be triggered, and no fetcher failure messages will reach the user.

The exception handler is then modified to be a pass so that it will loop and pull the rest of the files that that are requested.

Additionally, a check for the existance of the .sig file was added to the sstate_installpkg to avoid an error trying to load the .sig if it wasn't downloaded.

---
 meta/classes/sstate.bbclass | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
meta/classes/sstate.bbclass +5 -1
index c0329cd5d1..3528908727 100644
@@ -333,6 +333,9 @@ def sstate_installpkg(ss, d):
     d.setVar('SSTATE_INSTDIR', sstateinst)
 
     if bb.utils.to_boolean(d.getVar("SSTATE_VERIFY_SIG"), False):
+        if not os.path.isfile(sstatepkg + '.sig'):
+            bb.warn("Not signature file for sstate package %s, skipping acceleration..." % sstatepkg)
+            return False
         signer = get_signer(d, 'local')
         if not signer.verify(sstatepkg + '.sig'):
             bb.warn("Cannot verify signature on sstate package %s, skipping acceleration..." % sstatepkg)
@@ -703,10 +706,11 @@ def pstaging_fetch(sstatefetch, d):
         localdata.setVar('SRC_URI', srcuri)
         try:
             fetcher = bb.fetch2.Fetch([srcuri], localdata, cache=False)
+            fetcher.checkstatus()
             fetcher.download()
 
         except bb.fetch2.BBFetchException:
-            break
+            pass
 
 def sstate_setscene(d):
     shared_state = sstate_state_fromvars(d)
Signature · 1 line
2.17.1