Fix for two issues related to sstate-cache
cover Fix for two issues related to sstate-cache
Patch 1 is a backport from master (it's already in Dunfell). If you have a locked-sigs file, and the sstate for that item isn't available the .siginfo and sigdata files were not being written. Patch 1/2 resolved that with the fragment in "dump_sigtask".
Patch 2 was sent for master as well, and this is just to get it applied to Zeus. Due to problem #1, we were generating sstate-cache that did not have siginfo files. Due to this we were getting failures later on using that sstate-cache in a subsequent build. This patch ensures if this happens again (of if someone intentionally strips those) it won't fail.
Mark Hatle (1):
sstate.bbclass: When siginfo or sig files are missing, stop fetcher
errors
Richard Purdie (1):
sstatesig: Optimise get_taskhash for hashequiv
meta/classes/sstate.bbclass | 6 +++++- meta/lib/oe/sstatesig.py | 13 +++++++++++-- 2 files changed, 16 insertions(+), 3 deletions(-)
Signature · 1 line
2.17.1
PATCH 1/2 sstatesig: Optimise get_taskhash for hashequiv
From: Richard Purdie <[email protected]>
With hashequiv the get_taskhash function is called much more regularly and contains expensive operations. This these don't change based upon hash in a given build, improve the caching within the function to reduce overhead.
(From OE-Core rev: de98cfe3cde4b8d5f4b163b5fba3f129651ef06a)
--- meta/lib/oe/sstatesig.py | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-)
meta/lib/oe/sstatesig.py +11 -2
@@ -151,6 +151,13 @@ class SignatureGeneratorOEBasicHashMixIn(object): def get_taskhash(self, tid, deps, dataCache): h = super(bb.siggen.SignatureGeneratorBasicHash, self).get_taskhash(tid, deps, dataCache) + if tid in self.lockedhashes: + if self.lockedhashes[tid]: + return self.lockedhashes[tid] + else: + return h + + h = super(bb.siggen.SignatureGeneratorBasicHash, self).get_taskhash(tid, deps, dataCache) (mc, _, task, fn) = bb.runqueue.split_tid_mcfn(tid)
@@ -187,17 +194,19 @@ class SignatureGeneratorOEBasicHashMixIn(object): % (recipename, task, h, h_locked, var)) return h_locked + + self.lockedhashes[tid] = False #bb.warn("%s %s %s" % (recipename, task, h)) return h def get_unihash(self, tid): - if tid in self.lockedhashes: + if tid in self.lockedhashes and self.lockedhashes[tid]: return self.lockedhashes[tid] return super().get_unihash(tid) def dump_sigtask(self, fn, task, stampbase, runtime): tid = fn + ":" + task - if tid in self.lockedhashes: + if tid in self.lockedhashes and self.lockedhashes[tid]: return super(bb.siggen.SignatureGeneratorBasicHash, self).dump_sigtask(fn, task, stampbase, runtime)
Signature · 1 line
2.17.1
PATCH 2/2 sstate.bbclass: When siginfo or sig files are missing, stop fetcher errors
In reply to Mark HatleFrom: Mark Hatle <[email protected]>
Prior to fetching, the system checks if the sstate file is present either locally or on the mirror. If it is, then it goes to the fetch stage. Up to three files can be fetched, sstate, sstate.siginfo and sstate.sig (if signature validation is enabled).
The previous pstaging_fetch function would iterate over these, and if a download error occurred would spew forth a great amount of fetcher failure messages as well as stop fetching the next item in the set.
This was resolved by adding a fetcher.checkstatus() call prior to the download. If the file isn't present, then the exception will be triggered, and no fetcher failure messages will reach the user.
The exception handler is then modified to be a pass so that it will loop and pull the rest of the files that that are requested.
Additionally, a check for the existance of the .sig file was added to the sstate_installpkg to avoid an error trying to load the .sig if it wasn't downloaded.
--- meta/classes/sstate.bbclass | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
meta/classes/sstate.bbclass +5 -1
@@ -333,6 +333,9 @@ def sstate_installpkg(ss, d): d.setVar('SSTATE_INSTDIR', sstateinst) if bb.utils.to_boolean(d.getVar("SSTATE_VERIFY_SIG"), False): + if not os.path.isfile(sstatepkg + '.sig'): + bb.warn("Not signature file for sstate package %s, skipping acceleration..." % sstatepkg) + return False signer = get_signer(d, 'local') if not signer.verify(sstatepkg + '.sig'): bb.warn("Cannot verify signature on sstate package %s, skipping acceleration..." % sstatepkg)
@@ -703,10 +706,11 @@ def pstaging_fetch(sstatefetch, d): localdata.setVar('SRC_URI', srcuri) try: fetcher = bb.fetch2.Fetch([srcuri], localdata, cache=False) + fetcher.checkstatus() fetcher.download() except bb.fetch2.BBFetchException: - break + pass def sstate_setscene(d): shared_state = sstate_state_fromvars(d)
Signature · 1 line
2.17.1