Skip to contentSkip to conversation

sstate.bbclass: When siginfo or sig files are missing, stop fetcher errors

PATCH Mark Hatle 2 participants 2 messages May 27, 2020 – Jun 2, 2020 1 message not in the archive

lore
Share & tools
lore

Mark Hatle

author
#1 lore raw
In reply to a message not in the archive

Ping

I noticed this hasn't been integrated or commented on yet.

Quoted 63 lines · Mark Hatle

From: Mark Hatle <[email protected]>

Prior to fetching, the system checks if the sstate file is present either locally or on the mirror. If it is, then it goes to the fetch stage. Up to three files can be fetched, sstate, sstate.siginfo and sstate.sig (if signature validation is enabled).

The previous pstaging_fetch function would iterate over these, and if a download error occurred would spew forth a great amount of fetcher failure messages as well as stop fetching the next item in the set.

This was resolved by adding a fetcher.checkstatus() call prior to the download. If the file isn't present, then the exception will be triggered, and no fetcher failure messages will reach the user.

The exception handler is then modified to be a pass so that it will loop and pull the rest of the files that that are requested.

Additionally, a check for the existance of the .sig file was added to the sstate_installpkg to avoid an error trying to load the .sig if it wasn't downloaded.

---

v2 - fix a typo in the error message about signature not being
present.

 meta/classes/sstate.bbclass | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
meta/classes/sstate.bbclass +5 -1
index aa9c30b4e1..375196ef21 100644
@@ -355,6 +355,9 @@ def sstate_installpkg(ss, d):
     d.setVar('SSTATE_INSTDIR', sstateinst)
 
     if bb.utils.to_boolean(d.getVar("SSTATE_VERIFY_SIG"), False):
+        if not os.path.isfile(sstatepkg + '.sig'):
+            bb.warn("No signature file for sstate package %s, skipping acceleration..." % sstatepkg)
+            return False
         signer = get_signer(d, 'local')
         if not signer.verify(sstatepkg + '.sig'):
             bb.warn("Cannot verify signature on sstate package %s, skipping acceleration..." % sstatepkg)
@@ -733,10 +736,11 @@ def pstaging_fetch(sstatefetch, d):
         localdata.setVar('SRC_URI', srcuri)
         try:
             fetcher = bb.fetch2.Fetch([srcuri], localdata, cache=False)
+            fetcher.checkstatus()
             fetcher.download()
 
         except bb.fetch2.BBFetchException:
-            break
+            pass
 
 def sstate_setscene(d):
     shared_state = sstate_state_fromvars(d)
#2 lore raw
Quoted 63 lines · Mark Hatle

Ping

I noticed this hasn't been integrated or commented on yet.

Quoted 57 lines · Mark Hatle

From: Mark Hatle <[email protected]>

Prior to fetching, the system checks if the sstate file is present either locally or on the mirror. If it is, then it goes to the fetch stage. Up to three files can be fetched, sstate, sstate.siginfo and sstate.sig (if signature validation is enabled).

The previous pstaging_fetch function would iterate over these, and if a download error occurred would spew forth a great amount of fetcher failure messages as well as stop fetching the next item in the set.

This was resolved by adding a fetcher.checkstatus() call prior to the download. If the file isn't present, then the exception will be triggered, and no fetcher failure messages will reach the user.

The exception handler is then modified to be a pass so that it will loop and pull the rest of the files that that are requested.

Additionally, a check for the existance of the .sig file was added to the sstate_installpkg to avoid an error trying to load the .sig if it wasn't downloaded.

---

v2 - fix a typo in the error message about signature not being
present.

 meta/classes/sstate.bbclass | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
meta/classes/sstate.bbclass +5 -1
index aa9c30b4e1..375196ef21 100644
@@ -355,6 +355,9 @@ def sstate_installpkg(ss, d):
     d.setVar('SSTATE_INSTDIR', sstateinst)
 
     if bb.utils.to_boolean(d.getVar("SSTATE_VERIFY_SIG"), False):
+        if not os.path.isfile(sstatepkg + '.sig'):
+            bb.warn("No signature file for sstate package %s, skipping acceleration..." % sstatepkg)
+            return False
         signer = get_signer(d, 'local')
         if not signer.verify(sstatepkg + '.sig'):
             bb.warn("Cannot verify signature on sstate package %s, skipping acceleration..." % sstatepkg)
@@ -733,10 +736,11 @@ def pstaging_fetch(sstatefetch, d):
         localdata.setVar('SRC_URI', srcuri)
         try:
             fetcher = bb.fetch2.Fetch([srcuri], localdata, cache=False)
+            fetcher.checkstatus()
             fetcher.download()
 
         except bb.fetch2.BBFetchException:
-            break
+            pass
 
Quoted 6 lines

meta/classes/sstate.bbclass def pstaging_fetch(sstatefetch, d): · line 746

Quoted 6 lines
meta/classes/sstate.bbclass +0 -0
 def sstate_setscene(d):
     shared_state = sstate_state_fromvars(d)

I see this has now been committed to master.

I've previously discussed issues with the separation between time-of-check and time-of-use for sstate artifacts (see
https://lists.yoctoproject.org/g/yocto/topic/71426351 for the thread).

If checkstatus() succeeded in sstate_checkhashes() then the sstate artifact was marked as available. If the download then failed in sstate_setscene(), bitbake would fall back to running the real task instead of using sstate, however the build would be marked as a failure with bitbake returning non-zero even if the targets were successfully built.

Should this patch fix that issue? I'm not 100% clear on whether this is the same issue or if your patch is just addressing the case where one of the files is present but the others aren't.

Thanks,

Signature · 2 lines
Paul Barker
Konsulko Group