Skip to contentSkip to conversation

sstate.bbclass: exceptions on mirror fetcher is not fatal

PATCH In review Jose Quaresma 2 participants 4 messages Oct 29, 2021 – Nov 4, 2021

lore
Share & tools
lore
#0 lore raw

PATCH 1/2 sstate.bbclass: exceptions on mirror fetcher is not fatal

An exceptions in sstate_checkhashes sstate mirror fetcher is not fatal for the build and it can be a warning.

If we can't check status in the fetcher we can assume that sstate file don't exist on the mirror and build it from sources, ignoring the sstate cache.

---
 meta/classes/sstate.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/classes/sstate.bbclass +1 -1
index 6e4eb09f8e..54d38ed42d 100644
@@ -995,7 +995,7 @@ def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0, summary=True,
                 missed.add(tid)
                 bb.debug(2, "SState: Unsuccessful fetch test for %s (%s)" % (srcuri, e))
             except Exception as e:
-                bb.error("SState: cannot test %s: %s" % (srcuri, e))
+                bb.warn("SState: cannot test file %s from mirrors '%s': (%s)" % (srcuri, mirrors, e))
             if len(tasklist) >= min_tasks:
                 bb.event.fire(bb.event.ProcessProgress(msg, len(tasklist) - thread_worker.tasks.qsize()), d)
 
Signature · 1 line
2.33.1
#1 lore raw

PATCH 2/2 sstate.bbclass: add warning for exceptions on pstaging_fetch

Notify the user when there are connection issues with the mirror

---
 meta/classes/sstate.bbclass | 2 ++
 1 file changed, 2 insertions(+)
meta/classes/sstate.bbclass +2 -0
index 54d38ed42d..32cf3a9fe7 100644
@@ -785,6 +785,8 @@ def pstaging_fetch(sstatefetch, d):
 
         except bb.fetch2.BBFetchException:
             pass
+        except Exception as e:
+            bb.warn("SState: fetching file %s from mirrors '%s': (%s)" % (srcuri, mirrors, e))
 
 pstaging_fetch[vardepsexclude] += "SRCPV"
 
Signature · 1 line
2.33.1
#2 lore raw
In reply to Jose Quaresma
Quoted 17 lines · Jose Quaresma

An exceptions in sstate_checkhashes sstate mirror fetcher is not fatal for the build and it can be a warning.

If we can't check status in the fetcher we can assume that sstate file don't exist on the mirror and build it from sources, ignoring the sstate cache.

---
 meta/classes/sstate.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/classes/sstate.bbclass +0 -0
index 6e4eb09f8e..54d38ed42d 100644

meta/classes/sstate.bbclass def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0, summary=True, · line 1001

Quoted 8 lines
meta/classes/sstate.bbclass +1 -1
@@ -995,7 +995,7 @@ def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0, summary=True,
                 missed.add(tid)
                 bb.debug(2, "SState: Unsuccessful fetch test for %s (%s)" % (srcuri, e))
             except Exception as e:
-                bb.error("SState: cannot test %s: %s" % (srcuri, e))
+                bb.warn("SState: cannot test file %s from mirrors '%s': (%s)" % (srcuri, mirrors, e))
             if len(tasklist) >= min_tasks:
                 bb.event.fire(bb.event.ProcessProgress(msg, len(tasklist) - thread_worker.tasks.qsize()), d)
 

This has been discussed a bit before. Whilst the issue isn't fatal to the build I do think this should be an error and have bitbake return an exit code that there was an issue as there is something wrong somewhere if we're hitting this code path.

As an example:

https://autobuilder.yoctoproject.org/typhoon/#/builders/63/builds/4230/steps/15/logs/stdio

which might have not shown up if it wasn't an error.

Cheers,

Richard

#3 lore raw

Richard Purdie <[email protected]> escreveu no dia sábado, 30/10/2021 à(s) 22:34:

Quoted 39 lines · Jose Quaresma

meta/classes/sstate.bbclass def sstate_checkhashes(sq_data, d, siginfo=False,

Quoted 17 lines

An exceptions in sstate_checkhashes sstate mirror fetcher is not fatal for the build and it can be a warning.

If we can't check status in the fetcher we can assume that sstate file don't exist on the mirror and build it from sources, ignoring the sstate cache.

---
 meta/classes/sstate.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/classes/sstate.bbclass +0 -0
index 6e4eb09f8e..54d38ed42d 100644
@@ -995,7 +995,7 @@ def sstate_checkhashes(sq_data, d, siginfo=False,

currentcount=0, summary=True,

meta/classes/sstate.bbclass def sstate_checkhashes(sq_data, d, siginfo=False, · line 996

Quoted 2 lines
meta/classes/sstate.bbclass +0 -0
                 missed.add(tid)
                 bb.debug(2, "SState: Unsuccessful fetch test for %s

(%s)" % (srcuri, e))

meta/classes/sstate.bbclass def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0, summary=True,

Quoted 3 lines
meta/classes/sstate.bbclass +1 -1
             except Exception as e:
-                bb.error("SState: cannot test %s: %s" % (srcuri, e))
+                bb.warn("SState: cannot test file %s from mirrors '%s':

(%s)" % (srcuri, mirrors, e))

meta/classes/sstate.bbclass def sstate_checkhashes(sq_data, d, siginfo=False, · line 1000

Quoted 2 lines
meta/classes/sstate.bbclass +0 -0
             if len(tasklist) >= min_tasks:
                 bb.event.fire(bb.event.ProcessProgress(msg,

len(tasklist) - thread_worker.tasks.qsize()), d)

Quoted 1 line

This has been discussed a bit before. Whilst the issue isn't fatal to the build
I do think this should be an error and have bitbake return an exit code that
there was an issue as there is something wrong somewhere if we're hitting this
code path.

Quoted 8 lines

As an example:

https://autobuilder.yoctoproject.org/typhoon/#/builders/63/builds/4230/steps/15/logs/stdio

which might have not shown up if it wasn't an error.

I was not aware that this subject had already been discussed. But it makes sense that when we have the sstate mirrors, if we can't communicate with the server we have a problem.

My assumption in this patch can hide these issues as it will always build from the sources and the warning can go unnoticed.

Cheers,

Quoted 6 lines

Richard

Signature · 3 lines
Best regards,

José Quaresma