Skip to contentSkip to conversation

Checksum / error reporting improvements

PATCH Applied Paul Eggleton 2 participants 7 messages May 30, 2012

lore
Share & tools
lore
#0 lore raw

cover Checksum / error reporting improvements

Some checksum improvements and a few error reporting fixes that I made in the process. Note that the varflags change won't do anything without the list of exclusions; an appropriate list has been set in a patch which will shortly be set to the OE-Core list.

The following changes (against Poky, but which apply cleanly with -p2 against BitBake master) are available in the git repository at:

git://git.yoctoproject.org/poky-contrib paule/bb-checksum-fixes-2
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=paule/bb-checksum-fixes-2

Paul Eggleton (5):
lib/bb/data_smart.py: don't report variable in ExpansionError if not
set
cooker: fix UnboundLocalError when exception occurs during parsing
cooker: report recipe being parsed when ExpansionError occurs lib/bb/fetch2: ignore remote URIs when doing file checksums
bitbake: include varflags in checksums

 bitbake/lib/bb/cooker.py          |    8 +++++++-
 bitbake/lib/bb/data.py            |   20 +++++++++++++++++++-
 bitbake/lib/bb/data_smart.py      |    5 ++++-
 bitbake/lib/bb/fetch2/__init__.py |   26 ++++++++++++++++++++------
 bitbake/lib/bb/siggen.py          |    4 ++++
 5 files changed, 54 insertions(+), 9 deletions(-)
Signature · 1 line
1.7.9.5
#1 lore raw

PATCH 1/5 lib/bb/data_smart.py: don't report variable in ExpansionError if not set

If the variable name is not specified then don't confuse the error message by starting off with "Failure expanding variable None...".

---
 bitbake/lib/bb/data_smart.py |    5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
bitbake/lib/bb/data_smart.py +4 -1
index 27fb7d9..2c02cde 100644
@@ -102,7 +102,10 @@ class ExpansionError(Exception):
         self.expression = expression
         self.variablename = varname
         self.exception = exception
-        self.msg = "Failure expanding variable %s, expression was %s which triggered exception %s: %s" % (varname, expression, type(exception).__name__, exception)
+        if varname:
+            self.msg = "Failure expanding variable %s, expression was %s which triggered exception %s: %s" % (varname, expression, type(exception).__name__, exception)
+        else:
+            self.msg = "Failure expanding expression %s which triggered exception %s: %s" % (expression, type(exception).__name__, exception)
         Exception.__init__(self, self.msg)
         self.args = (varname, expression, exception)
     def __str__(self):
Signature · 1 line
1.7.9.5
#2 lore raw

PATCH 2/5 cooker: fix UnboundLocalError when exception occurs during parsing

In reply to Paul Eggleton

Fix a recent regression where we see the following additional error after an error occurs during parsing:

ERROR: Command execution failed: Traceback (most recent call last):
File "/home/paul/poky/poky/bitbake/lib/bb/command.py", line 84, in runAsyncCommand
self.cooker.updateCache()
File "/home/paul/poky/poky/bitbake/lib/bb/cooker.py", line 1202, in updateCache
if not self.parser.parse_next():
File "/home/paul/poky/poky/bitbake/lib/bb/cooker.py", line 1672, in parse_next

    self.virtuals += len(result)

UnboundLocalError: local variable 'result' referenced before assignment

---
 bitbake/lib/bb/cooker.py |    2 ++
 1 file changed, 2 insertions(+)
bitbake/lib/bb/cooker.py +2 -0
index d1de757..b30945c 100644
@@ -1646,6 +1646,8 @@ class CookerParser(object):
                     yield result
 
     def parse_next(self):
+        result = []
+        parsed = None
         try:
             parsed, result = self.results.next()
         except StopIteration:
Signature · 1 line
1.7.9.5
#3 lore raw

PATCH 3/5 cooker: report recipe being parsed when ExpansionError occurs

In reply to Paul Eggleton

When an ExpansionError occurs during parsing it is useful to know which recipe was being parsed when it occurred.

---
 bitbake/lib/bb/cooker.py |    6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
bitbake/lib/bb/cooker.py +5 -1
index b30945c..928b600 100644
@@ -1657,9 +1657,13 @@ class CookerParser(object):
             logger.error('Unable to parse %s: %s' %
                      (exc.recipe, bb.exceptions.to_string(exc.realexception)))
             self.shutdown(clean=False)
-        except (bb.parse.ParseError, bb.data_smart.ExpansionError) as exc:
+        except bb.parse.ParseError as exc:
             logger.error(str(exc))
             self.shutdown(clean=False)
+        except bb.data_smart.ExpansionError as exc:
+            _, value, _ = sys.exc_info()
+            logger.error('ExpansionError during parsing %s: %s', value.recipe, str(exc))
+            self.shutdown(clean=False)
         except SyntaxError as exc:
             logger.error('Unable to parse %s', exc.recipe)
             self.shutdown(clean=False)
Signature · 1 line
1.7.9.5
#4 lore raw

PATCH 4/5 lib/bb/fetch2: ignore remote URIs when doing file checksums

In reply to Paul Eggleton

Skip evaluating remote URIs when doing local file checksums, because we don't need to process them and doing so will trigger a parse failure if SRCREV is not fully specified. Whilst this is just delaying a check until runtime (when do_fetch runs for the recipe) we're only validating this here accidentally and if we did wish to check it during parsing it ought to be done explicitly.

Fixes [YOCTO #2512]

---
 bitbake/lib/bb/fetch2/__init__.py |   26 ++++++++++++++++++++------
 1 file changed, 20 insertions(+), 6 deletions(-)
bitbake/lib/bb/fetch2/__init__.py +20 -6
index 6ae69cd..83050e4 100644
@@ -112,6 +112,9 @@ class NetworkAccess(BBFetchException):
          BBFetchException.__init__(self, msg)
          self.args = (url, cmd)
 
+class NonLocalMethod(Exception):
+    def __init__(self):
+        Exception.__init__(self)
 
 def decodeurl(url):
     """Decodes an URL into the tokens (scheme, network location, path,
@@ -565,17 +568,17 @@ def srcrev_internal_helper(ud, d, name):
 def get_checksum_file_list(d):
     """ Get a list of files checksum in SRC_URI
 
-    Returns the all resolved local path of all local file entries in
+    Returns the resolved local paths of all local file entries in
     SRC_URI as a space-separated string
     """
-    fetch = Fetch([], d)
+    fetch = Fetch([], d, cache = False, localonly = True)
 
     dl_dir = d.getVar('DL_DIR', True)
     filelist = []
     for u in fetch.urls:
         ud = fetch.ud[u]
 
-        if isinstance(ud.method, local.Local):
+        if ud and isinstance(ud.method, local.Local):
             ud.setup_localpath(d)
             f = ud.localpath
             if f.startswith(dl_dir):
@@ -639,7 +642,7 @@ class FetchData(object):
     """
     A class which represents the fetcher state for a given URI.
     """
-    def __init__(self, url, d):
+    def __init__(self, url, d, localonly = False):
         # localpath is the location of a downloaded result. If not set, the file is local.
         self.donestamp = None
         self.localfile = ""
@@ -686,6 +689,9 @@ class FetchData(object):
         if not self.method:
             raise NoMethodError(url)
 
+        if localonly and not isinstance(self.method, local.Local):
+            raise NonLocalMethod()
+
         if hasattr(self.method, "urldata_init"):
             self.method.urldata_init(self, d)
 
@@ -1009,7 +1015,10 @@ class FetchMethod(object):
         return "%s-%s" % (key, d.getVar("PN", True) or "")
 
 class Fetch(object):
-    def __init__(self, urls, d, cache = True):
+    def __init__(self, urls, d, cache = True, localonly = False):
+        if localonly and cache:
+            raise Exception("bb.fetch2.Fetch.__init__: cannot set cache and localonly at same time")
+
         if len(urls) == 0:
             urls = d.getVar("SRC_URI", True).split()
         self.urls = urls
@@ -1022,7 +1031,12 @@ class Fetch(object):
 
         for url in urls:
             if url not in self.ud:
-                self.ud[url] = FetchData(url, d)
+                try:
+                    self.ud[url] = FetchData(url, d, localonly)
+                except NonLocalMethod:
+                    if localonly:
+                        self.ud[url] = None
+                        pass
 
         if cache:
             urldata_cache[fn] = self.ud
Signature · 1 line
1.7.9.5
#5 lore raw

PATCH 5/5 bitbake: include varflags in checksums

In reply to Paul Eggleton

Add a dependency to each variable on a filtered list of its varflags. This is intended to catch things such as SRC_URI checksums, varflags controlling extra functionality from classes (e.g. the recently updated update-alternatives class in OE-Core), etc. and ensure their values influence the sstate checksums.

There is an exclusion list which needs to be set via bitbake.conf (BB_SIGNATURE_EXCLUDE_FLAGS), if this is not set then the functionality is disabled. The existing vardepsexclude mechanism can also be used to exclude undesired varflags, but they must be fully specified, e.g.:

do_patch[vardepsexclude] += "do_patch[someflag]"

Implements [YOCTO #2517].

---
 bitbake/lib/bb/data.py   |   20 +++++++++++++++++++-
 bitbake/lib/bb/siggen.py |    4 ++++
 2 files changed, 23 insertions(+), 1 deletion(-)
bitbake/lib/bb/data.py +19 -1
index c0636e1..e3ffefe 100644
@@ -279,7 +279,12 @@ def build_dependencies(key, keys, shelldeps, vardepvals, d):
     deps = set()
     vardeps = d.getVarFlag(key, "vardeps", True)
     try:
-        value = d.getVar(key, False)
+        if key[-1] == ']':
+            vf = key[:-1].split('[')
+            value = d.getVarFlag(vf[0], vf[1], False)
+        else:
+            value = d.getVar(key, False)
+
         if key in vardepvals:
            value =  d.getVarFlag(key, "vardepvalue", True)
         elif d.getVarFlag(key, "func"):
@@ -301,6 +306,19 @@ def build_dependencies(key, keys, shelldeps, vardepvals, d):
             parser = d.expandWithRefs(value, key)
             deps |= parser.references
             deps = deps | (keys & parser.execs)
+
+        # Add varflags, assuming an exclusion list is set
+        varflagsexcl = d.getVar('BB_SIGNATURE_EXCLUDE_FLAGS', True)
+        if varflagsexcl:
+            varfdeps = []
+            varflags = d.getVarFlags(key)
+            if varflags:
+                for f in varflags:
+                    if f not in varflagsexcl:
+                        varfdeps.append('%s[%s]' % (key, f))
+            if varfdeps:
+                deps |= set(varfdeps)
+
         deps |= set((vardeps or "").split())
         deps -= set((d.getVarFlag(key, "vardepsexclude", True) or "").split())
     except:
bitbake/lib/bb/siggen.py +4 -0
index daf5677..c4b7c39 100644
@@ -108,6 +108,10 @@ class SignatureGeneratorBasic(SignatureGenerator):
                 data = data + dep
                 if dep in lookupcache:
                     var = lookupcache[dep]
+                elif dep[-1] == ']':
+                    vf = dep[:-1].split('[')
+                    var = d.getVarFlag(vf[0], vf[1], False)
+                    lookupcache[dep] = var
                 else:
                     var = d.getVar(dep, False)
                     lookupcache[dep] = var
Signature · 1 line
1.7.9.5
#6 lore raw
In reply to Paul Eggleton
Quoted 12 lines · Paul Eggleton

Some checksum improvements and a few error reporting fixes that I made in the process. Note that the varflags change won't do anything without the list of exclusions; an appropriate list has been set in a patch which will shortly be set to the OE-Core list.

The following changes (against Poky, but which apply cleanly with -p2 against BitBake master) are available in the git repository at:

git://git.yoctoproject.org/poky-contrib paule/bb-checksum-fixes-2
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=paule/bb-checksum-fixes-2

Quoted 7 lines

Paul Eggleton (5):
lib/bb/data_smart.py: don't report variable in ExpansionError if not
set
cooker: fix UnboundLocalError when exception occurs during parsing
cooker: report recipe being parsed when ExpansionError occurs lib/bb/fetch2: ignore remote URIs when doing file checksums
bitbake: include varflags in checksums

Merged to master, thanks.

Richard