Checksum / error reporting improvements
cover Checksum / error reporting improvements
Some checksum improvements and a few error reporting fixes that I made in the process. Note that the varflags change won't do anything without the list of exclusions; an appropriate list has been set in a patch which will shortly be set to the OE-Core list.
The following changes (against Poky, but which apply cleanly with -p2 against BitBake master) are available in the git repository at:
git://git.yoctoproject.org/poky-contrib paule/bb-checksum-fixes-2
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=paule/bb-checksum-fixes-2
Paul Eggleton (5):
lib/bb/data_smart.py: don't report variable in ExpansionError if not
set
cooker: fix UnboundLocalError when exception occurs during parsing
cooker: report recipe being parsed when ExpansionError occurs lib/bb/fetch2: ignore remote URIs when doing file checksums
bitbake: include varflags in checksums
bitbake/lib/bb/cooker.py | 8 +++++++- bitbake/lib/bb/data.py | 20 +++++++++++++++++++- bitbake/lib/bb/data_smart.py | 5 ++++- bitbake/lib/bb/fetch2/__init__.py | 26 ++++++++++++++++++++------ bitbake/lib/bb/siggen.py | 4 ++++ 5 files changed, 54 insertions(+), 9 deletions(-)
Signature · 1 line
1.7.9.5
PATCH 1/5 lib/bb/data_smart.py: don't report variable in ExpansionError if not set
If the variable name is not specified then don't confuse the error message by starting off with "Failure expanding variable None...".
--- bitbake/lib/bb/data_smart.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
bitbake/lib/bb/data_smart.py +4 -1
@@ -102,7 +102,10 @@ class ExpansionError(Exception): self.expression = expression self.variablename = varname self.exception = exception - self.msg = "Failure expanding variable %s, expression was %s which triggered exception %s: %s" % (varname, expression, type(exception).__name__, exception) + if varname: + self.msg = "Failure expanding variable %s, expression was %s which triggered exception %s: %s" % (varname, expression, type(exception).__name__, exception) + else: + self.msg = "Failure expanding expression %s which triggered exception %s: %s" % (expression, type(exception).__name__, exception) Exception.__init__(self, self.msg) self.args = (varname, expression, exception) def __str__(self):
Signature · 1 line
1.7.9.5
PATCH 2/5 cooker: fix UnboundLocalError when exception occurs during parsing
In reply to Paul EggletonFix a recent regression where we see the following additional error after an error occurs during parsing:
ERROR: Command execution failed: Traceback (most recent call last):
File "/home/paul/poky/poky/bitbake/lib/bb/command.py", line 84, in runAsyncCommand
self.cooker.updateCache()
File "/home/paul/poky/poky/bitbake/lib/bb/cooker.py", line 1202, in updateCache
if not self.parser.parse_next():
File "/home/paul/poky/poky/bitbake/lib/bb/cooker.py", line 1672, in parse_next
self.virtuals += len(result)
UnboundLocalError: local variable 'result' referenced before assignment
--- bitbake/lib/bb/cooker.py | 2 ++ 1 file changed, 2 insertions(+)
bitbake/lib/bb/cooker.py +2 -0
@@ -1646,6 +1646,8 @@ class CookerParser(object): yield result def parse_next(self): + result = [] + parsed = None try: parsed, result = self.results.next() except StopIteration:
Signature · 1 line
1.7.9.5
PATCH 3/5 cooker: report recipe being parsed when ExpansionError occurs
In reply to Paul EggletonWhen an ExpansionError occurs during parsing it is useful to know which recipe was being parsed when it occurred.
--- bitbake/lib/bb/cooker.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
bitbake/lib/bb/cooker.py +5 -1
@@ -1657,9 +1657,13 @@ class CookerParser(object): logger.error('Unable to parse %s: %s' % (exc.recipe, bb.exceptions.to_string(exc.realexception))) self.shutdown(clean=False) - except (bb.parse.ParseError, bb.data_smart.ExpansionError) as exc: + except bb.parse.ParseError as exc: logger.error(str(exc)) self.shutdown(clean=False) + except bb.data_smart.ExpansionError as exc: + _, value, _ = sys.exc_info() + logger.error('ExpansionError during parsing %s: %s', value.recipe, str(exc)) + self.shutdown(clean=False) except SyntaxError as exc: logger.error('Unable to parse %s', exc.recipe) self.shutdown(clean=False)
Signature · 1 line
1.7.9.5
PATCH 4/5 lib/bb/fetch2: ignore remote URIs when doing file checksums
In reply to Paul EggletonSkip evaluating remote URIs when doing local file checksums, because we don't need to process them and doing so will trigger a parse failure if SRCREV is not fully specified. Whilst this is just delaying a check until runtime (when do_fetch runs for the recipe) we're only validating this here accidentally and if we did wish to check it during parsing it ought to be done explicitly.
Fixes [YOCTO #2512]
--- bitbake/lib/bb/fetch2/__init__.py | 26 ++++++++++++++++++++------ 1 file changed, 20 insertions(+), 6 deletions(-)
bitbake/lib/bb/fetch2/__init__.py +20 -6
@@ -112,6 +112,9 @@ class NetworkAccess(BBFetchException): BBFetchException.__init__(self, msg) self.args = (url, cmd) +class NonLocalMethod(Exception): + def __init__(self): + Exception.__init__(self) def decodeurl(url): """Decodes an URL into the tokens (scheme, network location, path,
@@ -565,17 +568,17 @@ def srcrev_internal_helper(ud, d, name): def get_checksum_file_list(d): """ Get a list of files checksum in SRC_URI - Returns the all resolved local path of all local file entries in + Returns the resolved local paths of all local file entries in SRC_URI as a space-separated string """ - fetch = Fetch([], d) + fetch = Fetch([], d, cache = False, localonly = True) dl_dir = d.getVar('DL_DIR', True) filelist = [] for u in fetch.urls: ud = fetch.ud[u] - if isinstance(ud.method, local.Local): + if ud and isinstance(ud.method, local.Local): ud.setup_localpath(d) f = ud.localpath if f.startswith(dl_dir):
@@ -639,7 +642,7 @@ class FetchData(object): """ A class which represents the fetcher state for a given URI. """ - def __init__(self, url, d): + def __init__(self, url, d, localonly = False): # localpath is the location of a downloaded result. If not set, the file is local. self.donestamp = None self.localfile = ""
@@ -686,6 +689,9 @@ class FetchData(object): if not self.method: raise NoMethodError(url) + if localonly and not isinstance(self.method, local.Local): + raise NonLocalMethod() + if hasattr(self.method, "urldata_init"): self.method.urldata_init(self, d)
@@ -1009,7 +1015,10 @@ class FetchMethod(object): return "%s-%s" % (key, d.getVar("PN", True) or "") class Fetch(object): - def __init__(self, urls, d, cache = True): + def __init__(self, urls, d, cache = True, localonly = False): + if localonly and cache: + raise Exception("bb.fetch2.Fetch.__init__: cannot set cache and localonly at same time") + if len(urls) == 0: urls = d.getVar("SRC_URI", True).split() self.urls = urls
@@ -1022,7 +1031,12 @@ class Fetch(object): for url in urls: if url not in self.ud: - self.ud[url] = FetchData(url, d) + try: + self.ud[url] = FetchData(url, d, localonly) + except NonLocalMethod: + if localonly: + self.ud[url] = None + pass if cache: urldata_cache[fn] = self.ud
Signature · 1 line
1.7.9.5
PATCH 5/5 bitbake: include varflags in checksums
In reply to Paul EggletonAdd a dependency to each variable on a filtered list of its varflags. This is intended to catch things such as SRC_URI checksums, varflags controlling extra functionality from classes (e.g. the recently updated update-alternatives class in OE-Core), etc. and ensure their values influence the sstate checksums.
There is an exclusion list which needs to be set via bitbake.conf (BB_SIGNATURE_EXCLUDE_FLAGS), if this is not set then the functionality is disabled. The existing vardepsexclude mechanism can also be used to exclude undesired varflags, but they must be fully specified, e.g.:
do_patch[vardepsexclude] += "do_patch[someflag]"
Implements [YOCTO #2517].
--- bitbake/lib/bb/data.py | 20 +++++++++++++++++++- bitbake/lib/bb/siggen.py | 4 ++++ 2 files changed, 23 insertions(+), 1 deletion(-)
bitbake/lib/bb/data.py +19 -1
@@ -279,7 +279,12 @@ def build_dependencies(key, keys, shelldeps, vardepvals, d): deps = set() vardeps = d.getVarFlag(key, "vardeps", True) try: - value = d.getVar(key, False) + if key[-1] == ']': + vf = key[:-1].split('[') + value = d.getVarFlag(vf[0], vf[1], False) + else: + value = d.getVar(key, False) + if key in vardepvals: value = d.getVarFlag(key, "vardepvalue", True) elif d.getVarFlag(key, "func"):
@@ -301,6 +306,19 @@ def build_dependencies(key, keys, shelldeps, vardepvals, d): parser = d.expandWithRefs(value, key) deps |= parser.references deps = deps | (keys & parser.execs) + + # Add varflags, assuming an exclusion list is set + varflagsexcl = d.getVar('BB_SIGNATURE_EXCLUDE_FLAGS', True) + if varflagsexcl: + varfdeps = [] + varflags = d.getVarFlags(key) + if varflags: + for f in varflags: + if f not in varflagsexcl: + varfdeps.append('%s[%s]' % (key, f)) + if varfdeps: + deps |= set(varfdeps) + deps |= set((vardeps or "").split()) deps -= set((d.getVarFlag(key, "vardepsexclude", True) or "").split()) except:
bitbake/lib/bb/siggen.py +4 -0
@@ -108,6 +108,10 @@ class SignatureGeneratorBasic(SignatureGenerator): data = data + dep if dep in lookupcache: var = lookupcache[dep] + elif dep[-1] == ']': + vf = dep[:-1].split('[') + var = d.getVarFlag(vf[0], vf[1], False) + lookupcache[dep] = var else: var = d.getVar(dep, False) lookupcache[dep] = var
Signature · 1 line
1.7.9.5
Richard Purdie
AppliedQuoted 12 lines · Paul Eggleton
Some checksum improvements and a few error reporting fixes that I made in the process. Note that the varflags change won't do anything without the list of exclusions; an appropriate list has been set in a patch which will shortly be set to the OE-Core list.
The following changes (against Poky, but which apply cleanly with -p2 against BitBake master) are available in the git repository at:
git://git.yoctoproject.org/poky-contribpaule/bb-checksum-fixes-2
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=paule/bb-checksum-fixes-2
Quoted 7 lines
Paul Eggleton (5):
lib/bb/data_smart.py: don't report variable in ExpansionError if not
set
cooker: fix UnboundLocalError when exception occurs during parsing
cooker: report recipe being parsed when ExpansionError occurslib/bb/fetch2: ignore remote URIs when doing file checksums
bitbake: include varflags in checksums
Merged to master, thanks.
Richard