codeparser: don't interact with the cache for subshells
PATCH 1/1 codeparser: don't interact with the cache for subshells
Doing so was causing leakage between the execs of the main value and that of the subshell value, and was causing the cached subshell value to be used for the overall variable. At the least this could cause execs contamination between two variables that while differing, run the same subshell. Beyond that, it's possible we could have been using an incomplete cached value of a subshell for that of the main value.
Before this, bb_codeparser.dat would change between parses with differing bbfile parse order. After, it does not change.
The codeparser cache version is bumped, to ensure we don't use potentially incorrect cached values from previous runs.
This should hopefully resolve the difficult-to-reproduce issues we've seen at Mentor Graphics where bitbake emits a script to run a task and misses dependent functions, resulting in 'command not found' failures. This issue has also been mentioned on the oe devel list, where someone hit a case where oe_runmake was missing from a do_install task (IIRC). Adding debug information showed that bitbake's information about the variable dependencies for this task is inaccurate in the failure cases.
--- lib/bb/codeparser.py | 21 ++++++++++++--------- 1 file changed, 12 insertions(+), 9 deletions(-)
lib/bb/codeparser.py +12 -9
@@ -35,7 +35,7 @@ def check_indent(codestr): class CodeParserCache(MultiProcessCache): cache_file_name = "bb_codeparser.dat" - CACHE_VERSION = 4 + CACHE_VERSION = 5 def __init__(self): MultiProcessCache.__init__(self)
@@ -217,6 +217,15 @@ class ShellParser(): self.execs = codeparsercache.shellcacheextras[h]["execs"] return self.execs + self._parse_shell(value) + self.execs = set(cmd for cmd in self.allexecs if cmd not in self.funcdefs) + + codeparsercache.shellcacheextras[h] = {} + codeparsercache.shellcacheextras[h]["execs"] = self.execs + + return self.execs + + def _parse_shell(self, value): try: tokens, _ = pyshyacc.parse(value, eof=True, debug=False) except pyshlex.NeedMore:
@@ -224,12 +233,6 @@ class ShellParser(): for token in tokens: self.process_tokens(token) - self.execs = set(cmd for cmd in self.allexecs if cmd not in self.funcdefs) - - codeparsercache.shellcacheextras[h] = {} - codeparsercache.shellcacheextras[h]["execs"] = self.execs - - return self.execs def process_tokens(self, tokens): """Process a supplied portion of the syntax tree as returned by
@@ -303,7 +306,7 @@ class ShellParser(): if part[0] in ('`', '$('): command = pyshlex.wordtree_as_string(part[1:-1]) - self.parse_shell(command) + self._parse_shell(command) if word[0] in ("cmd_name", "cmd_word"): if word in words:
@@ -322,7 +325,7 @@ class ShellParser(): self.log.debug(1, self.unhandled_template % cmd) elif cmd == "eval": command = " ".join(word for _, word in words[1:]) - self.parse_shell(command) + self._parse_shell(command) else: self.allexecs.add(cmd) break
Signature · 1 line
1.8.3.4
Quoted 15 lines · Christopher Larson
Doing so was causing leakage between the execs of the main value and that of
the subshell value, and was causing the cached subshell value to be used for
the overall variable. At the least this could cause execs contamination between two variables that while differing, run the same subshell. Beyond that, it's possible we could have been using an incomplete cached value of a subshell for that of the main value.Before this, bb_codeparser.dat would change between parses with differing bbfile parse order. After, it does not change.
The codeparser cache version is bumped, to ensure we don't use potentially incorrect cached values from previous runs.
Quoted 11 lines
This should hopefully resolve the difficult-to-reproduce issues we've seen at
Mentor Graphics where bitbake emits a script to run a task and misses dependent functions, resulting in 'command not found' failures. This issue has
also been mentioned on the oe devel list, where someone hit a case whereoe_runmakewas missing from ado_installtask (IIRC). Adding debug information
showed that bitbake's information about the variable dependencies for this task is inaccurate in the failure cases.
We're trying to confirm the fix, but given it's not really reproducible, and only seemed
to be hit during automated builds, it'll be difficult to say with certainty that the problem
is resolved. That said, I think it's clear from reading the code that this is the correct
thing, and the previous beahvior was just an unintentional oversight when the cache
usage was added.
Thanks!
Signature · 5 lines
Christopher Larson clarson at kergoth dot com Founder - BitBake, OpenEmbedded, OpenZaurus Maintainer - Tslib Senior Software Engineer, Mentor Graphics
Quoted 29 lines · Chris Larson
Quoted 15 lines · Christopher Larson
Doing so was causing leakage between the execs of the main value and that of
the subshell value, and was causing the cached subshell value to be used for
the overall variable. At the least this could cause execs contamination between two variables that while differing, run the same subshell. Beyond that, it's possible we could have been using an incomplete cached value of a subshell for that of the main value.Before this, bb_codeparser.dat would change between parses with differing bbfile parse order. After, it does not change.
The codeparser cache version is bumped, to ensure we don't use potentially incorrect cached values from previous runs.
Quoted 11 lines
This should hopefully resolve the difficult-to-reproduce issues we've seen at
Mentor Graphics where bitbake emits a script to run a task and misses dependent functions, resulting in 'command not found' failures. This issue has
also been mentioned on the oe devel list, where someone hit a case whereoe_runmakewas missing from ado_installtask (IIRC). Adding debug information
showed that bitbake's information about the variable dependencies for this task is inaccurate in the failure cases.
Quoted 10 lines
We're trying to confirm the fix, but given it's not really reproducible, and only seemed
to be hit during automated builds, it'll be difficult to say with certainty that the problem
is resolved. That said, I think it's clear from reading the code that this is the correct
thing, and the previous beahvior was just an unintentional oversight when the cache
usage was added.
For reference, https://gist.github.com/kergoth/11301539 was used to track changes to
bb_codeparser.dat to investigate this.
Signature · 5 lines
Christopher Larson clarson at kergoth dot com Founder - BitBake, OpenEmbedded, OpenZaurus Maintainer - Tslib Senior Software Engineer, Mentor Graphics
Quoted 45 lines · Christopher Larson
Quoted 32 lines · Chris Larson
Quoted 17 lines · Christopher Larson
Doing so was causing leakage between the execs of the main value and that of
the subshell value, and was causing the cached subshell value to be used for
the overall variable. At the least this could cause execs contamination between two variables that while differing, run the same subshell. Beyond that, it's possible we could have been using an incomplete cached value of
a subshell for that of the main value.Before this, bb_codeparser.dat would change between parses with differing bbfile parse order. After, it does not change.
The codeparser cache version is bumped, to ensure we don't use potentially
incorrect cached values from previous runs.Quoted 12 lines
This should hopefully resolve the difficult-to-reproduce issues we've seen at
Mentor Graphics where bitbake emits a script to run a task and misses dependent functions, resulting in 'command not found' failures. This issue has
also been mentioned on the oe devel list, where someone hit a case whereoe_runmakewas missing from ado_installtask (IIRC). Adding debug information
showed that bitbake's information about the variable dependencies for this
task is inaccurate in the failure cases.Quoted 10 lines
We're trying to confirm the fix, but given it's not really reproducible, and only seemed
to be hit during automated builds, it'll be difficult to say with certainty that the problem
is resolved. That said, I think it's clear from reading the code that this is the correct
thing, and the previous beahvior was just an unintentional oversight when the cache
usage was added.
Quoted 4 lines
For reference, https://gist.github.com/kergoth/11301539 was used to track changes to
bb_codeparser.dat to investigate this.
Sorry for yet another email on this thread, but I wanted to thank Richard for his guidance
on this. His suggestion to look at the caching, specifically with an eye toward file parse
order, proved to be dead on, and valuable.
Signature · 5 lines
Christopher Larson clarson at kergoth dot com Founder - BitBake, OpenEmbedded, OpenZaurus Maintainer - Tslib Senior Software Engineer, Mentor Graphics
Quoted 21 lines · Christopher Larson
Doing so was causing leakage between the execs of the main value and that of the subshell value, and was causing the cached subshell value to be used for the overall variable. At the least this could cause execs contamination between two variables that while differing, run the same subshell. Beyond that, it's possible we could have been using an incomplete cached value of a subshell for that of the main value.
Before this, bb_codeparser.dat would change between parses with differing bbfile parse order. After, it does not change.
The codeparser cache version is bumped, to ensure we don't use potentially incorrect cached values from previous runs.
This should hopefully resolve the difficult-to-reproduce issues we've seen at Mentor Graphics where bitbake emits a script to run a task and misses dependent functions, resulting in 'command not found' failures. This issue has also been mentioned on the oe devel list, where someone hit a case where
oe_runmakewas missing from ado_installtask (IIRC). Adding debug information showed that bitbake's information about the variable dependencies for this task is inaccurate in the failure cases.
Quoted 4 lines
Signed-off-by Christopher Larson <[email protected]>--- lib/bb/codeparser.py | 21 ++++++++++++--------- 1 file changed, 12 insertions(+), 9 deletions(-)
That is nasty bug, nicely found!
Cheers,
Richard