shell parser: deep nesting

Hello!

I ran into a bug
(https://bugzilla.yoctoproject.org/show_bug.cgi?id=10668) where parsing a shell fragment like "echo hello & wait $!" triggers this error:

Traceback (most recent call last):
File "/work/poky/bitbake/lib/bb/codeparser.py", line 400, in ShellParser.process_tokens(tokens=[('if_clause', <bb.pysh.pyshyacc.IfCond object at 0x7fb480963390>)]):
if more_tokens:
> self.process_tokens(more_tokens)

File "/work/poky/bitbake/lib/bb/codeparser.py", line 400, in ShellParser.process_tokens(tokens=<itertools.chain object at 0x7fb480949278>):
if more_tokens:
> self.process_tokens(more_tokens)

File "/work/poky/bitbake/lib/bb/codeparser.py", line 400, in ShellParser.process_tokens(tokens=[('while_clause', <bb.pysh.pyshyacc.WhileLoop object at 0x7fb480949518>)]):
if more_tokens:
> self.process_tokens(more_tokens)

File "/work/poky/bitbake/lib/bb/codeparser.py", line 400, in ShellParser.process_tokens(tokens=<itertools.chain object at 0x7fb480446668>):
if more_tokens:
> self.process_tokens(more_tokens)

File "/work/poky/bitbake/lib/bb/codeparser.py", line 393, in ShellParser.process_tokens(tokens=[[('pipeline', <bb.pysh.pyshyacc.Pipeline object at 0x7fb480949c88>)]]):
for token in tokens:
> name, value = token
try:
ValueError: need more than 1 value to unpack

After thinking about this a bit more I *think* I understand the problem (expecting a certain level of nesting while the parser may also hand out tokens with deeper nesting in the pipeline case) and I'll send a patch fixing the issue for me.

But I'm not entirely sure whether that's really addressing the root cause of the problem - perhaps such deep nesting shouldn't occur?

So I hope that someone familiar with this code can have a look at the problem and the proposed solution.

Also, does it have performance implications that ShellParser.process_tokens() defines the same helper functions repeatedly? They don't depend on the current invocation context and thus could also be defined as private helper functions in ShellParser.

Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.

For shell constructs like
echo hello & wait $!
the process_tokens() method ended up with a situation where "token" in the "name, value = token" assignment was a list of tuples and not the expected tuple, causing the assignment to fail.

There were already two for loops (one in _parse_shell(), one in process_tokens()) which iterated over token lists. Apparently the actual nesting can also be deeper.

Now there is just one such loop in process_token_list() which calls itself recursively when it detects that a list entry is another list.

As a side effect (improvement?!) of the loop removal in _parse_shell(), the local function definitions in process_tokens() get executed less often.

---
 lib/bb/codeparser.py | 29 +++++++++++++++++------------
 1 file changed, 17 insertions(+), 12 deletions(-)

index 25938d6..5d2d440 100644
@@ -342,8 +342,7 @@ class ShellParser():
         except pyshlex.NeedMore:
             raise sherrors.ShellSyntaxError("Unexpected EOF")
 
-        for token in tokens:
-            self.process_tokens(token)
+        self.process_tokens(tokens)
 
     def process_tokens(self, tokens):
         """Process a supplied portion of the syntax tree as returned by

@@ -389,18 +388,24 @@ class ShellParser():
             "case_clause": case_clause,
         }
 
-        for token in tokens:
-            name, value = token
-            try:
-                more_tokens, words = token_handlers[name](value)
-            except KeyError:
-                raise NotImplementedError("Unsupported token type " + name)
+        def process_token_list(tokens):
+            for token in tokens:
+                if isinstance(token, list):
+                    process_token_list(token)
+                    continue
+                name, value = token
+                try:
+                    more_tokens, words = token_handlers[name](value)
+                except KeyError:
+                    raise NotImplementedError("Unsupported token type " + name)
+
+                if more_tokens:
+                    self.process_tokens(more_tokens)
 
-            if more_tokens:
-                self.process_tokens(more_tokens)
+                if words:
+                    self.process_words(words)
 
-            if words:
-                self.process_words(words)
+        process_token_list(tokens)
 
     def process_words(self, words):
         """Process a set of 'words' in pyshyacc parlance, which includes

2.1.4