Skip to contentSkip to conversation

Only try to expand refs to valid variable names

PATCH In review Christopher Larson 2 participants 4 messages Jan 18, 2019

lore
Share & tools
lore
#0 lore raw

cover Only try to expand refs to valid variable names

From: Christopher Larson <[email protected]>

This aligns the behavior of expansion with the recipe parser, only attempting to expand references to valid variable names. This avoids adding references for things like ${foo#${TOPDIR}} to our vardeps without imposing much additional processing overhead beyond the change to the expansion regexp.

YOCTO #12987

The following changes since commit 610dbee5634677f5055e2b36a3043cd197fb8c51:

gitsm.py: Refactor the functions and simplify the class (2019-01-15 22:22:29 +0000)

are available in the Git repository at:

[email protected]:kergoth/bitbake yocto-12987-new

for you to fetch changes up to 9cb64e9781c98f1aaa42571af54655eaf6816583:

bb.tests.codeparser: add parameter expansion modifiers test (2019-01-16 19:46:10 +0000)

----------------------------------------------------------------
Christopher Larson (2):
bb.data_smart: only try to expand refs to valid variable names bb.tests.codeparser: add parameter expansion modifiers test

 lib/bb/data_smart.py       | 2 +-
 lib/bb/tests/codeparser.py | 7 +++++++
 2 files changed, 8 insertions(+), 1 deletion(-)
Signature · 1 line
2.17.1
#1 lore raw

PATCH 1/2 bb.data_smart: only try to expand refs to valid variable names

From: Christopher Larson <[email protected]>

This aligns the behavior of expansion with the recipe parser, only attempting to expand references to valid variable names. This avoids adding references for things like ${foo#${TOPDIR}} to our vardeps without imposing much additional processing overhead beyond the change to the expansion regexp.

YOCTO #12987

---
 lib/bb/data_smart.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
lib/bb/data_smart.py +1 -1
index 1f45cd97..07db7be9 100644
@@ -40,7 +40,7 @@ logger = logging.getLogger("BitBake.Data")
 
 __setvar_keyword__ = ["_append", "_prepend", "_remove"]
 __setvar_regexp__ = re.compile(r'(?P<base>.*?)(?P<keyword>_append|_prepend|_remove)(_(?P<add>[^A-Z]*))?$')
-__expand_var_regexp__ = re.compile(r"\${[^{}@\n\t :]+}")
+__expand_var_regexp__ = re.compile(r"\${[a-zA-Z0-9\-_+./~]+?}")
 __expand_python_regexp__ = re.compile(r"\${@.+?}")
 __whitespace_split__ = re.compile(r'(\s)')
 __override_regexp__ = re.compile(r'[a-z0-9]+')
Signature · 1 line
2.17.1
#2 lore raw

PATCH 2/2 bb.tests.codeparser: add parameter expansion modifiers test

In reply to Christopher Larson

From: Christopher Larson <[email protected]>

We don't want references including shell parameter expansion modifiers (i.e. :-, #, %%, etc) to be added to our vardeps, so add a test to ensure this.

YOCTO #12987

---
 lib/bb/tests/codeparser.py | 7 +++++++
 1 file changed, 7 insertions(+)
lib/bb/tests/codeparser.py +7 -0
index e30e78c1..3fd76a8f 100644
@@ -123,6 +123,13 @@ ${D}${libdir}/pkgconfig/*.pc
         self.parseExpression("sed -i -e 's:IP{:I${:g' $pc")
         self.assertExecs(set(["sed"]))
 
+    def test_parameter_expansion_modifiers(self):
+        # - and + are also valid modifiers for parameter expansion, but are
+        # valid characters in bitbake variable names, so are not included here
+        for i in ('=', ':-', ':=', '?', ':?', ':+', '#', '%', '##', '%%'):
+            name = "foo%sbar" % i
+            self.parseExpression("${%s}" % name)
+            self.assertNotIn(name, self.references)
 
     def test_until(self):
         self.parseExpression("until false; do echo true; done")
Signature · 1 line
2.17.1
#3 lore raw
In reply to Christopher Larson
Quoted 10 lines · Christopher Larson

From: Christopher Larson <[email protected]>

This aligns the behavior of expansion with the recipe parser, only attempting to expand references to valid variable names. This avoids adding references for things like ${foo#${TOPDIR}} to our vardeps without imposing much additional processing overhead beyond the change to the expansion regexp.

YOCTO #12987

There’s no statistically significant performance impact to bitbake -p, bitbake-selftest, or my test image builds, only typical run-to-run variation.

Signature · 4 lines
Christopher Larson
kergoth at gmail dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Senior Software Engineer, Mentor Graphics