Skip to contentSkip to conversation

Make BBMASK support multiple regular expressions

PATCH In review Peter Kjellerstedt 2 participants 7 messages Jan 27, 2016

lore
Share & tools
lore
#0 lore raw

cover Make BBMASK support multiple regular expressions

[ This is a bitbake change, but since the documentation is all over
Poky I am cross-posting to the Poky list as well. ]

This series of patches changes BBMASK so that it can contain multiple, whitespace separated regular expressions. It should be fully backwards compatible with any existing values in BBMASK.

The first two patches are for bitbake, the third is for yocto-docs, the fourth is for OE-Core, and the fifth is for meta-yocto...

//Peter

The following changes since commit 3d2c0f5902cacf9d8544bf263b51ef0dd1a7218c:

cmake: update to 3.4.2 (2016-01-26 22:49:40 +0000)

are available in the git repository at:

git://git.yoctoproject.org/poky-contrib pkj/bbmask_multiple_re
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=pkj/bbmask_multiple_re

Peter Kjellerstedt (5):
cooker: Allow BBMASK to contain multiple regular expressions bitbake-user-manual-ref-variables: Update the help for BBMASK
ref-manual: Update the help for BBMASK documentation.conf: Update the help for BBMASK local.conf.sample.extended: Update the info about BBMASK

 .../bitbake-user-manual-ref-variables.xml          | 20 ++++++++---------
 bitbake/lib/bb/cooker.py                           | 17 ++++++++++++--
 documentation/ref-manual/ref-variables.xml         | 26 +++++++++-------------
 meta-yocto/conf/local.conf.sample.extended         |  4 ++--
 meta/conf/documentation.conf                       |  2 +-
 5 files changed, 37 insertions(+), 32 deletions(-)
Signature · 1 line
2.1.0
#1 lore raw

PATCH 1/5 cooker: Allow BBMASK to contain multiple regular expressions

Before, BBMASK was only permitted to contain one regular expression. This made it hard to add to the BBMASK in multiple places as one was supposed to separate the different regular expressions with a "|" rather than with whitespace as is customary in BitBake variables.

Now one can specify any number of regular expressions in BBMASK. This makes it possible to, e.g., mask out recipes in another layer from the layer.conf file.

This also properly ignores any regular expressions that do not compile (before an invalid regular expression would cause a ParseError in the first bbappend file found stating that it was not a BitBake file...)

---
 bitbake/lib/bb/cooker.py | 17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)
bitbake/lib/bb/cooker.py +15 -2
index 9c58d95..a02d143 100644
@@ -1759,11 +1759,24 @@ class CookerCollectFiles(object):
         bbmask = config.getVar('BBMASK', True)
 
         if bbmask:
+            # First validate the individual regular expressions and ignore any
+            # that do not compile
+            bbmasks = []
+            for mask in bbmask.split():
+                try:
+                    re.compile(mask)
+                    bbmasks.append(mask)
+                except sre_constants.error:
+                    collectlog.critical("BBMASK contains an invalid regular expression, ignoring: %s" % mask)
+
+            # Then validate the combined regular expressions. This should never
+            # fail, but better safe than sorry...
+            bbmask = "|".join(bbmasks)
             try:
                 bbmask_compiled = re.compile(bbmask)
             except sre_constants.error:
-                collectlog.critical("BBMASK is not a valid regular expression, ignoring.")
-                return list(newfiles), 0
+                collectlog.critical("BBMASK is not a valid regular expression, ignoring: %s" % bbmask)
+                bbmask = None
 
         bbfiles = []
         bbappend = []
Signature · 1 line
2.1.0
#2 lore raw

PATCH 2/5 bitbake-user-manual-ref-variables: Update the help for BBMASK

In reply to Peter Kjellerstedt

Update the help for BBMASK to reflect that it is now allowed to contain multiple regular expressions. Also changed the examples for BBMASK to be a bit more diverse.

---
 .../bitbake-user-manual-ref-variables.xml            | 20 +++++++++-----------
 1 file changed, 9 insertions(+), 11 deletions(-)
bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.xml +9 -11
index e5aeffc..ae7e4ce 100644
@@ -1142,14 +1142,14 @@
                     to "hide" these <filename>.bb</filename> and
                     <filename>.bbappend</filename> files.
                     BitBake ignores any recipe or recipe append files that
-                    match the expression.
+                    match any of the expressions.
                     It is as if BitBake does not see them at all.
                     Consequently, matching files are not parsed or otherwise
                     used by BitBake.</para>
                 <para>
-                    The value you provide is passed to Python's regular
+                    The values you provide are passed to Python's regular
                     expression compiler.
-                    The expression is compared against the full paths to
+                    The expressions are compared against the full paths to
                     the files.
                     For complete syntax information, see Python's
                     documentation at
@@ -1165,18 +1165,16 @@
      BBMASK = "meta-ti/recipes-misc/"
                     </literallayout>
                     If you want to mask out multiple directories or recipes,
-                    use the vertical bar to separate the regular expression
-                    fragments.
+                    you can specify multiple regular expression fragments.
                     This next example masks out multiple directories and
                     individual recipes:
                     <literallayout class='monospaced'>
-     BBMASK = "meta-ti/recipes-misc/|meta-ti/recipes-ti/packagegroup/"
-     BBMASK .= "|.*meta-oe/recipes-support/"
-     BBMASK .= "|.*openldap"
-     BBMASK .= "|.*opencv"
-     BBMASK .= "|.*lzma"
+     BBMASK += "/meta-ti/recipes-misc/ meta-ti/recipes-ti/packagegroup/"
+     BBMASK += "/meta-oe/recipes-support/"
+     BBMASK += "/meta-foo/.*/openldap"
+     BBMASK += "opencv.*\.bbappend"
+     BBMASK += "lzma"
                     </literallayout>
-                    Notice how the vertical bar is used to append the fragments.
                     <note>
                         When specifying a directory name, use the trailing
                         slash character to ensure you match just that directory
Signature · 1 line
2.1.0
#3 lore raw

PATCH 3/5 ref-manual: Update the help for BBMASK

In reply to Peter Kjellerstedt

Update the help for BBMASK to reflect that it is now allowed to contain multiple regular expressions. Also changed the examples for BBMASK to be a bit more diverse.

---
 documentation/ref-manual/ref-variables.xml | 26 ++++++++++----------------
 1 file changed, 10 insertions(+), 16 deletions(-)
documentation/ref-manual/ref-variables.xml +10 -16
index 673e431..49aea7a 100644
@@ -1200,17 +1200,13 @@
 
         <glossentry id='var-BBMASK'><glossterm>BBMASK</glossterm>
             <info>
-                BBMASK[doc] = "Prevents BitBake from processing specific recipes or recipe append files. Use the BBMASK variable from within conf/local.conf."
+                BBMASK[doc] = "Prevents BitBake from processing specific recipes or recipe append files."
             </info>
             <glossdef>
                 <para role="glossdeffirst">
 <!--                <para role="glossdeffirst"><imagedata fileref="figures/define-generic.png" /> -->
                     Prevents BitBake from processing recipes and recipe
                     append files.
-                    Use the <filename>BBMASK</filename> variable from within the
-                    <filename>conf/local.conf</filename> file found
-                    in the
-                    <ulink url='&YOCTO_DOCS_DEV_URL;#build-directory'>Build Directory</ulink>.
                 </para>
 
                 <para>
@@ -1218,14 +1214,14 @@
                     to "hide" these <filename>.bb</filename> and
                     <filename>.bbappend</filename> files.
                     BitBake ignores any recipe or recipe append files that
-                    match the expression.
+                    match any of the expressions.
                     It is as if BitBake does not see them at all.
                     Consequently, matching files are not parsed or otherwise
                     used by BitBake.</para>
                 <para>
-                    The value you provide is passed to Python's regular
+                    The values you provide are passed to Python's regular
                     expression compiler.
-                    The expression is compared against the full paths to
+                    The expressions are compared against the full paths to
                     the files.
                     For complete syntax information, see Python's
                     documentation at
@@ -1241,18 +1237,16 @@
      BBMASK = "meta-ti/recipes-misc/"
                     </literallayout>
                     If you want to mask out multiple directories or recipes,
-                    use the vertical bar to separate the regular expression
-                    fragments.
+                    you can specify multiple regular expression fragments.
                     This next example masks out multiple directories and
                     individual recipes:
                     <literallayout class='monospaced'>
-     BBMASK = "meta-ti/recipes-misc/|meta-ti/recipes-ti/packagegroup/"
-     BBMASK .= "|.*meta-oe/recipes-support/"
-     BBMASK .= "|.*openldap"
-     BBMASK .= "|.*opencv"
-     BBMASK .= "|.*lzma"
+     BBMASK += "/meta-ti/recipes-misc/ meta-ti/recipes-ti/packagegroup/"
+     BBMASK += "/meta-oe/recipes-support/"
+     BBMASK += "/meta-foo/.*/openldap"
+     BBMASK += "opencv.*\.bbappend"
+     BBMASK += "lzma"
                     </literallayout>
-                    Notice how the vertical bar is used to append the fragments.
                     <note>
                         When specifying a directory name, use the trailing
                         slash character to ensure you match just that directory
Signature · 1 line
2.1.0
#4 lore raw

Re: [poky] [PATCH 3/5] ref-manual: Update the help for BBMASK

This has been applied. see
http://www.yoctoproject.org/docs/2.1/ref-manual/ref-manual.html#var-BBMASK.

Thanks

Quoted 96 lines · Peter Kjellerstedt

Update the help for BBMASK to reflect that it is now allowed to contain multiple regular expressions. Also changed the examples for BBMASK to be a bit more diverse.

---
 documentation/ref-manual/ref-variables.xml | 26 ++++++++++----------------
 1 file changed, 10 insertions(+), 16 deletions(-)
documentation/ref-manual/ref-variables.xml +0 -1
index 673e431..49aea7a 100644
@@ -1200,17 +1200,13 @@
 
         <glossentry id='var-BBMASK'><glossterm>BBMASK</glossterm>
             <info>
-                BBMASK[doc] = "Prevents BitBake from processing specific

recipes or recipe append files. Use the BBMASK variable from within conf/local.conf."
+ BBMASK[doc] = "Prevents BitBake from processing specific
recipes or recipe append files."

             </info>
             <glossdef>
                 <para role="glossdeffirst">

<!-- <para role="glossdeffirst"><imagedata
fileref="figures/define-generic.png" /> -->
Prevents BitBake from processing recipes and recipe append files.
- Use the <filename>BBMASK</filename> variable from
within the

documentation/ref-manual/ref-variables.xml +0 -3
-                    <filename>conf/local.conf</filename> file found
-                    in the
-                    <ulink

url='&YOCTO_DOCS_DEV_URL;#build-directory'>Build Directory</ulink>.

                 </para>

                 <para>
documentation/ref-manual/ref-variables.xml +1 -1
@@ -1218,14 +1214,14 @@
                     to "hide" these <filename>.bb</filename> and
                     <filename>.bbappend</filename> files.
                     BitBake ignores any recipe or recipe append files that
-                    match the expression.
+                    match any of the expressions.
                     It is as if BitBake does not see them at all.
                     Consequently, matching files are not parsed or

otherwise
used by BitBake.</para>
<para>

documentation/ref-manual/ref-variables.xml +2 -3
-                    The value you provide is passed to Python's regular
+                    The values you provide are passed to Python's regular
                     expression compiler.
-                    The expression is compared against the full paths to
+                    The expressions are compared against the full paths to
                     the files.
                     For complete syntax information, see Python's
                     documentation at
@@ -1241,18 +1237,16 @@
      BBMASK = "meta-ti/recipes-misc/"
                     </literallayout>
                     If you want to mask out multiple directories or
 recipes,
-                    use the vertical bar to separate the regular

expression

documentation/ref-manual/ref-variables.xml +6 -7
-                    fragments.
+                    you can specify multiple regular expression fragments.
                     This next example masks out multiple directories and
                     individual recipes:
                     <literallayout class='monospaced'>
-     BBMASK = "meta-ti/recipes-misc/|meta-ti/recipes-ti/packagegroup/"
-     BBMASK .= "|.*meta-oe/recipes-support/"
-     BBMASK .= "|.*openldap"
-     BBMASK .= "|.*opencv"
-     BBMASK .= "|.*lzma"
+     BBMASK += "/meta-ti/recipes-misc/ meta-ti/recipes-ti/packagegroup/"
+     BBMASK += "/meta-oe/recipes-support/"
+     BBMASK += "/meta-foo/.*/openldap"
+     BBMASK += "opencv.*\.bbappend"
+     BBMASK += "lzma"
                     </literallayout>
-                    Notice how the vertical bar is used to append the

fragments.
<note>
When specifying a directory name, use the trailing slash character to ensure you match just that
directory
--
2.1.0

--
_______________________________________________
poky mailing list
[email protected]
https://lists.yoctoproject.org/listinfo/poky

#5 lore raw

PATCH 4/5 documentation.conf: Update the help for BBMASK

In reply to Peter Kjellerstedt

Since it is now possible to concatenate multiple regular expressions into BBMASK, there is no longer any real reason to limit it to be specified only in local.conf.

---
 meta/conf/documentation.conf | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/conf/documentation.conf +1 -1
index e09f7d8..a74da77 100644
@@ -91,7 +91,7 @@ BBINCLUDELOGS[doc] = "Variable that controls how BitBake displays logs on build
 BBINCLUDELOGS_LINES[doc] = "Amount of log lines printed on failure."
 BBLAYERS[doc] = "Lists the layers to enable during the build. This variable is defined in the bblayers.conf configuration file."
 BBLAYERS_NON_REMOVABLE[doc] = "Lists core layers that cannot be removed from the bblayers.conf file."
-BBMASK[doc] = "Prevents BitBake from processing specific recipes or recipe append files. Use the BBMASK variable from within conf/local.conf."
+BBMASK[doc] = "Prevents BitBake from processing specific recipes or recipe append files."
 BBPATH[doc] = "Used by BitBake to locate .bbclass and configuration files. This variable is analogous to the PATH variable."
 BBSERVER[doc] = "Points to the server that runs memory-resident BitBake."
 BINCONFIG_GLOB[doc] = "When inheriting binconfig.bbclass from a recipe, this variable specifies a wildcard for configuration scripts that need editing."
Signature · 1 line
2.1.0
#6 lore raw

PATCH 5/5 local.conf.sample.extended: Update the info about BBMASK

In reply to Peter Kjellerstedt

Update the info about BBMASK to reflect that it can now contain multiple regular expressions.

Also correct a misspelled PREFERRED_VERSION in a comment.

---
 meta-yocto/conf/local.conf.sample.extended | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
meta-yocto/conf/local.conf.sample.extended +2 -2
index 925270b..521cdcb 100644
@@ -1,4 +1,4 @@
-# BBMASK is a regular expression that can be used to tell BitBake to ignore
+# BBMASK contains regular expressions that can be used to tell BitBake to ignore
 # certain recipes.
 #BBMASK = ""
 
@@ -375,7 +375,7 @@
 #
 # Currently the rootfs_rpm code has a hard depends on rpmresolve:do_populate_sysroot,
 # when using rpm4 the rpmresolve code will not compile due to a missing header file.
-# That dependency needs to be removed when using RPM4, also the PREFRRED_VERSION needs
+# That dependency needs to be removed when using RPM4, also the PREFERRED_VERSION needs
 # to be set. This example shows how to enable rpm4
 # PREFERRED_VERSION_rpm = "4.11.2"
 # PREFERRED_VERSION_rpm-native = "4.11.2"
Signature · 1 line
2.1.0