Skip to contentSkip to conversation

sanity.bbclass: check for validity of TMPDIR

PATCH Superseded [email protected] 6 participants 13 messages Nov 14, 2013 – Nov 15, 2013

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

[email protected]

author
#0 lore raw

cover sanity.bbclass: check for validity of TMPDIR

From: Chen Qi <[email protected]>

This patch checks the validity of TMPDIR.

The TMPDIR must match the following regexp.
^/[a-zA-Z0-9\-_/.~]+$

Detailed tests about why I came up with this regexp are displayed at the end of this cover letter.

A major reason for why TMPDIR cannot support many special chars such as @ and # is that these special chars are used by the sed command as its field separator in our recipes.

And here are things I want to make sure of.
1. For now, TMPDIR must be an absolute path. So do we need to make it support relative path?
2. I included the tilde (~) into the valid char set because `bitbake world' succeeded. But it's still possible the the tilde (~) will serve as the field separator in sed. So is it OK that I include it into the valid char set?

********************** TESTING RECORDS **************************************** Test Case 1: use `bitbake minimal' on paths containing special chars
Here's the output.
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/`_contained"
RESULT: FAIL
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/^_contained"
RESULT: SUCCESS
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/~_contained"
RESULT: SUCCESS
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/<_contained"
RESULT: FAIL
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/=_contained"
RESULT: FAIL
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/>_contained"
RESULT: FAIL
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/|_contained"
RESULT: FAIL
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/ _contained"
RESULT: FAIL
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/,_contained"
RESULT: FAIL
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/;_contained"
RESULT: FAIL
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/:_contained"
RESULT: FAIL
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/!_contained"
RESULT: SUCCESS
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/?_contained"
RESULT: FAIL
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/'_contained"
RESULT: FAIL
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/\"_contained"
RESULT: FAIL
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/(_contained"
RESULT: FAIL
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/)_contained"
RESULT: FAIL
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/[_contained"
RESULT: FAIL
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/]_contained"
RESULT: SUCCESS
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/{_contained"
RESULT: SUCCESS
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/}_contained"
RESULT: FAIL
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/@_contained"
RESULT: SUCCESS
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/$_contained"
RESULT: FAIL
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/*_contained"
RESULT: FAIL
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/\_contained"
RESULT: FAIL
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/&_contained"
RESULT: FAIL
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/#_contained"
RESULT: FAIL
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/%_contained"
RESULT: FAIL
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/+_contained"
RESULT: SUCCESS

Test Case 2: TMPDIR containing ^
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/^_contained"
Command:
bitbake core-image-minimal -cpopulate_sdk
Result:
Fail
Formatted Error Message:
ERROR: Task 5 (virtual:nativesdk:/home/chenqi/poky/meta/recipes-core/eglibc/eglibc_2.18.bb, do_populate_sysroot) failed with exit code '1'
| checking whether the C compiler works... no
| configure: error: in `/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/^_contained/work/i686-nativesdk-pokysdk-linux/nativesdk-eglibc/2.18-r0/site_config_qemux86':
| configure: error: C compiler cannot create executables
| See `config.log' for more details

Test Case 3: TMPDIR containing +
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/+_contained"
Command:
bitbake world
Result:
Fail
Formatted Error Message:
ERROR: Task 9084 (virtual:native:/home/chenqi/poky/meta/recipes-devtools/tcltk/tcl_8.6.1.bb, do_install) failed with exit code '1' | oe_libinstall: cd /home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/+_contained/work/i686-linux/tcl-native/8.6.1-r0/build | sed: -e expression #1, char 117: unknown option to `s'
Reason:
'+' is used by the sed command as a separator in tcl recipe.
sed -i "s+${WORKDIR}+${STAGING_INCDIR}+g" tclConfig.sh

Test Case 4: TMPDIR containing ]
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/]_contained"
Command:
bitbake world
Result:
Fail
Formatted Error Message:
ERROR: Task 5 (virtual:native:/home/chenqi/poky/meta/recipes-graphics/xcb/xcb-proto_1.8.bb, do_configure) failed with exit code '1' | configure.ac:9: error: possibly undefined macro: AM_INIT_AUTOMAKE
| If this token and others are legitimate, please use m4_pattern_allow.
| See the Autoconf documentation.
| configure.ac:12: error: possibly undefined macro: AM_CONDITIONAL
| configure.ac:17: error: possibly undefined macro: AM_PATH_PYTHON
| autoreconf: /home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/]_contained/sysroots/i686-linux/usr/bin/autoconf failed with exit status: 1 | ERROR: autoreconf execution failed.
| WARNING: exit code 1 from a shell command.

Test Case 5: TMPDIR containing {
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/[_contained" Command:
bitbake world
Result:
Fail
Formatted Error Message:

    ERROR: Task 9237 (virtual:native:/home/chenqi/poky/meta/recipes-gnome/gnome/gnome-doc-utils_0.20.10.bb, do_compile) failed with exit code '1'
    | xsltproc -nonet -o "C/db-label.xml" \
    |      --stringparam basename "db-label" \
    |      --stringparam xsl_file "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/{_contained/work/i686-linux/gnome-doc-utils-native/0.20.10-r0/gnome-doc-utils-0.20.10/doc/xslt/../../xslt/\
docbook/common/db-label.xsl" \
    |      "./xsldoc-docbook.xsl" "C/db-label.xsldoc"
    | db-chunk: No stylesheet param for documented db.chunk.chunks
    | db-common: No named template for documented db.copyright
    | make[2]: *** [C/db-chunk.xml] Error 10

Test Case 6: TMPDIR containing @
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/@_contained"
Command:
bitbake qt-mobility-x11
Result:
Fail
Formatted Error Message:
ERROR: Task 2 (/home/chenqi/poky/meta/recipes-qt/qt4/qt-mobility-x11_1.2.0.bb, do_install) failed with exit code '1' | DEBUG: Executing shell function do_install
| sed: -e expression #1, char 75: unknown option to `s'

Test Case 7: TMPDIR containing ~
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/[_contained" Command:
bitbake world
Result:
Success

****************************************************************************************

The following changes since commit ea92671d9823e3667d6ced7ac2af20f991da404d:

bitbake: cooker: replace "w" file opening mode with "a" mode (2013-11-12 17:01:37 +0000)

are available in the git repository at:

git://git.pokylinux.org/poky-contrib ChenQi/TMPDIR
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=ChenQi/TMPDIR

Chen Qi (1):
sanity.bbclass: check for validity of TMPDIR

 meta/classes/sanity.bbclass |    7 +++++++
 1 file changed, 7 insertions(+)
Signature · 1 line
1.7.9.5
author
#1 lore raw

PATCH 1/1 sanity.bbclass: check for validity of TMPDIR

From: Chen Qi <[email protected]>

TMPDIR must be an absolute path, otherwise, the build will fail. Special characters in TMPDIR will also cause build failures.

This patch enables checking for the validity of TMPDIR.

Signed-off-by Chen Qi <[email protected]>
---
 meta/classes/sanity.bbclass |    7 +++++++
 1 file changed, 7 insertions(+)
meta/classes/sanity.bbclass +7 -0
index 83378b0..e45906e 100644
@@ -672,6 +672,13 @@ def check_sanity_everybuild(status, d):
         with open(checkfile, "w") as f:
             f.write(tmpdir)
 
+    # Check if TMPDIR contains invalid characters, and check if it is an absolute path
+    valid_tmpdir_regexp = "^/[a-zA-Z0-9\-_/.~]+$"
+    import re
+    valid_pattern = re.compile(valid_tmpdir_regexp)
+    if not valid_pattern.match(tmpdir):
+        status.addresult("Error, you have special characters in TMPDIR directory path or your TMPDIR is not an absolute path. The TMPDIR should match the this regexp: ^/[a-zA-Z0-9\-_/.~]$")
+
 def check_sanity(sanity_data):
     import subprocess
Signature · 1 line
1.7.9.5
#2 lore raw
Quoted 17 lines · On Thu, 2013-11-14 at 21:07 +0800, [email protected] wrote:

From: Chen Qi <[email protected]>

TMPDIR must be an absolute path, otherwise, the build will fail. Special characters in TMPDIR will also cause build failures.

This patch enables checking for the validity of TMPDIR.

Signed-off-by Chen Qi <[email protected]>
---
 meta/classes/sanity.bbclass |    7 +++++++
 1 file changed, 7 insertions(+)
meta/classes/sanity.bbclass +0 -0
index 83378b0..e45906e 100644

meta/classes/sanity.bbclass def check_sanity_everybuild(status, d): · line 681

Quoted 10 lines
meta/classes/sanity.bbclass +7 -0
@@ -672,6 +672,13 @@ def check_sanity_everybuild(status, d):
         with open(checkfile, "w") as f:
             f.write(tmpdir)
 
+    # Check if TMPDIR contains invalid characters, and check if it is an absolute path
+    valid_tmpdir_regexp = "^/[a-zA-Z0-9\-_/.~]+$"
+    import re
+    valid_pattern = re.compile(valid_tmpdir_regexp)
+    if not valid_pattern.match(tmpdir):
+        status.addresult("Error, you have special characters in TMPDIR directory path or your TMPDIR is not an absolute path. The TMPDIR should match the this regexp: ^/[a-zA-Z0-9\-_/.~]$")
+

I'm ok with the check itself however this does not need to be run every build. It needs to run when tmpdir changes only.

Cheers,

Richard

#4 lore raw
In reply to [email protected]
Quoted 19 lines · On 2013-11-14 06:07, [email protected] wrote:

From: Chen Qi <[email protected]>

TMPDIR must be an absolute path, otherwise, the build will fail. Special characters in TMPDIR will also cause build failures.

This patch enables checking for the validity of TMPDIR.

Signed-off-by Chen Qi <[email protected]>
---
  meta/classes/sanity.bbclass |    7 +++++++
  1 file changed, 7 insertions(+)
meta/classes/sanity.bbclass +0 -0
index 83378b0..e45906e 100644
@@ -672,6 +672,13 @@ def check_sanity_everybuild(status, d):
          with open(checkfile, "w") as f:
              f.write(tmpdir)

meta/classes/sanity.bbclass def check_sanity_everybuild(status, d): · line 683

Quoted 10 lines
meta/classes/sanity.bbclass +7 -0
 
+    # Check if TMPDIR contains invalid characters, and check if it is an absolute path
+    valid_tmpdir_regexp = "^/[a-zA-Z0-9\-_/.~]+$"
+    import re
+    valid_pattern = re.compile(valid_tmpdir_regexp)
+    if not valid_pattern.match(tmpdir):
+        status.addresult("Error, you have special characters in TMPDIR directory path or your TMPDIR is not an absolute path. The TMPDIR should match the this regexp: ^/[a-zA-Z0-9\-_/.~]$")
+
  def check_sanity(sanity_data):
      import subprocess

Check the wording of the warning "The TMPDIR should match the this regexp" should probably be "The TMPDIR should match this regexp" or maybe use a simpler version like "The TMPDIR name should must be an absolute path and contain only letters, digits and the characters -_"

Also, is "-" actually valid? I seem to recall having problems when my build tree had the hyphen ("-") in the path.

Signature · 4 lines
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------
#5 lore raw
Quoted 2 lines · Gary Thomas

Also, is "-" actually valid? I seem to recall having problems when my build tree had the hyphen ("-") in the path.

Paths with "-" in certainly work for me. If there are any recipes which break in that situation then we should just fix them.

OE already has quite a range of baroque restrictions on what sort of TMPDIR you are allowed to use (no nfs, no symlinks in the path, no spaces in the name) and every new prohibition represents a loss in usability. This patch as proposed already forbids a whole range of characters, including things like "+", and I think that disallowing "-" as well would be a step too far.

p.

#6 lore raw
Quoted 7 lines · Phil Blundell
Quoted 2 lines · Gary Thomas

Also, is "-" actually valid? I seem to recall having problems when my build tree had the hyphen ("-") in the path.

Paths with "-" in certainly work for me. If there are any recipes which break in that situation then we should just fix them.

Quoted 6 lines

OE already has quite a range of baroque restrictions on what sort of TMPDIR you are allowed to use (no nfs, no symlinks in the path, no spaces in the name) and every new prohibition represents a loss in usability. This patch as proposed already forbids a whole range of characters, including things like "+", and I think that disallowing "-" as well would be a step too far.

Fair enough. I don't recall exactly what problems I had (in the dark past), but I've just avoided trees with "-" in them. I'll give it another look if/when an opportunity comes by.

Signature · 4 lines
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------
#7 lore raw
In reply to Phil Blundell
Quoted 6 lines · Phil Blundell
Quoted 2 lines · Gary Thomas

Also, is "-" actually valid? I seem to recall having problems when my build tree had the hyphen ("-") in the path.

Paths with "-" in certainly work for me. If there are any recipes which break in that situation then we should just fix them.

We found an issue with paths that -start- with a '-', there are apparently many places where paths are passed into various shell, and the initial '-' can be read as an argument identifier.

Quoted 6 lines

OE already has quite a range of baroque restrictions on what sort of TMPDIR you are allowed to use (no nfs, no symlinks in the path, no spaces in the name) and every new prohibition represents a loss in usability. This patch as proposed already forbids a whole range of characters, including things like "+", and I think that disallowing "-" as well would be a step too far.

I don't believe Qi Chen sent it to the list, but we built path names with all of the special characters and tried to run builds. The specific list is based on the results of those tests. The other items are already broken, and we're trying to be explicit with this for end users. (Note, it's really the TMPDIR that matters.. the 'build' directory is a lot more flexible.)

--Mark

Quoted 8 lines

p.

_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

#8 lore raw
Quoted 11 lines · Mark Hatle
Quoted 6 lines · Phil Blundell
Quoted 2 lines · Gary Thomas

Also, is "-" actually valid? I seem to recall having problems when my build tree had the hyphen ("-") in the path.

Paths with "-" in certainly work for me. If there are any recipes which break in that situation then we should just fix them.

We found an issue with paths that -start- with a '-', there are apparently many places where paths are passed into various shell, and the initial '-' can be read as an argument identifier.

Right, I can imagine that this would break fairly badly. But if we're going to require TMPDIR to be an absolute path (which seems perfectly reasonable) then by definition it can't start with - so this will presumably become a non-issue.

Quoted 5 lines

I don't believe Qi Chen sent it to the list, but we built path names with all of the special characters and tried to run builds. The specific list is based on the results of those tests. The other items are already broken, and we're trying to be explicit with this for end users. (Note, it's really the TMPDIR that matters.. the 'build' directory is a lot more flexible.)

Yeah, I saw the results of those tests. My slight reservation with that methodology is that, if any single recipe fails with a given character in TMPDIR, no matter how obscure that recipe might be, it seems that this is enough to make that character verboten. For example, the rationale for excluding "+" seems to be that it breaks tcl; if this is the only recipe that fails then it seems like it might be worth at least trying to fix tcl rather than excluding "+" from TMPDIRs for ever more.

But, clearly, there are some characters in the list that are "just silly" to have in pathnames. I don't think anybody would be surprised to learn that "*" and "$" cause problems and blacklisting those ones does seem quite reasonable.

p.

#10 lore raw
In reply to Mark Hatle
Quoted 6 lines · Mark Hatle
Quoted 5 lines · Phil Blundell
Quoted 3 lines · Gary Thomas

Also, is "-" actually valid? I seem to recall having problems when my build
tree had the hyphen ("-") in the path.

Quoted 7 lines
Quoted 2 lines

Paths with "-" in certainly work for me. If there are any recipes which break in that situation then we should just fix them.

We found an issue with paths that -start- with a '-', there are apparently many places where paths are passed into various shell, and the initial '-' can be read as an argument identifier.

I think that I saw an email talking about "-D" in the TMPDIR caused some some recipes to pick that up as a CFLAG define and cause problems

Sau!

Quoted 31 lines
Quoted 6 lines

OE already has quite a range of baroque restrictions on what sort of TMPDIR you are allowed to use (no nfs, no symlinks in the path, no spaces in the name) and every new prohibition represents a loss in usability. This patch as proposed already forbids a whole range of characters, including things like "+", and I think that disallowing "-" as well would be a step too far.

I don't believe Qi Chen sent it to the list, but we built path names with all of the special characters and tried to run builds. The specific list is based on the results of those tests. The other items are already broken, and we're trying to be explicit with this for end users. (Note, it's really the TMPDIR that matters.. the 'build' directory is a lot more flexible.)

--Mark

Quoted 8 lines

p.

_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

#9 lore raw
In reply to Phil Blundell
Quoted 9 lines · Phil Blundell
Quoted 8 lines · Mark Hatle
Quoted 6 lines · Phil Blundell
Quoted 2 lines · Gary Thomas

Also, is "-" actually valid? I seem to recall having problems when my build tree had the hyphen ("-") in the path.

Paths with "-" in certainly work for me. If there are any recipes which break in that situation then we should just fix them.

Quoted 8 lines
Quoted 3 lines

We found an issue with paths that -start- with a '-', there are apparently many places where paths are passed into various shell, and the initial '-' can be read as an argument identifier.

Right, I can imagine that this would break fairly badly. But if we're going to require TMPDIR to be an absolute path (which seems perfectly reasonable) then by definition it can't start with - so this will presumably become a non-issue.

Except there are a number of packages that try to parse pieces of strings derived from TMPDIR.

Set your TMPDIR path to include "-Dmypath" some time. SVN will blow up.. :P

Quoted 6 lines
Quoted 5 lines

I don't believe Qi Chen sent it to the list, but we built path names with all of the special characters and tried to run builds. The specific list is based on the results of those tests. The other items are already broken, and we're trying to be explicit with this for end users. (Note, it's really the TMPDIR that matters.. the 'build' directory is a lot more flexible.)

Quoted 7 lines

Yeah, I saw the results of those tests. My slight reservation with that methodology is that, if any single recipe fails with a given character in TMPDIR, no matter how obscure that recipe might be, it seems that this is enough to make that character verboten. For example, the rationale for excluding "+" seems to be that it breaks tcl; if this is the only recipe that fails then it seems like it might be worth at least trying to fix tcl rather than excluding "+" from TMPDIRs for ever more.

I agree, in the case above, the SVN example I've asked our engineers to look into trying to fix that case. But what they're finding is that once you fix one case, then the next breaks.. and the next breaks.

Quoted 4 lines

But, clearly, there are some characters in the list that are "just silly" to have in pathnames. I don't think anybody would be surprised to learn that "*" and "$" cause problems and blacklisting those ones does seem quite reasonable.

:)

I'd like to see us blacklist the bunch, and then turn around and start opening the set backup. But I don't know how practical this actually is -- unless we can get automated test machines running each odd-ball combination on a nightly basis.

--Mark

Quoted 3 lines

p.

author
#3 lore raw
In reply to Richard Purdie
Quoted 31 lines · Richard Purdie
Quoted 17 lines · On Thu, 2013-11-14 at 21:07 +0800, [email protected] wrote:

From: Chen Qi <[email protected]>

TMPDIR must be an absolute path, otherwise, the build will fail. Special characters in TMPDIR will also cause build failures.

This patch enables checking for the validity of TMPDIR.

Signed-off-by Chen Qi <[email protected]>
---
  meta/classes/sanity.bbclass |    7 +++++++
  1 file changed, 7 insertions(+)
meta/classes/sanity.bbclass +0 -0
index 83378b0..e45906e 100644

meta/classes/sanity.bbclass def check_sanity_everybuild(status, d): · line 681

Quoted 10 lines
meta/classes/sanity.bbclass +7 -0
@@ -672,6 +672,13 @@ def check_sanity_everybuild(status, d):
          with open(checkfile, "w") as f:
              f.write(tmpdir)
 
+    # Check if TMPDIR contains invalid characters, and check if it is an absolute path
+    valid_tmpdir_regexp = "^/[a-zA-Z0-9\-_/.~]+$"
+    import re
+    valid_pattern = re.compile(valid_tmpdir_regexp)
+    if not valid_pattern.match(tmpdir):
+        status.addresult("Error, you have special characters in TMPDIR directory path or your TMPDIR is not an absolute path. The TMPDIR should match the this regexp: ^/[a-zA-Z0-9\-_/.~]$")
+

I'm ok with the check itself however this does not need to be run every build. It needs to run when tmpdir changes only.

Quoted 7 lines

Cheers,

Richard

Thanks for pointing it out.

I've changed the logic a little bit.
1. Create the directory hierarchy for TMPDIR only if TMPDIR is valid
2. Handle ABI version change only if TMPDIR has been created.

So only when we had a valid TMPDIR for last build and this time the TMPDIR hasn't changed location, we don't check the validity for TMPDIR.

I've sent out a V2.

Best Regards,
Chen Qi

author
#12 lore raw
In reply to [email protected]

Hi All,

This cover letter contains some testing records.

Best Regards,
Chen Qi

Quoted 212 lines · On 11/14/2013 09:07 PM, [email protected] wrote:

From: Chen Qi <[email protected]>

This patch checks the validity of TMPDIR.

The TMPDIR must match the following regexp.
^/[a-zA-Z0-9\-_/.~]+$

Detailed tests about why I came up with this regexp are displayed at the end of this cover letter.

A major reason for why TMPDIR cannot support many special chars such as @ and # is that these special chars are used by the sed command as its field separator in our recipes.

And here are things I want to make sure of.
1. For now, TMPDIR must be an absolute path. So do we need to make it support relative path?
2. I included the tilde (~) into the valid char set because `bitbake world' succeeded. But it's still possible the the tilde (~) will serve as the field separator in sed. So is it OK that I include it into the valid char set?

********************** TESTING RECORDS **************************************** Test Case 1: use `bitbake minimal' on paths containing special chars
Here's the output.
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/`_contained"
RESULT: FAIL
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/^_contained"
RESULT: SUCCESS
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/~_contained"
RESULT: SUCCESS
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/<_contained"
RESULT: FAIL
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/=_contained"
RESULT: FAIL
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/>_contained"
RESULT: FAIL
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/|_contained"
RESULT: FAIL
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/ _contained"
RESULT: FAIL
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/,_contained"
RESULT: FAIL
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/;_contained"
RESULT: FAIL
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/:_contained"
RESULT: FAIL
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/!_contained"
RESULT: SUCCESS
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/?_contained"
RESULT: FAIL
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/'_contained"
RESULT: FAIL
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/\"_contained"
RESULT: FAIL
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/(_contained"
RESULT: FAIL
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/)_contained"
RESULT: FAIL
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/[_contained"
RESULT: FAIL
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/]_contained"
RESULT: SUCCESS
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/{_contained"
RESULT: SUCCESS
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/}_contained"
RESULT: FAIL
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/@_contained"
RESULT: SUCCESS
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/$_contained"
RESULT: FAIL
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/*_contained"
RESULT: FAIL
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/\_contained"
RESULT: FAIL
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/&_contained"
RESULT: FAIL
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/#_contained"
RESULT: FAIL
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/%_contained"
RESULT: FAIL
===================
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/+_contained"
RESULT: SUCCESS

Test Case 2: TMPDIR containing ^
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/^_contained"
Command:
bitbake core-image-minimal -cpopulate_sdk
Result:
Fail
Formatted Error Message:
ERROR: Task 5 (virtual:nativesdk:/home/chenqi/poky/meta/recipes-core/eglibc/eglibc_2.18.bb, do_populate_sysroot) failed with exit code '1'
| checking whether the C compiler works... no
| configure: error: in `/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/^_contained/work/i686-nativesdk-pokysdk-linux/nativesdk-eglibc/2.18-r0/site_config_qemux86':
| configure: error: C compiler cannot create executables
| See `config.log' for more details

Test Case 3: TMPDIR containing +
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/+_contained"
Command:
bitbake world
Result:
Fail
Formatted Error Message:
ERROR: Task 9084 (virtual:native:/home/chenqi/poky/meta/recipes-devtools/tcltk/tcl_8.6.1.bb, do_install) failed with exit code '1' | oe_libinstall: cd /home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/+_contained/work/i686-linux/tcl-native/8.6.1-r0/build | sed: -e expression #1, char 117: unknown option to `s'
Reason:
'+' is used by the sed command as a separator in tcl recipe.
sed -i "s+${WORKDIR}+${STAGING_INCDIR}+g" tclConfig.sh

Test Case 4: TMPDIR containing ]
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/]_contained"
Command:
bitbake world
Result:
Fail
Formatted Error Message:
ERROR: Task 5 (virtual:native:/home/chenqi/poky/meta/recipes-graphics/xcb/xcb-proto_1.8.bb, do_configure) failed with exit code '1' | configure.ac:9: error: possibly undefined macro: AM_INIT_AUTOMAKE
| If this token and others are legitimate, please use m4_pattern_allow.
| See the Autoconf documentation.
| configure.ac:12: error: possibly undefined macro: AM_CONDITIONAL
| configure.ac:17: error: possibly undefined macro: AM_PATH_PYTHON
| autoreconf: /home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/]_contained/sysroots/i686-linux/usr/bin/autoconf failed with exit status: 1 | ERROR: autoreconf execution failed.
| WARNING: exit code 1 from a shell command.

Test Case 5: TMPDIR containing {
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/[_contained" Command:
bitbake world
Result:
Fail
Formatted Error Message:

     ERROR: Task 9237 (virtual:native:/home/chenqi/poky/meta/recipes-gnome/gnome/gnome-doc-utils_0.20.10.bb, do_compile) failed with exit code '1'
     | xsltproc -nonet -o "C/db-label.xml" \
     |      --stringparam basename "db-label" \
     |      --stringparam xsl_file "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/{_contained/work/i686-linux/gnome-doc-utils-native/0.20.10-r0/gnome-doc-utils-0.20.10/doc/xslt/../../xslt/\
docbook/common/db-label.xsl" \
     |      "./xsldoc-docbook.xsl" "C/db-label.xsldoc"
     | db-chunk: No stylesheet param for documented db.chunk.chunks
     | db-common: No named template for documented db.copyright
     | make[2]: *** [C/db-chunk.xml] Error 10

Test Case 6: TMPDIR containing @
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/@_contained"
Command:
bitbake qt-mobility-x11
Result:
Fail
Formatted Error Message:
ERROR: Task 2 (/home/chenqi/poky/meta/recipes-qt/qt4/qt-mobility-x11_1.2.0.bb, do_install) failed with exit code '1' | DEBUG: Executing shell function do_install
| sed: -e expression #1, char 75: unknown option to `s'

Test Case 7: TMPDIR containing ~
TMPDIR = "/home/chenqi/wrlinux-build/directory-names/dir-with-special-chars/[_contained" Command:
bitbake world
Result:
Success

****************************************************************************************

The following changes since commit ea92671d9823e3667d6ced7ac2af20f991da404d:

bitbake: cooker: replace "w" file opening mode with "a" mode (2013-11-12 17:01:37 +0000)

are available in the git repository at:

git://git.pokylinux.org/poky-contrib ChenQi/TMPDIR
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=ChenQi/TMPDIR

Chen Qi (1):
sanity.bbclass: check for validity of TMPDIR

  meta/classes/sanity.bbclass |    7 +++++++
  1 file changed, 7 insertions(+)
author
#11 lore raw
In reply to Saul Wold
Quoted 10 lines · Saul Wold
Quoted 9 lines · Mark Hatle
Quoted 7 lines · Phil Blundell
Quoted 3 lines · Gary Thomas

Also, is "-" actually valid? I seem to recall having problems when my build
tree had the hyphen ("-") in the path.

Paths with "-" in certainly work for me. If there are any recipes which break in that situation then we should just fix them.

Quoted 9 lines
Quoted 4 lines

We found an issue with paths that -start- with a '-', there are apparently many places where paths are passed into various shell, and the initial '-' can be read as an argument identifier.

I think that I saw an email talking about "-D" in the TMPDIR caused some some recipes to pick that up as a CFLAG define and cause problems

Sau!

Yes. It's actually caused by the following statement in subversion-native. ./build/ac-macros/neon.m4: SVN_NEON_INCLUDES=[$PKG_CONFIG neon --cflags | $SED -e 's/-D[^ ]*//g'] ./build/ac-macros/neon.m4: SVN_NEON_INCLUDES=[$neon_config --cflags | $SED -e 's/-D[^ ]*//g']

And there's a bug filed for it. https://bugzilla.yoctoproject.org/show_bug.cgi?id=5458

Best Regards,
Chen Qi

Quoted 37 lines
Quoted 25 lines
Quoted 6 lines

OE already has quite a range of baroque restrictions on what sort of TMPDIR you are allowed to use (no nfs, no symlinks in the path, no spaces in the name) and every new prohibition represents a loss in usability. This patch as proposed already forbids a whole range of characters, including things like "+", and I think that disallowing "-" as well would be a step too far.

I don't believe Qi Chen sent it to the list, but we built path names with all of the special characters and tried to run builds. The specific list is based on the results of those tests. The other items are already broken, and we're trying to be explicit with this for end users. (Note, it's really the TMPDIR that matters.. the 'build' directory is a lot more flexible.)

--Mark

Quoted 8 lines

p.

_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Quoted 6 lines

_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core