Fixes for square brackets in installed filenames
cover Fixes for square brackets in installed filenames
When attempting to create a recipe for something that installs files with square brackets in their name I stumbled across a couple of issues, here are the fixes.
The following changes since commit 5e21afc9395060b489156d3f90505a372b713f37:
Revert "selftest/wic: extending test coverage for WIC script options" (2016-12-20 17:06:01 +0000)
are available in the git repository at:
git://git.openembedded.org/openembedded-core-contrib paule/bracket-fixes
http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=paule/bracket-fixes
Paul Eggleton (4):
classes/package_rpm: handle square brackets in filenames lib/oe/path: add warning comment about oe.path.remove() with wildcarded filenames classes/sstate: handle filenames containing square brackets classes/sstate: fix file conflict message
meta/classes/package_rpm.bbclass | 6 ++++++ meta/classes/sstate.bbclass | 24 +++++++++++++----------- meta/lib/oe/path.py | 9 ++++++++- 3 files changed, 27 insertions(+), 12 deletions(-)
Signature · 1 line
2.5.5
PATCH 1/4 classes/package_rpm: handle square brackets in filenames
When constructing a spec file we list files and directory paths in the %files section. If ] or [ characters are in a file or directory name, rpm treats them as wildcards which will mean it won't properly match the filename. Instead, transform these into an ? wildcard so they don't cause a problem.
(This fixes packaging the npm package "file-set" and anything that happens to depend upon it, since it includes tests with files that contain unusual characters including ] and [).
--- meta/classes/package_rpm.bbclass | 6 ++++++ 1 file changed, 6 insertions(+)
meta/classes/package_rpm.bbclass +6 -0
@@ -197,6 +197,8 @@ python write_specfile () { if path.endswith("DEBIAN") or path.endswith("CONTROL"): continue path = path.replace("%", "%%%%%%%%") + path = path.replace("[", "?") + path = path.replace("]", "?") # Treat all symlinks to directories as normal files. # os.walk() lists them as directories.
@@ -216,6 +218,8 @@ python write_specfile () { if dir == "CONTROL" or dir == "DEBIAN": continue dir = dir.replace("%", "%%%%%%%%") + dir = dir.replace("[", "?") + dir = dir.replace("]", "?") # All packages own the directories their files are in... target.append('%dir "' + path + '/' + dir + '"') else:
@@ -230,6 +234,8 @@ python write_specfile () { if file == "CONTROL" or file == "DEBIAN": continue file = file.replace("%", "%%%%%%%%") + file = file.replace("[", "?") + file = file.replace("]", "?") if conffiles.count(path + '/' + file): target.append('%config "' + path + '/' + file + '"') else:
Signature · 1 line
2.5.5
PATCH 2/4 lib/oe/path: add warning comment about oe.path.remove() with wildcarded filenames
In reply to Paul EggletonAdd a warning in the doc comment for oe.path.remove() about using that function on paths that may contain wildcards in the actual file/directory names.
--- meta/lib/oe/path.py | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)
meta/lib/oe/path.py +8 -1
@@ -92,7 +92,14 @@ def copyhardlinktree(src, dst): copytree(src, dst) def remove(path, recurse=True): - """Equivalent to rm -f or rm -rf""" + """ + Equivalent to rm -f or rm -rf + NOTE: be careful about passing paths that may contain filenames with + wildcards in them (as opposed to passing an actual wildcarded path) - + since we use glob.glob() to expand the path. Filenames containing + square brackets are particularly problematic since the they may not + actually expand to match the original filename. + """ for name in glob.glob(path): try: os.unlink(name)
Signature · 1 line
2.5.5
PATCH 3/4 classes/sstate: handle filenames containing square brackets
In reply to Paul EggletonIf a recipe installs a file or directory whose name contains square brackets [ ] that form a valid glob expression and that file then they won't be correctly removed from the sysroot, because we pass each path in the sstate manifest to our oe.path.remove() function which calls glob.glob() on the path passed into it and the expression won't actually match the original filename. Since we don't expect to put any wildcarded expressions in the sstate manifests, and we already have a try...except around this, we can actually use os.remove() here instead.
Similarly, when we pass existing file paths to "grep" looking through the manifests, we don't want those paths to be treated as regexes - so use grep's -F command line switch.
Fixes [YOCTO #10836].
--- meta/classes/sstate.bbclass | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
meta/classes/sstate.bbclass +2 -2
@@ -213,7 +213,7 @@ def sstate_install(ss, d): break if realmatch: match.append(f) - sstate_search_cmd = "grep -rl '%s' %s --exclude=master.list | sed -e 's:^.*/::' -e 's:\.populate-sysroot::'" % (f, d.expand("${SSTATE_MANIFESTS}")) + sstate_search_cmd = "grep -rlF '%s' %s --exclude=master.list | sed -e 's:^.*/::' -e 's:\.populate-sysroot::'" % (f, d.expand("${SSTATE_MANIFESTS}")) search_output = subprocess.Popen(sstate_search_cmd, shell=True, stdout=subprocess.PIPE).communicate()[0] if search_output != "": match.append("Matched in %s" % search_output.rstrip())
@@ -406,7 +406,7 @@ def sstate_clean_manifest(manifest, d): elif os.path.exists(entry) and len(os.listdir(entry)) == 0: os.rmdir(entry[:-1]) else: - oe.path.remove(entry) + os.remove(entry) except OSError: pass
Signature · 1 line
2.5.5
PATCH 4/4 classes/sstate: fix file conflict message
In reply to Paul Eggleton* Fix not decoding output from grep ("Matched in b'manifest...')
* Fix showing "Matched in b''" if no match (show "not matched to any task" instead)
* Drop the filtering out of .populate-sysroot from matched manifest names - it should have been .populate_sysroot so it doesn't work, and in any case the value of removing the task name is questionable given that we aren't removing it for any other task, and that the rest of the filename isn't only the task name, we might as well have the whole thing. At least then you can do a find on that exact name without wildcards and find it.
* Fix indenting of file list entries and indent "matched in" further underneath
* Minor punctuation fixes
--- meta/classes/sstate.bbclass | 22 ++++++++++++---------- 1 file changed, 12 insertions(+), 10 deletions(-)
meta/classes/sstate.bbclass +12 -10
@@ -213,25 +213,27 @@ def sstate_install(ss, d): break if realmatch: match.append(f) - sstate_search_cmd = "grep -rlF '%s' %s --exclude=master.list | sed -e 's:^.*/::' -e 's:\.populate-sysroot::'" % (f, d.expand("${SSTATE_MANIFESTS}")) + sstate_search_cmd = "grep -rlF '%s' %s --exclude=master.list | sed -e 's:^.*/::'" % (f, d.expand("${SSTATE_MANIFESTS}")) search_output = subprocess.Popen(sstate_search_cmd, shell=True, stdout=subprocess.PIPE).communicate()[0] - if search_output != "": - match.append("Matched in %s" % search_output.rstrip()) + if search_output: + match.append(" (matched in %s)" % search_output.decode('utf-8').rstrip()) + else: + match.append(" (not matched to any task)") if match: bb.error("The recipe %s is trying to install files into a shared " \ "area when those files already exist. Those files and their manifest " \ - "location are:\n %s\nPlease verify which recipe should provide the " \ - "above files.\nThe build has stopped as continuing in this scenario WILL " \ - "break things, if not now, possibly in the future (we've seen builds fail " \ + "location are:\n %s\nPlease verify which recipe should provide the " \ + "above files.\n\nThe build has stopped, as continuing in this scenario WILL " \ + "break things - if not now, possibly in the future (we've seen builds fail " \ "several months later). If the system knew how to recover from this " \ - "automatically it would however there are several different scenarios " \ + "automatically it would, however there are several different scenarios " \ "which can result in this and we don't know which one this is. It may be " \ "you have switched providers of something like virtual/kernel (e.g. from " \ "linux-yocto to linux-yocto-dev), in that case you need to execute the " \ "clean task for both recipes and it will resolve this error. It may be " \ "you changed DISTRO_FEATURES from systemd to udev or vice versa. Cleaning " \ - "those recipes should again resolve this error however switching " \ - "DISTRO_FEATURES on an existing build directory is not supported, you " \ + "those recipes should again resolve this error, however switching " \ + "DISTRO_FEATURES on an existing build directory is not supported - you " \ "should really clean out tmp and rebuild (reusing sstate should be safe). " \ "It could be the overlapping files detected are harmless in which case " \ "adding them to SSTATE_DUPWHITELIST may be the correct solution. It could " \
@@ -239,7 +241,7 @@ def sstate_install(ss, d): "things (e.g. bluez 4 and bluez 5 and the correct solution for that would " \ "be to resolve the conflict. If in doubt, please ask on the mailing list, " \ "sharing the error and filelist above." % \ - (d.getVar('PN'), "\n ".join(match))) + (d.getVar('PN'), "\n ".join(match))) bb.fatal("If the above message is too much, the simpler version is you're advised to wipe out tmp and rebuild (reusing sstate is fine). That will likely fix things in most (but not all) cases.") # Write out the manifest
Signature · 1 line
2.5.5