Support symbolic links in paths in PSEUDO_IGNORE_PATHS
v4 cover Support symbolic links in paths in PSEUDO_IGNORE_PATHS
The changes in this patch series aim to rectify a problem with pseudo and its support for ignoring paths. When pseudo compares a path to the paths specified in PSEUDO_IGNORE_PATHS, it uses a path that has been canonicalized. However, it does not canonicalize the paths in PSEUDO_IGNORE_PATHS itself and unless they are canonicalized, they will not match as intended. Thus the paths in PSEUDO_IGNORE_PATHS needs to be canonicalized. These patches do that by adding a new function oe.path.canonicalize(), which is then used when passing ${PSEUDO_IGNORE_PATHS} to pseudo.
There is also one patch that adds two patches to pseudo to clean up pseudo_client_ignore_path_chroot(), and they also plug a memory leak. The patches were brought about as I initially intended to do the canonicalization in pseudo itself in this function.
I have not tested the change to wic as we do not use it, though I do not expect it to be problematic.
PATCHv2:
* Removed some unrelated changes in the second patch.
PATCHv3:
* Handle an edge case in pseudo_client_ignore_path_chroot() in case PSEUDO_IGNORE_PATHS contains ",," or leading/trailing commas.
* Introduce oe.path.canonicalize() and use it instead of adding PSEUDO_IGNORE_REAL_PATHS as was previously done.
* Add similar code to wic.
PATCHv4:
* Rewrote oe.path.canonicalize() to maintain trailing slashes present in the paths.
//Peter
The following changes since commit 9826881036191be6ffba98c9bc8a86d1b852ff41:
build-appliance-image: Update to master head revision (2020-12-09 12:36:32 +0000)
are available in the Git repository at:
git://push.yoctoproject.org/poky-contrib pkj/pseudo-ignore
Peter Kjellerstedt (3):
lib/oe/path: Add canonicalize()
bitbake.conf: Canonicalize paths in PSEUDO_IGNORE_PATHS
wic: Pass canonicalized paths in PSEUDO_IGNORE_PATHS
meta/conf/bitbake.conf | 4 ++-- meta/lib/oe/path.py | 21 +++++++++++++++++++++ scripts/lib/wic/partition.py | 12 +++++++++++- 3 files changed, 34 insertions(+), 3 deletions(-)
PATCH v4 1/3 lib/oe/path: Add canonicalize()
oe.path.canonicalize() is used to canonicalize paths (i.e., remove symbolic links and "..", and make them absolute). It takes a string with paths separated by commas, and returns the canonicalized path in the same format.
--- meta/lib/oe/path.py | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+)
meta/lib/oe/path.py +21 -0
@@ -320,3 +320,24 @@ def which_wild(pathname, path=None, mode=os.F_OK, *, reverse=False, candidates=F return files +def canonicalize(paths, sep=','): + """Given a string with paths (separated by commas by default), expand + each path using os.path.realpath() and return the resulting paths as a + string (separated using the same separator a the original string). + """ + # Ignore paths containing "$" as they are assumed to be unexpanded bitbake + # variables. Normally they would be ignored, e.g., when passing the paths + # through the shell they would expand to empty strings. However, when they + # are passed through os.path.realpath(), it will cause them to be prefixed + # with the absolute path to the current directory and thus not be empty + # anymore. + # + # Also maintain trailing slashes, as the paths may actually be used as + # prefixes in sting compares later on, where the slashes then are important. + canonical_paths = [] + for path in (paths or '').split(sep): + if '$' not in path: + trailing_slash = path.endswith('/') and '/' or '' + canonical_paths.append(os.path.realpath(path) + trailing_slash) + + return sep.join(canonical_paths)
PATCH v4 2/3 bitbake.conf: Canonicalize paths in PSEUDO_IGNORE_PATHS
In reply to Peter KjellerstedtUse oe.path.canonicalize() to canonicalize the paths in PSEUDO_IGNORE_PATHS before passing them to pseudo. This is needed since pseudo will compare them to paths that are canonicalized.
--- meta/conf/bitbake.conf | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
meta/conf/bitbake.conf +2 -2
@@ -691,9 +691,9 @@ export PSEUDO_DISABLED = "1" #export PSEUDO_PREFIX = "${STAGING_DIR_NATIVE}${prefix_native}" #export PSEUDO_BINDIR = "${STAGING_DIR_NATIVE}${bindir_native}" #export PSEUDO_LIBDIR = "${STAGING_DIR_NATIVE}$PSEUDOBINDIR/../lib/pseudo/lib -FAKEROOTBASEENV = "PSEUDO_BINDIR=${PSEUDO_SYSROOT}${bindir_native} PSEUDO_LIBDIR=${PSEUDO_SYSROOT}${prefix_native}/lib/pseudo/lib PSEUDO_PREFIX=${PSEUDO_SYSROOT}${prefix_native} PSEUDO_IGNORE_PATHS=${PSEUDO_IGNORE_PATHS} PSEUDO_DISABLED=1" +FAKEROOTBASEENV = "PSEUDO_BINDIR=${PSEUDO_SYSROOT}${bindir_native} PSEUDO_LIBDIR=${PSEUDO_SYSROOT}${prefix_native}/lib/pseudo/lib PSEUDO_PREFIX=${PSEUDO_SYSROOT}${prefix_native} PSEUDO_IGNORE_PATHS=${@oe.path.canonicalize(d.getVar('PSEUDO_IGNORE_PATHS'))} PSEUDO_DISABLED=1" FAKEROOTCMD = "${PSEUDO_SYSROOT}${bindir_native}/pseudo" -FAKEROOTENV = "PSEUDO_PREFIX=${PSEUDO_SYSROOT}${prefix_native} PSEUDO_LOCALSTATEDIR=${PSEUDO_LOCALSTATEDIR} PSEUDO_PASSWD=${PSEUDO_PASSWD} PSEUDO_NOSYMLINKEXP=1 PSEUDO_IGNORE_PATHS=${PSEUDO_IGNORE_PATHS} PSEUDO_DISABLED=0" +FAKEROOTENV = "PSEUDO_PREFIX=${PSEUDO_SYSROOT}${prefix_native} PSEUDO_LOCALSTATEDIR=${PSEUDO_LOCALSTATEDIR} PSEUDO_PASSWD=${PSEUDO_PASSWD} PSEUDO_NOSYMLINKEXP=1 PSEUDO_IGNORE_PATHS=${@oe.path.canonicalize(d.getVar('PSEUDO_IGNORE_PATHS'))} PSEUDO_DISABLED=0" FAKEROOTNOENV = "PSEUDO_UNLOAD=1" FAKEROOTDIRS = "${PSEUDO_LOCALSTATEDIR}" PREFERRED_PROVIDER_virtual/fakeroot-native ?= "pseudo-native"
PATCH v4 3/3 wic: Pass canonicalized paths in PSEUDO_IGNORE_PATHS
In reply to Peter Kjellerstedt--- scripts/lib/wic/partition.py | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-)
scripts/lib/wic/partition.py +11 -1
@@ -207,11 +207,21 @@ class Partition(): p_prefix = os.environ.get("PSEUDO_PREFIX", "%s/usr" % native_sysroot) if (pseudo_dir): + # Canonicalize the ignore paths. This corresponds to + # calling oe.path.canonicalize(), which is used in bitbake.conf. + ignore_paths = [rootfs] + (get_bitbake_var("PSEUDO_IGNORE_PATHS") or "").split(",") + canonical_paths = [] + for path in ignore_paths: + if "$" not in path: + trailing_slash = path.endswith("/") and "/" or "" + canonical_paths.append(os.path.realpath(path) + trailing_slash) + ignore_paths = ",".join(canonical_paths) + pseudo = "export PSEUDO_PREFIX=%s;" % p_prefix pseudo += "export PSEUDO_LOCALSTATEDIR=%s;" % pseudo_dir pseudo += "export PSEUDO_PASSWD=%s;" % rootfs_dir pseudo += "export PSEUDO_NOSYMLINKEXP=1;" - pseudo += "export PSEUDO_IGNORE_PATHS=%s;" % (rootfs + "," + (get_bitbake_var("PSEUDO_IGNORE_PATHS") or "")) + pseudo += "export PSEUDO_IGNORE_PATHS=%s;" % ignore_paths pseudo += "%s " % get_bitbake_var("FAKEROOTCMD") else: pseudo = None