Skip to contentSkip to conversation

Support symbolic links in paths in PSEUDO_IGNORE_PATHS

PATCH New Peter Kjellerstedt 1 participant 4 messages Dec 9, 2020

lore
Share & tools
lore
#0 lore raw

v4 cover Support symbolic links in paths in PSEUDO_IGNORE_PATHS

The changes in this patch series aim to rectify a problem with pseudo and its support for ignoring paths. When pseudo compares a path to the paths specified in PSEUDO_IGNORE_PATHS, it uses a path that has been canonicalized. However, it does not canonicalize the paths in PSEUDO_IGNORE_PATHS itself and unless they are canonicalized, they will not match as intended. Thus the paths in PSEUDO_IGNORE_PATHS needs to be canonicalized. These patches do that by adding a new function oe.path.canonicalize(), which is then used when passing ${PSEUDO_IGNORE_PATHS} to pseudo.

There is also one patch that adds two patches to pseudo to clean up pseudo_client_ignore_path_chroot(), and they also plug a memory leak. The patches were brought about as I initially intended to do the canonicalization in pseudo itself in this function.

I have not tested the change to wic as we do not use it, though I do not expect it to be problematic.

PATCHv2:
* Removed some unrelated changes in the second patch.
PATCHv3:
* Handle an edge case in pseudo_client_ignore_path_chroot() in case PSEUDO_IGNORE_PATHS contains ",," or leading/trailing commas.
* Introduce oe.path.canonicalize() and use it instead of adding PSEUDO_IGNORE_REAL_PATHS as was previously done.
* Add similar code to wic.
PATCHv4:
* Rewrote oe.path.canonicalize() to maintain trailing slashes present in the paths.

//Peter

The following changes since commit 9826881036191be6ffba98c9bc8a86d1b852ff41:

build-appliance-image: Update to master head revision (2020-12-09 12:36:32 +0000)

are available in the Git repository at:

git://push.yoctoproject.org/poky-contrib pkj/pseudo-ignore

Peter Kjellerstedt (3):
lib/oe/path: Add canonicalize()
bitbake.conf: Canonicalize paths in PSEUDO_IGNORE_PATHS
wic: Pass canonicalized paths in PSEUDO_IGNORE_PATHS

 meta/conf/bitbake.conf       |  4 ++--
 meta/lib/oe/path.py          | 21 +++++++++++++++++++++
 scripts/lib/wic/partition.py | 12 +++++++++++-
 3 files changed, 34 insertions(+), 3 deletions(-)
#1 lore raw

PATCH v4 1/3 lib/oe/path: Add canonicalize()

oe.path.canonicalize() is used to canonicalize paths (i.e., remove symbolic links and "..", and make them absolute). It takes a string with paths separated by commas, and returns the canonicalized path in the same format.

---
 meta/lib/oe/path.py | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)
meta/lib/oe/path.py +21 -0
index 082972457b..c8d8ad05b9 100644
@@ -320,3 +320,24 @@ def which_wild(pathname, path=None, mode=os.F_OK, *, reverse=False, candidates=F
 
     return files
 
+def canonicalize(paths, sep=','):
+    """Given a string with paths (separated by commas by default), expand
+    each path using os.path.realpath() and return the resulting paths as a
+    string (separated using the same separator a the original string).
+    """
+    # Ignore paths containing "$" as they are assumed to be unexpanded bitbake
+    # variables. Normally they would be ignored, e.g., when passing the paths
+    # through the shell they would expand to empty strings. However, when they
+    # are passed through os.path.realpath(), it will cause them to be prefixed
+    # with the absolute path to the current directory and thus not be empty
+    # anymore.
+    #
+    # Also maintain trailing slashes, as the paths may actually be used as
+    # prefixes in sting compares later on, where the slashes then are important.
+    canonical_paths = []
+    for path in (paths or '').split(sep):
+        if '$' not in path:
+            trailing_slash = path.endswith('/') and '/' or ''
+            canonical_paths.append(os.path.realpath(path) + trailing_slash)
+
+    return sep.join(canonical_paths)
#2 lore raw

PATCH v4 2/3 bitbake.conf: Canonicalize paths in PSEUDO_IGNORE_PATHS

In reply to Peter Kjellerstedt

Use oe.path.canonicalize() to canonicalize the paths in PSEUDO_IGNORE_PATHS before passing them to pseudo. This is needed since pseudo will compare them to paths that are canonicalized.

---
 meta/conf/bitbake.conf | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
meta/conf/bitbake.conf +2 -2
index 0d38eac094..812cd05102 100644
@@ -691,9 +691,9 @@ export PSEUDO_DISABLED = "1"
 #export PSEUDO_PREFIX = "${STAGING_DIR_NATIVE}${prefix_native}"
 #export PSEUDO_BINDIR = "${STAGING_DIR_NATIVE}${bindir_native}"
 #export PSEUDO_LIBDIR = "${STAGING_DIR_NATIVE}$PSEUDOBINDIR/../lib/pseudo/lib
-FAKEROOTBASEENV = "PSEUDO_BINDIR=${PSEUDO_SYSROOT}${bindir_native} PSEUDO_LIBDIR=${PSEUDO_SYSROOT}${prefix_native}/lib/pseudo/lib PSEUDO_PREFIX=${PSEUDO_SYSROOT}${prefix_native} PSEUDO_IGNORE_PATHS=${PSEUDO_IGNORE_PATHS} PSEUDO_DISABLED=1"
+FAKEROOTBASEENV = "PSEUDO_BINDIR=${PSEUDO_SYSROOT}${bindir_native} PSEUDO_LIBDIR=${PSEUDO_SYSROOT}${prefix_native}/lib/pseudo/lib PSEUDO_PREFIX=${PSEUDO_SYSROOT}${prefix_native} PSEUDO_IGNORE_PATHS=${@oe.path.canonicalize(d.getVar('PSEUDO_IGNORE_PATHS'))} PSEUDO_DISABLED=1"
 FAKEROOTCMD = "${PSEUDO_SYSROOT}${bindir_native}/pseudo"
-FAKEROOTENV = "PSEUDO_PREFIX=${PSEUDO_SYSROOT}${prefix_native} PSEUDO_LOCALSTATEDIR=${PSEUDO_LOCALSTATEDIR} PSEUDO_PASSWD=${PSEUDO_PASSWD} PSEUDO_NOSYMLINKEXP=1 PSEUDO_IGNORE_PATHS=${PSEUDO_IGNORE_PATHS} PSEUDO_DISABLED=0"
+FAKEROOTENV = "PSEUDO_PREFIX=${PSEUDO_SYSROOT}${prefix_native} PSEUDO_LOCALSTATEDIR=${PSEUDO_LOCALSTATEDIR} PSEUDO_PASSWD=${PSEUDO_PASSWD} PSEUDO_NOSYMLINKEXP=1 PSEUDO_IGNORE_PATHS=${@oe.path.canonicalize(d.getVar('PSEUDO_IGNORE_PATHS'))} PSEUDO_DISABLED=0"
 FAKEROOTNOENV = "PSEUDO_UNLOAD=1"
 FAKEROOTDIRS = "${PSEUDO_LOCALSTATEDIR}"
 PREFERRED_PROVIDER_virtual/fakeroot-native ?= "pseudo-native"
#3 lore raw

PATCH v4 3/3 wic: Pass canonicalized paths in PSEUDO_IGNORE_PATHS

In reply to Peter Kjellerstedt
---
 scripts/lib/wic/partition.py | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)
scripts/lib/wic/partition.py +11 -1
index ebe250b00d..286c7867cb 100644
@@ -207,11 +207,21 @@ class Partition():
 
         p_prefix = os.environ.get("PSEUDO_PREFIX", "%s/usr" % native_sysroot)
         if (pseudo_dir):
+            # Canonicalize the ignore paths. This corresponds to
+            # calling oe.path.canonicalize(), which is used in bitbake.conf.
+            ignore_paths = [rootfs] + (get_bitbake_var("PSEUDO_IGNORE_PATHS") or "").split(",")
+            canonical_paths = []
+            for path in ignore_paths:
+                if "$" not in path:
+                    trailing_slash = path.endswith("/") and "/" or ""
+                    canonical_paths.append(os.path.realpath(path) + trailing_slash)
+            ignore_paths = ",".join(canonical_paths)
+
             pseudo = "export PSEUDO_PREFIX=%s;" % p_prefix
             pseudo += "export PSEUDO_LOCALSTATEDIR=%s;" % pseudo_dir
             pseudo += "export PSEUDO_PASSWD=%s;" % rootfs_dir
             pseudo += "export PSEUDO_NOSYMLINKEXP=1;"
-            pseudo += "export PSEUDO_IGNORE_PATHS=%s;" % (rootfs + "," + (get_bitbake_var("PSEUDO_IGNORE_PATHS") or ""))
+            pseudo += "export PSEUDO_IGNORE_PATHS=%s;" % ignore_paths
             pseudo += "%s " % get_bitbake_var("FAKEROOTCMD")
         else:
             pseudo = None