oe/path: three fixes plus selftest coverage
v2 cover oe/path: three fixes plus selftest coverage
While adding unit coverage for the copy of oe/path.py vendored into the standalone wic repository, three real bugs turned up in the shared meta/lib/oe/path.py. They are fixed here at the source so the whole project benefits, each as its own standalone commit:
- __realpath()'s isdir guard assigns a bare false, which is not a Python name; when os.path.isdir() raises, the handler meant to absorb the error raises NameError instead. Use the builtin False.
- symlink(force=True) cleared the destination through remove(), which globs its argument; a destination containing glob metacharacters could fail to match itself (leaving a stale entry) or match unrelated files. Remove the literal destination instead.
- canonicalize('') and canonicalize(None) returned the current working directory, because os.path.realpath('') does; a stray separator injected a spurious cwd entry too. Skip empty tokens.
The final commit extends meta/lib/oeqa/selftest/cases/liboe.py with a PathTests class covering oe.path's own logic (join(), is_path_parent(), symlink(), make_relative_symlink(), canonicalize(), which_wild(), realpath()) and locking in the three fixes; backing any fix out turns the matching test red. oe-selftest -r liboe.PathTests runs 46 tests, all passing.
changes in v2:
- No source changes; the three oe/path.py fixes are unchanged from v1.
- Expand liboe.PathTests from 37 to 46 cases, mostly around the harder realpath() contracts (self-referential and A->B->A symlink loops both raising ELOOP, multi-component resolution under use_physdir, and the assume_dir ENOENT-vs-tolerated contract), plus a dangling symlink(), a make_relative_symlink() several directories below its target, and two more canonicalize() cases.
Trevor Woerner (4):
oe/path: fix bare false NameError in __realpath's isdir guard oe/path: don't glob-expand the destination in symlink(force=True) oe/path: canonicalize('') should return '' rather than the cwd oeqa/selftest/liboe: cover oe.path's own path logic
meta/lib/oe/path.py | 21 +- meta/lib/oeqa/selftest/cases/liboe.py | 376 ++++++++++++++++++++++++++ 2 files changed, 394 insertions(+), 3 deletions(-)
Signature · 1 line
2.50.0.173.g8b6f19ccfc3a
PATCH v2 1/4 oe/path: fix bare `false` NameError in __realpath's isdir guard
__realpath() wraps its os.path.isdir() probe in a bare except that assigns is_dir = false. false is not a Python name, so when os.path.isdir() does raise (for example on an ELOOP path), the handler meant to absorb the error instead raises NameError and aborts the walk.
Use the builtin False so the guard degrades to "not a directory" as intended.
AI-Generated: codex/claude-opus 4.8 (xhigh)
---
changes in v2:
- No change; carried over from v1.
--- meta/lib/oe/path.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
meta/lib/oe/path.py +1 -1
@@ -234,7 +234,7 @@ def __realpath(file, root, loop_cnt, assume_dir): try: is_dir = os.path.isdir(file) except: - is_dir = false + is_dir = False return (file, is_dir)
Signature · 1 line
2.50.0.173.g8b6f19ccfc3a
PATCH v2 2/4 oe/path: don't glob-expand the destination in symlink(force=True)
In reply to Trevor Woernersymlink(source, destination, force=True) cleared an existing destination by calling remove(destination). remove() treats its argument as a glob pattern (it iterates glob.glob(path)), so a destination whose name contains glob metacharacters is mishandled: a name such as "foo[bar]" may fail to match itself and be left in place, or a pattern could match and delete unrelated files.
Remove the literal destination instead: unlink it directly, and fall back to rmtree() for a directory, ignoring ENOENT. This keeps the force=True semantics without passing the path through glob.
AI-Generated: codex/claude-opus 4.8 (xhigh)
---
changes in v2:
- No change; carried over from v1.
--- meta/lib/oe/path.py | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-)
meta/lib/oe/path.py +12 -1
@@ -169,7 +169,18 @@ def symlink(source, destination, force=False): """Create a symbolic link""" try: if force: - remove(destination) + # Remove the exact destination path. Do not route this through + # remove(), which treats its argument as a glob pattern: a + # destination containing glob metacharacters (for example a + # '[' in the name) could fail to match, or match and delete + # unrelated files. + try: + os.unlink(destination) + except OSError as exc: + if exc.errno == errno.EISDIR: + shutil.rmtree(destination) + elif exc.errno != errno.ENOENT: + raise os.symlink(source, destination) except OSError as e: if e.errno != errno.EEXIST or os.readlink(destination) != source:
Signature · 1 line
2.50.0.173.g8b6f19ccfc3a
PATCH v2 3/4 oe/path: canonicalize('') should return '' rather than the cwd
In reply to Trevor Woernercanonicalize() splits its input on the separator and runs each token through os.path.realpath(). os.path.realpath('') returns the current working directory, so canonicalize('') and canonicalize(None) wrongly produced the cwd instead of an empty string, and a stray separator (for example "a,,b") injected a spurious cwd entry into the result.
Skip empty tokens alongside the existing unexpanded-variable skip so only real paths are canonicalized.
AI-Generated: codex/claude-opus 4.8 (xhigh)
---
changes in v2:
- No change; carried over from v1.
--- meta/lib/oe/path.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
meta/lib/oe/path.py +5 -1
@@ -356,7 +356,11 @@ def canonicalize(paths, sep=','): # prefixes in sting compares later on, where the slashes then are important. canonical_paths = [] for path in (paths or '').split(sep): - if '$' not in path: + # Skip empty tokens as well as unexpanded bitbake variables: an + # empty path would otherwise reach os.path.realpath(''), which + # returns the current working directory, so canonicalize('') or + # canonicalize(None) would wrongly produce the cwd instead of ''. + if path and '$' not in path: trailing_slash = path.endswith('/') and '/' or '' canonical_paths.append(os.path.realpath(path) + trailing_slash)
Signature · 1 line
2.50.0.173.g8b6f19ccfc3a
PATCH v2 4/4 oeqa/selftest/liboe: cover oe.path's own path logic
In reply to Trevor WoernerAdd in-process coverage for oe.path's join(), is_path_parent(), symlink(), make_relative_symlink(), canonicalize(), which_wild() and realpath(), alongside the existing copytree tests.
The cases pin behaviour that is easy to regress: __realpath() falls back to "not a directory" when os.path.isdir() raises; symlink(force=True) replaces the literal destination even when its name contains glob metacharacters, and never removes a pattern sibling; and canonicalize() skips empty tokens so '', None, and a stray separator do not inject the current working directory.
realpath() also gets its harder contracts covered: a self-referential link and an A->B->A cycle both raise ELOOP rather than spinning, a symlink several components deep still resolves under use_physdir, and a missing trailing component raises ENOENT by default but is tolerated with assume_dir=True. symlink() is shown to stage a dangling link whose target does not exist yet, make_relative_symlink() is exercised several directories below its target so the multi-level "../" loop runs, and canonicalize() drops a run of "$" tokens and preserves a trailing slash.
AI-Generated: codex/claude-opus 4.8 (xhigh)
---
changes in v2:
- Expand PathTests from 37 to 46 cases; no source changes.
- realpath(): cover ELOOP on a self-referential link and on an A->B->A cycle, multi-component resolution under use_physdir, and the assume_dir ENOENT-vs-tolerated contract.
- symlink(): cover a dangling link whose target does not exist yet.
- make_relative_symlink(): cover a link several directories below its target (multi-level '../').
- canonicalize(): cover a run of '$' tokens all dropped, and a dropped '$' token before a trailing-slash path.
--- meta/lib/oeqa/selftest/cases/liboe.py | 376 ++++++++++++++++++++++++++ 1 file changed, 376 insertions(+)
meta/lib/oeqa/selftest/cases/liboe.py +376 -0
@@ -7,7 +7,10 @@ from oeqa.selftest.case import OESelftestTestCase from oeqa.utils.commands import get_bb_var, get_bb_vars, bitbake, runCmd import oe.path +import errno import os +import tempfile +import unittest.mock class CopyTreeTests(OESelftestTestCase):
@@ -135,3 +138,376 @@ exit 42 self.assertIn("exit status 42", string) self.assertIn("Standard Output: Via stdout", string) self.assertIn("Standard Error: Via stderr", string) + +class PathTests(OESelftestTestCase): + """ + In-process coverage for oe.path's own path logic: join, is_path_parent, + symlink, make_relative_symlink, canonicalize, which_wild and realpath. + Thin standard-library wrappers (relative -> os.path.relpath, + find -> os.walk) are left to the standard library. + """ + + def setUp(self): + super().setUp() + self._tmp = tempfile.TemporaryDirectory(prefix="oeqa-oepath-") + self.addCleanup(self._tmp.cleanup) + self.tmp_path = self._tmp.name + + # --- join -------------------------------------------------------------- + # join() is os.path.normpath("/".join(paths)); unlike os.path.join it + # does not treat an absolute right-hand component specially. + + def test_join_two_paths(self): + self.assertEqual(oe.path.join("a", "b"), "a/b") + + def test_join_absolute_rhs_is_not_special(self): + # os.path.join("a", "/b") == "/b"; oe.path.join() keeps it relative. + self.assertEqual(oe.path.join("a", "/b"), "a/b") + + def test_join_redundant_separators_normalised(self): + self.assertEqual(oe.path.join("a//", "b"), "a/b") + + def test_join_pardir_normalised(self): + self.assertEqual(oe.path.join("a", "..", "b"), "b") + + def test_join_empty_leading_component_is_absolute(self): + # "/".join(["", "b"]) is "/b", which normpath leaves absolute. + self.assertEqual(oe.path.join("", "b"), "/b") + + # --- is_path_parent ---------------------------------------------------- + + def test_is_path_parent_direct_child(self): + self.assertTrue(oe.path.is_path_parent("/usr", "/usr/bin")) + + def test_is_path_parent_unrelated_path(self): + self.assertFalse(oe.path.is_path_parent("/usr", "/tmp")) + + def test_is_path_parent_child_is_not_parent_of_its_parent(self): + self.assertFalse(oe.path.is_path_parent("/usr/bin", "/usr")) + + def test_is_path_parent_prefix_string_is_not_a_path_parent(self): + # "/usrlocal" shares a string prefix with "/usr" but is not below it; + # the trailing-separator handling must reject it. + self.assertFalse(oe.path.is_path_parent("/usr", "/usrlocal")) + + def test_is_path_parent_no_paths_returns_false(self): + self.assertFalse(oe.path.is_path_parent("/usr")) + + def test_is_path_parent_all_paths_must_be_below(self): + self.assertTrue(oe.path.is_path_parent("/usr", "/usr/bin", "/usr/lib")) + + def test_is_path_parent_one_path_outside_fails(self): + self.assertFalse(oe.path.is_path_parent("/usr", "/usr/bin", "/tmp")) + + # --- symlink ----------------------------------------------------------- + + def test_symlink_creates_symlink(self): + src = os.path.join(self.tmp_path, "target.txt") + with open(src, "w") as f: + f.write("content") + dst = os.path.join(self.tmp_path, "link") + oe.path.symlink(src, dst) + self.assertTrue(os.path.islink(dst)) + self.assertEqual(os.readlink(dst), src) + + def test_symlink_matching_existing_is_idempotent(self): + src = os.path.join(self.tmp_path, "target.txt") + with open(src, "w") as f: + f.write("content") + dst = os.path.join(self.tmp_path, "link") + oe.path.symlink(src, dst) + oe.path.symlink(src, dst) + self.assertEqual(os.readlink(dst), src) + + def test_symlink_conflicting_existing_raises(self): + src1 = os.path.join(self.tmp_path, "t1.txt") + src2 = os.path.join(self.tmp_path, "t2.txt") + for p in (src1, src2): + with open(p, "w") as f: + f.write(p) + dst = os.path.join(self.tmp_path, "link") + oe.path.symlink(src1, dst) + with self.assertRaises(OSError): + oe.path.symlink(src2, dst) + + def test_symlink_force_overwrites_existing_symlink(self): + src1 = os.path.join(self.tmp_path, "t1") + src2 = os.path.join(self.tmp_path, "t2") + for p in (src1, src2): + with open(p, "w") as f: + f.write(p) + dst = os.path.join(self.tmp_path, "link") + oe.path.symlink(src1, dst) + oe.path.symlink(src2, dst, force=True) + self.assertEqual(os.readlink(dst), src2) + + def test_symlink_force_replaces_destination_with_glob_metacharacters(self): + # A destination whose name contains glob metacharacters must still be + # replaced. Routing the removal through remove() (which globs) fails + # to match "link[1]" against itself, leaving the stale entry in place. + src = os.path.join(self.tmp_path, "target") + with open(src, "w") as f: + f.write("content") + dst = os.path.join(self.tmp_path, "link[1]") + with open(dst, "w") as f: + f.write("stale regular file") + oe.path.symlink(src, dst, force=True) + self.assertTrue(os.path.islink(dst)) + self.assertEqual(os.readlink(dst), src) + + def test_symlink_force_does_not_delete_glob_siblings(self): + # The destination "keep?.txt" does not exist, but a sibling + # "keepX.txt" matches it as a glob pattern. force=True must remove + # only the literal destination, never a pattern sibling. + sibling = os.path.join(self.tmp_path, "keepX.txt") + with open(sibling, "w") as f: + f.write("do not delete me") + src = os.path.join(self.tmp_path, "target") + with open(src, "w") as f: + f.write("content") + dst = os.path.join(self.tmp_path, "keep?.txt") + oe.path.symlink(src, dst, force=True) + self.assertTrue(os.path.exists(sibling)) + self.assertEqual(os.readlink(dst), src) + + def test_symlink_force_replaces_existing_directory(self): + # An existing directory at the destination is torn down (EISDIR -> + # rmtree) before the link is created. + src = os.path.join(self.tmp_path, "target") + with open(src, "w") as f: + f.write("content") + dst = os.path.join(self.tmp_path, "dir") + os.mkdir(dst) + with open(os.path.join(dst, "child"), "w") as f: + f.write("x") + oe.path.symlink(src, dst, force=True) + self.assertTrue(os.path.islink(dst)) + self.assertEqual(os.readlink(dst), src) + + def test_symlink_creates_dangling_symlink(self): + # symlink() does not require the source to exist; callers rely on + # being able to stage a link before its target is populated. + src = os.path.join(self.tmp_path, "not_created_yet") + dst = os.path.join(self.tmp_path, "link") + oe.path.symlink(src, dst) + self.assertTrue(os.path.islink(dst)) + self.assertEqual(os.readlink(dst), src) + self.assertFalse(os.path.exists(dst)) # dangling: target is absent + + # --- make_relative_symlink --------------------------------------------- + + def test_make_relative_symlink_non_symlink_is_ignored(self): + regular = os.path.join(self.tmp_path, "file.txt") + with open(regular, "w") as f: + f.write("content") + oe.path.make_relative_symlink(regular) + self.assertTrue(os.path.isfile(regular)) + + def test_make_relative_symlink_already_relative_unchanged(self): + target = os.path.join(self.tmp_path, "target.txt") + with open(target, "w") as f: + f.write("content") + link = os.path.join(self.tmp_path, "link") + os.symlink("target.txt", link) + oe.path.make_relative_symlink(link) + self.assertEqual(os.readlink(link), "target.txt") + + def test_make_relative_symlink_absolute_becomes_relative_and_resolves(self): + target = os.path.join(self.tmp_path, "target.txt") + with open(target, "w") as f: + f.write("content") + link = os.path.join(self.tmp_path, "link") + os.symlink(target, link) + self.assertTrue(os.path.isabs(os.readlink(link))) + oe.path.make_relative_symlink(link) + result = os.readlink(link) + self.assertFalse(os.path.isabs(result)) + self.assertTrue(os.path.exists(link)) + with open(link) as f: + self.assertEqual(f.read(), "content") + + def test_make_relative_symlink_several_levels_deep_becomes_relative(self): + # The link sits several directories below the target, so the depth + # loop has to prepend more than one '../' segment. A single-level + # case never exercises that loop body. + target = os.path.join(self.tmp_path, "target.txt") + with open(target, "w") as f: + f.write("content") + deep = os.path.join(self.tmp_path, "a", "b", "c") + os.makedirs(deep) + link = os.path.join(deep, "link") + os.symlink(target, link) + oe.path.make_relative_symlink(link) + result = os.readlink(link) + self.assertFalse(os.path.isabs(result)) + self.assertEqual(result, "../../../target.txt") + with open(link) as f: + self.assertEqual(f.read(), "content") + + # --- canonicalize ------------------------------------------------------ + + def test_canonicalize_real_path_returned(self): + self.assertEqual(oe.path.canonicalize(self.tmp_path), + os.path.realpath(self.tmp_path)) + + def test_canonicalize_unexpanded_variable_is_skipped(self): + self.assertEqual(oe.path.canonicalize("$SOME_VAR"), "") + + def test_canonicalize_variable_token_dropped_leaving_real_path(self): + result = oe.path.canonicalize("$VAR," + self.tmp_path) + # The "$VAR" token is dropped entirely: no empty placeholder, no + # leading separator, just the canonical real path. + self.assertEqual(result, os.path.realpath(self.tmp_path)) + + def test_canonicalize_empty_string_returns_empty_string(self): + # os.path.realpath("") is the cwd; canonicalize("") must not leak it. + self.assertEqual(oe.path.canonicalize(""), "") + + def test_canonicalize_none_returns_empty_string(self): + self.assertEqual(oe.path.canonicalize(None), "") + + def test_canonicalize_empty_token_between_paths_is_dropped(self): + p1 = os.path.join(self.tmp_path, "a") + p2 = os.path.join(self.tmp_path, "b") + os.mkdir(p1) + os.mkdir(p2) + # The stray separator in "a,,b" must not inject a cwd entry. + result = oe.path.canonicalize("%s,,%s" % (p1, p2)) + self.assertEqual(result, "%s,%s" % (os.path.realpath(p1), + os.path.realpath(p2))) + + def test_canonicalize_trailing_slash_preserved(self): + result = oe.path.canonicalize(self.tmp_path + "/") + self.assertEqual(result, os.path.realpath(self.tmp_path) + "/") + + def test_canonicalize_multiple_variable_tokens_all_skipped(self): + # Every "$"-bearing token is dropped, so a list made only of them + # canonicalizes to the empty string (no separators, no cwd). + self.assertEqual(oe.path.canonicalize("$A,$B,$C"), "") + + def test_canonicalize_variable_and_trailing_slash_path(self): + # A dropped "$" token followed by a real path with a trailing slash: + # the slash is preserved and no phantom leading separator appears. + result = oe.path.canonicalize("$VAR,%s/" % self.tmp_path) + self.assertEqual(result, os.path.realpath(self.tmp_path) + "/") + + # --- which_wild -------------------------------------------------------- + + def test_which_wild_missing_tool_returns_empty(self): + self.assertEqual(oe.path.which_wild("totally_nonexistent_tool_xyz"), []) + + def test_which_wild_explicit_search_path(self): + tool = os.path.join(self.tmp_path, "mytool") + with open(tool, "w") as f: + f.write("#!/bin/sh\n") + os.chmod(tool, 0o755) + self.assertEqual(oe.path.which_wild("mytool", path=self.tmp_path), + [tool]) + + def test_which_wild_wildcard_pattern(self): + for name in ("foo-a", "foo-b", "bar"): + with open(os.path.join(self.tmp_path, name), "w") as f: + f.write("") + results = oe.path.which_wild("foo-*", path=self.tmp_path) + self.assertEqual(sorted(os.path.basename(p) for p in results), + ["foo-a", "foo-b"]) + + def test_which_wild_first_match_per_name_wins(self): + # A name found in an earlier PATH element shadows the same name later; + # reverse=True walks PATH the other way, so the other copy wins. + first = os.path.join(self.tmp_path, "first") + second = os.path.join(self.tmp_path, "second") + os.mkdir(first) + os.mkdir(second) + for d in (first, second): + with open(os.path.join(d, "tool"), "w") as f: + f.write("") + search = "%s:%s" % (first, second) + self.assertEqual(oe.path.which_wild("tool", path=search), + [os.path.join(first, "tool")]) + self.assertEqual(oe.path.which_wild("tool", path=search, reverse=True), + [os.path.join(second, "tool")]) + + # --- realpath ---------------------------------------------------------- + + def test_realpath_resolves_symlink_below_root(self): + target = os.path.join(self.tmp_path, "real") + os.mkdir(target) + link = os.path.join(self.tmp_path, "lnk") + os.symlink("real", link) + self.assertEqual(oe.path.realpath(link, self.tmp_path), target) + + def test_realpath_plain_path_is_returned(self): + sub = os.path.join(self.tmp_path, "d") + os.mkdir(sub) + self.assertEqual(oe.path.realpath(sub, self.tmp_path), sub) + + def test_realpath_path_outside_root_raises(self): + root = os.path.join(self.tmp_path, "root") + outside = os.path.join(self.tmp_path, "outside") + os.mkdir(root) + os.mkdir(outside) + with self.assertRaises(OSError): + oe.path.realpath(outside, root) + + def test_realpath_resolves_symlink_several_components_deep(self): + # With use_physdir=True realpath resolves each intermediate + # component, so a link buried under real subdirectories must still + # resolve to its final target. + os.makedirs(os.path.join(self.tmp_path, "a", "b", "c")) + link = os.path.join(self.tmp_path, "link_to_c") + os.symlink("a/b/c", link) + result = oe.path.realpath(os.path.join(link, "file"), + self.tmp_path, assume_dir=True) + self.assertEqual(result, + os.path.join(self.tmp_path, "a", "b", "c", "file")) + + def test_realpath_self_referential_symlink_raises_eloop(self): + # A link that points at itself must be reported as ELOOP rather than + # spinning until the recursion limit or the stack blows. + link = os.path.join(self.tmp_path, "loop") + os.symlink("loop", link) + with self.assertRaises(OSError) as cm: + oe.path.realpath(link, self.tmp_path) + self.assertEqual(cm.exception.errno, errno.ELOOP) + + def test_realpath_mutual_symlink_cycle_raises_eloop(self): + # An A -> B -> A cycle is the same contract as the self-loop. + a = os.path.join(self.tmp_path, "a") + b = os.path.join(self.tmp_path, "b") + os.symlink("b", a) + os.symlink("a", b) + with self.assertRaises(OSError) as cm: + oe.path.realpath(a, self.tmp_path) + self.assertEqual(cm.exception.errno, errno.ELOOP) + + def test_realpath_missing_component_raises_enoent_without_assume_dir(self): + # By default a missing path component is an error. + with self.assertRaises(OSError) as cm: + oe.path.realpath(os.path.join(self.tmp_path, "nope", "missing"), + self.tmp_path) + self.assertEqual(cm.exception.errno, errno.ENOENT) + + def test_realpath_missing_component_tolerated_with_assume_dir(self): + # assume_dir=True lets realpath resolve a path whose trailing + # components do not exist yet, returning the composed path. + result = oe.path.realpath( + os.path.join(self.tmp_path, "nope", "missing"), + self.tmp_path, assume_dir=True) + self.assertEqual(result, + os.path.join(self.tmp_path, "nope", "missing")) + + def test_realpath_isdir_failure_falls_back_without_nameerror(self): + # The final stanza of __realpath tolerates any failure from + # os.path.isdir by treating the path as not-a-directory. Force isdir + # to raise and confirm the fallback returns the resolved path rather + # than raising NameError from an undefined fallback value. + sub = os.path.join(self.tmp_path, "d") + os.mkdir(sub) + + def boom(_path): + raise OSError("synthetic stat failure") + + with unittest.mock.patch("os.path.isdir", boom): + self.assertEqual( + oe.path.realpath(sub, self.tmp_path, use_physdir=False), sub)
Signature · 1 line
2.50.0.173.g8b6f19ccfc3a