Skip to contentSkip to conversation

sstate-cache-management: find work-shared stamps too

PATCH New Alex Kiernan 1 participant 3 messages Oct 5

lore
Share & tools
lore
#0 lore raw

PATCH 1/3 sstate-cache-management: find work-shared stamps too

remove_by_stamps() collects the hashes still in use by globbing the stamps directory at a fixed depth, "*/*/*.do_*", matching <arch>/<pn>/<file>; the fixed depth is deliberate, so that the walk does not stat() the whole tree, which over NFS costs a round trip per entry. But the shared-source recipes - gcc-source, llvm-project-source and rust-source - set STAMP to ${STAMPS_DIR}/work-shared/<pn>-<pv>-<pr>, one level shallower, so their hashes are never collected and --stamps-dir plans their sstate objects for removal while a build still uses them. (externalsrc's work-shared/<pn>/ is at the usual depth and was already found.)

Glob that layout too, at its own fixed depth, rather than recursively.

Against a cache shared by eight build directories, the stock script planned 740 of 91486 files for removal; 43 of them, all gcc-source, llvm-project-source and rust-source objects, had hashes in live work-shared stamps. With this change it plans 697, the same list as a recursive "**" glob produces. Filesystem calls during the run (strace):
353084 before, 353196 after, against 377568 for the recursive glob.

The shell script this replaced used "find $stamps -maxdepth 3", which did include them.

Fixes 2fa1b25d7485 ("sstate-cache-management: Rewrite in python")

AI-Generated: Claude Opus 5.5

---
 scripts/sstate-cache-management.py | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)
scripts/sstate-cache-management.py +13 -2
index f5737004c544..d088878a92df 100755
@@ -142,6 +142,15 @@ def collect_sstate_paths(args):
     return paths
 
 
+# Where stamps live under STAMPS_DIR, each at a fixed depth: globbing those
+# depths exactly, rather than recursively, keeps the walk from stat()ing
+# everything in the tree, which over NFS is slow with a round trip per entry.
+STAMP_LAYOUTS = (
+    "*/*/",  # <arch>/<pn>/<pv>... (bitbake.conf); also externalsrc's work-shared/<pn>/
+    "work-shared/",  # work-shared/<pn>-<pv>-<pr>... (gcc-source, llvm-project-source, rust-source)
+)
+
+
 def remove_by_stamps(args, paths):
     all_sums = set()
     for stamps_dir in args.stamps_dir:
@@ -151,14 +160,16 @@ def remove_by_stamps(args, paths):
         all_sums |= set(
             [
                 re_sigdata.search(x.parts[-1]).group(1)
-                for x in stamps_path.glob("*/*/*.do_*.sigdata.*")
+                for layout in STAMP_LAYOUTS
+                for x in stamps_path.glob(layout + "*.do_*.sigdata.*")
             ]
         )
         re_setscene = re.compile(r"do_.*_setscene\.([^.]*)")
         all_sums |= set(
             [
                 re_setscene.search(x.parts[-1]).group(1)
-                for x in stamps_path.glob("*/*/*.do_*_setscene.*")
+                for layout in STAMP_LAYOUTS
+                for x in stamps_path.glob(layout + "*.do_*_setscene.*")
             ]
         )
     return [p for p in paths if p.bb_unihash not in all_sums]
#1 lore raw

PATCH 2/3 sstate-cache-management: skip directories it cannot read

collect_sstate_paths() walks the cache assuming every entry is a directory, and only tolerates NotADirectoryError. A directory the user cannot read raises PermissionError instead, which aborts the whole run with a traceback. The usual one is lost+found, whenever SSTATE_DIR is the root of a filesystem - a dedicated disk, a volume, an NFS export - and the script is run as the build user rather than root.

Warn on stderr and carry on, as find did in the shell script this replaced. stderr keeps the warning out of the list -D prints.

Run as a non-root user against a cache at the root of an ext4
filesystem: before, PermissionError on lost+found and exit 1; after, "Skipping .../lost+found: Permission denied" and a complete run.

Fixes 2fa1b25d7485 ("sstate-cache-management: Rewrite in python")

AI-Generated: Claude Opus 5.5

---
 scripts/sstate-cache-management.py | 3 +++
 1 file changed, 3 insertions(+)
scripts/sstate-cache-management.py +3 -0
index d088878a92df..c361c3d5a07f 100755
@@ -125,6 +125,9 @@ def collect_sstate_paths(args):
 
         except NotADirectoryError:
             pass
+        except PermissionError as e:
+            # e.g. lost+found when the cache is the root of a filesystem
+            print(f"Skipping {path}: {e.strerror}", file=sys.stderr)
 
     paths = set()
     # TODO: parellise scandir
#2 lore raw

PATCH 3/3 sstate-cache-management: drop unused functions

In reply to Alex Kiernan

find_archs() and find_tasks() were implemented when converting from sstate-cache-management.sh; neither have ever been used, so drop them.

---
 scripts/sstate-cache-management.py | 44 ------------------------------
 1 file changed, 44 deletions(-)
scripts/sstate-cache-management.py +0 -44
index c361c3d5a07f..e77f0b0efb60 100755
@@ -63,50 +63,6 @@ class SstateEntry:
         return self.match.group(name)
 
 
-# this is what's in the original script; as far as I can tell, it's an
-# implementation artefact which we don't need?
-def find_archs():
-    # all_archs
-    builder_arch = os.uname().machine
-
-    # FIXME
-    layer_paths = [Path("../..")]
-
-    tune_archs = set()
-    re_tune = re.compile(r'AVAILTUNES .*=.*"(.*)"')
-    for path in layer_paths:
-        for tunefile in [
-            p for p in path.glob("meta*/conf/machine/include/**/*") if p.is_file()
-        ]:
-            with open(tunefile) as f:
-                for line in f:
-                    m = re_tune.match(line)
-                    if m:
-                        tune_archs.update(m.group(1).split())
-
-    # all_machines
-    machine_archs = set()
-    for path in layer_paths:
-        for machine_file in path.glob("meta*/conf/machine/*.conf"):
-            machine_archs.add(machine_file.parts[-1][:-5])
-
-    extra_archs = set()
-    all_archs = (
-        set(
-            arch.replace("-", "_")
-            for arch in machine_archs | tune_archs | set(["allarch", builder_arch])
-        )
-        | extra_archs
-    )
-
-    print(all_archs)
-
-
-# again, not needed?
-def find_tasks():
-    print(set([p.bb_task for p in paths]))
-
-
 def collect_sstate_paths(args):
     def scandir(path, paths):
         # Assume everything is a directory; by not checking we avoid needing an