Skip to contentSkip to conversation

dev-manual: disk-space: mention faster "find" command to trim sstate cache

PATCH Superseded [email protected] 2 participants 3 messages Aug 11, 2023

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 dev-manual: disk-space: mention faster "find" command to trim sstate cache

From: Michael Opdenacker <[email protected]>

[YOCTO #15182]

---
 documentation/dev-manual/disk-space.rst | 24 +++++++++++++++---------
 1 file changed, 15 insertions(+), 9 deletions(-)
documentation/dev-manual/disk-space.rst +15 -9
index c63591cc7a..670f3d2792 100644
@@ -27,19 +27,25 @@ Purging Duplicate Shared State Cache Files
 ==========================================
 
 After multiple build iterations, the Shared State (sstate) cache can contain
-duplicate cache files for a given package, while only the most recent one
-is likely to be reusable. The following command purges all but the
-newest sstate cache file for each package::
+duplicate cache files for a given package, consuming a substantial amount of
+disk space. However, only the most recent cache files are likeky to be reusable.
 
-   sstate-cache-management.sh --remove-duplicated --cache-dir=build/sstate-cache
+The following command is a quick way to purge all the cache files which
+are older than a specified number of days::
 
-This command will ask you to confirm the deletions it identifies.
+   find build/sstate-cache -type f -atime +$DAYS -delete
 
-.. note::
+OpenEmbedded-Core also offers a command which can be used to look for
+cache files only for enabled architectures, and purge all but the newest
+ones on each architecture::
 
-   The duplicated sstate cache files of one package must have the same
-   architecture, which means that sstate cache files with multiple
-   architectures are not considered as duplicate.
+   sstate-cache-management.sh --remove-duplicated --cache-dir=build/sstate-cache
 
+This command will ask you to confirm the deletions it identifies.
 Run ``sstate-cache-management.sh`` for more details about this script.
 
+.. note::
+
+   As this command is much more cautious and selective, removing only cache files,
+   it will execute much slower than the simple ``find`` command described above.
+   Therefore, it may not be your best option to trim huge cache directories.
Signature · 1 line
2.34.1
#1 lore raw
Quoted 35 lines · Michael Opdenacker via lists.yoctoproject.org

From: Michael Opdenacker <[email protected]>

[YOCTO #15182]

---
 documentation/dev-manual/disk-space.rst | 24 +++++++++++++++---------
 1 file changed, 15 insertions(+), 9 deletions(-)
documentation/dev-manual/disk-space.rst +6 -6
index c63591cc7a..670f3d2792 100644
@@ -27,19 +27,25 @@ Purging Duplicate Shared State Cache Files
 ==========================================
 
 After multiple build iterations, the Shared State (sstate) cache can contain
-duplicate cache files for a given package, while only the most recent one
-is likely to be reusable. The following command purges all but the
-newest sstate cache file for each package::
+duplicate cache files for a given package, consuming a substantial amount of
+disk space. However, only the most recent cache files are likeky to be reusable.
 
-   sstate-cache-management.sh --remove-duplicated --cache-dir=build/sstate-cache
+The following command is a quick way to purge all the cache files which
+are older than a specified number of days::
 
-This command will ask you to confirm the deletions it identifies.
+   find build/sstate-cache -type f -atime +$DAYS -delete
 
-.. note::
+OpenEmbedded-Core also offers a command which can be used to look for

documentation/dev-manual/disk-space.rst Purging Duplicate Shared State Cache Files · line 45

Quoted 10 lines
documentation/dev-manual/disk-space.rst +4 -3
+cache files only for enabled architectures, and purge all but the newest
+ones on each architecture::
 
-   The duplicated sstate cache files of one package must have the same
-   architecture, which means that sstate cache files with multiple
-   architectures are not considered as duplicate.
+   sstate-cache-management.sh --remove-duplicated --cache-dir=build/sstate-cache
 
+This command will ask you to confirm the deletions it identifies.
 Run ``sstate-cache-management.sh`` for more details about this script.

As long as bitbake builds have writable access to the sstate cache, files are touched as they are accessed so it is easy to know which ones are being used and which ones are not. As such you can use mtime in the above example even for a partition mounted as "noatime" for performance reasons.

documentation/dev-manual/disk-space.rst Purging Duplicate Shared State Cache Files · line 51

Quoted 6 lines
documentation/dev-manual/disk-space.rst +5 -0
+.. note::
+
+   As this command is much more cautious and selective, removing only cache files,
+   it will execute much slower than the simple ``find`` command described above.
+   Therefore, it may not be your best option to trim huge cache directories.

It doesn't remove only cache files, it removes files that it considers to be unreachable by exploring a set of build configurations. It requires full build environment to be available and doesn't work well covering multiple releases. It also doesn't work in a limited environment like a BSD based NAS where the above find command would still likely work easily.

Can we get rid of sstate-cache-management.sh? :)

Cheers,

Richard

#2 lore raw

Hi Richard,

Quoted 45 lines · Richard Purdie
Quoted 43 lines · Michael Opdenacker via lists.yoctoproject.org

From: Michael Opdenacker <[email protected]>

[YOCTO #15182]

---
  documentation/dev-manual/disk-space.rst | 24 +++++++++++++++---------
  1 file changed, 15 insertions(+), 9 deletions(-)
documentation/dev-manual/disk-space.rst +9 -9
index c63591cc7a..670f3d2792 100644
@@ -27,19 +27,25 @@ Purging Duplicate Shared State Cache Files
  ==========================================
 
  After multiple build iterations, the Shared State (sstate) cache can contain
-duplicate cache files for a given package, while only the most recent one
-is likely to be reusable. The following command purges all but the
-newest sstate cache file for each package::
+duplicate cache files for a given package, consuming a substantial amount of
+disk space. However, only the most recent cache files are likeky to be reusable.
 
-   sstate-cache-management.sh --remove-duplicated --cache-dir=build/sstate-cache
+The following command is a quick way to purge all the cache files which
+are older than a specified number of days::
 
-This command will ask you to confirm the deletions it identifies.
+   find build/sstate-cache -type f -atime +$DAYS -delete
 
-.. note::
+OpenEmbedded-Core also offers a command which can be used to look for
+cache files only for enabled architectures, and purge all but the newest
+ones on each architecture::
 
-   The duplicated sstate cache files of one package must have the same
-   architecture, which means that sstate cache files with multiple
-   architectures are not considered as duplicate.
+   sstate-cache-management.sh --remove-duplicated --cache-dir=build/sstate-cache
 
Quoted 7 lines

documentation/dev-manual/disk-space.rst Purging Duplicate Shared State Cache Files · line 45

Quoted 2 lines
documentation/dev-manual/disk-space.rst +1 -0
+This command will ask you to confirm the deletions it identifies.
  Run ``sstate-cache-management.sh`` for more details about this script.

As long as bitbake builds have writable access to the sstate cache, files are touched as they are accessed so it is easy to know which ones are being used and which ones are not. As such you can use mtime in the above example even for a partition mounted as "noatime" for performance reasons.

Good idea, this way the solution is a little broader as "noatime" is frequently used.

Quoted 7 lines

documentation/dev-manual/disk-space.rst Purging Duplicate Shared State Cache Files · line 51

Quoted 7 lines
documentation/dev-manual/disk-space.rst +5 -0
+.. note::
+
+   As this command is much more cautious and selective, removing only cache files,
+   it will execute much slower than the simple ``find`` command described above.
+   Therefore, it may not be your best option to trim huge cache directories.
Quoted 8 lines

It doesn't remove only cache files, it removes files that it considers to be unreachable by exploring a set of build configurations. It requires full build environment to be available and doesn't work well covering multiple releases. It also doesn't work in a limited environment like a BSD based NAS where the above find command would still likely work easily.

Can we get rid of sstate-cache-management.sh? :)

Thanks for these useful details about this script!

Are you suggesting we could at least remove sstate-cache-management.sh from the docs? Thanks again,
Michael.

Signature · 3 lines
Michael Opdenacker, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com