Skip to contentSkip to conversation

dev-manual: how to purge duplicate sstate cache files

PATCH New Michael Opdenacker 3 participants 7 messages Nov 19, 2021 – Nov 26, 2021 2 messages not in the archive

lore
Share & tools
lore
#1 lore raw

PATCH 1/1 dev-manual: how to purge duplicate sstate cache files

In reply to a message not in the archive
---
 documentation/dev-manual/common-tasks.rst | 25 ++++++++++++++++++++++-
 1 file changed, 24 insertions(+), 1 deletion(-)
documentation/dev-manual/common-tasks.rst +24 -1
index 3eead147a3..37612c84b7 100644
@@ -6242,8 +6242,11 @@ Changing the listed common targets is as easy as editing your version of
 ``conf-notes.txt`` in your custom template configuration directory and
 making sure you have ``TEMPLATECONF`` set to your directory.
 
+Conserving Disk Space
+=====================
+
 Conserving Disk Space During Builds
-===================================
+-----------------------------------
 
 To help conserve disk space during builds, you can add the following
 statement to your project's ``local.conf`` configuration file found in
@@ -6257,6 +6260,26 @@ building a recipe once the recipe is built. For more information on
 :ref:`rm_work <ref-classes-rm-work>` class in the
 Yocto Project Reference Manual.
 
+Purging Duplicate Shared State Cache Files
+-------------------------------------------
+
+After multiple build iterations, the Shared State (sstate) cache can contain
+duplicate cache files for a given package, while only the most recent one
+is likely to be reusable. The following command purges all but the
+newest sstate cache file for each package::
+
+   sstate-cache-management.sh --remove-duplicated --cache-dir=build/sstate-cache
+
+This command will ask you to confirm the deletions it identifies.
+
+Note::
+
+   The duplicated sstate cache files of one package must have the same
+   architecture, which means that sstate cache files with multiple
+   architectures are not considered as duplicate.
+
+Run ``sstate-cache-management.sh`` for more details about this script.
+
 Working with Packages
 =====================
 
Signature · 1 line
2.25.1
#3 lore raw
In reply to a message not in the archive

Greetings,

Quoted 39 lines · Michael Opdenacker
---
 documentation/dev-manual/common-tasks.rst | 25 ++++++++++++++++++++++-
 1 file changed, 24 insertions(+), 1 deletion(-)
documentation/dev-manual/common-tasks.rst +17 -1
index 3eead147a3..37612c84b7 100644
@@ -6242,8 +6242,11 @@ Changing the listed common targets is as easy as editing your version of
 ``conf-notes.txt`` in your custom template configuration directory and
 making sure you have ``TEMPLATECONF`` set to your directory.
 
+Conserving Disk Space
+=====================
+
 Conserving Disk Space During Builds
-===================================
+-----------------------------------
 
 To help conserve disk space during builds, you can add the following
 statement to your project's ``local.conf`` configuration file found in
@@ -6257,6 +6260,26 @@ building a recipe once the recipe is built. For more information on
 :ref:`rm_work <ref-classes-rm-work>` class in the
 Yocto Project Reference Manual.
 
+Purging Duplicate Shared State Cache Files
+-------------------------------------------
+
+After multiple build iterations, the Shared State (sstate) cache can contain
+duplicate cache files for a given package, while only the most recent one
+is likely to be reusable. The following command purges all but the
+newest sstate cache file for each package::
+
+   sstate-cache-management.sh --remove-duplicated --cache-dir=build/sstate-cache
+
+This command will ask you to confirm the deletions it identifies.
+
+Note::

documentation/dev-manual/common-tasks.rst building a recipe once the recipe is built. For more information on · line 6285

Quoted 10 lines
documentation/dev-manual/common-tasks.rst +7 -0
+
+   The duplicated sstate cache files of one package must have the same
+   architecture, which means that sstate cache files with multiple
+   architectures are not considered as duplicate.
+
+Run ``sstate-cache-management.sh`` for more details about this script.
+
 Working with Packages
 =====================
 

Did anyone have time to have a look at what I wrote ? There may be better ways to purge the sstate cache files.
Thanks in advance
Michael.

Signature · 3 lines
Michael Opdenacker, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#4 lore raw

Hi Michael

Quoted 48 lines · Michael Opdenacker

Greetings,

Quoted 45 lines · Michael Opdenacker
---
  documentation/dev-manual/common-tasks.rst | 25 ++++++++++++++++++++++-
  1 file changed, 24 insertions(+), 1 deletion(-)
documentation/dev-manual/common-tasks.rst +23 -1
index 3eead147a3..37612c84b7 100644
@@ -6242,8 +6242,11 @@ Changing the listed common targets is as easy as editing your version of
  ``conf-notes.txt`` in your custom template configuration directory and
  making sure you have ``TEMPLATECONF`` set to your directory.
 
+Conserving Disk Space
+=====================
+
  Conserving Disk Space During Builds
-===================================
+-----------------------------------
 
  To help conserve disk space during builds, you can add the following
  statement to your project's ``local.conf`` configuration file found in
@@ -6257,6 +6260,26 @@ building a recipe once the recipe is built. For more information on
  :ref:`rm_work <ref-classes-rm-work>` class in the
  Yocto Project Reference Manual.
 
+Purging Duplicate Shared State Cache Files
+-------------------------------------------
+
+After multiple build iterations, the Shared State (sstate) cache can contain
+duplicate cache files for a given package, while only the most recent one
+is likely to be reusable. The following command purges all but the
+newest sstate cache file for each package::
+
+   sstate-cache-management.sh --remove-duplicated --cache-dir=build/sstate-cache
+
+This command will ask you to confirm the deletions it identifies.
+
+Note::
+
+   The duplicated sstate cache files of one package must have the same
+   architecture, which means that sstate cache files with multiple
+   architectures are not considered as duplicate.
+
+Run ``sstate-cache-management.sh`` for more details about this script.
Quoted 8 lines

documentation/dev-manual/common-tasks.rst building a recipe once the recipe is built. For more information on · line 6285

Quoted 4 lines
documentation/dev-manual/common-tasks.rst +1 -0
+
  Working with Packages
  =====================
 

Did anyone have time to have a look at what I wrote ? There may be better ways to purge the sstate cache files.
Thanks in advance
Michael.

The description looks good to me. By other ways you mean other usage scenarios of sstate-cache-management.sh script or by means of something else?

Vyacheslav

#5 lore raw

Hi Michael,

Quoted 63 lines · Vyacheslav Yurkov

Hi Michael

Quoted 51 lines · Michael Opdenacker

Greetings,

Quoted 48 lines · Michael Opdenacker

---
  documentation/dev-manual/common-tasks.rst | 25 ++++++++++++++++++++++-

  1 file changed, 24 insertions(+), 1 deletion(-)
documentation/dev-manual/common-tasks.rst +0 -0
index 3eead147a3..37612c84b7 100644
@@ -6242,8 +6242,11 @@ Changing the listed common targets is as easy

as editing your version of
  ``conf-notes.txt`` in your custom template configuration directory and making sure you have ``TEMPLATECONF`` set to your directory. +Conserving Disk Space
+=====================
+
  Conserving Disk Space During Builds
-===================================
+-----------------------------------
    To help conserve disk space during builds, you can add the following statement to your project's ``local.conf`` configuration file found in

documentation/dev-manual/common-tasks.rst +0 -0
@@ -6257,6 +6260,26 @@ building a recipe once the recipe is built.

For more information on
  :ref:rm_work <ref-classes-rm-work> class in the Yocto Project Reference Manual.
  +Purging Duplicate Shared State Cache Files
+-------------------------------------------

documentation/dev-manual/common-tasks.rst +2 -0
+
+After multiple build iterations, the Shared State (sstate) cache can

contain
+duplicate cache files for a given package, while only the most recent one

documentation/dev-manual/common-tasks.rst +14 -1
+is likely to be reusable. The following command purges all but the
+newest sstate cache file for each package::
+
+   sstate-cache-management.sh --remove-duplicated
--cache-dir=build/sstate-cache
+
+This command will ask you to confirm the deletions it identifies.
+
+Note::
+
+   The duplicated sstate cache files of one package must have the same
+   architecture, which means that sstate cache files with multiple
+   architectures are not considered as duplicate.
+
+Run ``sstate-cache-management.sh`` for more details about this script.
Quoted 8 lines
Quoted 4 lines

+
  Working with Packages
  =====================

Did anyone have time to have a look at what I wrote ? There may be better ways to purge the sstate cache files.
Thanks in advance
Michael.

Quoted 4 lines

The description looks good to me. By other ways you mean other usage scenarios of sstate-cache-management.sh script or by means of something else?

I also use the --stamps-dir method since that removes additional files. You just gotta remember to do a build of everything you want to keep first.

/Jacob

#6 lore raw

Hi Jacob,

Thank you for the suggestion!

Quoted 2 lines · Jacob Kroon

I also use the --stamps-dir method since that removes additional files. You just gotta remember to do a build of everything you want to keep first.

I don't understand the value of this option in the simple case when you have a single build directory.  Here's the description of this option:

  --stamps-dir=<dir1>,<dir2>...<dirn>
        Specify the build directory's stamps directories, the sstate cache file which IS USED by these build diretories will be KEPT, other sstate cache files in cache-dir will be removed. Use "," as the separator. For example:
        --stamps-dir=build1/tmp/stamps,build2/tmp/stamps

What am I missing?

Thanks again

Michael.

Signature · 3 lines
Michael Opdenacker, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#8 lore raw
In reply to Vyacheslav Yurkov

Hi Slava,

On 11/26/21 10:13 AM, Vyacheslav Yurkov wrote:Did anyone have time to have a look at what I wrote ? There may be

Quoted 7 lines
Quoted 3 lines

better ways to purge the sstate cache files.
Thanks in advance
Michael.

The description looks good to me. By other ways you mean other usage scenarios of sstate-cache-management.sh script or by means of something else?

Thanks for the review. I was interested in any advise from real life experience, with or without this script, to get rid of obsolete sstate cache files.
Cheers
Michael.

Signature · 3 lines
Michael Opdenacker, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#7 lore raw
In reply to Michael Opdenacker
Quoted 12 lines · Michael Opdenacker

Hi Jacob,

Thank you for the suggestion!

Quoted 2 lines · Jacob Kroon

I also use the --stamps-dir method since that removes additional files. You just gotta remember to do a build of everything you want to keep first.

I don't understand the value of this option in the simple case when you have a single build directory.  Here's the description of this option:

Quoted 9 lines

  --stamps-dir=<dir1>,<dir2>...<dirn>
        Specify the build directory's stamps directories, the sstate cache file which IS USED by these build diretories will be KEPT, other sstate cache files in cache-dir will be removed. Use "," as the separator. For example:
        --stamps-dir=build1/tmp/stamps,build2/tmp/stamps

What am I missing?

--stamps-dir=${TMPDIR}/stamps

in a default build dir I think that would be "build/tmp-glibc/stamps"