sstate-cache-management fixes

Removing do_package was reported by Jacob Kroon and then I've found few more issues, the most surprising is the one with populate_lic.

The following changes since commit b16de9af5c7108396a347e7107c73608a2f8d74f:

packagegroup-toolset-native: Update after ocf-linux -> cryptodev-linux change (2014-03-27 19:53:07 +0000)

are available in the git repository at:

git://git.openembedded.org/openembedded-core-contrib jansa/sstate-management
http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=jansa/sstate-management

Martin Jansa (4):
sstate-cache-management: rm_by_stamps - include signatures of
.sigdata. files
sstate-cache-management: rm_by_stamps - remove .siginfo and .done
files
sstate-cache-management: skip populate_lic archives when removing
duplicates
sstate-cache-management: rm_by_stamps - discover all suffixes like
remove_duplicated does

 scripts/sstate-cache-management.sh | 34 ++++++++++++++++++++--------------
 1 file changed, 20 insertions(+), 14 deletions(-)

1.9.1

* there are .siginfo files in sstate-cache, but STAMPS_DIR calls them .sigdata, make sure that such signatures are kept, because some tasks like do_package, don't have _setscene or main task entry and are removed:

stamps:
tmp-eglibc/stamps/i586-oe-linux/busybox/1.22.1-r0.do_package.sigdata.cd6f625471ef1b20a9379e90519db6f1 tmp-eglibc/stamps/i586-oe-linux/busybox/1.22.1-r0.do_package_write_ipk.f20fe66285219f23373fc64d5de1d412 tmp-eglibc/stamps/i586-oe-linux/busybox/1.22.1-r0.do_package_write_ipk.sigdata.f20fe66285219f23373fc64d5de1d412 tmp-eglibc/stamps/i586-oe-linux/busybox/1.22.1-r0.do_packagedata.sigdata.c55a3dbe90010c15aa3294753fbd402e tmp-eglibc/stamps/i586-oe-linux/busybox/1.22.1-r0.do_packagedata_setscene.c55a3dbe90010c15aa3294753fbd402e.qemux86 sstate (we want to keep all in this case):
sstate-cache/c5/sstate:busybox:i586-oe-linux:1.22.1:r0:i586:3:c55a3dbe90010c15aa3294753fbd402e_packagedata.tgz sstate-cache/c5/sstate:busybox:i586-oe-linux:1.22.1:r0:i586:3:c55a3dbe90010c15aa3294753fbd402e_packagedata.tgz.siginfo sstate-cache/cd/sstate:busybox:i586-oe-linux:1.22.1:r0:i586:3:cd6f625471ef1b20a9379e90519db6f1_package.tgz sstate-cache/cd/sstate:busybox:i586-oe-linux:1.22.1:r0:i586:3:cd6f625471ef1b20a9379e90519db6f1_package.tgz.siginfo sstate-cache/f2/sstate:busybox:i586-oe-linux:1.22.1:r0:i586:3:f20fe66285219f23373fc64d5de1d412_package_write_ipk.tgz sstate-cache/f2/sstate:busybox:i586-oe-linux:1.22.1:r0:i586:3:f20fe66285219f23373fc64d5de1d412_package_write_ipk.tgz.siginfo

---
 scripts/sstate-cache-management.sh | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

index d741ec9..34bde01 100755
@@ -330,11 +330,11 @@ rm_by_stamps (){
   # Figure out all the md5sums in the stamps dir.
   echo "Figuring out all the md5sums in stamps dir ... "
   for i in $suffixes; do
-      # There is no "\.siginfo" but "_setcene" when it is mirrored
+      # There is no "\.sigdata" but "_setcene" when it is mirrored
       # from the SSTATE_MIRRORS, use them to figure out the sum.
       sums=`find $stamps -maxdepth 3 -name "*.do_$i.*" \
         -o -name "*.do_${i}_setscene.*" | \
-        sed -ne 's#.*_setscene\.##p' -e 's#.*\.siginfo\.##p' | \
+        sed -ne 's#.*_setscene\.##p' -e 's#.*\.sigdata\.##p' | \
         sed -e 's#\..*##' | sort -u`
       all_sums="$all_sums $sums"
   done

1.9.1

* it's possible that corresponding .tgz files were already removed (e.g. with -d option and older version of this script) and this won't find orphaned .siginfo or .done files to remove
* add sort -u to count files found multiple times only once

---
 scripts/sstate-cache-management.sh | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

index 34bde01..ffda05e 100755
@@ -341,9 +341,9 @@ rm_by_stamps (){
   echo "Done"
 
   # Total number of files including sstate-, .siginfo and .done files
-  total_files=`find $cache_dir -name 'sstate*' | wc -l`
+  total_files=`find $cache_dir -type f -name 'sstate*' | wc -l`
   # Save all the state file list to a file
-  find $cache_dir -name 'sstate*.tgz' | sort -u -o $cache_list
+  find $cache_dir -type f -name 'sstate*' | sort -u -o $cache_list
 
   echo "Figuring out the files which will be removed ... "
   for i in $all_sums; do

@@ -355,14 +355,14 @@ rm_by_stamps (){
       sort -u $keep_list -o $keep_list
       to_del=`comm -1 -3 $keep_list $cache_list`
       gen_rmlist $rm_list "$to_del"
-      let total_deleted=`cat $rm_list | wc -w`
+      let total_deleted=`cat $rm_list | sort -u | wc -w`
       if [ $total_deleted -gt 0 ]; then
-          [ $debug -gt 0 ] && cat $rm_list
+          [ $debug -gt 0 ] && cat $rm_list | sort -u
           read_confirm
           if [ "$confirm" = "y" -o "$confirm" = "Y" ]; then
               echo "Removing sstate cache files ... ($total_deleted files)"
               # Remove them one by one to avoid the argument list too long error
-              for i in `cat $rm_list`; do
+              for i in `cat $rm_list | sort -u`; do
                   rm -f $verbose $i
               done
               echo "$total_deleted files have been removed"

1.9.1

* there are 2 issues

* all *gcc* recipes produce "duplicate" files and we cannot keep just the newest one:
OE qemux86@ ~/build/oe-core $ find sstate-cache/ | grep gcc.*populate_lic sstate-cache/cd/sstate:gcc::4.8.2:r0::3:cd8d1a33b832662b624f5e4bd9f55154_populate_lic.tgz sstate-cache/cd/sstate:gcc::4.8.2:r0::3:cd8d1a33b832662b624f5e4bd9f55154_populate_lic.tgz.siginfo sstate-cache/d5/sstate:gcc::4.8.2:r0::3:d591af812395cc386e017fa2c8fa0722_populate_lic.tgz sstate-cache/d5/sstate:gcc::4.8.2:r0::3:d591af812395cc386e017fa2c8fa0722_populate_lic.tgz.siginfo sstate-cache/f7/sstate:gcc::4.8.2:r0::3:f74cfd4982fd7c349a5418f07297f955_populate_lic.tgz sstate-cache/f7/sstate:gcc::4.8.2:r0::3:f74cfd4982fd7c349a5418f07297f955_populate_lic.tgz.siginfo sstate-cache/74/sstate:gcc::4.8.2:r0::3:7489e4502cb446506b9829cda13c7630_populate_lic.tgz sstate-cache/74/sstate:gcc::4.8.2:r0::3:7489e4502cb446506b9829cda13c7630_populate_lic.tgz.siginfo

OE qemux86@ ~/build/oe-core $ find tmp-eglibc/stamps/ | grep gcc.*populate_lic tmp-eglibc/stamps/i586-oe-linux/gcc-cross-initial/4.8.2-r0.do_populate_lic_setscene.cd8d1a33b832662b624f5e4bd9f55154 tmp-eglibc/stamps/i586-oe-linux/libgcc/4.8.2-r0.do_populate_lic_setscene.f74cfd4982fd7c349a5418f07297f955 tmp-eglibc/stamps/i586-oe-linux/gcc-cross/4.8.2-r0.do_populate_lic_setscene.d591af812395cc386e017fa2c8fa0722 tmp-eglibc/stamps/i586-oe-linux/gcc-runtime/4.8.2-r0.do_populate_lic_setscene.7489e4502cb446506b9829cda13c7630

* the same for target and native versions of the same recipe:
OE qemux86@ ~/build/oe-core $ find tmp-eglibc/stamps/ | grep opkg-utils.*populate_lic tmp-eglibc/stamps/x86_64-linux/opkg-utils-native/0.1.8+gitAUTOINC+c33b217016-r0.do_populate_lic_setscene.dd21a3c5444482ce90be4c9a33d806f1 tmp-eglibc/stamps/i586-oe-linux/opkg-utils/0.1.8+gitAUTOINC+c33b217016-r0.do_populate_lic_setscene.8571422f9e311dc41bb6b21e71a09bc0

OE qemux86@ ~/build/oe-core $ find sstate-cache/ | grep opkg-utils.*populate_lic sstate-cache/dd/sstate:opkg-utils::0.1.8+gitAUTOINC+c33b217016:r0::3:dd21a3c5444482ce90be4c9a33d806f1_populate_lic.tgz.siginfo sstate-cache/dd/sstate:opkg-utils::0.1.8+gitAUTOINC+c33b217016:r0::3:dd21a3c5444482ce90be4c9a33d806f1_populate_lic.tgz sstate-cache/85/sstate:opkg-utils::0.1.8+gitAUTOINC+c33b217016:r0::3:8571422f9e311dc41bb6b21e71a09bc0_populate_lic.tgz.siginfo sstate-cache/85/sstate:opkg-utils::0.1.8+gitAUTOINC+c33b217016:r0::3:8571422f9e311dc41bb6b21e71a09bc0_populate_lic.tgz

* similar problem for .siginfo file of do_patch:
OE qemux86@ ~/build/oe-core $ find sstate-cache/ | grep eglibc.*patch sstate-cache/99/sstate:eglibc::2.19:r0::3:99b7e1d688e4c6c659b458c9c57611df_patch.tgz.siginfo sstate-cache/11/sstate:eglibc::2.19:r0::3:11401ddd208a753fee17696b14dc1e52_patch.tgz.siginfo

OE qemux86@ ~/build/oe-core $ find tmp-eglibc/stamps/ | grep /eglibc.*patch tmp-eglibc/stamps/i586-oe-linux/eglibc-initial/2.19-r0.do_patch.sigdata.99b7e1d688e4c6c659b458c9c57611df tmp-eglibc/stamps/i586-oe-linux/eglibc/2.19-r0.do_patch.sigdata.11401ddd208a753fee17696b14dc1e52

* maybe these should have ideally identical signatures, but before it's fixed lets just skip removing them

---
 scripts/sstate-cache-management.sh | 4 ++++
 1 file changed, 4 insertions(+)

index ffda05e..77afd68 100755
@@ -225,6 +225,10 @@ remove_duplicated () {
   # Save the file list which needs to be removed
   local remove_listdir=`mktemp -d` || exit 1
   for suffix in $sstate_suffixes; do
+      if [ "$suffix" = "populate_lic" ] ; then
+          echo "Skipping populate_lic, because removing duplicates doesn't work correctly for them (use --stamps-dir instead)"
+          continue
+      fi
       # Total number of files including .siginfo and .done files
       total_files_suffix=`grep ".*/sstate:[^:]*:[^:]*:[^:]*:[^:]*:[^:]*:[^:]*:[^:_]*_$suffix\.tgz.*" $sstate_files_list | wc -l 2>/dev/null`
       total_tgz_suffix=`grep ".*/sstate:[^:]*:[^:]*:[^:]*:[^:]*:[^:]*:[^:]*:[^:_]*_$suffix\.tgz$" $sstate_files_list | wc -l 2>/dev/null`

1.9.1

* instead of hard coding list of sstate enabled task use the same function as remove_duplicated to find them in sstate-cache directory

---
 scripts/sstate-cache-management.sh | 20 +++++++++++---------
 1 file changed, 11 insertions(+), 9 deletions(-)

index 77afd68..e5c57d3 100755
@@ -324,16 +324,23 @@ rm_by_stamps (){
   local cache_list=`mktemp` || exit 1
   local keep_list=`mktemp` || exit 1
   local rm_list=`mktemp` || exit 1
-  local suffixes
   local sums
   local all_sums
 
-  suffixes="populate_sysroot populate_lic package_write_ipk \
-            package_write_rpm package_write_deb package packagedata deploy"
+  # Total number of files including sstate-, .siginfo and .done files
+  total_files=`find $cache_dir -type f -name 'sstate*' | wc -l`
+  # Save all the state file list to a file
+  find $cache_dir -type f -name 'sstate*' | sort -u -o $cache_list
+
+  echo "Figuring out the suffixes in the sstate cache dir ... "
+  local sstate_suffixes="`sed 's%.*/sstate:[^:]*:[^:]*:[^:]*:[^:]*:[^:]*:[^:]*:[^_]*_\([^:]*\)\.tgz.*%\1%g' $cache_list | sort -u`"
+  echo "Done"
+  echo "The following suffixes have been found in the cache dir:"
+  echo $sstate_suffixes
 
   # Figure out all the md5sums in the stamps dir.
   echo "Figuring out all the md5sums in stamps dir ... "
-  for i in $suffixes; do
+  for i in $sstate_suffixes; do
       # There is no "\.sigdata" but "_setcene" when it is mirrored
       # from the SSTATE_MIRRORS, use them to figure out the sum.
       sums=`find $stamps -maxdepth 3 -name "*.do_$i.*" \

@@ -344,11 +351,6 @@ rm_by_stamps (){
   done
   echo "Done"
 
-  # Total number of files including sstate-, .siginfo and .done files
-  total_files=`find $cache_dir -type f -name 'sstate*' | wc -l`
-  # Save all the state file list to a file
-  find $cache_dir -type f -name 'sstate*' | sort -u -o $cache_list
-
   echo "Figuring out the files which will be removed ... "
   for i in $all_sums; do
       grep ".*/sstate:[^:]*:[^:]*:[^:]*:[^:]*:[^:]*:[^:]*:${i}_.*" $cache_list >>$keep_list

1.9.1

For the whole patch series: