Improve performance for native/cross/crosssdk's cleansstate

This can improve the performance a lot for "bitbake <recipe-native/cross/crosssdk> -ccleansstate" when there are a lot of sstate files.

For example:
* Before

  $ bitbake quilt-native -ccleansstate
  - Check log.do_cleansstate:
  Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_package.tgz*
  Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_package_qa.tgz*
  Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_package_write_rpm.tgz*
  Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_packagedata.tgz*
  Removing /sstate-cache/*/sstate:quilt-native::0.65:r0::3:*_populate_lic.tgz*
  Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_populate_sysroot.tgz*

There are no package tasks for quilt-native, so the first 4 lines doesn't make any sense, but the glob pattern "sstate-cache/*/*" is very time consuming when there are no disk caches. E.g., I have more than 600,000 sstate files:
- Without disk caches

  # echo 3 >/proc/sys/vm/drop_caches
  $ time python3 -c 'import glob; glob.glob("/sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_package.tgz*")'
    real    4m32.583s
    user    0m5.768s
    sys     0m12.892s

- With disk caches (e.g., run it in the second time)

  $ time python3 -c 'import glob; glob.glob("/sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_package.tgz*")'
    real    0m5.128s
    user    0m2.772s
    sys     0m2.308s

So the 4 removing *package* commands cost more than 20s or 272s in theory.

* After

  $ bitbake quilt-native -ccleansstate
  - Check log.do_cleansstate:
  Removing /sstate-cache/*/sstate:quilt-native::0.65:r0::3:*_populate_lic.tgz*
  Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_populate_sysroot.tgz*

We can see that it saved 20s or 272s in theory.

---
 meta/classes/sstate.bbclass | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

index efb0096..9f059a0 100644
@@ -423,8 +423,9 @@ def sstate_clean_cachefile(ss, d):
     import oe.path
 
     sstatepkgfile = d.getVar('SSTATE_PATHSPEC') + "*_" + ss['task'] + ".tgz*"
-    bb.note("Removing %s" % sstatepkgfile)
-    oe.path.remove(sstatepkgfile)
+    if d.getVarFlag('do_%s' % ss['task'], 'task'):
+        bb.note("Removing %s" % sstatepkgfile)
+        oe.path.remove(sstatepkgfile)
 
 def sstate_clean_cachefiles(d):
     for task in (d.getVar('SSTATETASKS') or "").split():

2.7.4

The glob.glob("/sstate/*/*/") is very time consuming, set SSTATE_EXTRAPATHWILDCARD explicity to avoid that. This can save a lot of time when there are many sstate files.

For example, I have more than 600,000 sstate files:
* Before
- Without disk caches

  $ time python3 -c 'import glob; glob.glob("/sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_populate_sysroot.tgz*")'
    real    4m32.583s
    user    0m5.768s
    sys     0m12.892s

- With disk caches

  $ time python3 -c 'import glob; glob.glob("/sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_populate_sysroot.tgz*")'
  real    0m4.111s
  user    0m2.348s
  sys     0m1.756s

* After

  $ time python3 -c 'import glob; glob.glob("/sstate-cache.bak/universal/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_populate_sysroot.tgz*")'
  - Without disk caches:
  real    0m7.928s
  user    0m0.172s
  sys     0m0.124s

- With disk caches:

  real    0m0.131s
  user    0m0.088s
  sys     0m0.044s

We can see that it saves about 3.8s with disk caches, and saves about 264s without disk caches.

---
 meta/classes/sstate.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

index 9f059a0..8b48ab4 100644
@@ -101,7 +101,7 @@ python () {
     if bb.data.inherits_class('native', d) or bb.data.inherits_class('crosssdk', d) or bb.data.inherits_class('cross', d):
         d.setVar('SSTATE_EXTRAPATH', "${NATIVELSBSTRING}/")
         d.setVar('BB_HASHFILENAME', "True ${SSTATE_PKGSPEC} ${SSTATE_SWSPEC}")
-        d.setVar('SSTATE_EXTRAPATHWILDCARD', "*/")
+        d.setVar('SSTATE_EXTRAPATHWILDCARD', "${NATIVELSBSTRING}/")
 
     unique_tasks = sorted(set((d.getVar('SSTATETASKS') or "").split()))
     d.setVar('SSTATETASKS', " ".join(unique_tasks))

2.7.4

* V2
- Teach sstate.bbclass rather than edit nopackages.bbclass to fix the problem.

* V1
These 2 patches can improve performance a lot for native/cross/crosssdk's cleansstate:

This test is based on more than 600,000 sstate files
* Without disk caches

  # echo 3 >/proc/sys/vm/drop_caches
  $ bitbake -p
  $ time bitbake quilt-native -ccleansstate
  - Before:
    real    4m53.815s
    user    0m0.820s
    sys     0m0.128s

- After

    real    0m58.483s
    user    0m0.744s
    sys     0m0.108s

Saved 293 - 58 = 235s (80% improvement)

* With disk caches (26.6s -> 5.8s, 78% improvement)

  $ bitbake -p
  $ time bitbake quilt-native -ccleansstate
  - Before:
    real    0m26.682s
    user    0m0.748s
    sys     0m0.088s

- After

    real    0m5.846s
    user    0m0.716s
    sys     0m0.116s

Saved 26.6 - 5.8 = 20.8s (78% improvement)

======
I have another idea is put sstate files into the directory named by the recipe, e.g.:

sstate-cache/quilt-native/sstate:quilt-native::0.65:r0::3:773341447192feb6c1519cace0e6be35_populate_lic.tgz sstate-cache/quilt-native/sstate:quilt-native::0.65:r0::3:773341447192feb6c1519cace0e6be35_populate_lic.tgz.siginfo sstate-cache/quilt-native/sstate:quilt-native::0.65:r0::3:ab4a82a07b10926bb42d2f37242413c7_patch.tgz.siginfo sstate-cache/quilt-native/sstate:quilt-native::0.65:r0::3:e8a4c952a66942653e36f289eaf68ca5_fetch.tgz.siginfo sstate-cache/quilt-native/sstate:quilt-native::0.65:r0::3:fd35240c437cefa66e24cfe65d3fb335_unpack.tgz.siginfo

This can make it easier to manage sstate files, and can reduce the load of glob.glob(), currently, the remove command is something like:

$ rm -f glob.glob(sstate-cache/*/sstate:foo.*bar.*)

If we put them into recipe named directory, then we can run:

$ rm -f glob.glob(sstate-cache/<recipe>/sstate:foo.*bar.*)

Which should be faster, what's your opinion, please ?

The following changes since commit fc5418e7bbdecfb27bafe595084e0fd0f991a388:

meta/icecc.bbclass: Update system blacklists (2018-11-21 11:48:18 +0000)

are available in the git repository at:

git://git.openembedded.org/openembedded-core-contrib rbt/sstate
http://cgit.openembedded.org/openembedded-core-contrib/log/?h=rbt/sstate

Robert Yang (2):
sstate.bbclass: Only remove sstate file when task is existed sstate.bbclass: set SSTATE_EXTRAPATHWILDCARD explicitly

 meta/classes/sstate.bbclass | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

2.7.4