Skip to content

Improve performance for native/cross/crosssdk's cleansstate

Robert Yang openembedded-core 2 versions – New

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v2 Latest2 patches 0 repliesNew
  2. v12 patches 2 repliesSuperseded

Patches in v2 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/2sstate.bbclass: Only remove sstate file when task is existedmeta/classes/sstate.bbclass+3 -20 replies
2/2sstate.bbclass: set SSTATE_EXTRAPATHWILDCARD explicitlymeta/classes/sstate.bbclass+1 -10 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v2 yet.

Read the v2 thread

Cover letter [PATCH V2 0/2] Improve performance for native/cross/crosssdk's cleansstate

* V2
- Teach sstate.bbclass rather than edit nopackages.bbclass to fix the problem.

* V1
These 2 patches can improve performance a lot for native/cross/crosssdk's cleansstate:

This test is based on more than 600,000 sstate files
* Without disk caches

  # echo 3 >/proc/sys/vm/drop_caches
  $ bitbake -p
  $ time bitbake quilt-native -ccleansstate
  - Before:
    real    4m53.815s
    user    0m0.820s
    sys     0m0.128s

- After

    real    0m58.483s
    user    0m0.744s
    sys     0m0.108s

Saved 293 - 58 = 235s (80% improvement)

* With disk caches (26.6s -> 5.8s, 78% improvement)

  $ bitbake -p
  $ time bitbake quilt-native -ccleansstate
  - Before:
    real    0m26.682s
    user    0m0.748s
    sys     0m0.088s

- After

    real    0m5.846s
    user    0m0.716s
    sys     0m0.116s

Saved 26.6 - 5.8 = 20.8s (78% improvement)

======
I have another idea is put sstate files into the directory named by the recipe, e.g.:

sstate-cache/quilt-native/sstate:quilt-native::0.65:r0::3:773341447192feb6c1519cace0e6be35_populate_lic.tgz sstate-cache/quilt-native/sstate:quilt-native::0.65:r0::3:773341447192feb6c1519cace0e6be35_populate_lic.tgz.siginfo sstate-cache/quilt-native/sstate:quilt-native::0.65:r0::3:ab4a82a07b10926bb42d2f37242413c7_patch.tgz.siginfo sstate-cache/quilt-native/sstate:quilt-native::0.65:r0::3:e8a4c952a66942653e36f289eaf68ca5_fetch.tgz.siginfo sstate-cache/quilt-native/sstate:quilt-native::0.65:r0::3:fd35240c437cefa66e24cfe65d3fb335_unpack.tgz.siginfo

This can make it easier to manage sstate files, and can reduce the load of glob.glob(), currently, the remove command is something like:

$ rm -f glob.glob(sstate-cache/*/sstate:foo.*bar.*)

If we put them into recipe named directory, then we can run:

$ rm -f glob.glob(sstate-cache/<recipe>/sstate:foo.*bar.*)

Which should be faster, what's your opinion, please ?

The following changes since commit fc5418e7bbdecfb27bafe595084e0fd0f991a388:

meta/icecc.bbclass: Update system blacklists (2018-11-21 11:48:18 +0000)

are available in the git repository at:

git://git.openembedded.org/openembedded-core-contrib rbt/sstate
http://cgit.openembedded.org/openembedded-core-contrib/log/?h=rbt/sstate

Robert Yang (2):
sstate.bbclass: Only remove sstate file when task is existed sstate.bbclass: set SSTATE_EXTRAPATHWILDCARD explicitly

 meta/classes/sstate.bbclass | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)
Signature · 1 line
2.7.4