Improve performance for native/cross/crosssdk's cleansstate
cover Improve performance for native/cross/crosssdk's cleansstate
Hi RP and Ross,
These 2 patches can improve performance a lot for native/cross/crosssdk's cleansstate:
This test is based on more than 600,000 sstate files
* Without disk caches
# echo 3 >/proc/sys/vm/drop_caches
$ bitbake -p
$ time bitbake quilt-native -ccleansstate
- Before:
real 4m53.815s
user 0m0.820s
sys 0m0.128s- After
real 0m58.483s
user 0m0.744s
sys 0m0.108sSaved 293 - 58 = 235s (80% improvement)
* With disk caches (26.6s -> 5.8s, 78% improvement)
$ bitbake -p
$ time bitbake quilt-native -ccleansstate
- Before:
real 0m26.682s
user 0m0.748s
sys 0m0.088s- After
real 0m5.846s
user 0m0.716s
sys 0m0.116sSaved 26.6 - 5.8 = 20.8s (78% improvement)
======
I have another idea is put sstate files into the directory named by the recipe, e.g.:
sstate-cache/quilt-native/sstate:quilt-native::0.65:r0::3:773341447192feb6c1519cace0e6be35_populate_lic.tgz sstate-cache/quilt-native/sstate:quilt-native::0.65:r0::3:773341447192feb6c1519cace0e6be35_populate_lic.tgz.siginfo sstate-cache/quilt-native/sstate:quilt-native::0.65:r0::3:ab4a82a07b10926bb42d2f37242413c7_patch.tgz.siginfo sstate-cache/quilt-native/sstate:quilt-native::0.65:r0::3:e8a4c952a66942653e36f289eaf68ca5_fetch.tgz.siginfo sstate-cache/quilt-native/sstate:quilt-native::0.65:r0::3:fd35240c437cefa66e24cfe65d3fb335_unpack.tgz.siginfo
This can make it easier to manage sstate files, and can reduce the load of glob.glob(), currently, the remove command is something like:
$ rm -f glob.glob(sstate-cache/*/sstate:foo.*bar.*)
If we put them into recipe named directory, then we can run:
$ rm -f glob.glob(sstate-cache/<recipe>/sstate:foo.*bar.*)
Which should be faster, what's your opinion, please ?
// Robert
The following changes since commit fc5418e7bbdecfb27bafe595084e0fd0f991a388:
meta/icecc.bbclass: Update system blacklists (2018-11-21 11:48:18 +0000)
are available in the git repository at:
git://git.openembedded.org/openembedded-core-contrib rbt/sstate
http://cgit.openembedded.org/openembedded-core-contrib/log/?h=rbt/sstate
Robert Yang (2):
nopackages.bbclass: improve performance for cleansstate sstate.bbclass: set SSTATE_EXTRAPATHWILDCARD explicitly
meta/classes/nopackages.bbclass | 35 +++++++++++++++++++++++------------ meta/classes/sstate.bbclass | 2 +- 2 files changed, 24 insertions(+), 13 deletions(-)
Signature · 1 line
2.7.4
PATCH 1/2 nopackages.bbclass: improve performance for cleansstate
Leave the task in SSTATETASKS doesn't make sense when it is removed, so also remove it from SSTATETASKS.
This can improve the performance a lot for "bitbake <recipe-native/cross/crosssdk> -ccleansstate" when there are a lot of sstate files.
For example:
* Before
$ bitbake quilt-native -ccleansstate - Check log.do_cleansstate: Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_package.tgz* Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_package_qa.tgz* Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_package_write_rpm.tgz* Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_packagedata.tgz* Removing /sstate-cache/*/sstate:quilt-native::0.65:r0::3:*_populate_lic.tgz* Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_populate_sysroot.tgz*
There are no package tasks for quilt-native, so the first 4 lines doesn't make any sense, but the glob pattern "sstate-cache/*/*" is very time consuming when there are no disk caches. E.g., I have more than 600,000 sstate files:
- Without disk caches
# echo 3 >/proc/sys/vm/drop_caches
$ time python3 -c 'import glob; glob.glob("/sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_package.tgz*")'
real 4m32.583s
user 0m5.768s
sys 0m12.892s- With disk caches (e.g., run it in the second time)
$ time python3 -c 'import glob; glob.glob("/sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_package.tgz*")'
real 0m5.128s
user 0m2.772s
sys 0m2.308sSo the 4 removing *package* commands cost more than 20s or 272s in theory.
* After
$ bitbake quilt-native -ccleansstate - Check log.do_cleansstate: Removing /sstate-cache/*/sstate:quilt-native::0.65:r0::3:*_populate_lic.tgz* Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_populate_sysroot.tgz*
Now the package tasks are gone when cleanssate for native recipes, which saved 20s or 272s in theory.
--- meta/classes/nopackages.bbclass | 35 +++++++++++++++++++++++------------ 1 file changed, 23 insertions(+), 12 deletions(-)
meta/classes/nopackages.bbclass +23 -12
@@ -1,12 +1,23 @@ -deltask do_package -deltask do_package_write_rpm -deltask do_package_write_ipk -deltask do_package_write_deb -deltask do_package_qa -deltask do_packagedata -deltask do_package_setscene -deltask do_package_write_rpm_setscene -deltask do_package_write_ipk_setscene -deltask do_package_write_deb_setscene -deltask do_package_qa_setscene -deltask do_packagedata_setscene +NO_PACKAGES_TASKS = " \ + do_package \ + do_package_write_rpm \ + do_package_write_ipk \ + do_package_write_deb \ + do_package_qa \ + do_packagedata \ +" + +python() { + sstatetasks = (d.getVar('SSTATETASKS') or '').split() + for task in d.getVar('NO_PACKAGES_TASKS').split(): + setscene_task = '%s_setscene' % task + tasks = (task, setscene_task) + for t in tasks: + bb.build.deltask(t, d) + + if task in sstatetasks: + bb.debug(2, 'Removing sstate task %s from SSTATETASKS' % task) + sstatetasks.remove(task) + d.setVar('SSTATETASKS', ' '.join(sstatetasks)) + +}
Signature · 1 line
2.7.4
PATCH 2/2 sstate.bbclass: set SSTATE_EXTRAPATHWILDCARD explicitly
In reply to Robert YangThe glob.glob("/sstate/*/*/") is very time consuming, set SSTATE_EXTRAPATHWILDCARD explicity to avoid that. This can save a lot of time when there are many sstate files.
For example, I have more than 600,000 sstate files:
* Before
- Without disk caches
$ time python3 -c 'import glob; glob.glob("/sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_populate_sysroot.tgz*")'
real 4m32.583s
user 0m5.768s
sys 0m12.892s- With disk caches
$ time python3 -c 'import glob; glob.glob("/sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_populate_sysroot.tgz*")'
real 0m4.111s
user 0m2.348s
sys 0m1.756s* After
$ time python3 -c 'import glob; glob.glob("/sstate-cache.bak/universal/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_populate_sysroot.tgz*")'
- Without disk caches:
real 0m7.928s
user 0m0.172s
sys 0m0.124s- With disk caches:
real 0m0.131s user 0m0.088s sys 0m0.044s
We can see that it saves about 3.8s with disk caches, and saves about 264s without disk caches.
--- meta/classes/sstate.bbclass | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
meta/classes/sstate.bbclass +1 -1
@@ -101,7 +101,7 @@ python () { if bb.data.inherits_class('native', d) or bb.data.inherits_class('crosssdk', d) or bb.data.inherits_class('cross', d): d.setVar('SSTATE_EXTRAPATH', "${NATIVELSBSTRING}/") d.setVar('BB_HASHFILENAME', "True ${SSTATE_PKGSPEC} ${SSTATE_SWSPEC}") - d.setVar('SSTATE_EXTRAPATHWILDCARD', "*/") + d.setVar('SSTATE_EXTRAPATHWILDCARD', "${NATIVELSBSTRING}/") unique_tasks = sorted(set((d.getVar('SSTATETASKS') or "").split())) d.setVar('SSTATETASKS', " ".join(unique_tasks))
Signature · 1 line
2.7.4
Re: [PATCH 1/2] nopackages.bbclass: improve performance for cleansstate
In reply to Robert YangQuoted 108 lines · Robert Yang
Leave the task in SSTATETASKS doesn't make sense when it is removed, so also
remove it from SSTATETASKS.This can improve the performance a lot for "bitbake <recipe- native/cross/crosssdk>
-ccleansstate" when there are a lot of sstate files.For example:
* Before$ bitbake quilt-native -ccleansstate - Check log.do_cleansstate: Removing /sstate-cache/*/*/sstate:quilt-native:x86_64- linux:0.65:r0:x86_64:3:*_package.tgz* Removing /sstate-cache/*/*/sstate:quilt-native:x86_64- linux:0.65:r0:x86_64:3:*_package_qa.tgz* Removing /sstate-cache/*/*/sstate:quilt-native:x86_64- linux:0.65:r0:x86_64:3:*_package_write_rpm.tgz* Removing /sstate-cache/*/*/sstate:quilt-native:x86_64- linux:0.65:r0:x86_64:3:*_packagedata.tgz* Removing /sstate-cache/*/sstate:quilt- native::0.65:r0::3:*_populate_lic.tgz* Removing /sstate-cache/*/*/sstate:quilt-native:x86_64- linux:0.65:r0:x86_64:3:*_populate_sysroot.tgz*There are no package tasks for quilt-native, so the first 4 lines
doesn't
make any sense, but the glob pattern "sstate-cache/*/*" is very
time
consuming when there are no disk caches. E.g., I have more than
600,000
sstate files:
- Without disk caches# echo 3 >/proc/sys/vm/drop_caches $ time python3 -c 'import glob; glob.glob("/sstate- cache/*/*/sstate:quilt-native:x86_64- linux:0.65:r0:x86_64:3:*_package.tgz*")' real 4m32.583s user 0m5.768s sys 0m12.892s- With disk caches (e.g., run it in the second time)
$ time python3 -c 'import glob; glob.glob("/sstate- cache/*/*/sstate:quilt-native:x86_64- linux:0.65:r0:x86_64:3:*_package.tgz*")' real 0m5.128s user 0m2.772s sys 0m2.308sSo the 4 removing *package* commands cost more than 20s or 272s in
theory.* After
$ bitbake quilt-native -ccleansstate - Check log.do_cleansstate: Removing /sstate-cache/*/sstate:quilt- native::0.65:r0::3:*_populate_lic.tgz* Removing /sstate-cache/*/*/sstate:quilt-native:x86_64- linux:0.65:r0:x86_64:3:*_populate_sysroot.tgz*Now the package tasks are gone when cleanssate for native recipes,
which
saved 20s or 272s in theory.Signed-off-by Robert Yang <[email protected]>--- meta/classes/nopackages.bbclass | 35 +++++++++++++++++++++++------ ------ 1 file changed, 23 insertions(+), 12 deletions(-)meta/classes/nopackages.bbclass +19
-12@@ -1,12 +1,23 @@ -deltask do_package -deltask do_package_write_rpm -deltask do_package_write_ipk -deltask do_package_write_deb -deltask do_package_qa -deltask do_packagedata -deltask do_package_setscene -deltask do_package_write_rpm_setscene -deltask do_package_write_ipk_setscene -deltask do_package_write_deb_setscene -deltask do_package_qa_setscene -deltask do_packagedata_setscene +NO_PACKAGES_TASKS = " \ + do_package \ + do_package_write_rpm \ + do_package_write_ipk \ + do_package_write_deb \ + do_package_qa \ + do_packagedata \ +" + +python() { + sstatetasks = (d.getVar('SSTATETASKS') or '').split() + for task in d.getVar('NO_PACKAGES_TASKS').split(): + setscene_task = '%s_setscene' % task + tasks = (task, setscene_task) + for t in tasks: + bb.build.deltask(t, d) + + if task in sstatetasks: + bb.debug(2, 'Removing sstate task %s from SSTATETASKS' %task)
meta/classes/nopackages.bbclass line 23
Quoted 4 lines
meta/classes/nopackages.bbclass +4
-0+ sstatetasks.remove(task) + d.setVar('SSTATETASKS', ' '.join(sstatetasks)) + +}
I really don't want to turn more comparatively readable classes into a mess of anonymous python if we can help it.
Can we not just teach cleansstate to check if a task exists before trying to delete the sstate? It should be possible with d.getVarFlag(taskname, "task")?
Cheers,
Richard
Re: [PATCH 1/2] nopackages.bbclass: improve performance for cleansstate
Sent from mobile phone
Quoted 119 lines · 在 2018年11月22日，20:15，Richard Purdie <[email protected]> 写道：
Quoted 110 lines · Robert Yang
Leave the task in SSTATETASKS doesn't make sense when it is removed, so also
remove it from SSTATETASKS.This can improve the performance a lot for "bitbake <recipe- native/cross/crosssdk>
-ccleansstate" when there are a lot of sstate files.For example:
* Before$ bitbake quilt-native -ccleansstate - Check log.do_cleansstate: Removing /sstate-cache/*/*/sstate:quilt-native:x86_64- linux:0.65:r0:x86_64:3:*_package.tgz* Removing /sstate-cache/*/*/sstate:quilt-native:x86_64- linux:0.65:r0:x86_64:3:*_package_qa.tgz* Removing /sstate-cache/*/*/sstate:quilt-native:x86_64- linux:0.65:r0:x86_64:3:*_package_write_rpm.tgz* Removing /sstate-cache/*/*/sstate:quilt-native:x86_64- linux:0.65:r0:x86_64:3:*_packagedata.tgz* Removing /sstate-cache/*/sstate:quilt- native::0.65:r0::3:*_populate_lic.tgz* Removing /sstate-cache/*/*/sstate:quilt-native:x86_64- linux:0.65:r0:x86_64:3:*_populate_sysroot.tgz*There are no package tasks for quilt-native, so the first 4 lines
doesn't
make any sense, but the glob pattern "sstate-cache/*/*" is very
time
consuming when there are no disk caches. E.g., I have more than
600,000
sstate files:
- Without disk caches# echo 3 >/proc/sys/vm/drop_caches $ time python3 -c 'import glob; glob.glob("/sstate- cache/*/*/sstate:quilt-native:x86_64- linux:0.65:r0:x86_64:3:*_package.tgz*")' real 4m32.583s user 0m5.768s sys 0m12.892s- With disk caches (e.g., run it in the second time)
$ time python3 -c 'import glob; glob.glob("/sstate- cache/*/*/sstate:quilt-native:x86_64- linux:0.65:r0:x86_64:3:*_package.tgz*")' real 0m5.128s user 0m2.772s sys 0m2.308sSo the 4 removing *package* commands cost more than 20s or 272s in
theory.* After
$ bitbake quilt-native -ccleansstate - Check log.do_cleansstate: Removing /sstate-cache/*/sstate:quilt- native::0.65:r0::3:*_populate_lic.tgz* Removing /sstate-cache/*/*/sstate:quilt-native:x86_64- linux:0.65:r0:x86_64:3:*_populate_sysroot.tgz*Now the package tasks are gone when cleanssate for native recipes,
which
saved 20s or 272s in theory.Signed-off-by Robert Yang <[email protected]>---
meta/classes/nopackages.bbclass | 35 +++++++++++++++++++++++------
------1 file changed, 23 insertions(+), 12 deletions(-)meta/classes/nopackages.bbclass +19
-12@@ -1,12 +1,23 @@ -deltask do_package -deltask do_package_write_rpm -deltask do_package_write_ipk -deltask do_package_write_deb -deltask do_package_qa -deltask do_packagedata -deltask do_package_setscene -deltask do_package_write_rpm_setscene -deltask do_package_write_ipk_setscene -deltask do_package_write_deb_setscene -deltask do_package_qa_setscene -deltask do_packagedata_setscene +NO_PACKAGES_TASKS = " \ + do_package \ + do_package_write_rpm \ + do_package_write_ipk \ + do_package_write_deb \ + do_package_qa \ + do_packagedata \ +" + +python() { + sstatetasks = (d.getVar('SSTATETASKS') or '').split() + for task in d.getVar('NO_PACKAGES_TASKS').split(): + setscene_task = '%s_setscene' % task + tasks = (task, setscene_task) + for t in tasks: + bb.build.deltask(t, d) + + if task in sstatetasks: + bb.debug(2, 'Removing sstate task %s from SSTATETASKS' %task)
meta/classes/nopackages.bbclass line 23
Quoted 4 lines
meta/classes/nopackages.bbclass +4
-0+ sstatetasks.remove(task) + d.setVar('SSTATETASKS', ' '.join(sstatetasks)) + +}I really don't want to turn more comparatively readable classes into a mess of anonymous python if we can help it.
Quoted 3 lines
Can we not just teach cleansstate to check if a task exists before trying to delete the sstate? It should be possible with
d.getVarFlag(taskname, "task")?
Thanks, yes, this is my first thought, but I only knew __BBTASKS which is an internal var of bitbake, I will use d.getVarFlag(taskname, "task") to send a v2.
// Robert
Quoted 5 lines
Cheers,
Richard