About the sstate cache directory hierarchy
Hi All,
I'm sending out this email because I'm wondering if we can change the sstate cache directory to use ${PN} and taskname as subditories. Hope to hear your opinions.
Below is the long story.
Recently I noticed that running `bitbake xxx -c cleansstate' usually takes more than 10 minutes.
After some investigation, I can see that most of the time is spent on file searching. This is because we have:
SSTATE_PATHSPEC = "${SSTATE_DIR}/${SSTATE_EXTRAPATHWILDCARD}${PN}/${SSTATE_PATH_CURRTASK}/${SSTATE_PKGSPEC}*_${SSTATE_PATH_CURRTASK}.tar.zst*"
And our sstate cache directory's hierarchy uses hash[:2]/hash[2:4]/ as sub-directories.
This essentially means that all sub-directories are searched. This would take a long time, especially when run for the first time. I made some changes to output the time and the logs are as below.
$ bitbake glibc -c cleansstate WARNING: glibc-2.34-r0 do_cleansstate: Removing /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_deploy_source_date_epoch.tar.zst* WARNING: glibc-2.34-r0 do_cleansstate: Took 611.8865714073181 seconds WARNING: glibc-2.34-r0 do_cleansstate: Removing /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_package.tar.zst* WARNING: glibc-2.34-r0 do_cleansstate: Took 1.3219327926635742 seconds WARNING: glibc-2.34-r0 do_cleansstate: Removing /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_package_qa.tar.zst* WARNING: glibc-2.34-r0 do_cleansstate: Took 1.470815658569336 seconds WARNING: glibc-2.34-r0 do_cleansstate: Removing /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_package_write_rpm.tar.zst* WARNING: glibc-2.34-r0 do_cleansstate: Took 1.251939058303833 seconds WARNING: glibc-2.34-r0 do_cleansstate: Removing /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_packagedata.tar.zst* WARNING: glibc-2.34-r0 do_cleansstate: Took 1.2369801998138428 seconds WARNING: glibc-2.34-r0 do_cleansstate: Removing /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc::2.34:r0::7:*_populate_lic.tar.zst* WARNING: glibc-2.34-r0 do_cleansstate: Took 1.1668426990509033 seconds WARNING: glibc-2.34-r0 do_cleansstate: Removing /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_populate_sysroot.tar.zst* WARNING: glibc-2.34-r0 do_cleansstate: Took 1.385568380355835 seconds WARNING: glibc-2.34-r0 do_cleansstate: Removing /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_stash_locale.tar.zst* WARNING: glibc-2.34-r0 do_cleansstate: Took 1.4884181022644043 seconds
I figured that unlike git, we do have knowledge on our sstate objects. It does not seem necessary to use hash value as sub directory. So I changed the sstate directory hierarchy to use ${PN}/taskname/ as sub directories, and here's the result.
$ bitbake libgcc -c cleansstate
WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/deploy_source_date_epoch/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_deploy_source_date_epoch.tar.zst*
WARNING: libgcc-11.2.0-r0 do_cleansstate: Took 0.020630598068237305 seconds
WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/package/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_package.tar.zst*
WARNING: libgcc-11.2.0-r0 do_cleansstate: Took 0.0011608600616455078 seconds
WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/package_qa/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_package_qa.tar.zst*
WARNING: libgcc-11.2.0-r0 do_cleansstate: Took 0.0007557868957519531 seconds
WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/package_write_rpm/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_package_write_rpm.tar.zst*
WARNING: libgcc-11.2.0-r0 do_cleansstate: Took 0.0013995170593261719 seconds
WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/packagedata/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_packagedata.tar.zst*
WARNING: libgcc-11.2.0-r0 do_cleansstate: Took 0.0007488727569580078 seconds
WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/populate_lic/sstate:libgcc::11.2.0:r0::7:*_populate_lic.tar.zst*
WARNING: libgcc-11.2.0-r0 do_cleansstate: Took 0.0005896091461181641 seconds
WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/populate_sysroot/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_populate_sysroot.tar.zst*
WARNING: libgcc-11.2.0-r0 do_cleansstate: Took 0.00080108642578125 seconds
It's much faster.
In addition, the sub dirs now give more info, which should potentially make sstate cache easier to manage.
Attached is the patch to quickly try things out. Hope to hear your opinions.
Best Regards,
Chen Qi
From 6f148d399c2ac575a9079eae682a56802dfac3ca Mon Sep 17 00:00:00 2001
From: Chen Qi <[email protected]>
Date: Thu, 27 Jan 2022 22:57:57 -0800
Subject: [PATCH] sstate.bbclasss: use /taskname as sub dirs
--- meta/classes/sstate.bbclass | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-)
meta/classes/sstate.bbclass +8 -3
@@ -17,6 +17,7 @@ def generate_sstatefn(spec, hash, taskname, siginfo, d): if not hash: hash = "INVALID" fn = spec + hash + "_" + taskname + extension + pn = spec.split(':')[1] # If the filename is too long, attempt to reduce it if len(fn) > limit: components = spec.split(":")
@@ -30,7 +31,7 @@ def generate_sstatefn(spec, hash, taskname, siginfo, d): fn = spec + hash + "_" + taskname + extension if len(fn) > limit: bb.fatal("Unable to reduce sstate name to less than 255 chararacters") - return hash[:2] + "/" + hash[2:4] + "/" + fn + return pn + "/" + taskname + "/" + fn SSTATE_PKGARCH = "${PACKAGE_ARCH}" SSTATE_PKGSPEC = "sstate:${PN}:${PACKAGE_ARCH}${TARGET_VENDOR}-${TARGET_OS}:${PV}:${PR}:${SSTATE_PKGARCH}:${SSTATE_VERSION}:"
@@ -39,7 +40,7 @@ SSTATE_PKGNAME = "${SSTATE_EXTRAPATH}${@generate_sstatefn(d.getVar('SSTATE_PK SSTATE_PKG = "${SSTATE_DIR}/${SSTATE_PKGNAME}" SSTATE_EXTRAPATH = "" SSTATE_EXTRAPATHWILDCARD = "" -SSTATE_PATHSPEC = "${SSTATE_DIR}/${SSTATE_EXTRAPATHWILDCARD}*/*/${SSTATE_PKGSPEC}*_${SSTATE_PATH_CURRTASK}.tar.zst*" +SSTATE_PATHSPEC = "${SSTATE_DIR}/${SSTATE_EXTRAPATHWILDCARD}${PN}/${SSTATE_PATH_CURRTASK}/${SSTATE_PKGSPEC}*_${SSTATE_PATH_CURRTASK}.tar.zst*" # explicitly make PV to depend on evaluated value of PV variable PV[vardepvalue] = "${PV}"
@@ -482,12 +483,16 @@ python sstate_hardcode_path_unpack () { def sstate_clean_cachefile(ss, d): import oe.path + import time if d.getVarFlag('do_%s' % ss['task'], 'task'): d.setVar("SSTATE_PATH_CURRTASK", ss['task']) sstatepkgfile = d.getVar('SSTATE_PATHSPEC') - bb.note("Removing %s" % sstatepkgfile) + bb.warn("Removing %s" % sstatepkgfile) + start = time.time() oe.path.remove(sstatepkgfile) + end = time.time() + bb.warn("Took %s seconds" % (end-start)) def sstate_clean_cachefiles(d): for task in (d.getVar('SSTATETASKS') or "").split():
Signature · 1 line
2.33.0
I do like the idea; not everyone has a pcie gen4/5 ssd for builds, or rigorously trims sstate on a schedule. But there may be consequences or regressions, maybe RP will immediately shoot it down :)
I would however still place a single level of hash[:2] *under* the pn/task/, to avoid too many files piling up in a single directory.
Can you send this as an RFC patch?
Alex
Quoted 124 lines · Chen Qi
Hi All,
I'm sending out this email because I'm wondering if we can change the sstate cache directory to use ${PN} and taskname as subditories. Hope to hear your opinions.
Below is the long story.
Recently I noticed that running `bitbake xxx -c cleansstate' usually takes more than 10 minutes.
After some investigation, I can see that most of the time is spent on file searching. This is because we have:
SSTATE_PATHSPEC="${SSTATE_DIR}/${SSTATE_EXTRAPATHWILDCARD}${PN}/${SSTATE_PATH_CURRTASK}/${SSTATE_PKGSPEC}*_${SSTATE_PATH_CURRTASK}.tar.zst*"
And our sstate cache directory's hierarchy uses hash[:2]/hash[2:4]/ as sub-directories.
This essentially means that all sub-directories are searched. This would take a long time, especially when run for the first time. I made some changes to output the time and the logs are as below.
$ bitbake glibc -c cleansstate WARNING: glibc-2.34-r0 do_cleansstate: Removing/ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_deploy_source_date_epoch.tar.zst*
WARNING: glibc-2.34-r0do_cleansstate: Took 611.8865714073181 seconds
WARNING: glibc-2.34-r0do_cleansstate: Removing/ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_package.tar.zst*
WARNING: glibc-2.34-r0do_cleansstate: Took 1.3219327926635742 seconds
WARNING: glibc-2.34-r0do_cleansstate: Removing/ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_package_qa.tar.zst*
WARNING: glibc-2.34-r0do_cleansstate: Took 1.470815658569336 seconds
WARNING: glibc-2.34-r0do_cleansstate: Removing/ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_package_write_rpm.tar.zst*
WARNING: glibc-2.34-r0do_cleansstate: Took 1.251939058303833 seconds
WARNING: glibc-2.34-r0do_cleansstate: Removing/ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_packagedata.tar.zst*
WARNING: glibc-2.34-r0do_cleansstate: Took 1.2369801998138428 seconds
WARNING: glibc-2.34-r0do_cleansstate: Removing/ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc::2.34:r0::7:*_populate_lic.tar.zst*
WARNING: glibc-2.34-r0do_cleansstate: Took 1.1668426990509033 seconds
WARNING: glibc-2.34-r0do_cleansstate: Removing/ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_populate_sysroot.tar.zst*
WARNING: glibc-2.34-r0do_cleansstate: Took 1.385568380355835 seconds
WARNING: glibc-2.34-r0do_cleansstate: Removing/ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_stash_locale.tar.zst*
WARNING: glibc-2.34-r0do_cleansstate: Took 1.4884181022644043 secondsI figured that unlike git, we do have knowledge on our sstate objects. It does not seem necessary to use hash value as sub directory. So I changed the sstate directory hierarchy to use ${PN}/taskname/ as sub directories, and here's the result.
$ bitbake libgcc -c cleansstateWARNING: libgcc-11.2.0-r0
do_cleansstate: Removing/ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/deploy_source_date_epoch/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_deploy_source_date_epoch.tar.zst*
WARNING: libgcc-11.2.0-r0do_cleansstate: Took 0.020630598068237305 seconds
WARNING: libgcc-11.2.0-r0do_cleansstate: Removing/ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/package/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_package.tar.zst*
WARNING: libgcc-11.2.0-r0do_cleansstate: Took 0.0011608600616455078
seconds
WARNING: libgcc-11.2.0-r0do_cleansstate: Removing/ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/package_qa/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_package_qa.tar.zst*
WARNING: libgcc-11.2.0-r0do_cleansstate: Took 0.0007557868957519531
seconds
WARNING: libgcc-11.2.0-r0do_cleansstate: Removing/ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/package_write_rpm/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_package_write_rpm.tar.zst*
WARNING: libgcc-11.2.0-r0do_cleansstate: Took 0.0013995170593261719
seconds
WARNING: libgcc-11.2.0-r0do_cleansstate: Removing/ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/packagedata/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_packagedata.tar.zst*
WARNING: libgcc-11.2.0-r0do_cleansstate: Took 0.0007488727569580078
seconds
WARNING: libgcc-11.2.0-r0do_cleansstate: Removing/ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/populate_lic/sstate:libgcc::11.2.0:r0::7:*_populate_lic.tar.zst*
WARNING: libgcc-11.2.0-r0do_cleansstate: Took 0.0005896091461181641
seconds
WARNING: libgcc-11.2.0-r0do_cleansstate: Removing/ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/populate_sysroot/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_populate_sysroot.tar.zst*
WARNING: libgcc-11.2.0-r0do_cleansstate: Took 0.00080108642578125 secondsIt's much faster.
In addition, the sub dirs now give more info, which should potentially make sstate cache easier to manage.
Attached is the patch to quickly try things out. Hope to hear your opinions.
Best Regards,
Chen Qi
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#161067):
https://lists.openembedded.org/g/openembedded-core/message/161067
Mute This Topic: https://lists.openembedded.org/mt/88740516/1686489
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ [email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Quoted 7 lines · Alexander Kanavin
I do like the idea; not everyone has a pcie gen4/5 ssd for builds, or rigorously trims sstate on a schedule. But there may be consequences or regressions, maybe RP will immediately shoot it down :)
I would however still place a single level of hash[:2] *under* the pn/task/, to avoid too many files piling up in a single directory.
Can you send this as an RFC patch?
Alex
Yes. I will ensure that all oe selftests pass before sending out patch.
Regards,
Qi
Quoted 123 lines · On Fri, 28 Jan 2022 at 08:06, Chen Qi <[email protected]…
Hi All,
I'm sending out this email because I'm wondering if we can change the
sstate cache directory to use ${PN} and taskname as subditories.
Hope to
hear your opinions.
Below is the long story.
Recently I noticed that running `bitbake xxx -c cleansstate' usually
takes more than 10 minutes.
After some investigation, I can see that most of the time is spent on
file searching. This is because we have:
SSTATE_PATHSPEC =
"${SSTATE_DIR}/${SSTATE_EXTRAPATHWILDCARD}${PN}/${SSTATE_PATH_CURRTASK}/${SSTATE_PKGSPEC}*_${SSTATE_PATH_CURRTASK}.tar.zst*"
And our sstate cache directory's hierarchy uses
hash[:2]/hash[2:4]/ as
sub-directories.
This essentially means that all sub-directories are searched. This
would
take a long time, especially when run for the first time. I made some
changes to output the time and the logs are as below.
$ bitbake glibc -c cleansstate
WARNING: glibc-2.34-r0 do_cleansstate: Removing
/ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_deploy_source_date_epoch.tar.zst*
WARNING: glibc-2.34-r0 do_cleansstate: Took 611.8865714073181 seconds
WARNING: glibc-2.34-r0 do_cleansstate: Removing
/ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_package.tar.zst*
WARNING: glibc-2.34-r0 do_cleansstate: Took 1.3219327926635742 seconds
WARNING: glibc-2.34-r0 do_cleansstate: Removing
/ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_package_qa.tar.zst*
WARNING: glibc-2.34-r0 do_cleansstate: Took 1.470815658569336 seconds
WARNING: glibc-2.34-r0 do_cleansstate: Removing
/ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_package_write_rpm.tar.zst*
WARNING: glibc-2.34-r0 do_cleansstate: Took 1.251939058303833 seconds
WARNING: glibc-2.34-r0 do_cleansstate: Removing
/ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_packagedata.tar.zst*
WARNING: glibc-2.34-r0 do_cleansstate: Took 1.2369801998138428 seconds
WARNING: glibc-2.34-r0 do_cleansstate: Removing
/ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc::2.34:r0::7:*_populate_lic.tar.zst*
WARNING: glibc-2.34-r0 do_cleansstate: Took 1.1668426990509033 seconds
WARNING: glibc-2.34-r0 do_cleansstate: Removing
/ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_populate_sysroot.tar.zst*
WARNING: glibc-2.34-r0 do_cleansstate: Took 1.385568380355835 seconds
WARNING: glibc-2.34-r0 do_cleansstate: Removing
/ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_stash_locale.tar.zst*
WARNING: glibc-2.34-r0 do_cleansstate: Took 1.4884181022644043 seconds
I figured that unlike git, we do have knowledge on our sstate
objects.
It does not seem necessary to use hash value as sub directory. So I
changed the sstate directory hierarchy to use ${PN}/taskname/ as sub
directories, and here's the result.
$ bitbake libgcc -c cleansstate
WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing
/ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/deploy_source_date_epoch/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_deploy_source_date_epoch.tar.zst*
WARNING: libgcc-11.2.0-r0 do_cleansstate: Took
0.020630598068237305 seconds
WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing
/ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/package/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_package.tar.zst*
WARNING: libgcc-11.2.0-r0 do_cleansstate: Took
0.0011608600616455078 seconds
WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing
/ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/package_qa/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_package_qa.tar.zst*
WARNING: libgcc-11.2.0-r0 do_cleansstate: Took
0.0007557868957519531 seconds
WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing
/ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/package_write_rpm/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_package_write_rpm.tar.zst*
WARNING: libgcc-11.2.0-r0 do_cleansstate: Took
0.0013995170593261719 seconds
WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing
/ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/packagedata/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_packagedata.tar.zst*
WARNING: libgcc-11.2.0-r0 do_cleansstate: Took
0.0007488727569580078 seconds
WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing
/ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/populate_lic/sstate:libgcc::11.2.0:r0::7:*_populate_lic.tar.zst*
WARNING: libgcc-11.2.0-r0 do_cleansstate: Took
0.0005896091461181641 seconds
WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing
/ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/populate_sysroot/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_populate_sysroot.tar.zst*
WARNING: libgcc-11.2.0-r0 do_cleansstate: Took 0.00080108642578125
seconds
It's much faster.
In addition, the sub dirs now give more info, which should
potentially
make sstate cache easier to manage.
Attached is the patch to quickly try things out. Hope to hear your
opinions.
Best Regards,
Chen Qi
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#161067):
https://lists.openembedded.org/g/openembedded-core/message/161067
<https://urldefense.com/v3/__https://lists.openembedded.org/g/openembedded-core/message/161067__;!!AjveYdw8EvQ!N5Dyvu4eenTGK1TY1BrreaZ7uMlcahVqvjTbg01fXC5Sptlu1wS-nKbk5PGtw6ODQA$>
Mute This Topic:
https://lists.openembedded.org/mt/88740516/1686489
<https://urldefense.com/v3/__https://lists.openembedded.org/mt/88740516/1686489__;!!AjveYdw8EvQ!N5Dyvu4eenTGK1TY1BrreaZ7uMlcahVqvjTbg01fXC5Sptlu1wS-nKbk5PGWaXA87A$>
Group Owner: [email protected]
<mailto:openembedded-core%[email protected]>
Unsubscribe:
https://lists.openembedded.org/g/openembedded-core/unsub
<https://urldefense.com/v3/__https://lists.openembedded.org/g/openembedded-core/unsub__;!!AjveYdw8EvQ!N5Dyvu4eenTGK1TY1BrreaZ7uMlcahVqvjTbg01fXC5Sptlu1wS-nKbk5PEkTurZng$>
[[email protected] <mailto:[email protected]>]
-=-=-=-=-=-=-=-=-=-=-=-Quoted 71 lines · Chen Qi
Hi All,
I'm sending out this email because I'm wondering if we can change the sstate cache directory to use ${PN} and taskname as subditories. Hope to hear your opinions.
Below is the long story.
Recently I noticed that running `bitbake xxx -c cleansstate' usually takes more than 10 minutes.
After some investigation, I can see that most of the time is spent on file searching. This is because we have:
SSTATE_PATHSPEC = "${SSTATE_DIR}/${SSTATE_EXTRAPATHWILDCARD}${PN}/${SSTATE_PATH_CURRTASK}/${SSTATE_PKGSPEC}*_${SSTATE_PATH_CURRTASK}.tar.zst*"
And our sstate cache directory's hierarchy uses hash[:2]/hash[2:4]/ as sub-directories.
This essentially means that all sub-directories are searched. This would take a long time, especially when run for the first time. I made some changes to output the time and the logs are as below.
$ bitbake glibc -c cleansstate WARNING: glibc-2.34-r0 do_cleansstate: Removing /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_deploy_source_date_epoch.tar.zst*WARNING: glibc-2.34-r0
do_cleansstate: Took 611.8865714073181 seconds
WARNING: glibc-2.34-r0do_cleansstate: Removing /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_package.tar.zst*WARNING: glibc-2.34-r0
do_cleansstate: Took 1.3219327926635742 seconds
WARNING: glibc-2.34-r0do_cleansstate: Removing /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_package_qa.tar.zst*WARNING: glibc-2.34-r0
do_cleansstate: Took 1.470815658569336 seconds
WARNING: glibc-2.34-r0do_cleansstate: Removing /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_package_write_rpm.tar.zst*WARNING: glibc-2.34-r0
do_cleansstate: Took 1.251939058303833 seconds
WARNING: glibc-2.34-r0do_cleansstate: Removing /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_packagedata.tar.zst*WARNING: glibc-2.34-r0
do_cleansstate: Took 1.2369801998138428 seconds
WARNING: glibc-2.34-r0do_cleansstate: Removing /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc::2.34:r0::7:*_populate_lic.tar.zst*WARNING: glibc-2.34-r0
do_cleansstate: Took 1.1668426990509033 seconds
WARNING: glibc-2.34-r0do_cleansstate: Removing /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_populate_sysroot.tar.zst*WARNING: glibc-2.34-r0
do_cleansstate: Took 1.385568380355835 seconds
WARNING: glibc-2.34-r0do_cleansstate: Removing /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_stash_locale.tar.zst*WARNING: glibc-2.34-r0
do_cleansstate: Took 1.4884181022644043 secondsI figured that unlike git, we do have knowledge on our sstate objects. It does not seem necessary to use hash value as sub directory. So I changed the sstate directory hierarchy to use ${PN}/taskname/ as sub directories, and here's the result.
$ bitbake libgcc -c cleansstateWARNING: libgcc-11.2.0-r0
do_cleansstate: Removing /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/deploy_source_date_epoch/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_deploy_source_date_epoch.tar.zst*WARNING: libgcc-11.2.0-r0
do_cleansstate: Took 0.020630598068237305 seconds
WARNING: libgcc-11.2.0-r0do_cleansstate: Removing /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/package/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_package.tar.zst*WARNING: libgcc-11.2.0-r0
do_cleansstate: Took 0.0011608600616455078 seconds
WARNING: libgcc-11.2.0-r0do_cleansstate: Removing /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/package_qa/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_package_qa.tar.zst*WARNING: libgcc-11.2.0-r0
do_cleansstate: Took 0.0007557868957519531 seconds
WARNING: libgcc-11.2.0-r0do_cleansstate: Removing /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/package_write_rpm/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_package_write_rpm.tar.zst*WARNING: libgcc-11.2.0-r0
do_cleansstate: Took 0.0013995170593261719 seconds
WARNING: libgcc-11.2.0-r0do_cleansstate: Removing /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/packagedata/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_packagedata.tar.zst*WARNING: libgcc-11.2.0-r0
do_cleansstate: Took 0.0007488727569580078 seconds
WARNING: libgcc-11.2.0-r0do_cleansstate: Removing /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/populate_lic/sstate:libgcc::11.2.0:r0::7:*_populate_lic.tar.zst*WARNING: libgcc-11.2.0-r0
do_cleansstate: Took 0.0005896091461181641 seconds
WARNING: libgcc-11.2.0-r0do_cleansstate: Removing /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/populate_sysroot/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_populate_sysroot.tar.zst*WARNING: libgcc-11.2.0-r0
do_cleansstate: Took 0.00080108642578125 secondsIt's much faster.
Quoted 8 lines
In addition, the sub dirs now give more info, which should potentially make sstate cache easier to manage.
Attached is the patch to quickly try things out. Hope to hear your opinions.
Best Regards,
Chen Qi
I thought this sounded interesting. Has anything come out of this that I missed ?
Jacob
Hi Jacob,
No. I haven't done anything since then, because I've been involved in something else.
Regards,
Qi
-----Original Message-----
From: Jacob Kroon <[email protected]>
Sent: Monday, May 16, 2022 1:30 PM
To: Chen, Qi <[email protected]>; [email protected]; Richard Purdie <[email protected]>
Subject: Re: [OE-core] About the sstate cache directory hierarchy
Quoted 128 lines · Chen Qi
Hi All,
I'm sending out this email because I'm wondering if we can change the sstate cache directory to use ${PN} and taskname as subditories. Hope to hear your opinions.
Below is the long story.
Recently I noticed that running `bitbake xxx -c cleansstate' usually takes more than 10 minutes.
After some investigation, I can see that most of the time is spent on file searching. This is because we have:
SSTATE_PATHSPEC = "${SSTATE_DIR}/${SSTATE_EXTRAPATHWILDCARD}${PN}/${SSTATE_PATH_CURRTASK}/${SSTATE_PKGSPEC}*_${SSTATE_PATH_CURRTASK}.tar.zst*"
And our sstate cache directory's hierarchy uses hash[:2]/hash[2:4]/ as sub-directories.
This essentially means that all sub-directories are searched. This would take a long time, especially when run for the first time. I made some changes to output the time and the logs are as below.
$ bitbake glibc -c cleansstate WARNING: glibc-2.34-r0 do_cleansstate: Removing /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:cor e2-64-poky-linux:2.34:r0:core2-64:7:*_deploy_source_date_epoch.tar.zst *WARNING: glibc-2.34-r0
do_cleansstate: Took 611.8865714073181 seconds
WARNING: glibc-2.34-r0do_cleansstate: Removing /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:cor e2-64-poky-linux:2.34:r0:core2-64:7:*_package.tar.zst*WARNING: glibc-2.34-r0
do_cleansstate: Took 1.3219327926635742 seconds
WARNING: glibc-2.34-r0do_cleansstate: Removing /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:cor e2-64-poky-linux:2.34:r0:core2-64:7:*_package_qa.tar.zst*WARNING: glibc-2.34-r0
do_cleansstate: Took 1.470815658569336 seconds
WARNING: glibc-2.34-r0do_cleansstate: Removing /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:cor e2-64-poky-linux:2.34:r0:core2-64:7:*_package_write_rpm.tar.zst*WARNING: glibc-2.34-r0
do_cleansstate: Took 1.251939058303833 seconds
WARNING: glibc-2.34-r0do_cleansstate: Removing /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:cor e2-64-poky-linux:2.34:r0:core2-64:7:*_packagedata.tar.zst*WARNING: glibc-2.34-r0
do_cleansstate: Took 1.2369801998138428 seconds
WARNING: glibc-2.34-r0do_cleansstate: Removing /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc::2. 34:r0::7:*_populate_lic.tar.zst*WARNING: glibc-2.34-r0
do_cleansstate: Took 1.1668426990509033 seconds
WARNING: glibc-2.34-r0do_cleansstate: Removing /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:cor e2-64-poky-linux:2.34:r0:core2-64:7:*_populate_sysroot.tar.zst*WARNING: glibc-2.34-r0
do_cleansstate: Took 1.385568380355835 seconds
WARNING: glibc-2.34-r0do_cleansstate: Removing /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:cor e2-64-poky-linux:2.34:r0:core2-64:7:*_stash_locale.tar.zst*WARNING: glibc-2.34-r0
do_cleansstate: Took 1.4884181022644043 secondsI figured that unlike git, we do have knowledge on our sstate objects. It does not seem necessary to use hash value as sub directory. So I changed the sstate directory hierarchy to use ${PN}/taskname/ as sub directories, and here's the result.
$ bitbake libgcc -c cleansstateWARNING: libgcc-11.2.0-r0
do_cleansstate: Removing /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/deploy_source _date_epoch/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_d eploy_source_date_epoch.tar.zst*WARNING: libgcc-11.2.0-r0
do_cleansstate: Took 0.020630598068237305 seconds
WARNING: libgcc-11.2.0-r0do_cleansstate: Removing /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/package/sstat e:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_package.tar.zst*WARNING: libgcc-11.2.0-r0
do_cleansstate: Took 0.0011608600616455078 seconds
WARNING: libgcc-11.2.0-r0do_cleansstate: Removing /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/package_qa/ss tate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_package_qa.tar. zst*WARNING: libgcc-11.2.0-r0
do_cleansstate: Took 0.0007557868957519531 seconds
WARNING: libgcc-11.2.0-r0do_cleansstate: Removing /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/package_write _rpm/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_package_ write_rpm.tar.zst*WARNING: libgcc-11.2.0-r0
do_cleansstate: Took 0.0013995170593261719 seconds
WARNING: libgcc-11.2.0-r0do_cleansstate: Removing /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/packagedata/s state:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_packagedata.ta r.zst*WARNING: libgcc-11.2.0-r0
do_cleansstate: Took 0.0007488727569580078 seconds
WARNING: libgcc-11.2.0-r0do_cleansstate: Removing /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/populate_lic/ sstate:libgcc::11.2.0:r0::7:*_populate_lic.tar.zst*WARNING: libgcc-11.2.0-r0
do_cleansstate: Took 0.0005896091461181641 seconds
WARNING: libgcc-11.2.0-r0do_cleansstate: Removing /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/populate_sysr oot/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_populate_ sysroot.tar.zst*WARNING: libgcc-11.2.0-r0
do_cleansstate: Took 0.00080108642578125 secondsIt's much faster.
In addition, the sub dirs now give more info, which should potentially make sstate cache easier to manage.
Quoted 7 lines
Attached is the patch to quickly try things out. Hope to hear your opinions.
Best Regards,
Chen Qi
I thought this sounded interesting. Has anything come out of this that I missed ?
Jacob