bitbake.conf: make ZSTD_COMPRESSION_LEVEL a number rather than a dash-prefixed option
PATCH v2 1/3 bitbake.conf: make ZSTD_COMPRESSION_LEVEL a number rather than a dash-prefixed option
From: Alexander Kanavin <[email protected]>
This allows using it where only a number is needed, such as rpm compression setting.
--- meta/conf/bitbake.conf | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
meta/conf/bitbake.conf +2 -2
@@ -881,8 +881,8 @@ XZ_DEFAULTS[vardepsexclude] += "XZ_MEMLIMIT XZ_THREADS" ZSTD_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}" ZSTD_THREADS[vardepvalue] = "1" -ZSTD_COMPRESSION_LEVEL ?= "-3" -ZSTD_DEFAULTS ?= "--threads=${ZSTD_THREADS} ${ZSTD_COMPRESSION_LEVEL}" +ZSTD_COMPRESSION_LEVEL ?= "3" +ZSTD_DEFAULTS ?= "--threads=${ZSTD_THREADS} -${ZSTD_COMPRESSION_LEVEL}" ZSTD_DEFAULTS[vardepsexclude] = "ZSTD_THREADS" # Limit the number of threads that OpenMP libraries will use. Otherwise they
Signature · 1 line
2.39.5
PATCH v2 2/3 package_rpm: use zstd's default compression level
From: Alexander Kanavin <[email protected]>
zstd uses 3 by default (and ZSTD_COMPRESSION_LEVEL is set to that), while 19 is the highest and slowest.
It's not clear why 19 was picked to begin with, possibly I copy-pasted it from rpm's examples without thinking:
https://git.yoctoproject.org/poky/commit/?h=master-next&id=4a4d5f78a6962dda5f63e9891825c80a8a87bf66
This brings significant speedups in rpm's compression step:
for example compressing webkitgtk takes 11s instead of 36s.
The rpm size increases from 175648k to 234860k. I think it's a worthy default tradeoff.
zstd fixup
--- meta/classes-global/package_rpm.bbclass | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
meta/classes-global/package_rpm.bbclass +1 -1
@@ -10,7 +10,7 @@ IMAGE_PKGTYPE ?= "rpm" RPM = "rpm" RPMBUILD = "rpmbuild" -RPMBUILD_COMPMODE ?= "${@'w19T%d.zstdio' % int(d.getVar('ZSTD_THREADS'))}" +RPMBUILD_COMPMODE ?= "${@'w%dT%d.zstdio' % (int(d.getVar('ZSTD_COMPRESSION_LEVEL')), int(d.getVar('ZSTD_THREADS')))}" PKGWRITEDIRRPM = "${WORKDIR}/deploy-rpms"
Signature · 1 line
2.39.5
Re: [OE-core] [PATCH v2 2/3] package_rpm: use zstd's default compression level
Hi Alex,
Quoted 12 lines · Alexander Kanavin via lists.openembedded.org
From: Alexander Kanavin <[email protected]>
zstd uses 3 by default (and
ZSTD_COMPRESSION_LEVELis set to that), while 19 is the highest and slowest.It's not clear why 19 was picked to begin with, possibly I copy-pasted it from rpm's examples without thinking:
https://git.yoctoproject.org/poky/commit/?h=master-next&id=4a4d5f78a6962dda5f63e9891825c80a8a87bf66This brings significant speedups in rpm's compression step:
for example compressing webkitgtk takes 11s instead of 36s.
Quoted 8 lines
The rpm size increases from 175648k to 234860k. I think it's a worthy default tradeoff.
Signed-off-by Alexander Kanavin <[email protected]>zstd fixup
Signed-off-by Alexander Kanavin <[email protected]>
Small oopsie there :)
Looks ok to me otherwise,
Thanks!
Quentin
Re: [OE-core] [PATCH v2 2/3] package_rpm: use zstd's default compression level
In reply to Alexander KanavinQuoted 34 lines · Alexander Kanavin via lists.openembedded.org
From: Alexander Kanavin <[email protected]>
zstd uses 3 by default (and
ZSTD_COMPRESSION_LEVELis set to that), while 19 is the highest and slowest.It's not clear why 19 was picked to begin with, possibly I copy-pasted it from rpm's examples without thinking:
https://git.yoctoproject.org/poky/commit/?h=master-next&id=4a4d5f78a6962dda5f63e9891825c80a8a87bf66This brings significant speedups in rpm's compression step:
for example compressing webkitgtk takes 11s instead of 36s.The rpm size increases from 175648k to 234860k. I think it's a worthy default tradeoff.
Signed-off-by Alexander Kanavin <[email protected]>zstd fixup
Signed-off-by Alexander Kanavin <[email protected]>---
meta/classes-global/package_rpm.bbclass | 2 +-1 file changed, 1 insertion(+), 1 deletion(-)meta/classes-global/package_rpm.bbclass +0
-0@@ -10,7 +10,7 @@ IMAGE_PKGTYPE ?= "rpm"
RPM = "rpm"
RPMBUILD = "rpmbuild"meta/classes-global/package_rpm.bbclass +1
-1-RPMBUILD_COMPMODE ?= "${@'w19T%d.zstdio' % int(d.getVar('ZSTD_THREADS'))}" +RPMBUILD_COMPMODE ?= "${@'w%dT%d.zstdio' % (int(d.getVar('ZSTD_COMPRESSION_LEVEL')), int(d.getVar('ZSTD_THREADS')))}"
Quoted 2 lines
PKGWRITEDIRRPM = "${WORKDIR}/deploy-rpms"
There are some nice steps on the performance charts:
https://valkyrie.yocto.io/pub/non-release/20241130-20/testresults/buildperf-alma8/perf-alma8-vk_master_20241130030029_2a163834b0.html
https://valkyrie.yocto.io/pub/non-release/20241130-17/testresults/buildperf-debian11/perf-debian12-vk_master_20241130040023_2a163834b0.html
I'm trying to trigger some specific builds so we can more precisely point to the change but this is the most likely candidate in that range :)
Nice to see things speed up for a change, thanks!
Cheers,
Richard
PATCH v2 3/3 package_rpm: restrict rpm to 4 threads
In reply to Alexander KanavinFrom: Alexander Kanavin <[email protected]>
TL;DR version:
with this, and the previous compression level changes
I am seeing drastic speedups in package_write_rpm completion times:
webkitgtk goes from 78 seconds to 37 seconds glibc-locale goes from 399 seconds to 58 seconds (!)
The long version:
rpm uses multithreading for two purposes:
- spawning compressors (which are nowadays themselves multi-threaded, so the feature is not as useful as it once was)
- parallel file classification
While the former behaves well on massively parallel CPUs (it was written and verified here :), the latter was then added by upstream and only benchmarked on their very old, slow laptop, apparently:
https://github.com/rpm-software-management/rpm/commit/41f0e214f2266f02d6185ba11f797716de8125d4
On anything more capable it starts showing pathologic behavior, presumably from spawning massive amount of very short-lived threads, and then having to synchronize them. For example classifying glibc-locale takes
5m20s with 256 threads (default on my machine!)
1m49s with 64 threads
59s with 16 threads
48s with 8 threads
Even a more typical recipe like webkitgtk is affected:
47s with 256 threads
32s with 64 threads
27s with 16 or 8 threads
I have found that the optimal amount is actually four: this also means that only four compressors are running at a time, but as they're themselves using threads, and typical recipes are dominated by just two or three large packages, this does not affect overall completion time.
--- meta/classes-global/package_rpm.bbclass | 1 + 1 file changed, 1 insertion(+)
meta/classes-global/package_rpm.bbclass +1 -0
@@ -696,6 +696,7 @@ python do_package_rpm () { cmd = cmd + " --define '_use_internal_dependency_generator 0'" cmd = cmd + " --define '_binaries_in_noarch_packages_terminate_build 0'" cmd = cmd + " --define '_build_id_links none'" + cmd = cmd + " --define '_smp_ncpus_max 4'" cmd = cmd + " --define '_source_payload %s'" % rpmbuild_compmode cmd = cmd + " --define '_binary_payload %s'" % rpmbuild_compmode cmd = cmd + " --define 'clamp_mtime_to_source_date_epoch 1'"
Signature · 1 line
2.39.5
Hi Alex,
Quoted 5 lines · Alexander Kanavin via lists.openembedded.org
From: Alexander Kanavin <[email protected]>
This allows using it where only a number is needed, such as rpm compression setting.
Sounds good to me except now we need to document this change in content format (if someone was modifying it, they need to put a number instead of the number prefixed with a dash) in the migration manual for walnascar, would you mind sending a patch to the docs for that (or at the very least open a bug ticket so it's tracked) once this gets merged?
Looks good to me,
Thanks!
Quentin
Quoted 5 lines · Quentin Schulz
Sounds good to me except now we need to document this change in content format (if someone was modifying it, they need to put a number instead of the number prefixed with a dash) in the migration manual for walnascar, would you mind sending a patch to the docs for that (or at the very least open a bug ticket so it's tracked) once this gets merged?
I just sent a patch to docs@ for that.
Alex