Skip to contentSkip to conversation

bitbake.conf: make ZSTD_COMPRESSION_LEVEL a number rather than a dash-prefixed option

PATCH In review Alexander Kanavin 3 participants 7 messages Nov 20, 2024 – Nov 30, 2024

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/3 bitbake.conf: make ZSTD_COMPRESSION_LEVEL a number rather than a dash-prefixed option

From: Alexander Kanavin <[email protected]>

This allows using it where only a number is needed, such as rpm compression setting.

---
 meta/conf/bitbake.conf | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
meta/conf/bitbake.conf +2 -2
index 02bbf0e7a52..b8b321dd1b7 100644
@@ -881,8 +881,8 @@ XZ_DEFAULTS[vardepsexclude] += "XZ_MEMLIMIT XZ_THREADS"
 ZSTD_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}"
 ZSTD_THREADS[vardepvalue] = "1"
 
-ZSTD_COMPRESSION_LEVEL ?= "-3"
-ZSTD_DEFAULTS ?= "--threads=${ZSTD_THREADS} ${ZSTD_COMPRESSION_LEVEL}"
+ZSTD_COMPRESSION_LEVEL ?= "3"
+ZSTD_DEFAULTS ?= "--threads=${ZSTD_THREADS} -${ZSTD_COMPRESSION_LEVEL}"
 ZSTD_DEFAULTS[vardepsexclude] = "ZSTD_THREADS"
 
 # Limit the number of threads that OpenMP libraries will use. Otherwise they
Signature · 1 line
2.39.5
#1 lore raw

PATCH v2 2/3 package_rpm: use zstd's default compression level

From: Alexander Kanavin <[email protected]>

zstd uses 3 by default (and ZSTD_COMPRESSION_LEVEL is set to that), while 19 is the highest and slowest.

It's not clear why 19 was picked to begin with, possibly I copy-pasted it from rpm's examples without thinking:
https://git.yoctoproject.org/poky/commit/?h=master-next&id=4a4d5f78a6962dda5f63e9891825c80a8a87bf66

This brings significant speedups in rpm's compression step:
for example compressing webkitgtk takes 11s instead of 36s.

The rpm size increases from 175648k to 234860k. I think it's a worthy default tradeoff.

zstd fixup

---
 meta/classes-global/package_rpm.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/classes-global/package_rpm.bbclass +1 -1
index ddc4bf3a6a5..fab8f568781 100644
@@ -10,7 +10,7 @@ IMAGE_PKGTYPE ?= "rpm"
 
 RPM = "rpm"
 RPMBUILD = "rpmbuild"
-RPMBUILD_COMPMODE ?= "${@'w19T%d.zstdio' % int(d.getVar('ZSTD_THREADS'))}"
+RPMBUILD_COMPMODE ?= "${@'w%dT%d.zstdio' % (int(d.getVar('ZSTD_COMPRESSION_LEVEL')), int(d.getVar('ZSTD_THREADS')))}"
 
 PKGWRITEDIRRPM = "${WORKDIR}/deploy-rpms"
 
Signature · 1 line
2.39.5
Reviewed-by on v2 2/3
#2 lore raw

Re: [OE-core] [PATCH v2 2/3] package_rpm: use zstd's default compression level

Hi Alex,

Quoted 12 lines · Alexander Kanavin via lists.openembedded.org

From: Alexander Kanavin <[email protected]>

zstd uses 3 by default (and ZSTD_COMPRESSION_LEVEL is set to that), while 19 is the highest and slowest.

It's not clear why 19 was picked to begin with, possibly I copy-pasted it from rpm's examples without thinking:
https://git.yoctoproject.org/poky/commit/?h=master-next&id=4a4d5f78a6962dda5f63e9891825c80a8a87bf66

This brings significant speedups in rpm's compression step:
for example compressing webkitgtk takes 11s instead of 36s.

Quoted 8 lines

The rpm size increases from 175648k to 234860k. I think it's a worthy default tradeoff.

zstd fixup

Small oopsie there :)

Looks ok to me otherwise,

Thanks!
Quentin

#3 lore raw

Re: [OE-core] [PATCH v2 2/3] package_rpm: use zstd's default compression level

In reply to Alexander Kanavin
Quoted 34 lines · Alexander Kanavin via lists.openembedded.org

From: Alexander Kanavin <[email protected]>

zstd uses 3 by default (and ZSTD_COMPRESSION_LEVEL is set to that), while 19 is the highest and slowest.

It's not clear why 19 was picked to begin with, possibly I copy-pasted it from rpm's examples without thinking:
https://git.yoctoproject.org/poky/commit/?h=master-next&id=4a4d5f78a6962dda5f63e9891825c80a8a87bf66

This brings significant speedups in rpm's compression step:
for example compressing webkitgtk takes 11s instead of 36s.

The rpm size increases from 175648k to 234860k. I think it's a worthy default tradeoff.

zstd fixup

---
 meta/classes-global/package_rpm.bbclass | 2 +-

 1 file changed, 1 insertion(+), 1 deletion(-)
meta/classes-global/package_rpm.bbclass +0 -0
index ddc4bf3a6a5..fab8f568781 100644
@@ -10,7 +10,7 @@ IMAGE_PKGTYPE ?= "rpm"


 RPM = "rpm"
 RPMBUILD = "rpmbuild"

meta/classes-global/package_rpm.bbclass +1 -1
-RPMBUILD_COMPMODE ?= "${@'w19T%d.zstdio' % int(d.getVar('ZSTD_THREADS'))}"
+RPMBUILD_COMPMODE ?= "${@'w%dT%d.zstdio' % (int(d.getVar('ZSTD_COMPRESSION_LEVEL')), int(d.getVar('ZSTD_THREADS')))}"
Quoted 2 lines


 PKGWRITEDIRRPM = "${WORKDIR}/deploy-rpms"

There are some nice steps on the performance charts:

https://valkyrie.yocto.io/pub/non-release/20241130-20/testresults/buildperf-alma8/perf-alma8-vk_master_20241130030029_2a163834b0.html
https://valkyrie.yocto.io/pub/non-release/20241130-17/testresults/buildperf-debian11/perf-debian12-vk_master_20241130040023_2a163834b0.html

I'm trying to trigger some specific builds so we can more precisely point to the change but this is the most likely candidate in that range :)

Nice to see things speed up for a change, thanks!

Cheers,

Richard

#4 lore raw

PATCH v2 3/3 package_rpm: restrict rpm to 4 threads

In reply to Alexander Kanavin

From: Alexander Kanavin <[email protected]>

TL;DR version:

with this, and the previous compression level changes
I am seeing drastic speedups in package_write_rpm completion times:

webkitgtk goes from 78 seconds to 37 seconds glibc-locale goes from 399 seconds to 58 seconds (!)

The long version:

rpm uses multithreading for two purposes:

- spawning compressors (which are nowadays themselves multi-threaded, so the feature is not as useful as it once was)
- parallel file classification

While the former behaves well on massively parallel CPUs (it was written and verified here :), the latter was then added by upstream and only benchmarked on their very old, slow laptop, apparently:
https://github.com/rpm-software-management/rpm/commit/41f0e214f2266f02d6185ba11f797716de8125d4

On anything more capable it starts showing pathologic behavior, presumably from spawning massive amount of very short-lived threads, and then having to synchronize them. For example classifying glibc-locale takes
5m20s with 256 threads (default on my machine!)
1m49s with 64 threads
59s with 16 threads
48s with 8 threads

Even a more typical recipe like webkitgtk is affected:
47s with 256 threads
32s with 64 threads
27s with 16 or 8 threads

I have found that the optimal amount is actually four: this also means that only four compressors are running at a time, but as they're themselves using threads, and typical recipes are dominated by just two or three large packages, this does not affect overall completion time.

---
 meta/classes-global/package_rpm.bbclass | 1 +
 1 file changed, 1 insertion(+)
meta/classes-global/package_rpm.bbclass +1 -0
index fab8f568781..a3be823c9d8 100644
@@ -696,6 +696,7 @@ python do_package_rpm () {
     cmd = cmd + " --define '_use_internal_dependency_generator 0'"
     cmd = cmd + " --define '_binaries_in_noarch_packages_terminate_build 0'"
     cmd = cmd + " --define '_build_id_links none'"
+    cmd = cmd + " --define '_smp_ncpus_max 4'"
     cmd = cmd + " --define '_source_payload %s'" % rpmbuild_compmode
     cmd = cmd + " --define '_binary_payload %s'" % rpmbuild_compmode
     cmd = cmd + " --define 'clamp_mtime_to_source_date_epoch 1'"
Signature · 1 line
2.39.5
Reviewed-by on v2 1/3
#5 lore raw
In reply to Alexander Kanavin

Hi Alex,

Quoted 5 lines · Alexander Kanavin via lists.openembedded.org

From: Alexander Kanavin <[email protected]>

This allows using it where only a number is needed, such as rpm compression setting.

Sounds good to me except now we need to document this change in content format (if someone was modifying it, they need to put a number instead of the number prefixed with a dash) in the migration manual for walnascar, would you mind sending a patch to the docs for that (or at the very least open a bug ticket so it's tracked) once this gets merged?

Looks good to me,

Thanks!
Quentin

#6 lore raw
Quoted 5 lines · Quentin Schulz

Sounds good to me except now we need to document this change in content format (if someone was modifying it, they need to put a number instead of the number prefixed with a dash) in the migration manual for walnascar, would you mind sending a patch to the docs for that (or at the very least open a bug ticket so it's tracked) once this gets merged?

I just sent a patch to docs@ for that.

Alex