Skip to contentSkip to conversation

package_rpm: use zstd's default compression level

PATCH In review Alexander Kanavin 2 participants 4 messages Nov 20, 2024

lore
Share & tools
lore
#0 lore raw

PATCH 1/2 package_rpm: use zstd's default compression level

From: Alexander Kanavin <[email protected]>

zstd uses 3 by default, while 19 is the highest and slowest. It's not clear why 19 was picked to begin with, possibly I copy-pasted it from rpm's examples without thinking:
https://git.yoctoproject.org/poky/commit/?h=master-next&id=4a4d5f78a6962dda5f63e9891825c80a8a87bf66

This brings significant speedups in rpm's compression step:
for example compressing webkitgtk takes 11s instead of 36s.

The rpm size increases from 175648k to 234860k. I think it's a worthy default tradeoff.

---
 meta/classes-global/package_rpm.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/classes-global/package_rpm.bbclass +1 -1
index ddc4bf3a6a5..b2b7fafa177 100644
@@ -10,7 +10,7 @@ IMAGE_PKGTYPE ?= "rpm"
 
 RPM = "rpm"
 RPMBUILD = "rpmbuild"
-RPMBUILD_COMPMODE ?= "${@'w19T%d.zstdio' % int(d.getVar('ZSTD_THREADS'))}"
+RPMBUILD_COMPMODE ?= "${@'w3T%d.zstdio' % int(d.getVar('ZSTD_THREADS'))}"
 
 PKGWRITEDIRRPM = "${WORKDIR}/deploy-rpms"
 
Signature · 1 line
2.39.5
#1 lore raw

PATCH 2/2 package_rpm: restrict rpm to 4 threads

From: Alexander Kanavin <[email protected]>

TL;DR version:

with this, and the previous compression level changes
I am seeing drastic speedups in package_write_rpm completion times:

webkitgtk goes from 78 seconds to 37 seconds glibc-locale goes from 399 seconds to 58 seconds (!)

The long version:

rpm uses multithreading for two purposes:

- spawning compressors (which are nowadays themselves multi-threaded, so the feature is not as useful as it once was)
- parallel file classification

While the former behaves well on massively parallel CPUs (it was written and verified here :), the latter was then added by upstream and only benchmarked on their very old, slow laptop, apparently:
https://github.com/rpm-software-management/rpm/commit/41f0e214f2266f02d6185ba11f797716de8125d4

On anything more capable it starts showing pathologic behavior, presumably from spawning massive amount of very short-lived threads, and then having to synchronize them. For example classifying glibc-locale takes
5m20s with 256 threads (default on my machine!)
1m49s with 64 threads
59s with 16 threads
48s with 8 threads

Even a more typical recipe like webkitgtk is affected:
47s with 256 threads
32s with 64 threads
27s with 16 or 8 threads

I have found that the optimal amount is actually four: this also means that only four compressors are running at a time, but as they're themselves using threads, and typical recipes are dominated by just two or three large packages, this does not affect overall completion time.

---
 meta/classes-global/package_rpm.bbclass | 1 +
 1 file changed, 1 insertion(+)
meta/classes-global/package_rpm.bbclass +1 -0
index b2b7fafa177..021c53593fd 100644
@@ -696,6 +696,7 @@ python do_package_rpm () {
     cmd = cmd + " --define '_use_internal_dependency_generator 0'"
     cmd = cmd + " --define '_binaries_in_noarch_packages_terminate_build 0'"
     cmd = cmd + " --define '_build_id_links none'"
+    cmd = cmd + " --define '_smp_ncpus_max 4'"
     cmd = cmd + " --define '_source_payload %s'" % rpmbuild_compmode
     cmd = cmd + " --define '_binary_payload %s'" % rpmbuild_compmode
     cmd = cmd + " --define 'clamp_mtime_to_source_date_epoch 1'"
Signature · 1 line
2.39.5
#2 lore raw
In reply to Alexander Kanavin

Hi Alex,

Quoted 23 lines · Alexander Kanavin via lists.openembedded.org

From: Alexander Kanavin <[email protected]>

zstd uses 3 by default, while 19 is the highest and slowest. It's not clear why 19 was picked to begin with, possibly I copy-pasted it from rpm's examples without thinking:
https://git.yoctoproject.org/poky/commit/?h=master-next&id=4a4d5f78a6962dda5f63e9891825c80a8a87bf66

This brings significant speedups in rpm's compression step:
for example compressing webkitgtk takes 11s instead of 36s.

The rpm size increases from 175648k to 234860k. I think it's a worthy default tradeoff.

---
  meta/classes-global/package_rpm.bbclass | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
meta/classes-global/package_rpm.bbclass +0 -0
index ddc4bf3a6a5..b2b7fafa177 100644

meta/classes-global/package_rpm.bbclass IMAGE_PKGTYPE ?= "rpm" · line 14

Quoted 6 lines
meta/classes-global/package_rpm.bbclass +1 -1
@@ -10,7 +10,7 @@ IMAGE_PKGTYPE ?= "rpm"
 
  RPM = "rpm"
  RPMBUILD = "rpmbuild"
-RPMBUILD_COMPMODE ?= "${@'w19T%d.zstdio' % int(d.getVar('ZSTD_THREADS'))}"
+RPMBUILD_COMPMODE ?= "${@'w3T%d.zstdio' % int(d.getVar('ZSTD_THREADS'))}"
 

Since we have a variable containing the compression level for ZSTD (ZSTD_COMPRESSION_LEVEL), should we use it? The downside is that it's prefixed with a dash, which would need to remove before using it (or update the variable to not contain it and add it where the variable is currently used.

Cheers,
Quentin

#3 lore raw
Quoted 5 lines · Quentin Schulz

Since we have a variable containing the compression level for ZSTD (ZSTD_COMPRESSION_LEVEL), should we use it? The downside is that it's prefixed with a dash, which would need to remove before using it (or update the variable to not contain it and add it where the variable is currently used.

Sure. I can send a v2.

Alex