Skip to content

Multi-threaded RPM support

Alexander Kanavin openembedded-core 3 versions – In review

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v3 Latest2 patches 2 repliesIn review
  2. v22 patches 7 repliesSuperseded
  3. v12 patches 21 repliesSuperseded

Patches in v3 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/2package_rpm.bbclass: use multithreaded xz compressionmeta/classes/package_rpm.bbclass+2 -00 replies
2/2rpm: run binary package generation via thread poolsmeta/recipes-devtools/rpm/files/0001-Split-binary-package-building-into-a-separate-functi.patch meta/recipes-devtools/rpm/files/0002-Run-binary-package-creation-via-thread-pools.patch meta/recipes-devtools/rpm/files/0003-rpmstrpool.c-make-operations-over-string-pools-threa.patch +2 more+637 -02 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v3 yet.

Read the v3 thread

Cover letter [PATCHv3 0/2] Multi-threaded RPM support

Changes in version 3: fix up indentation inconsistencies, fix a nasty variable name typo

Changes in version 2: use openmp for multi-processing instead of NSPR library, as requested by upstream. Add a link to the upstream submission to patches.

These two patches add multi-threaded features to RPM to speed up the do_package_write_rpm task.

Specifically:

1) Multi-threaded xz compressor is used instead of default single-threaded gz. This has the most dramatic effect when a recipe produces a smaller number of large-sized packages.

2) Packages creation is run in multiple threads via thread pools. This is most beneficial when a recipe produces a large amount of small packages.

Some not very scientific benchmarks for time and .rpm sizes (time is measured for do_package_write_rpm tasks only):

webkitgtk
before: 9m12s 1550M
after: 1m40s 858M

glibc-locale
before: 2m52s 125M
after: 30s 56M

glibc
before: 46s 54M
after: 13s 38M

perl
before: 1m09s 63M
after: 45s 42M

python3
before: 30s 38M
after: 18s 24M

The following changes since commit 576821ea0a7558b626ccc87e9ae0e9ee40864956:

bitbake: bitbake-layers: check layer dependencies before adding (2017-06-06 19:52:51 +0100)

are available in the git repository at:

git://git.yoctoproject.org/poky-contrib akanavin/parallel-rpm
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=akanavin/parallel-rpm

Alexander Kanavin (2):
package_rpm.bbclass: use multithreaded xz compression
rpm: run binary package generation via thread pools

 meta/classes/package_rpm.bbclass                   |   2 +
 ...y-package-building-into-a-separate-functi.patch |  84 ++++++++
 ...-binary-package-creation-via-thread-pools.patch | 126 ++++++++++++
 ...c-make-operations-over-string-pools-threa.patch | 207 ++++++++++++++++++++
 ...c-remove-static-local-variables-from-buil.patch | 216 +++++++++++++++++++++
 meta/recipes-devtools/rpm/rpm_git.bb               |   4 +
 6 files changed, 639 insertions(+)
 create mode 100644 meta/recipes-devtools/rpm/files/0001-Split-binary-package-building-into-a-separate-functi.patch
 create mode 100644 meta/recipes-devtools/rpm/files/0002-Run-binary-package-creation-via-thread-pools.patch
 create mode 100644 meta/recipes-devtools/rpm/files/0003-rpmstrpool.c-make-operations-over-string-pools-threa.patch
 create mode 100644 meta/recipes-devtools/rpm/files/0004-build-pack.c-remove-static-local-variables-from-buil.patch
Signature · 1 line
2.11.0