Skip to contentSkip to conversation

Multi-threaded RPM support

PATCH In review Alexander Kanavin 2 participants 5 messages Jun 9, 2017

lore
Share & tools
lore
#0 lore raw

v3 cover Multi-threaded RPM support

Changes in version 3: fix up indentation inconsistencies, fix a nasty variable name typo

Changes in version 2: use openmp for multi-processing instead of NSPR library, as requested by upstream. Add a link to the upstream submission to patches.

These two patches add multi-threaded features to RPM to speed up the do_package_write_rpm task.

Specifically:

1) Multi-threaded xz compressor is used instead of default single-threaded gz. This has the most dramatic effect when a recipe produces a smaller number of large-sized packages.

2) Packages creation is run in multiple threads via thread pools. This is most beneficial when a recipe produces a large amount of small packages.

Some not very scientific benchmarks for time and .rpm sizes (time is measured for do_package_write_rpm tasks only):

webkitgtk
before: 9m12s 1550M
after: 1m40s 858M

glibc-locale
before: 2m52s 125M
after: 30s 56M

glibc
before: 46s 54M
after: 13s 38M

perl
before: 1m09s 63M
after: 45s 42M

python3
before: 30s 38M
after: 18s 24M

The following changes since commit 576821ea0a7558b626ccc87e9ae0e9ee40864956:

bitbake: bitbake-layers: check layer dependencies before adding (2017-06-06 19:52:51 +0100)

are available in the git repository at:

git://git.yoctoproject.org/poky-contrib akanavin/parallel-rpm
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=akanavin/parallel-rpm

Alexander Kanavin (2):
package_rpm.bbclass: use multithreaded xz compression
rpm: run binary package generation via thread pools

 meta/classes/package_rpm.bbclass                   |   2 +
 ...y-package-building-into-a-separate-functi.patch |  84 ++++++++
 ...-binary-package-creation-via-thread-pools.patch | 126 ++++++++++++
 ...c-make-operations-over-string-pools-threa.patch | 207 ++++++++++++++++++++
 ...c-remove-static-local-variables-from-buil.patch | 216 +++++++++++++++++++++
 meta/recipes-devtools/rpm/rpm_git.bb               |   4 +
 6 files changed, 639 insertions(+)
 create mode 100644 meta/recipes-devtools/rpm/files/0001-Split-binary-package-building-into-a-separate-functi.patch
 create mode 100644 meta/recipes-devtools/rpm/files/0002-Run-binary-package-creation-via-thread-pools.patch
 create mode 100644 meta/recipes-devtools/rpm/files/0003-rpmstrpool.c-make-operations-over-string-pools-threa.patch
 create mode 100644 meta/recipes-devtools/rpm/files/0004-build-pack.c-remove-static-local-variables-from-buil.patch
Signature · 1 line
2.11.0
#1 lore raw

PATCH v3 1/2 package_rpm.bbclass: use multithreaded xz compression

RPM's default is single-threaded gz; the change greatly helps with both buildtimes (when there is a small number of large-sized packages) and disk space taken by resulting rpms.

---
 meta/classes/package_rpm.bbclass | 2 ++
 1 file changed, 2 insertions(+)
meta/classes/package_rpm.bbclass +2 -0
index a844c4d4360..dc241975c3b 100644
@@ -644,6 +644,8 @@ python do_package_rpm () {
     cmd = cmd + " --define '_build_name_fmt %%{NAME}-%%{VERSION}-%%{RELEASE}.%%{ARCH}.rpm'"
     cmd = cmd + " --define '_use_internal_dependency_generator 0'"
     cmd = cmd + " --define '_binaries_in_noarch_packages_terminate_build 0'"
+    cmd = cmd + " --define '_binary_payload w6T.xzdio'"
+    cmd = cmd + " --define '_source_payload w6T.xzdio'"
     if perfiledeps:
         cmd = cmd + " --define '__find_requires " + outdepends + "'"
         cmd = cmd + " --define '__find_provides " + outprovides + "'"
Signature · 1 line
2.11.0
#2 lore raw

PATCH v3 2/2 rpm: run binary package generation via thread pools

In reply to Alexander Kanavin

This greatly reduces build times when there is a large amount of small rpm packages to produce. The patches are rather invasive, and so will be submitted upstream.

---
 ...y-package-building-into-a-separate-functi.patch |  84 ++++++++
 ...-binary-package-creation-via-thread-pools.patch | 126 ++++++++++++
 ...c-make-operations-over-string-pools-threa.patch | 207 ++++++++++++++++++++
 ...c-remove-static-local-variables-from-buil.patch | 216 +++++++++++++++++++++
 meta/recipes-devtools/rpm/rpm_git.bb               |   4 +
 5 files changed, 637 insertions(+)
 create mode 100644 meta/recipes-devtools/rpm/files/0001-Split-binary-package-building-into-a-separate-functi.patch
 create mode 100644 meta/recipes-devtools/rpm/files/0002-Run-binary-package-creation-via-thread-pools.patch
 create mode 100644 meta/recipes-devtools/rpm/files/0003-rpmstrpool.c-make-operations-over-string-pools-threa.patch
 create mode 100644 meta/recipes-devtools/rpm/files/0004-build-pack.c-remove-static-local-variables-from-buil.patch
meta/recipes-devtools/rpm/files/0001-Split-binary-package-building-into-a-separate-functi.patch new file +84 -0
new file mode 100644
index 00000000000..6e44f0b7fc9
@@ -0,0 +1,84 @@
+From 721a660a507d6d062e7aecafad886c643970a5d5 Mon Sep 17 00:00:00 2001
+From: Alexander Kanavin <[email protected]>
+Date: Thu, 25 May 2017 18:15:27 +0300
+Subject: [PATCH 1/4] Split binary package building into a separate function
+
+So that it can be run as a thread pool task.
+
+Upstream-Status: Submitted [https://github.com/rpm-software-management/rpm/pull/226]
+Signed-off-by: Alexander Kanavin <[email protected]>
+
+---
+ build/pack.c | 33 +++++++++++++++++++++------------
+ 1 file changed, 21 insertions(+), 12 deletions(-)
+
+diff --git a/build/pack.c b/build/pack.c
+index 518f4e92a..ccfd614cc 100644
+--- a/build/pack.c
++++ b/build/pack.c
+@@ -546,18 +546,13 @@ static rpmRC checkPackages(char *pkgcheck)
+     return RPMRC_OK;
+ }
+ 
+-rpmRC packageBinaries(rpmSpec spec, const char *cookie, int cheating)
++static rpmRC packageBinary(rpmSpec spec, Package pkg, const char *cookie, int cheating, char** filename)
+ {
+-    rpmRC rc;
+-    const char *errorString;
+-    Package pkg;
+-    char *pkglist = NULL;
+-
+-    for (pkg = spec->packages; pkg != NULL; pkg = pkg->next) {
+-	char *fn;
++	const char *errorString;
++	rpmRC rc = RPMRC_OK;
+ 
+ 	if (pkg->fileList == NULL)
+-	    continue;
++	    return rc;
+ 
+ 	if ((rc = processScriptFiles(spec, pkg)))
+ 	    return rc;
+@@ -587,7 +582,7 @@ rpmRC packageBinaries(rpmSpec spec, const char *cookie, int cheating)
+ 		     headerGetString(pkg->header, RPMTAG_NAME), errorString);
+ 		return RPMRC_FAIL;
+ 	    }
+-	    fn = rpmGetPath("%{_rpmdir}/", binRpm, NULL);
++	    *filename = rpmGetPath("%{_rpmdir}/", binRpm, NULL);
+ 	    if ((binDir = strchr(binRpm, '/')) != NULL) {
+ 		struct stat st;
+ 		char *dn;
+@@ -609,14 +604,28 @@ rpmRC packageBinaries(rpmSpec spec, const char *cookie, int cheating)
+ 	    free(binRpm);
+ 	}
+ 
+-	rc = writeRPM(pkg, NULL, fn, NULL);
++	rc = writeRPM(pkg, NULL, *filename, NULL);
+ 	if (rc == RPMRC_OK) {
+ 	    /* Do check each written package if enabled */
+-	    char *pkgcheck = rpmExpand("%{?_build_pkgcheck} ", fn, NULL);
++	    char *pkgcheck = rpmExpand("%{?_build_pkgcheck} ", *filename, NULL);
+ 	    if (pkgcheck[0] != ' ') {
+ 		rc = checkPackages(pkgcheck);
+ 	    }
+ 	    free(pkgcheck);
++	}
++	return rc;
++}
++
++rpmRC packageBinaries(rpmSpec spec, const char *cookie, int cheating)
++{
++    rpmRC rc;
++    Package pkg;
++    char *pkglist = NULL;
++
++    for (pkg = spec->packages; pkg != NULL; pkg = pkg->next) {
++	char *fn = NULL;
++	rc = packageBinary(spec, pkg, cookie, cheating, &fn);
++	if (rc == RPMRC_OK) {
+ 	    rstrcat(&pkglist, fn);
+ 	    rstrcat(&pkglist, " ");
+ 	}
+-- 
+2.11.0
+
meta/recipes-devtools/rpm/files/0002-Run-binary-package-creation-via-thread-pools.patch new file +126 -0

126 lines not on this page · raw message

meta/recipes-devtools/rpm/files/0003-rpmstrpool.c-make-operations-over-string-pools-threa.patch new file +207 -0

207 lines not on this page · raw message

meta/recipes-devtools/rpm/files/0004-build-pack.c-remove-static-local-variables-from-buil.patch new file +216 -0

216 lines not on this page · raw message

meta/recipes-devtools/rpm/rpm_git.bb +4 -0
index 2310ee6b09e..b95b4719c19 100644
@@ -35,6 +35,10 @@ SRC_URI = "git://github.com/rpm-software-management/rpm \
            file://0001-Fix-build-with-musl-C-library.patch \
            file://0001-Add-a-color-setting-for-mips64_n32-binaries.patch \
            file://0001-Add-PYTHON_ABI-when-searching-for-python-libraries.patch \
+           file://0001-Split-binary-package-building-into-a-separate-functi.patch \
+           file://0002-Run-binary-package-creation-via-thread-pools.patch \
+           file://0003-rpmstrpool.c-make-operations-over-string-pools-threa.patch \
+           file://0004-build-pack.c-remove-static-local-variables-from-buil.patch \
            "
 
 PV = "4.13.90+git${SRCPV}"
Signature · 1 line
2.11.0
#3 lore raw

Re: [PATCHv3 2/2] rpm: run binary package generation via thread pools

Quoted 4 lines · Alexander Kanavin

This greatly reduces build times when there is a large amount of small rpm packages to produce. The patches are rather invasive, and so will be submitted upstream.

Sorry but I broke this in the same way as I did before...

Ross

#4 lore raw

Re: [PATCHv3 2/2] rpm: run binary package generation via thread pools

Quoted 11 lines · Ross

On 9 June 2017 at 12:13, Alexander Kanavin <[email protected] <mailto:[email protected]>> wrote:

This greatly reduces build times when there is a large amount of small rpm packages to produce. The patches are rather invasive, and so will be submitted upstream.

Sorry but I broke this in the same way as I did before...

Needless to say, this worked flawlessly here, many times over. It seems that failures might be specific to gcc 4.x (I have 6.x here), so I'll try to get a failure using that. The nasty bit about openmp is that

#pragma omp unsupported_directive_from_latest_standard

will trigger nothing with compilers that are too old to support it.

Alex