Multi-threaded RPM support
cover Multi-threaded RPM support
These two patches add multi-threaded features to RPM to speed up the do_package_write_rpm task.
Specifically:
1) Multi-threaded xz compressor is used instead of default single-threaded gz. This has the most dramatic effect when a recipe produces a smaller number of large-sized packages.
2) Packages creation is run in multiple threads via thread pools. This is most beneficial when a recipe produces a large amount of small packages.
Some not very scientific benchmarks for time and .rpm sizes (time is measured for do_package_write_rpm tasks only):
webkitgtk
before: 9m22s 1550M
after: 1m50s 858M
glibc-locale
before: 3m2s 125M
after: 40s 56M
glibc
before: 56s 54M
after: 23s 38M
perl
before: 1m19s 63M
after: 55s 42M
python3
before: 40s 38M
after: 28s 24M
The following changes since commit bd063fa288b49b6e3ea77982d0ccc46610feb1ad:
tune-mips32*.inc: use consistent comments across all three .inc files (2017-05-30 10:15:20 +0100)
are available in the git repository at:
git://git.yoctoproject.org/poky-contrib akanavin/parallel-rpm
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=akanavin/parallel-rpm
Alexander Kanavin (2):
package_rpm.bbclass: use multithreaded xz compression
rpm: run binary package generation via thread pools
meta/classes/package_rpm.bbclass | 2 + ...y-package-building-into-a-separate-functi.patch | 83 ++++++++ ...-binary-package-creation-via-thread-pools.patch | 137 +++++++++++++ ...c-make-operations-over-string-pools-threa.patch | 209 +++++++++++++++++++ ...c-remove-static-local-variables-from-buil.patch | 227 +++++++++++++++++++++ meta/recipes-devtools/rpm/rpm_git.bb | 4 + 6 files changed, 662 insertions(+) create mode 100644 meta/recipes-devtools/rpm/files/0001-Split-binary-package-building-into-a-separate-functi.patch create mode 100644 meta/recipes-devtools/rpm/files/0002-Run-binary-package-creation-via-thread-pools.patch create mode 100644 meta/recipes-devtools/rpm/files/0003-rpmstrpool.c-make-operations-over-string-pools-threa.patch create mode 100644 meta/recipes-devtools/rpm/files/0004-build-pack.c-remove-static-local-variables-from-buil.patch
Signature · 1 line
2.11.0
PATCH 1/2 package_rpm.bbclass: use multithreaded xz compression
RPM's default is single-threaded gz; the change greatly helps with both buildtimes (when there is a small number of large-sized packages) and disk space taken by resulting rpms.
--- meta/classes/package_rpm.bbclass | 2 ++ 1 file changed, 2 insertions(+)
meta/classes/package_rpm.bbclass +2 -0
@@ -644,6 +644,8 @@ python do_package_rpm () { cmd = cmd + " --define '_build_name_fmt %%{NAME}-%%{VERSION}-%%{RELEASE}.%%{ARCH}.rpm'" cmd = cmd + " --define '_use_internal_dependency_generator 0'" cmd = cmd + " --define '_binaries_in_noarch_packages_terminate_build 0'" + cmd = cmd + " --define '_binary_payload w6T.xzdio'" + cmd = cmd + " --define '_source_payload w6T.xzdio'" if perfiledeps: cmd = cmd + " --define '__find_requires " + outdepends + "'" cmd = cmd + " --define '__find_provides " + outprovides + "'"
Signature · 1 line
2.11.0
PATCH 2/2 rpm: run binary package generation via thread pools
In reply to Alexander KanavinThis greatly reduces build times when there is a large amount of small rpm packages to produce. The patches are rather invasive, and so will be submitted upstream.
--- ...y-package-building-into-a-separate-functi.patch | 83 ++++++++ ...-binary-package-creation-via-thread-pools.patch | 137 +++++++++++++ ...c-make-operations-over-string-pools-threa.patch | 209 +++++++++++++++++++ ...c-remove-static-local-variables-from-buil.patch | 227 +++++++++++++++++++++ meta/recipes-devtools/rpm/rpm_git.bb | 4 + 5 files changed, 660 insertions(+) create mode 100644 meta/recipes-devtools/rpm/files/0001-Split-binary-package-building-into-a-separate-functi.patch create mode 100644 meta/recipes-devtools/rpm/files/0002-Run-binary-package-creation-via-thread-pools.patch create mode 100644 meta/recipes-devtools/rpm/files/0003-rpmstrpool.c-make-operations-over-string-pools-threa.patch create mode 100644 meta/recipes-devtools/rpm/files/0004-build-pack.c-remove-static-local-variables-from-buil.patch
meta/recipes-devtools/rpm/files/0001-Split-binary-package-building-into-a-separate-functi.patch new file +83 -0
@@ -0,0 +1,83 @@ +From b841b699e519438a66b661247c94efff63d0700e Mon Sep 17 00:00:00 2001 +From: Alexander Kanavin <[email protected]> +Date: Thu, 25 May 2017 18:15:27 +0300 +Subject: [PATCH 01/14] Split binary package building into a separate function + +So that it can be run as a thread pool task. + +Upstream-Status: Pending +Signed-off-by: Alexander Kanavin <[email protected]> +--- + build/pack.c | 33 +++++++++++++++++++++------------ + 1 file changed, 21 insertions(+), 12 deletions(-) + +diff --git a/build/pack.c b/build/pack.c +index 497300b96..891e6bdc3 100644 +--- a/build/pack.c ++++ b/build/pack.c +@@ -546,18 +546,13 @@ static rpmRC checkPackages(char *pkgcheck) + return RPMRC_OK; + } + +-rpmRC packageBinaries(rpmSpec spec, const char *cookie, int cheating) ++static rpmRC packageBinary(rpmSpec spec, Package pkg, const char *cookie, int cheating, char** filename) + { +- rpmRC rc; +- const char *errorString; +- Package pkg; +- char *pkglist = NULL; +- +- for (pkg = spec->packages; pkg != NULL; pkg = pkg->next) { +- char *fn; ++ const char *errorString; ++ rpmRC rc = RPMRC_OK; + + if (pkg->fileList == NULL) +- continue; ++ return rc; + + if ((rc = processScriptFiles(spec, pkg))) + return rc; +@@ -591,7 +586,7 @@ rpmRC packageBinaries(rpmSpec spec, const char *cookie, int cheating) + headerGetString(pkg->header, RPMTAG_NAME), errorString); + return RPMRC_FAIL; + } +- fn = rpmGetPath("%{_rpmdir}/", binRpm, NULL); ++ *filename = rpmGetPath("%{_rpmdir}/", binRpm, NULL); + if ((binDir = strchr(binRpm, '/')) != NULL) { + struct stat st; + char *dn; +@@ -613,14 +608,28 @@ rpmRC packageBinaries(rpmSpec spec, const char *cookie, int cheating) + free(binRpm); + } + +- rc = writeRPM(pkg, NULL, fn, NULL); ++ rc = writeRPM(pkg, NULL, *filename, NULL); + if (rc == RPMRC_OK) { + /* Do check each written package if enabled */ +- char *pkgcheck = rpmExpand("%{?_build_pkgcheck} ", fn, NULL); ++ char *pkgcheck = rpmExpand("%{?_build_pkgcheck} ", *filename, NULL); + if (pkgcheck[0] != ' ') { + rc = checkPackages(pkgcheck); + } + free(pkgcheck); ++ } ++ return rc; ++} ++ ++rpmRC packageBinaries(rpmSpec spec, const char *cookie, int cheating) ++{ ++ rpmRC rc; ++ Package pkg; ++ char *pkglist = NULL; ++ ++ for (pkg = spec->packages; pkg != NULL; pkg = pkg->next) { ++ char *fn = NULL; ++ rc = packageBinary(spec, pkg, cookie, cheating, &fn); ++ if (rc == RPMRC_OK) { + rstrcat(&pkglist, fn); + rstrcat(&pkglist, " "); + } +-- +2.11.0 +
meta/recipes-devtools/rpm/files/0002-Run-binary-package-creation-via-thread-pools.patch new file +137 -0
137 lines not on this page · raw message
meta/recipes-devtools/rpm/files/0003-rpmstrpool.c-make-operations-over-string-pools-threa.patch new file +209 -0
209 lines not on this page · raw message
meta/recipes-devtools/rpm/files/0004-build-pack.c-remove-static-local-variables-from-buil.patch new file +227 -0
227 lines not on this page · raw message
meta/recipes-devtools/rpm/rpm_git.bb +4 -0
@@ -35,6 +35,10 @@ SRC_URI = "git://github.com/rpm-software-management/rpm \ file://0001-Fix-build-with-musl-C-library.patch \ file://0001-Add-a-color-setting-for-mips64_n32-binaries.patch \ file://0001-Add-PYTHON_ABI-when-searching-for-python-libraries.patch \ + file://0001-Split-binary-package-building-into-a-separate-functi.patch \ + file://0002-Run-binary-package-creation-via-thread-pools.patch \ + file://0003-rpmstrpool.c-make-operations-over-string-pools-threa.patch \ + file://0004-build-pack.c-remove-static-local-variables-from-buil.patch \ " PV = "4.13.90+git${SRCPV}"
Signature · 1 line
2.11.0
Re: [PATCH 2/2] rpm: run binary package generation via thread pools
Quoted 3 lines · Alexander Kanavin
This greatly reduces build times when there is a large amount of small rpm packages to produce. The patches are rather invasive, and so will be submitted upstream.
And now the pull request is open:
https://github.com/rpm-software-management/rpm/pull/226
https://github.com/rpm-software-management/rpm/issues/211
Alex
Quoted 25 lines · Alexander Kanavin
These two patches add multi-threaded features to RPM to speed up the
do_package_write_rpmtask.Specifically:
1) Multi-threaded xz compressor is used instead of default single-threaded gz. This has the most dramatic effect when a recipe produces a smaller number of large-sized packages.
2) Packages creation is run in multiple threads via thread pools. This is most beneficial when a recipe produces a large amount of small packages.
Some not very scientific benchmarks for time and .rpm sizes (time is measured for
do_package_write_rpmtasks only):webkitgtk
before: 9m22s 1550M
after: 1m50s 858Mglibc-locale
before: 3m2s 125M
after: 40s 56Mglibc
before: 56s 54M
after: 23s 38M
Quoted 7 lines
perl
before: 1m19s 63M
after: 55s 42Mpython3
before: 40s 38M
after: 28s 24M
it is interesting to see that perl and python does not benefit 'much'. buildstats should give us a better picture per recipe.
Leo
Quoted 29 lines
The following changes since commit
bd063fa288b49b6e3ea77982d0ccc46610feb1ad:tune-mips32*.inc: use consistent comments across all three .inc files (2017-05-30 10:15:20 +0100)
are available in the git repository at:
git://git.yoctoproject.org/poky-contribakanavin/parallel-rpm
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=akanavin/parallel-rpmAlexander Kanavin (2):
package_rpm.bbclass: use multithreaded xz compression
rpm: run binary package generation via thread poolsmeta/classes/package_rpm.bbclass | 2 + ...y-package-building-into-a-separate-functi.patch | 83 ++++++++ ...-binary-package-creation-via-thread-pools.patch | 137 +++++++++++++ ...c-make-operations-over-string-pools-threa.patch | 209 +++++++++++++++++++ ...c-remove-static-local-variables-from-buil.patch | 227 +++++++++++++++++++++ meta/recipes-devtools/rpm/rpm_git.bb | 4 + 6 files changed, 662 insertions(+) create mode 100644 meta/recipes-devtools/rpm/files/0001-Split-binary-package-building-into-a-separate-functi.patch create mode 100644 meta/recipes-devtools/rpm/files/0002-Run-binary-package-creation-via-thread-pools.patch create mode 100644 meta/recipes-devtools/rpm/files/0003-rpmstrpool.c-make-operations-over-string-pools-threa.patch create mode 100644 meta/recipes-devtools/rpm/files/0004-build-pack.c-remove-static-local-variables-from-buil.patchSignature · 1 line
2.11.0
Re: [PATCH 1/2] package_rpm.bbclass: use multithreaded xz compression
In reply to Alexander KanavinQuoted 3 lines · Alexander Kanavin
RPM's default is single-threaded gz; the change greatly helps with both buildtimes (when there is a small number of large-sized packages) and disk space taken by resulting rpms.
If you wnable XZ here, you need to make sure XZ compression is enabled within rpm itself. (I know how that worked in rpm5, not sure how rpm4 does it -- it may be enable that the binary be present to call out to.)
Similarly to XZ, you can use pigz or other parallel compression tools to do similar with other schemes.
Finally I have not investigated parallel XZ, but in some cases the resulting compressed file is different depending on compression artifacts due to parallelization. (The compressed file is different, not the uncompressed content.)
This can be a concern to some who want fully reproducible builds (from source). So we need to make sure we have a way to disable this type of parallelization --if-- it causes a problem with different contents.
(Note the fact RPM package metadata changes each time a new RPM is constructed, this is far -less- of a concern to me.. but something to be aware of.)
--Mark
Quoted 19 lines
Signed-off-by Alexander Kanavin <[email protected]>--- meta/classes/package_rpm.bbclass | 2 ++ 1 file changed, 2 insertions(+)meta/classes/package_rpm.bbclass +2
-0@@ -644,6 +644,8 @@ python do_package_rpm () { cmd = cmd + " --define '_build_name_fmt %%{NAME}-%%{VERSION}-%%{RELEASE}.%%{ARCH}.rpm'" cmd = cmd + " --define '_use_internal_dependency_generator 0'" cmd = cmd + " --define '_binaries_in_noarch_packages_terminate_build 0'" + cmd = cmd + " --define '_binary_payload w6T.xzdio'" + cmd = cmd + " --define '_source_payload w6T.xzdio'" if perfiledeps: cmd = cmd + " --define '__find_requires " + outdepends + "'" cmd = cmd + " --define '__find_provides " + outprovides + "'"
Quoted 1 line · Leonardo Sandoval
On Thu, 2017-06-01 at 18:15 +0300, Alexander Kanavin wrote:
...
Quoted 10 lines
Quoted 7 lines
perl
before: 1m19s 63M
after: 55s 42Mpython3
before: 40s 38M
after: 28s 24Mit is interesting to see that perl and python does not benefit 'much'. buildstats should give us a better picture per recipe.
This may be due to how we process rpmdeps within OpenEmbedded.. but I'm not sure. I'd have expected a bigger time difference with large packages (lots of sub packages) like perl and python3.
(kernel-modules would be another to look at, as it can get quite large.)
Quoted 8 lines · Mark Hatle
Quoted 8 lines
Quoted 7 lines
perl
before: 1m19s 63M
after: 55s 42Mpython3
before: 40s 38M
after: 28s 24M
Quoted 6 lines
Quoted 2 lines
it is interesting to see that perl and python does not benefit 'much'. buildstats should give us a better picture per recipe.
This may be due to how we process rpmdeps within OpenEmbedded.. but I'm not sure. I'd have expected a bigger time difference with large packages (lots of sub packages) like perl and python3.
The numbers include the overhead of starting bitbake (some 10 seconds). Subtract that (from both 'before' and 'after') and then it's more impressive :) I tested via
time bitbake -f -c do_package_write_rpm perl
Alex
Re: [PATCH 1/2] package_rpm.bbclass: use multithreaded xz compression
In reply to Mark HatleQuoted 8 lines · Mark Hatle
Quoted 3 lines · Alexander Kanavin
RPM's default is single-threaded gz; the change greatly helps with both buildtimes (when there is a small number of large-sized packages) and disk space taken by resulting rpms.
If you wnable XZ here, you need to make sure XZ compression is enabled within rpm itself. (I know how that worked in rpm5, not sure how rpm4 does it -- it may be enable that the binary be present to call out to.)
rpm4 does not call out to any binary, it is linking against liblzma from xz package, if it can find it during configure. Xz was already in list of dependencies, so nothing needed to be done. Same for other compression schemes.
Quoted 7 lines
Finally I have not investigated parallel XZ, but in some cases the resulting compressed file is different depending on compression artifacts due to parallelization. (The compressed file is different, not the uncompressed content.)
This can be a concern to some who want fully reproducible builds (from source). So we need to make sure we have a way to disable this type of parallelization --if-- it causes a problem with different contents.
Let's first see if it is an actual concern for anyone. If the uncompressed contents is exactly same, I have difficulty seeing what is the problem specifically.
Alex
Re: [PATCH 1/2] package_rpm.bbclass: use multithreaded xz compression
In reply to Alexander KanavinQuoted 3 lines · Alexander Kanavin
RPM's default is single-threaded gz; the change greatly helps with both buildtimes (when there is a small number of large-sized packages) and disk space taken by resulting rpms.
Do you have any test results for that? Obviously build times will be longer when building with a single CPU since xz compression needs more CPU than gzip, but I'm curious how many CPUs are required before the multiple threads win over and build times actually get shorter. 4? 8?
One other concern with parallel xz compression is memory usage. Single threaded xz needs more memory than gzip and running multiple xz compression threads together will push the memory usage up again. Maybe not an issue if the peak is still acceptable, but something to be aware of (I've given up trying to build in 4GB, but hopefully 8GB build machines still have some life left in them...).
Quoted 26 lines
Signed-off-by Alexander Kanavin <[email protected]>--- meta/classes/package_rpm.bbclass | 2 ++ 1 file changed, 2 insertions(+)meta/classes/package_rpm.bbclass +2
-1@@ -644,6 +644,8 @@ python do_package_rpm () { cmd = cmd + " --define '_build_name_fmt %%{NAME}-%%{VERSION}-%%{RELEASE}.%%{ARCH}.rpm'" cmd = cmd + " --define '_use_internal_dependency_generator 0'" cmd = cmd + " --define '_binaries_in_noarch_packages_terminate_build 0'" + cmd = cmd + " --define '_binary_payload w6T.xzdio'" + cmd = cmd + " --define '_source_payload w6T.xzdio'" if perfiledeps: cmd = cmd + " --define '__find_requires " + outdepends + "'" cmd = cmd + " --define '__find_provides " + outprovides + "'" --2.11.0
--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [PATCH 1/2] package_rpm.bbclass: use multithreaded xz compression
Quoted 8 lines · Andre McCurdy
Quoted 3 lines
RPM's default is single-threaded gz; the change greatly helps with both buildtimes (when there is a small number of large-sized packages) and disk space taken by resulting rpms.
Do you have any test results for that? Obviously build times will be longer when building with a single CPU since xz compression needs more CPU than gzip, but I'm curious how many CPUs are required before the multiple threads win over and build times actually get shorter. 4? 8?
Yes; it's in the cover letter.
Quoted 6 lines
One other concern with parallel xz compression is memory usage. Single threaded xz needs more memory than gzip and running multiple xz compression threads together will push the memory usage up again. Maybe not an issue if the peak is still acceptable, but something to be aware of (I've given up trying to build in 4GB, but hopefully 8GB build machines still have some life left in them...).
I haven't seen anything resembling low-memory or OOM in my testing, even when running 44 rpm instances at the same time, each with their own several threads, each thread handling a package, and compression for the package again being done in a multi-threaded way. But then I have 22 cores and 32G RAM here.
So do try the patches please, and report the findings, YMMV.
Alex
Re: [PATCH 2/2] rpm: run binary package generation via thread pools
In reply to Alexander KanavinQuoted 9 lines · Alexander Kanavin
Quoted 3 lines · Alexander Kanavin
This greatly reduces build times when there is a large amount of small rpm packages to produce. The patches are rather invasive, and so will be submitted upstream.
And now the pull request is open:
https://github.com/rpm-software-management/rpm/pull/226
https://github.com/rpm-software-management/rpm/issues/211
So the upstream's reaction to the pull request is that they're not against this, but they want the multiprocessing to be done with openmp (and not nss/nspr), given that they now actually support openssl as an alternative to nss, and don't want any hard dependencies on netscape stuff. I'll spend some time studying that.
Alex
Re: [PATCH 1/2] package_rpm.bbclass: use multithreaded xz compression
In reply to Alexander KanavinQuoted 6 lines · Alexander Kanavin
Quoted 5 lines · Andre McCurdy
Quoted 4 lines
RPM's default is single-threaded gz; the change greatly helps with both buildtimes (when there is a small number of large-sized packages) and disk space taken by resulting rpms.
Quoted 6 lines
Quoted 4 lines
Do you have any test results for that? Obviously build times will be longer when building with a single CPU since xz compression needs more CPU than gzip, but I'm curious how many CPUs are required before the multiple threads win over and build times actually get shorter. 4? 8?
Yes; it's in the cover letter.
Yes, I only saw that after sending my reply. I shouldn't read emails in reverse order... sorry about that.
The cover letter benchmarks didn't mention the 22 CPU cores though, so it's good to have that clarified. Seeing some results from a 4 or 8 core machine would be interesting too.
Quoted 17 lines
Quoted 6 lines
One other concern with parallel xz compression is memory usage. Single threaded xz needs more memory than gzip and running multiple xz compression threads together will push the memory usage up again. Maybe not an issue if the peak is still acceptable, but something to be aware of (I've given up trying to build in 4GB, but hopefully 8GB build machines still have some life left in them...).>
I haven't seen anything resembling low-memory or OOM in my testing, even when running 44 rpm instances at the same time, each with their own several threads, each thread handling a package, and compression for the package again being done in a multi-threaded way. But then I have 22 cores and 32G RAM here.
So do try the patches please, and report the findings, YMMV.
Alex
Alex,
I get this when bitbake core-image-minimal
http://errors.yoctoproject.org/Errors/Build/38347/
HEAD is at
http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=akanavin/parallel-rpm
Quoted 24 lines · Alexander Kanavin
Quoted 8 lines · Mark Hatle
Quoted 8 lines
Quoted 7 lines
perl
before: 1m19s 63M
after: 55s 42Mpython3
before: 40s 38M
after: 28s 24MQuoted 6 lines
Quoted 2 lines
it is interesting to see that perl and python does not benefit 'much'. buildstats should give us a better picture per recipe.
This may be due to how we process rpmdeps within OpenEmbedded.. but I'm not sure. I'd have expected a bigger time difference with large packages (lots of sub packages) like perl and python3.
The numbers include the overhead of starting bitbake (some 10 seconds). Subtract that (from both 'before' and 'after') and then it's more impressive :) I tested via
time bitbake -f -c
do_package_write_rpmperlAlex
I did get a laugh when this showed up in my mail client screen as:
"Multi-threaded RP" :)
Seriously, I am looking forward to trying and hopefully merging this, should make for some nice speedups.
Cheers,
RP
Quoted 17 lines · Alexander Kanavin
These two patches add multi-threaded features to RPM to speed up the
do_package_write_rpmtask.Specifically:
1) Multi-threaded xz compressor is used instead of default single- threaded gz. This has the most
dramatic effect when a recipe produces a smaller number of large- sized packages.2) Packages creation is run in multiple threads via thread pools. This is most beneficial when
a recipe produces a large amount of small packages.The following changes since commit
bd063fa288b49b6e3ea77982d0ccc46610feb1ad:
Quoted 8 lines
tune-mips32*.inc: use consistent comments across all three .inc files (2017-05-30 10:15:20 +0100)
are available in the git repository at:
git://git.yoctoproject.org/poky-contribakanavin/parallel-rpm
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=akanavin/p
arallel-rpm
https://autobuilder.yoctoproject.org/main/builders/nightly-x86/builds/1
163/steps/BuildImages/logs/stdio
https://autobuilder.yoctoproject.org/main/builders/nightly-x86-64/build
s/1203/steps/BuildImages/logs/stdio
https://autobuilder.yoctoproject.org/main/builders/nightly-ipk/builds/1
112/steps/BuildImages/logs/stdio
I suspect the same issue causing failures in multiple places.
Cheers,
Richard
Quoted 10 lines · Richard Purdie
https://autobuilder.yoctoproject.org/main/builders/nightly-x86/builds/1
163/steps/BuildImages/logs/stdiohttps://autobuilder.yoctoproject.org/main/builders/nightly-x86-64/build
s/1203/steps/BuildImages/logs/stdiohttps://autobuilder.yoctoproject.org/main/builders/nightly-ipk/builds/1
112/steps/BuildImages/logs/stdioI suspect the same issue causing failures in multiple places.
The patches as they are now require that the C compiler on the host is recent; specifically that it defaults to C11 mode. I'm not sure when GCC made the switch, but it works that way with GCC 6.3.0.
In any case, I'll rework the patches to use openmp, as requested by upstream - this may however cause a different set of compatibility problems with older GCC versions. What are the oldest host compiler versions/distro versions that we still need to support in master?
Alex
Quoted 3 lines · Alexander Kanavin
The patches as they are now require that the C compiler on the host is recent; specifically that it defaults to C11 mode. I'm not sure when GCC made the switch, but it works that way with GCC 6.3.0.
I researched; the change happened in gcc 5 (released april 22 2015).
Alex
Quoted 4 lines · Alexander Kanavin
The patches as they are now require that the C compiler on the host is recent; specifically that it defaults to C11 mode. I'm not sure when GCC made the switch, but it works that way with GCC 6.3.0.
Isn't that just a compiler flag away from being fixed?
Ross
Quoted 11 lines · Ross
On 5 June 2017 at 13:14, Alexander Kanavin <[email protected] <mailto:[email protected]>> wrote:
The patches as they are now require that the C compiler on the host is recent; specifically that it defaults to C11 mode. I'm not sure when GCC made the switch, but it works that way with GCC 6.3.0.
Isn't that just a compiler flag away from being fixed?
Not if the compiler is 4.8 or earlier; it has to support _Atomic (a C11 feature), which only appeared in 4.9.
Alex
Quoted 22 lines · Alexander Kanavin
Quoted 8 lines · Richard Purdie
https://autobuilder.yoctoproject.org/main/builders/nightly-x86/buil
ds/1
163/steps/BuildImages/logs/stdiohttps://autobuilder.yoctoproject.org/main/builders/nightly-x86-64/b
uild
s/1203/steps/BuildImages/logs/stdioQuoted 5 lines
https://autobuilder.yoctoproject.org/main/builders/nightly-ipk/buil
ds/1
112/steps/BuildImages/logs/stdioI suspect the same issue causing failures in multiple places.
The patches as they are now require that the C compiler on the host is
recent; specifically that it defaults to C11 mode. I'm not sure when GCC
made the switch, but it works that way with GCC 6.3.0.
Quoted 4 lines
In any case, I'll rework the patches to use openmp, as requested by upstream - this may however cause a different set of compatibility problems with older GCC versions. What are the oldest host compiler versions/distro versions that we still need to support in master?
On the Yocto Autobuilder clusters we have:
* centos7 / gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-11)
* debian8 / gcc (Debian 4.9.2-10) 4.9.2
* debian-testing / gcc (Debian 6.3.0-14) 6.3.0 20170415
* fedora24 / gcc (GCC) 6.3.1 20161221 (Red Hat 6.3.1-1)
* fedora25 / gcc (GCC) 6.3.1 20161221 (Red Hat 6.3.1-1)
* opensuse132 / gcc (SUSE Linux) 4.8.3 20140627 [gcc-4_8-branch revision 212064]
* opensuse422 / gcc (SUSE Linux) 4.8.5
* ubuntu1604 / gcc (Ubuntu 5.4.0-6ubuntu1~16.04.4) 5.4.0 20160609
* ubuntu1610 / gcc (Ubuntu 6.2.0-5ubuntu12) 6.2.0 20161005
We are aiming to replace openSUSE 13.2 (as it was discontinued in January of this year) but CentOS 7 has a good long lifetime ahead of it, therefore supporting it and RHEL 7 will require gcc 4.8.5 support.
Joshua
Quoted 12 lines · Joshua Lock
On the Yocto Autobuilder clusters we have:
* centos7 / gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-11)
* debian8 / gcc (Debian 4.9.2-10) 4.9.2
* debian-testing / gcc (Debian 6.3.0-14) 6.3.0 20170415
* fedora24 / gcc (GCC) 6.3.1 20161221 (Red Hat 6.3.1-1)
* fedora25 / gcc (GCC) 6.3.1 20161221 (Red Hat 6.3.1-1)
* opensuse132 / gcc (SUSE Linux) 4.8.3 20140627 [gcc-4_8-branch revision 212064]
* opensuse422 / gcc (SUSE Linux) 4.8.5
* ubuntu1604 / gcc (Ubuntu 5.4.0-6ubuntu1~16.04.4) 5.4.0 20160609
* ubuntu1610 / gcc (Ubuntu 6.2.0-5ubuntu12) 6.2.0 20161005
Quoted 3 lines
We are aiming to replace openSUSE 13.2 (as it was discontinued in January of this year) but CentOS 7 has a good long lifetime ahead of it, therefore supporting it and RHEL 7 will require gcc 4.8.5 support.
Thanks; in that case using atomic types and other C11 features is not possible at all for quite some time, and openmp support, while available, is restricted to version 3.1 of the spec (which should be just about enough - earlier versions do not have task support).
I actually have gcc 4.8 here (on debian testing, which will become debian 9 in a couple of weeks), even though the default version is 6.3, so I can test 4.8 compatibility locally.
Alex
I just wanted to note that in my quest to get newer chrome to build I have used centos7 with the official devtoolset-6 scl.
It was able to build my pyro based distro fine in gcc6. So Centos7 has an officially supported path that may be viable if it comes down to being the last gcc 4.x distro.
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Alexander Kanavin
Sent: Monday, June 05, 2017 8:35 AM
To: Joshua Lock; Richard Purdie; [email protected]; Leonardo Sandoval
Subject: Re: [OE-core] [PATCH 0/2] Multi-threaded RPM support
Quoted 12 lines · Joshua Lock
On the Yocto Autobuilder clusters we have:
* centos7 / gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-11)
* debian8 / gcc (Debian 4.9.2-10) 4.9.2
* debian-testing / gcc (Debian 6.3.0-14) 6.3.0 20170415
* fedora24 / gcc (GCC) 6.3.1 20161221 (Red Hat 6.3.1-1)
* fedora25 / gcc (GCC) 6.3.1 20161221 (Red Hat 6.3.1-1)
* opensuse132 / gcc (SUSE Linux) 4.8.3 20140627 [gcc-4_8-branch revision 212064]
* opensuse422 / gcc (SUSE Linux) 4.8.5
* ubuntu1604 / gcc (Ubuntu 5.4.0-6ubuntu1~16.04.4) 5.4.0 20160609
* ubuntu1610 / gcc (Ubuntu 6.2.0-5ubuntu12) 6.2.0 20161005
Quoted 3 lines
We are aiming to replace openSUSE 13.2 (as it was discontinued in January of this year) but CentOS 7 has a good long lifetime ahead of it, therefore supporting it and RHEL 7 will require gcc 4.8.5 support.
Thanks; in that case using atomic types and other C11 features is not possible at all for quite some time, and openmp support, while available, is restricted to version 3.1 of the spec (which should be just about enough - earlier versions do not have task support).
I actually have gcc 4.8 here (on debian testing, which will become debian 9 in a couple of weeks), even though the default version is 6.3, so I can test 4.8 compatibility locally.
Alex
Signature · 4 lines
_______________________________________________ Openembedded-core mailing list [email protected] http://lists.openembedded.org/mailman/listinfo/openembedded-core
Quoted 2 lines · Michael
I just wanted to note that in my quest to get newer chrome to build I have used centos7 with the official devtoolset-6 scl.
It was able to build my pyro based distro fine in gcc6. So Centos7 has an officially supported path that may be viable if it comes down to being the last gcc 4.x distro.
I guess at some point we'll have to officially do this as well, as the upstream projects will adopt C/C++11 (or even newer) and otherwise become incompatible with 4.x.
Alex