Skip to contentSkip to conversation

package/libopenssl: add (proposed) upstream fix for parallel installation

PATCH In review Peter Korsgaard 3 participants 6 messages Apr 15, 2025

lore
Share & tools
lore

Peter Korsgaard

author
#0 lore raw

PATCH 1/2 package/libopenssl: add (proposed) upstream fix for parallel installation

The openssl developers have proposed a fix for the parallel installation issue worked around by commit 27ab880ebb9a (package/libopenssl do not build in parallel).

Add the fix here so the workaround can dropped again.

Signed-off-by Peter Korsgaard <[email protected]>
---
 ...l-process-to-avoid-multiple-make-dep.patch | 44 +++++++++++++++++++
 1 file changed, 44 insertions(+)
 create mode 100644 package/libopenssl/0004-Serialize-install-process-to-avoid-multiple-make-dep.patch
package/libopenssl/0004-Serialize-install-process-to-avoid-multiple-make-dep.patch new file +44 -0
new file mode 100644
index 0000000000..feb0e21ef7
@@ -0,0 +1,44 @@
+From 4562b002366535b3d387fae7fa1dbd8e315ae142 Mon Sep 17 00:00:00 2001
+From: Neil Horman <[email protected]>
+Date: Mon, 14 Apr 2025 12:23:15 -0400
+Subject: [PATCH] Serialize install process to avoid multiple make depend
+ operations
+
+If make install is run with a large -j value (make install -j N , where
+N > 1)
+
+We can run into a situation in which the install fails because multiple
+make depend operations are running in parallel, which will fail due to
+makefile rewriting.
+
+Serialize the install process to guarantee that those operations don't
+step on one another
+
+Fixes # 27074
+
+Upstream: https://github.com/openssl/openssl/pull/27388
+Signed-off-by: Peter Korsgaard <[email protected]>
+---
+ Configurations/unix-Makefile.tmpl | 6 +++++-
+ 1 file changed, 5 insertions(+), 1 deletion(-)
+
+diff --git a/Configurations/unix-Makefile.tmpl b/Configurations/unix-Makefile.tmpl
+index e85763ccf8..b671723813 100644
+--- a/Configurations/unix-Makefile.tmpl
++++ b/Configurations/unix-Makefile.tmpl
+@@ -650,7 +650,11 @@ depend: Makefile
+ # Install helper targets #############################################
+ ##@ Installation
+ 
+-install: install_sw install_ssldirs {- "install_docs" if !$disabled{docs}; -} {- $disabled{fips} ? "" : "install_fips" -} ## Install software and documentation, create OpenSSL directories
++install: Makefile ## Install software and documentation, create OpenSSL directories
++	$(MAKE) install_sw
++	$(MAKE) install_ssldirs
++	{- "\$(MAKE) install_docs" if !$disabled{docs} -}
++	{- "\$(MAKE) install_fips" if !$disabled{fips} -}
+ 
+ uninstall: {- "uninstall_docs" if !$disabled{docs}; -} uninstall_sw {- $disabled{fips} ? "" : "uninstall_fips" -} ## Uninstall software and documentation
+ 
+-- 
+2.39.5
+
Signature · 6 lines
2.39.5

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
author
#1 lore raw

PATCH 2/2 Revert "package/libopenssl do not build in parallel"

This reverts commit 27ab880ebb9a3efdec15733841f7b88063604247.

With the (proposed) fix from the openssl developers added as 0004-Serialize-install-process-to-avoid-multiple-make-dep.patch, the workaround can now be dropped so openssl can again be built and installed in parallel, significantly speeding up builds.

Signed-off-by Peter Korsgaard <[email protected]>
---
 package/libopenssl/libopenssl.mk | 12 +++++-------
 1 file changed, 5 insertions(+), 7 deletions(-)
package/libopenssl/libopenssl.mk +5 -7
index 663e8d9b9f..c1c5b818d7 100644
@@ -132,26 +132,24 @@ endef
 LIBOPENSSL_POST_CONFIGURE_HOOKS += LIBOPENSSL_FIXUP_STATIC_MAKEFILE
 endif
 
-# Parallel build broken: https://github.com/openssl/openssl/issues/27074
 define HOST_LIBOPENSSL_BUILD_CMDS
-	$(HOST_MAKE_ENV) $(MAKE1) -C $(@D)
+	$(HOST_MAKE_ENV) $(MAKE) -C $(@D)
 endef
 
-# Parallel build broken: https://github.com/openssl/openssl/issues/27074
 define LIBOPENSSL_BUILD_CMDS
-	$(TARGET_MAKE_ENV) $(MAKE1) -C $(@D)
+	$(TARGET_MAKE_ENV) $(MAKE) -C $(@D)
 endef
 
 define LIBOPENSSL_INSTALL_STAGING_CMDS
-	$(TARGET_MAKE_ENV) $(MAKE1) -C $(@D) DESTDIR=$(STAGING_DIR) install
+	$(TARGET_MAKE_ENV) $(MAKE) -C $(@D) DESTDIR=$(STAGING_DIR) install
 endef
 
 define HOST_LIBOPENSSL_INSTALL_CMDS
-	$(HOST_MAKE_ENV) $(MAKE1) -C $(@D) install
+	$(HOST_MAKE_ENV) $(MAKE) -C $(@D) install
 endef
 
 define LIBOPENSSL_INSTALL_TARGET_CMDS
-	$(TARGET_MAKE_ENV) $(MAKE1) -C $(@D) DESTDIR=$(TARGET_DIR) install
+	$(TARGET_MAKE_ENV) $(MAKE) -C $(@D) DESTDIR=$(TARGET_DIR) install
 	$(RM) -rf $(TARGET_DIR)/usr/lib/ssl
 	$(RM) -f $(TARGET_DIR)/usr/bin/c_rehash
 endef
Signature · 6 lines
2.39.5

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
Reviewed-by on v1 2/2
#2 lore raw

Re: [Buildroot] [PATCH 2/2] Revert "package/libopenssl do not build in parallel"

Pter, All,

On 2025-04-15 22:00 +0200, Peter Korsgaard spake thusly:

Quoted 1 line

This reverts commit 27ab880ebb9a3efdec15733841f7b88063604247.

I don't think this really is a "revert". Reverting a commit is for when that commit was wrong when it was applied. But here, the commit was not wrong, so it should not be "reverted". Even if the change is technically undone, that's semantically a different thing. But oh well... ;-)

Quoted 4 lines

With the (proposed) fix from the openssl developers added as 0004-Serialize-install-process-to-avoid-multiple-make-dep.patch, the workaround can now be dropped so openssl can again be built and installed in parallel, significantly speeding up builds.

Yes, that was a shame that we had to lose parallelism...

Note that I also hit a hihly-parallel highly-loaded build (not install) issue that is still not fixed. Alas, that one is also very, very difficult to hit. We can probably ignore it until I get bitten again and get more traces to inspect...

Quoted 1 line
Signed-off-by Peter Korsgaard <[email protected]>
Reviewed-by Yann E. MORIN <[email protected]>

Regards,
Yann E. MORIN.

Quoted 43 lines
---
 package/libopenssl/libopenssl.mk | 12 +++++-------
 1 file changed, 5 insertions(+), 7 deletions(-)
package/libopenssl/libopenssl.mk +5 -7
index 663e8d9b9f..c1c5b818d7 100644
@@ -132,26 +132,24 @@ endef
 LIBOPENSSL_POST_CONFIGURE_HOOKS += LIBOPENSSL_FIXUP_STATIC_MAKEFILE
 endif
 
-# Parallel build broken: https://github.com/openssl/openssl/issues/27074
 define HOST_LIBOPENSSL_BUILD_CMDS
-	$(HOST_MAKE_ENV) $(MAKE1) -C $(@D)
+	$(HOST_MAKE_ENV) $(MAKE) -C $(@D)
 endef
 
-# Parallel build broken: https://github.com/openssl/openssl/issues/27074
 define LIBOPENSSL_BUILD_CMDS
-	$(TARGET_MAKE_ENV) $(MAKE1) -C $(@D)
+	$(TARGET_MAKE_ENV) $(MAKE) -C $(@D)
 endef
 
 define LIBOPENSSL_INSTALL_STAGING_CMDS
-	$(TARGET_MAKE_ENV) $(MAKE1) -C $(@D) DESTDIR=$(STAGING_DIR) install
+	$(TARGET_MAKE_ENV) $(MAKE) -C $(@D) DESTDIR=$(STAGING_DIR) install
 endef
 
 define HOST_LIBOPENSSL_INSTALL_CMDS
-	$(HOST_MAKE_ENV) $(MAKE1) -C $(@D) install
+	$(HOST_MAKE_ENV) $(MAKE) -C $(@D) install
 endef
 
 define LIBOPENSSL_INSTALL_TARGET_CMDS
-	$(TARGET_MAKE_ENV) $(MAKE1) -C $(@D) DESTDIR=$(TARGET_DIR) install
+	$(TARGET_MAKE_ENV) $(MAKE) -C $(@D) DESTDIR=$(TARGET_DIR) install
 	$(RM) -rf $(TARGET_DIR)/usr/lib/ssl
 	$(RM) -f $(TARGET_DIR)/usr/bin/c_rehash
 endef
Signature · 1 line
2.39.5
Signature · 10 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
author
#3 lore raw

Re: [Buildroot] [PATCH 2/2] Revert "package/libopenssl do not build in parallel"

Quoted 10 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Yann" == Yann E MORIN <[email protected]> writes:

Pter, All,
On 2025-04-15 22:00 +0200, Peter Korsgaard spake thusly:

Quoted 1 line

This reverts commit 27ab880ebb9a3efdec15733841f7b88063604247.

I don't think this really is a "revert". Reverting a commit is for when that commit was wrong when it was applied. But here, the commit was not wrong, so it should not be "reverted". Even if the change is technically undone, that's semantically a different thing. But oh well... ;-)

Well, it reverted the workaround?

Quoted 6 lines
Quoted 4 lines

With the (proposed) fix from the openssl developers added as 0004-Serialize-install-process-to-avoid-multiple-make-dep.patch, the workaround can now be dropped so openssl can again be built and installed in parallel, significantly speeding up builds.

Yes, that was a shame that we had to lose parallelism...

I agree, especially since openssl takes so long to build. I have a defconfig at $WORK where openssl is the package taking longest to build. This defconfig builds both libopenssl and host-libopenssl, and together they take over 25% of the build time.

Quoted 4 lines

Note that I also hit a hihly-parallel highly-loaded build (not install) issue that is still not fixed. Alas, that one is also very, very difficult to hit. We can probably ignore it until I get bitten again and get more traces to inspect...

OK. I have so far not been able to trigger that here.

Quoted 3 lines
Quoted 1 line
Signed-off-by Peter Korsgaard <[email protected]>
Reviewed-by Yann E. MORIN <[email protected]>

Thanks!

Signature · 5 lines
Bye, Peter Korsgaard
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
Reviewed-by on v1 1/2
#4 lore raw
In reply to Peter Korsgaard

Peter, All,

On 2025-04-15 22:00 +0200, Peter Korsgaard spake thusly:

Quoted 7 lines

The openssl developers have proposed a fix for the parallel installation issue worked around by commit 27ab880ebb9a (package/libopenssl do not build in parallel).

Add the fix here so the workaround can dropped again.

Signed-off-by Peter Korsgaard <[email protected]>
Reviewed-by Yann E. MORIN <[email protected]>

Regards,
Yann E. MORIN.

Quoted 58 lines
---
 ...l-process-to-avoid-multiple-make-dep.patch | 44 +++++++++++++++++++
 1 file changed, 44 insertions(+)
 create mode 100644 package/libopenssl/0004-Serialize-install-process-to-avoid-multiple-make-dep.patch
package/libopenssl/0004-Serialize-install-process-to-avoid-multiple-make-dep.patch new file +44 -0
new file mode 100644
index 0000000000..feb0e21ef7
@@ -0,0 +1,44 @@
+From 4562b002366535b3d387fae7fa1dbd8e315ae142 Mon Sep 17 00:00:00 2001
+From: Neil Horman <[email protected]>
+Date: Mon, 14 Apr 2025 12:23:15 -0400
+Subject: [PATCH] Serialize install process to avoid multiple make depend
+ operations
+
+If make install is run with a large -j value (make install -j N , where
+N > 1)
+
+We can run into a situation in which the install fails because multiple
+make depend operations are running in parallel, which will fail due to
+makefile rewriting.
+
+Serialize the install process to guarantee that those operations don't
+step on one another
+
+Fixes # 27074
+
+Upstream: https://github.com/openssl/openssl/pull/27388
+Signed-off-by: Peter Korsgaard <[email protected]>
+---
+ Configurations/unix-Makefile.tmpl | 6 +++++-
+ 1 file changed, 5 insertions(+), 1 deletion(-)
+
+diff --git a/Configurations/unix-Makefile.tmpl b/Configurations/unix-Makefile.tmpl
+index e85763ccf8..b671723813 100644
+--- a/Configurations/unix-Makefile.tmpl
++++ b/Configurations/unix-Makefile.tmpl
+@@ -650,7 +650,11 @@ depend: Makefile
+ # Install helper targets #############################################
+ ##@ Installation
+ 
+-install: install_sw install_ssldirs {- "install_docs" if !$disabled{docs}; -} {- $disabled{fips} ? "" : "install_fips" -} ## Install software and documentation, create OpenSSL directories
++install: Makefile ## Install software and documentation, create OpenSSL directories
++	$(MAKE) install_sw
++	$(MAKE) install_ssldirs
++	{- "\$(MAKE) install_docs" if !$disabled{docs} -}
++	{- "\$(MAKE) install_fips" if !$disabled{fips} -}
+ 
+ uninstall: {- "uninstall_docs" if !$disabled{docs}; -} uninstall_sw {- $disabled{fips} ? "" : "uninstall_fips" -} ## Uninstall software and documentation
+ 
+-- 
+2.39.5
+
Signature · 1 line
2.39.5
Signature · 10 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#5 lore raw
In reply to Peter Korsgaard
Quoted 7 lines · Peter Korsgaard

The openssl developers have proposed a fix for the parallel installation issue worked around by commit 27ab880ebb9a (package/libopenssl do not build in parallel).

Add the fix here so the workaround can dropped again.

Signed-off-by Peter Korsgaard <[email protected]>

Series applied to master, thanks.
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot