Skip to contentSkip to conversation

openssl: parallel build

PATCH New Ryan Barnett 6 participants 11 messages Sep 16, 2015 – Nov 24, 2015

lore
Share & tools
lore

Ryan Barnett

author
#0 lore raw

v3 cover openssl: parallel build

I have adopted Thomas P's patches that we submitted but in May in order enable parallel build of openssl. I have updated the patches to use the latest patches available from gentoo and have tested builds only on i386, ARM, and MIPS.

Original Patches:

http://patchwork.ozlabs.org/patch/473086/
http://patchwork.ozlabs.org/patch/473087/

Ryan Barnett (2):
openssl: use git formatted patches
openssl: enable parallel build and installation

 ...time-building-manpages-if-we-re-not-going.patch | 27 ++++++++
 ...odev-Fix-issue-with-signature-generation.patch} | 71 +++++++++++++---------
 package/openssl/001-do-not-build-docs.patch        | 13 ----
 package/openssl/openssl.hash                       |  2 +
 package/openssl/openssl.mk                         | 12 ++--
 5 files changed, 78 insertions(+), 47 deletions(-)
 create mode 100644 package/openssl/0001-Dont-waste-time-building-manpages-if-we-re-not-going.patch
 rename package/openssl/{003-cryptodev-Fix-issue-with-signature-generation.patch => 0002-cryptodev-Fix-issue-with-signature-generation.patch} (82%)
 delete mode 100644 package/openssl/001-do-not-build-docs.patch
Signature · 1 line
1.9.1
author
#1 lore raw

PATCH v3 1/2 openssl: use git formatted patches

Using Git formatted patches makes it easier to adjust the patches when needed.

Signed-off-by Ryan Barnett <[email protected]>
CC Thomas Petazzoni <[email protected]>
CC Gustavo Zacarias <[email protected]>
CC Arnout Vandecappelle <[email protected]>

---
Changes v2 to v3:
- Added Gustavo's SOB back to his patch after accidently droping it (suggested by Baurch)

Changes v1 to v2:
- Move changes from patch 2/2 to here (suggested by Arnout)
- Note: git automatically detects a delete and new file for patch 001 to 0001. So there is no rename since the changes to the file are greater than 50%.

Signed-off-by Ryan Barnett <[email protected]>
---
 ...time-building-manpages-if-we-re-not-going.patch | 26 ++++++++
 ...odev-Fix-issue-with-signature-generation.patch} | 69 +++++++++++++---------
 package/openssl/001-do-not-build-docs.patch        | 13 ----
 3 files changed, 67 insertions(+), 41 deletions(-)
 create mode 100644 package/openssl/0001-Dont-waste-time-building-manpages-if-we-re-not-going.patch
 rename package/openssl/{003-cryptodev-Fix-issue-with-signature-generation.patch => 0002-cryptodev-Fix-issue-with-signature-generation.patch} (82%)
 delete mode 100644 package/openssl/001-do-not-build-docs.patch
package/openssl/0001-Dont-waste-time-building-manpages-if-we-re-not-going.patch new file +26 -0
new file mode 100644
index 0000000..8a0081e
@@ -0,0 +1,26 @@
+From 389efb564fa1453a9da835393eec9006bfae2a52 Mon Sep 17 00:00:00 2001
+From: Mike Frysinger <[email protected]>
+Date: Sat, 16 May 2015 18:53:51 +0200
+Subject: [1/2] Dont waste time building manpages if we're not going to use em.
+
+Signed-off-by: Ryan Barnett <[email protected]>
+---
+ Makefile.org | 2 +-
+ 1 file changed, 1 insertion(+), 1 deletion(-)
+
+diff --git a/Makefile.org b/Makefile.org
+index 60f07cc..976ceaf 100644
+--- a/Makefile.org
++++ b/Makefile.org
+@@ -527,7 +527,7 @@ dist:
+ dist_pem_h:
+ 	(cd crypto/pem; $(MAKE) -e $(BUILDENV) pem.h; $(MAKE) clean)
+ 
+-install: all install_docs install_sw
++install: all install_sw
+ 
+ install_sw:
+ 	@$(PERL) $(TOP)/util/mkdir-p.pl $(INSTALL_PREFIX)$(INSTALLTOP)/bin \
+-- 
+1.9.1
+
package/openssl/0002-cryptodev-Fix-issue-with-signature-generation.patch renamed from package/openssl/003-cryptodev-Fix-issue-with-signature-generation.patch +41 -28

232 lines not on this page · raw message

package/openssl/001-do-not-build-docs.patch deleted +0 -13
deleted file mode 100644
index 135a9d3..0000000
@@ -1,13 +0,0 @@
-Dont waste time building manpages if we're not going to use em.
-
---- openssl/Makefile.org
-+++ openssl/Makefile.org
-@@ -792,7 +792,7 @@
- dist_pem_h:
- 	(cd crypto/pem; $(MAKE) CC='${CC}' SDIRS='${SDIRS}' CFLAG='${CFLAG}' pem.h; $(MAKE) clean)
- 
--install: all install_docs install_sw
-+install: all install_sw
- 
- install_sw:
- 	@$(PERL) $(TOP)/util/mkdir-p.pl $(INSTALL_PREFIX)$(INSTALLTOP)/bin \
Signature · 1 line
1.9.1
Applied
#2 lore raw

PATCH v3 1/2 openssl: use git formatted patches

Ryan,

Quoted 1 line · Ryan Barnett

+Subject: [1/2] Dont waste time building manpages if we're not going to use em.

Arnout also suggested to get rid of [1/2] (and [2/2] for the second patch), since this information becomes obsolete when we add a new patch.

I've applied after doing this change. Thanks!

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com
author
#3 lore raw

PATCH v3 2/2 openssl: enable parallel build and installation

In reply to Ryan Barnett

This is a patch that is originally based on a patch Thomas P. submitted for an earlier version of this package. I have adopted this patch to use the latest available Gentoo parallel patch. I have also seen about a minute improvement on my build times of openssl.

Part of Thomas P's original message:

On my build server, the current build of OpenSSL takes 1 minutes and 20 seconds. With this commit applied, enabling parallel build and installation, the build only takes 28 seconds.

All the patches are downloaded from Gentoo.

There is apparently some interest in upstream OpenSSL to enable
parallel build, see for example commit
https://github.com/openssl/openssl/commit/c3f22253b139793ff3b91ff7e6969e180cf06815. This commit is not part of any OpenSSL release, but we can hope that the problem will resolved in the future.

Signed-off-by Ryan Barnett <[email protected]>
CC Thomas Petazzoni <[email protected]>
CC Gustavo Zacarias <[email protected]>
CC Arnout Vandecappelle <[email protected]>

---
Changes from v2 to v3:
- No changes

Changes from v1 to v2:
- Removed changes to existing patches in openssl (suggested by Arnout)
- Updated parallel patch URL to use Gentoo gitweb (suggested by Mike)

Signed-off-by Ryan Barnett <[email protected]>
---
 package/openssl/openssl.hash |  2 ++
 package/openssl/openssl.mk   | 12 +++++++-----
 2 files changed, 9 insertions(+), 5 deletions(-)
package/openssl/openssl.hash +2 -0
index 355be79..9eaa363 100644
@@ -1,2 +1,4 @@
 # From https://www.openssl.org/source/openssl-1.0.2d.tar.gz.sha256
 sha256	671c36487785628a703374c652ad2cebea45fa920ae5681515df25d9f2c9a8c8	openssl-1.0.2d.tar.gz
+# Locally computed
+sha256	deaf6f3af41874ecc6d63841ea14b8e6c71cea81d4a511a754bc90c9a993147f	openssl-1.0.2d-parallel-build.patch?id=56bd759df1d0c750a065b8c845e93d5dfa6b549d
package/openssl/openssl.mk +7 -5
index 3d87919..c1fc420 100644
@@ -13,6 +13,8 @@ OPENSSL_DEPENDENCIES = zlib
 HOST_OPENSSL_DEPENDENCIES = host-zlib
 OPENSSL_TARGET_ARCH = generic32
 OPENSSL_CFLAGS = $(TARGET_CFLAGS)
+OPENSSL_PATCH = \
+	https://gitweb.gentoo.org/repo/gentoo.git/plain/dev-libs/openssl/files/openssl-1.0.2d-parallel-build.patch?id=56bd759df1d0c750a065b8c845e93d5dfa6b549d
 
 ifeq ($(BR2_USE_MMU),)
 OPENSSL_CFLAGS += -DHAVE_FORK=0

@@ -99,23 +101,23 @@ OPENSSL_POST_CONFIGURE_HOOKS += OPENSSL_FIXUP_STATIC_MAKEFILE
 endif
 
 define HOST_OPENSSL_BUILD_CMDS
-	$(MAKE1) -C $(@D)
+	$(MAKE) -C $(@D)
 endef
 
 define OPENSSL_BUILD_CMDS
-	$(MAKE1) -C $(@D)
+	$(MAKE) -C $(@D)
 endef
 
 define OPENSSL_INSTALL_STAGING_CMDS
-	$(MAKE1) -C $(@D) INSTALL_PREFIX=$(STAGING_DIR) install
+	$(MAKE) -C $(@D) INSTALL_PREFIX=$(STAGING_DIR) install
 endef
 
 define HOST_OPENSSL_INSTALL_CMDS
-	$(MAKE1) -C $(@D) install
+	$(MAKE) -C $(@D) install
 endef
 
 define OPENSSL_INSTALL_TARGET_CMDS
-	$(MAKE1) -C $(@D) INSTALL_PREFIX=$(TARGET_DIR) install
+	$(MAKE) -C $(@D) INSTALL_PREFIX=$(TARGET_DIR) install
 	rm -rf $(TARGET_DIR)/usr/lib/ssl
 	rm -f $(TARGET_DIR)/usr/bin/c_rehash
 endef
Signature · 1 line
1.9.1
Applied
#4 lore raw

PATCH v3 2/2 openssl: enable parallel build and installation

Dear Ryan Barnett,

Quoted 19 lines · Ryan Barnett

This is a patch that is originally based on a patch Thomas P. submitted for an earlier version of this package. I have adopted this patch to use the latest available Gentoo parallel patch. I have also seen about a minute improvement on my build times of openssl.

Part of Thomas P's original message:

On my build server, the current build of OpenSSL takes 1 minutes and 20 seconds. With this commit applied, enabling parallel build and installation, the build only takes 28 seconds.

All the patches are downloaded from Gentoo.

There is apparently some interest in upstream OpenSSL to enable
parallel build, see for example commit
https://github.com/openssl/openssl/commit/c3f22253b139793ff3b91ff7e6969e180cf06815. This commit is not part of any OpenSSL release, but we can hope that the problem will resolved in the future.

Quoted 8 lines
Signed-off-by Ryan Barnett <[email protected]>
CC Thomas Petazzoni <[email protected]>
CC Gustavo Zacarias <[email protected]>
CC Arnout Vandecappelle <[email protected]>

---
Changes from v2 to v3:
- No changes

Applied, thanks.

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com
#5 lore raw

v3 cover openssl: parallel build

In reply to Ryan Barnett

Ryan,

Quoted 5 lines · Ryan Barnett

I have adopted Thomas P's patches that we submitted but in May in order enable parallel build of openssl. I have updated the patches to use the latest patches available from gentoo and have tested builds only on i386, ARM, and MIPS.

Quoted 8 lines

Original Patches:

http://patchwork.ozlabs.org/patch/473086/
http://patchwork.ozlabs.org/patch/473087/

Ryan Barnett (2):
openssl: use git formatted patches
openssl: enable parallel build and installation

I have the feeling that the parallel build patches for OpenSSL are not working properly. Look at the recent build failures we've got for the openssl and host-openssl builds:

http://autobuild.buildroot.org/?reason=openssl-1.0.2d
http://autobuild.buildroot.org/?reason=host-openssl-1.0.2d

They are pretty much all due to a similar error:

ln: failed to create symbolic link 'libssl.so': File exists
ln: creating symbolic link `libcrypto.so': File exists

I could spot only one that is different:

In file included from apps.h:117:0,

                 from req.c:74:
../include/openssl/bio.h:521:34: error: expected ';' before 'BIO_ctrl'
 # define BIO_set_fp(b,fp,c)      BIO_ctrl(b,BIO_C_SET_FILE_PTR,c,(char *)fp)

Do you have some ideas? Does Gentoo has some more patches? Should we revert back to non-parallel build for OpenSSL ?

Thanks,

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com
#6 lore raw

v3 cover openssl: parallel build

Hi Thomas,

Quoted 13 lines · Thomas Petazzoni

I have the feeling that the parallel build patches for OpenSSL are not working properly. Look at the recent build failures we've got for the openssl and host-openssl builds:

http://autobuild.buildroot.org/?reason=openssl-1.0.2d
http://autobuild.buildroot.org/?reason=host-openssl-1.0.2d

They are pretty much all due to a similar error:

ln: failed to create symbolic link 'libssl.so': File exists
ln: creating symbolic link `libcrypto.so': File exists

Quoted 8 lines

I could spot only one that is different:

In file included from apps.h:117:0,

                 from req.c:74:
../include/openssl/bio.h:521:34: error: expected ';' before 'BIO_ctrl'
 # define BIO_set_fp(b,fp,c)      BIO_ctrl(b,BIO_C_SET_FILE_PTR,c,(char *)fp)

Do you have some ideas?

I agree with you in that it looks like are some issues with the parallel build. It appears there is a hard to track down race condition due a symbolic being created twice during the build but as to way I don't have any clue.

Quoted 1 line

Does Gentoo has some more patches?

No this is the only patch for openssl 1.0.2d. There also hasn't been any updates to this patch other then the one that is currently there.

Quoted 1 line

Should we revert back to non-parallel build for OpenSSL ?

I guess I will leave that up to either yourself or Peter if you feel that it should. I don't have time in the next week or two to investigate the problem further so I'm OK with reverting this patch.

Thanks,
-Ryan

Signature · 5 lines
Ryan Barnett / Sr Software Engineer
Airborne Information Systems / Secure Platforms
MS 131-100, C Ave NE, Cedar Rapids, IA, 52498, USA
ryan.barnett at rockwellcollins.com
www.rockwellcollins.com
#7 lore raw

v3 cover openssl: parallel build

Quoted 24 lines · Ryan Barnett

Hi Thomas,

Quoted 20 lines · Thomas Petazzoni

I have the feeling that the parallel build patches for OpenSSL are not working properly. Look at the recent build failures we've got for the openssl and host-openssl builds:

http://autobuild.buildroot.org/?reason=openssl-1.0.2d
http://autobuild.buildroot.org/?reason=host-openssl-1.0.2d

They are pretty much all due to a similar error:

ln: failed to create symbolic link 'libssl.so': File exists
ln: creating symbolic link `libcrypto.so': File exists

I could spot only one that is different:

In file included from apps.h:117:0,

                 from req.c:74:
../include/openssl/bio.h:521:34: error: expected ';' before 'BIO_ctrl'
 # define BIO_set_fp(b,fp,c)      BIO_ctrl(b,BIO_C_SET_FILE_PTR,c,(char *)fp)
Quoted 6 lines
Quoted 1 line

Do you have some ideas?

I agree with you in that it looks like are some issues with the parallel build. It appears there is a hard to track down race condition due a symbolic being created twice during the build but as to way I don't have any clue.

I can see where the problem is, but not what the solution is... The
build-shared target depends on do_crypto and link-shared, which will be executed in parallel. do_crypto calls link_a.linux_shared -> link_a.gnu which does SYMLINK_SO; in parallel, link-shared calls symlink.linux_shared which also does SYMLINK_SO. Before the symlink is created, it is rm'ed, but there is a tiny chance that the second one is created after the rm has been called. Bam.

Maybe just replace the rm with an ln -sf? Don't know if upstream will like that
though.

Regards,
Arnout

Quoted 15 lines
Quoted 2 lines

Does Gentoo has some more patches?

No this is the only patch for openssl 1.0.2d. There also hasn't been any updates to this patch other then the one that is currently there.

Quoted 1 line

Should we revert back to non-parallel build for OpenSSL ?

I guess I will leave that up to either yourself or Peter if you feel that it should. I don't have time in the next week or two to investigate the problem further so I'm OK with reverting this patch.

Thanks,
-Ryan

Signature · 6 lines
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
#8 lore raw

v3 cover openssl: parallel build

In reply to Ryan Barnett
Quoted 5 lines · Ryan Barnett
Quoted 1 line · Thomas Petazzoni

Does Gentoo has some more patches?

No this is the only patch for openssl 1.0.2d. There also hasn't been any updates to this patch other then the one that is currently there.

that's incorrect. you have to look at the ebuild and then see that we apply four parallel patches for 1.0.2d, one of which specifically for this issue.

https://gitweb.gentoo.org/repo/gentoo.git/tree/dev-libs/openssl/openssl-1.0.2d-r2.ebuild?id=c8abcbe8de5d3b6cdd68c162f398c011ff6e2d9d

		epatch "${FILESDIR}"/${PN}-1.0.2d-parallel-build.patch
		epatch "${FILESDIR}"/${PN}-1.0.2a-parallel-obj-headers.patch
		epatch "${FILESDIR}"/${PN}-1.0.2a-parallel-install-dirs.patch
		epatch "${FILESDIR}"/${PN}-1.0.2a-parallel-symlinking.patch #545028

-mike
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: Digital signature
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20151124/dccb3f2f/attachment.asc>

#9 lore raw

v3 cover openssl: parallel build

In reply to Thomas Petazzoni

Thomas, All,

On 2015-11-18 13:57 +0100, Thomas Petazzoni spake thusly:

Quoted 15 lines · Ryan Barnett
Quoted 5 lines

I have adopted Thomas P's patches that we submitted but in May in order enable parallel build of openssl. I have updated the patches to use the latest patches available from gentoo and have tested builds only on i386, ARM, and MIPS.

Quoted 8 lines

Original Patches:

http://patchwork.ozlabs.org/patch/473086/
http://patchwork.ozlabs.org/patch/473087/

Ryan Barnett (2):
openssl: use git formatted patches
openssl: enable parallel build and installation

Quoted 6 lines

I have the feeling that the parallel build patches for OpenSSL are not working properly. Look at the recent build failures we've got for the openssl and host-openssl builds:

http://autobuild.buildroot.org/?reason=openssl-1.0.2d
http://autobuild.buildroot.org/?reason=host-openssl-1.0.2d

[--SNIP--]

Quoted 2 lines

Do you have some ideas? Does Gentoo has some more patches? Should we revert back to non-parallel build for OpenSSL ?

I believe we should just revert the patch.

It's unfortunate that we can't buid OpenSSL in parallel, but we can't seriously go and patch their broken buildsystem, not at least until they are willing to review and apply those patches upstream first.

Regards,
Yann E. MORIN.

Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#10 lore raw

v3 cover openssl: parallel build

Hello,

Quoted 8 lines · Yann E. MORIN
Quoted 2 lines

Do you have some ideas? Does Gentoo has some more patches? Should we revert back to non-parallel build for OpenSSL ?

I believe we should just revert the patch.

It's unfortunate that we can't buid OpenSSL in parallel, but we can't seriously go and patch their broken buildsystem, not at least until they are willing to review and apply those patches upstream first.

Agreed, we should revert the patch.

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com