openssl: fix race condition when symlink shared libs
PATCH v2 1/1 openssl: fix race condition when symlink shared libs
The build-shared target depends on do_crypto and link-shared, which will be executed in parallel. do_crypto calls link_a.linux_shared -> link_a.gnu which does SYMLINK_SO; in parallel, link-shared calls symlink.linux_shared which also does SYMLINK_SO. Before the symlink is created, it is rm'ed, but there is a tiny chance that the second one is created after the rm has been called.
Fix this by using 'ln -sf' instead of 'ln -s' so the build doesn't error out.
Patch submitted upstream at:
https://bugs.gentoo.org/show_bug.cgi?id=566260
Thanks to Arnout for explain the issue (wording used above).
---
Note this is a temporary fix until a gentoo maintainer weighs in and updates their parallel build patch.
v1 -> v2
Remove the 'rm -f' since we are using 'ln -sf' (Arnout originally)
--- ...ared-fix-race-condition-when-symlinking-s.patch | 48 ++++++++++++++++++++++ 1 file changed, 48 insertions(+) create mode 100644 package/openssl/0003-makefile.shared-fix-race-condition-when-symlinking-s.patch
package/openssl/0003-makefile.shared-fix-race-condition-when-symlinking-s.patch new file +48 -0
@@ -0,0 +1,48 @@ +From 878793890a9b84ff313397c56478d7c6f770b2d2 Mon Sep 17 00:00:00 2001 +From: Ryan Barnett <[email protected]> +Date: Thu, 19 Nov 2015 10:47:00 -0600 +Subject: [PATCH] makefile.shared: fix race condition when symlinking shared + libs + +The build-shared target depends on do_crypto and link-shared, which +will be executed in parallel. do_crypto calls +link_a.linux_shared -> link_a.gnu which does SYMLINK_SO; in parallel, +link-shared calls symlink.linux_shared which also does SYMLINK_SO. +Before the symlink is created, it is rm'ed, but there is a tiny chance +that the second one is created after the rm has been called. + +Fix this race condition by just using ln -sf since it will be the same +symlink regards and not cause the build to error out. + +Signed-off-by: Ryan Barnett <[email protected]> +--- + Makefile.shared | 8 ++++---- + 1 file changed, 4 insertions(+), 4 deletions(-) + +diff --git a/Makefile.shared b/Makefile.shared +index 8d57163..8ab3d18 100644 +--- a/Makefile.shared ++++ b/Makefile.shared +@@ -117,15 +117,15 @@ SYMLINK_SO= \ + prev=$$SHLIB$$SHLIB_SOVER$$SHLIB_SUFFIX; \ + if [ -n "$$SHLIB_COMPAT" ]; then \ + for x in $$SHLIB_COMPAT; do \ +- ( $(SET_X); rm -f $$SHLIB$$x$$SHLIB_SUFFIX; \ +- ln -s $$prev $$SHLIB$$x$$SHLIB_SUFFIX ); \ ++ ( $(SET_X); \ ++ ln -sf $$prev $$SHLIB$$x$$SHLIB_SUFFIX ); \ + prev=$$SHLIB$$x$$SHLIB_SUFFIX; \ + done; \ + fi; \ + if [ -n "$$SHLIB_SOVER" ]; then \ + [ -e "$$SHLIB$$SHLIB_SUFFIX" ] || \ +- ( $(SET_X); rm -f $$SHLIB$$SHLIB_SUFFIX; \ +- ln -s $$prev $$SHLIB$$SHLIB_SUFFIX ); \ ++ ( $(SET_X); \ ++ ln -sf $$prev $$SHLIB$$SHLIB_SUFFIX ); \ + fi; \ + fi + +-- +1.9.1 +
Signature · 1 line
1.9.1
PATCH v2 1/1 openssl: fix race condition when symlink shared libs
All,
Quoted 7 lines · Ryan Barnett
The build-shared target depends on
do_cryptoand link-shared, which will be executed in parallel.do_cryptocalls link_a.linux_shared -> link_a.gnu which doesSYMLINK_SO; in parallel, link-shared calls symlink.linux_shared which also doesSYMLINK_SO. Before the symlink is created, it is rm'ed, but there is a tiny chance that the second one is created after the rm has been called.
Quoted 5 lines
Fix this by using 'ln -sf' instead of 'ln -s' so the build doesn't error out.
Patch submitted upstream at:
https://bugs.gentoo.org/show_bug.cgi?id=566260
So it doesn't appear that using 'ln -sf' is a valid fix as pointed out in the bug report on the gentoo forum:
-- Comment #2 from SpanKY <[email protected]> ---
ln -sf is no more atomic than rm; ln. there is still a window where things can fail. you can easily check this:
i=0; while [ $((i++)) -lt 500 ]; do ln -sf a b & :; done
a good number of those will fail with:
ln: failed to create symbolic link ?b?: File exists
the deps need to be fixed up
So how do we want to proceed with this? Do we revert the patch for parallel builds for openssl?
Thanks,
-Ryan
PATCH v2 1/1 openssl: fix race condition when symlink shared libs
Quoted 26 lines · Ryan Barnett
All,
Quoted 7 lines · Ryan Barnett
The build-shared target depends on
do_cryptoand link-shared, which will be executed in parallel.do_cryptocalls link_a.linux_shared -> link_a.gnu which doesSYMLINK_SO; in parallel, link-shared calls symlink.linux_shared which also doesSYMLINK_SO. Before the symlink is created, it is rm'ed, but there is a tiny chance that the second one is created after the rm has been called.Quoted 5 lines
Fix this by using 'ln -sf' instead of 'ln -s' so the build doesn't error out.
Patch submitted upstream at:
https://bugs.gentoo.org/show_bug.cgi?id=566260So it doesn't appear that using 'ln -sf' is a valid fix as pointed out in the bug report on the gentoo forum:
-- Comment #2 from SpanKY <[email protected]> ---
ln -sfis no more atomic thanrm; ln. there is still a window where things can fail. you can easily check this:i=0; while [ $((i++)) -lt 500 ]; do ln -sf a b & :; done
Quoted 7 lines
a good number of those will fail with:
ln: failed to create symbolic link ?b?: File existsthe deps need to be fixed up
So how do we want to proceed with this? Do we revert the patch for parallel builds for openssl?
I think for 2015.11 at least we should indeed revert it.
The symlink can be created atomically by going through a temp file, but that's
a bit crazy...
Regards,
Arnout
Quoted 4 lines
Thanks,
-Ryan
Signature · 6 lines
Arnout Vandecappelle arnout at mind be Senior Embedded Software Architect +32-16-286500 Essensium/Mind http://www.mind.be G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle GPG fingerprint: 7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
PATCH v2 1/1 openssl: fix race condition when symlink shared libs
In reply to Ryan BarnettQuoted 7 lines · Ryan Barnett
The build-shared target depends on
do_cryptoand link-shared, which will be executed in parallel.do_cryptocalls link_a.linux_shared -> link_a.gnu which doesSYMLINK_SO; in parallel, link-shared calls symlink.linux_shared which also doesSYMLINK_SO. Before the symlink is created, it is rm'ed, but there is a tiny chance that the second one is created after the rm has been called.
Quoted 5 lines
Fix this by using 'ln -sf' instead of 'ln -s' so the build doesn't error out.
Patch submitted upstream at:
https://bugs.gentoo.org/show_bug.cgi?id=566260
Ahem, gentoo is not exactly upstream :-) From [1]:
To report a bug or make an enhancement request, send email to rt at openssl.org. In the subject line, please make sure to indicate if it's a bug or a fix, and a brief description of the issue. In the body of your mail, please include the versions of the operating system and OpenSSL you are using. If you have a patch or diff, please send it as an attachment, and not inline in the message body.
As far as I can see, this patch should still apply (in slightly adapted form)
to the real upstream.
Quoted 2 lines
Thanks to Arnout for explain the issue (wording used above).
You're a native English speaker, right? explain -> explaining :-)
Some more minor things below, but anyway more for the upstreams. So
Quoted 28 lines
Signed-off-by Ryan Barnett <[email protected]>CC Arnout Vandecappelle <[email protected]>CC Thomas Petazzoni <[email protected]>---
Note this is a temporary fix until a gentoo maintainer weighs in and updates their parallel build patch.
v1 -> v2
Remove the 'rm -f' since we are using 'ln -sf' (Arnout originally)--- ...ared-fix-race-condition-when-symlinking-s.patch | 48 ++++++++++++++++++++++ 1 file changed, 48 insertions(+) create mode 100644 package/openssl/0003-makefile.shared-fix-race-condition-when-symlinking-s.patchpackage/openssl/0003-makefile.shared-fix-race-condition-when-symlinking-s.patch new file +5
-0@@ -0,0 +1,48 @@ +From 878793890a9b84ff313397c56478d7c6f770b2d2 Mon Sep 17 00:00:00 2001 +From: Ryan Barnett <[email protected]> +Date: Thu, 19 Nov 2015 10:47:00 -0600 +Subject: [PATCH] makefile.shared: fix race condition when symlinking shared + libs
package/openssl/0003-makefile.shared-fix-race-condition-when-symlinking-s.patch line 15
Quoted 10 lines
package/openssl/0003-makefile.shared-fix-race-condition-when-symlinking-s.patch new file +10
-0+ +The build-shared target depends on do_crypto and link-shared, which +will be executed in parallel. do_crypto calls +link_a.linux_shared -> link_a.gnu which does SYMLINK_SO; in parallel, +link-shared calls symlink.linux_shared which also does SYMLINK_SO. +Before the symlink is created, it is rm'ed, but there is a tiny chance +that the second one is created after the rm has been called. + +Fix this race condition by just using ln -sf since it will be the same +symlink regards and not cause the build to error out.
regards -> regardless
Quoted 13 lines
package/openssl/0003-makefile.shared-fix-race-condition-when-symlinking-s.patch new file +13
-0+ +Signed-off-by: Ryan Barnett <[email protected]> +--- + Makefile.shared | 8 ++++---- + 1 file changed, 4 insertions(+), 4 deletions(-) + +diff --git a/Makefile.shared b/Makefile.shared +index 8d57163..8ab3d18 100644 +--- a/Makefile.shared ++++ b/Makefile.shared +@@ -117,15 +117,15 @@ SYMLINK_SO= \ + prev=$$SHLIB$$SHLIB_SOVER$$SHLIB_SUFFIX; \ + if [ -n "$$SHLIB_COMPAT" ]; then \
package/openssl/0003-makefile.shared-fix-race-condition-when-symlinking-s.patch line 38
Quoted 10 lines
package/openssl/0003-makefile.shared-fix-race-condition-when-symlinking-s.patch new file +10
-0+ for x in $$SHLIB_COMPAT; do \ +- ( $(SET_X); rm -f $$SHLIB$$x$$SHLIB_SUFFIX; \ +- ln -s $$prev $$SHLIB$$x$$SHLIB_SUFFIX ); \ ++ ( $(SET_X); \ ++ ln -sf $$prev $$SHLIB$$x$$SHLIB_SUFFIX ); \ + prev=$$SHLIB$$x$$SHLIB_SUFFIX; \ + done; \ + fi; \ + if [ -n "$$SHLIB_SOVER" ]; then \ + [ -e "$$SHLIB$$SHLIB_SUFFIX" ] || \
This check is not really needed anymore.
Regards,
Arnout
Quoted 11 lines
package/openssl/0003-makefile.shared-fix-race-condition-when-symlinking-s.patch new file +10
-0+- ( $(SET_X); rm -f $$SHLIB$$SHLIB_SUFFIX; \ +- ln -s $$prev $$SHLIB$$SHLIB_SUFFIX ); \ ++ ( $(SET_X); \ ++ ln -sf $$prev $$SHLIB$$SHLIB_SUFFIX ); \ + fi; \ + fi + +-- +1.9.1 +
Signature · 6 lines
Arnout Vandecappelle arnout at mind be Senior Embedded Software Architect +32-16-286500 Essensium/Mind http://www.mind.be G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle GPG fingerprint: 7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
PATCH v2 1/1 openssl: fix race condition when symlink shared libs
Arnout,
Quoted 16 lines · Arnout Vandecappelle
Quoted 7 lines · Ryan Barnett
The build-shared target depends on
do_cryptoand link-shared, which will be executed in parallel.do_cryptocalls link_a.linux_shared -> link_a.gnu which doesSYMLINK_SO; in parallel, link-shared calls symlink.linux_shared which also doesSYMLINK_SO. Before the symlink is created, it is rm'ed, but there is a tiny chance that the second one is created after the rm has been called.Quoted 5 lines
Fix this by using 'ln -sf' instead of 'ln -s' so the build doesn't error out.
Patch submitted upstream at:
https://bugs.gentoo.org/show_bug.cgi?id=566260Ahem, gentoo is not exactly upstream :-) From [1]:
Quoted 8 lines
To report a bug or make an enhancement request, send email to rt at openssl.org. In the subject line, please make sure to indicate if it's a bug or a fix, and a brief description of the issue. In the body of your mail, please include the versions of the operating system and OpenSSL you are using. If you have a patch or diff, please send it as an attachment, and not inline in the message body.
As far as I can see, this patch should still apply (in slightly adapted form)
to the real upstream.
Yes I agree to submit it upstream directly to openssl. However, I started with Gentoo since there are the ones that seem most interested in making OpenSSL work in parallel. I have also just sent an email to OpenSSL about the issue.
I think this patch is pointless now since it doesn't actually fix the problem that existed before. Please see my reply to this patch earlier describing that 'ln -sf' isn't an atomic operation and the error from the build failures still will exist:
http://lists.busybox.net/pipermail/buildroot/2015-November/144882.html
Quoted 4 lines
Quoted 2 lines
Thanks to Arnout for explain the issue (wording used above).
You're a native English speaker, right? explain -> explaining :-)
Yes but I never claimed to be great at translating my English thoughts to properly written sentence ;) This is especially true when I type up something in a hurry which this commit message was.
Thanks,
-Ryan
Signature · 5 lines
Ryan Barnett / Sr Software Engineer Airborne Information Systems / Secure Platforms MS 131-100, C Ave NE, Cedar Rapids, IA, 52498, USA ryan.barnett at rockwellcollins.com www.rockwellcollins.com
PATCH v2 1/1 openssl: fix race condition when symlink shared libs
In reply to Arnout VandecappelleQuoted 11 lines · Arnout Vandecappelle
Quoted 10 lines · Ryan Barnett
The build-shared target depends on
do_cryptoand link-shared, which will be executed in parallel.do_cryptocalls link_a.linux_shared -> link_a.gnu which doesSYMLINK_SO; in parallel, link-shared calls symlink.linux_shared which also doesSYMLINK_SO. Before the symlink is created, it is rm'ed, but there is a tiny chance that the second one is created after the rm has been called.Fix this by using 'ln -sf' instead of 'ln -s' so the build doesn't error out.
Quoted 6 lines
Quoted 2 lines
Patch submitted upstream at:
https://bugs.gentoo.org/show_bug.cgi?id=566260Ahem, gentoo is not exactly upstream :-) From [1]:
To report a bug or make an enhancement request, send email to rt at openssl.org.
which these have been already. they don't respond. in fact, you can see
a link to the relevant upstream report in every Gentoo parallel patch.
-mike
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: Digital signature
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20151125/729563ba/attachment.asc>