Skip to contentSkip to conversation

boost: fix static detection of lock-free atomic ints

PATCH Superseded Fabrice Fontaine 2 participants 3 messages Sep 4, 2018 – Sep 5, 2018

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 boost: fix static detection of lock-free atomic ints

When build statically, boost is unable to detect lock-free atomics ints because it tries to link dynamically, see output/build/boost-1.67.0/bin.v2/config.log (with -d5 option):

Using shell: /bin/sh -c

    argv[0] = '/bin/sh'
    argv[1] = '-c'
    argv[2] = '
    "/home/fabrice/buildroot/output/host/bin/arm-linux-g++"   -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64  -Os   -Wl,-elf2flt -static  -Wl,-elf2flt -static -fPIC -pthread -O0 -fno-inline -Wall -pedantic -g -Wextra -Wno-long-long -Wno-unused-parameter -Wunused-function -pedantic -DBOOST_ALL_NO_LIB=1 -DBOOST_SYSTEM_DYN_LINK=1 -DBOOST_THREAD_BUILD_DLL=1  -I"." -c -o "bin.v2/libs/thread/build/gcc-6.4.0/debug/threading-multi/has_atomic_flag_lockfree_test.o" "libs/thread/src/../build/has_atomic_flag_lockfree_test.cpp"

'
gcc.compile.c++ bin.v2/libs/thread/build/gcc-6.4.0/debug/threading-multi/has_atomic_flag_lockfree_test.o

"/home/fabrice/buildroot/output/host/bin/arm-linux-g++" -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -Os -Wl,-elf2flt -static -Wl,-elf2flt -static -fPIC -pthread -O0 -fno-inline -Wall -pedantic -g -Wextra -Wno-long-long -Wno-unused-parameter -Wunused-function -pedantic -DBOOST_ALL_NO_LIB=1 -DBOOST_SYSTEM_DYN_LINK=1 -DBOOST_THREAD_BUILD_DLL=1 -I"." -c -o "bin.v2/libs/thread/build/gcc-6.4.0/debug/threading-multi/has_atomic_flag_lockfree_test.o" "libs/thread/src/../build/has_atomic_flag_lockfree_test.cpp"

0.033561 sec system; 0.126314 sec user; 288.682473 sec clock gcc.compile.c++ bin.v2/libs/system/build/gcc-6.4.0/debug/threading-multi/error_code.o

"/home/fabrice/buildroot/output/host/bin/arm-linux-g++" -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -Os -Wl,-elf2flt -static -Wl,-elf2flt -static -fPIC -pthread -O0 -fno-inline -Wall -pedantic -g -DBOOST_ALL_NO_LIB=1 -DBOOST_SYSTEM_DYN_LINK=1 -I"." -c -o "bin.v2/libs/system/build/gcc-6.4.0/debug/threading-multi/error_code.o" "libs/system/src/error_code.cpp"

0.084060 sec system; 0.644133 sec user; 8.858824 sec clock
SEM: <s>gcc-link-semaphore now used by <pbin.v2/libs/system/build/gcc-6.4.0/debug/threading-multi>libboost_system.so.1.67.0 Using shell: /bin/sh -c

    argv[0] = '/bin/sh'
    argv[1] = '-c'
    argv[2] = '
    "/home/fabrice/buildroot/output/host/bin/arm-linux-g++"    -o "bin.v2/libs/system/build/gcc-6.4.0/debug/threading-multi/libboost_system.so.1.67.0" -Wl,-h -Wl,libboost_system.so.1.67.0 -shared -Wl,--start-group "bin.v2/libs/system/build/gcc-6.4.0/debug/threading-multi/error_code.o"  -Wl,-Bstatic  -Wl,-Bdynamic -lrt -Wl,--end-group -fPIC -pthread -g  -Wl,-elf2flt -static

'
gcc.link.dll bin.v2/libs/system/build/gcc-6.4.0/debug/threading-multi/libboost_system.so.1.67.0

    "/home/fabrice/buildroot/output/host/bin/arm-linux-g++"    -o "bin.v2/libs/system/build/gcc-6.4.0/debug/threading-multi/libboost_system.so.1.67.0" -Wl,-h -Wl,libboost_system.so.1.67.0 -shared -Wl,--start-group "bin.v2/libs/system/build/gcc-6.4.0/debug/threading-multi/error_code.o"  -Wl,-Bstatic  -Wl,-Bdynamic -lrt -Wl,--end-group -fPIC -pthread -g  -Wl,-elf2flt -static

ld (ld-elf2flt): -shared used without passing a shared library ID
collect2: error: ld a retourn? le statut de sortie 1
0.003123 sec system; 0.004732 sec user; 15.646509 sec clock
...failed gcc.link.dll bin.v2/libs/system/build/gcc-6.4.0/debug/threading-multi/libboost_system.so.1.67.0...

Due to this wrong detection, boost_atomic is wrongly selected by boost_thread which cause a build failure on missing select

Add a hack to fix this behavior in libs/thread/build/Jamfile.v2 as I'm not sufficiently knowledgeable in bjam to know why the configuration is not correctly passed

Fixes:
- http://autobuild.buildroot.org/results/f46d38991385cbc2a4fa14eb31074e770cd79803

---
Changes v1 -> v2 (after review of Arnout Vandecappelle):
- Fix URL of build failure

 package/boost/boost.mk | 10 ++++++++++
 1 file changed, 10 insertions(+)
package/boost/boost.mk +10 -0
index b2605a70b7..fe986edb00 100644
@@ -106,6 +106,16 @@ endif
 # variants. Override that if we want static only or shared only.
 ifeq ($(BR2_STATIC_LIBS),y)
 BOOST_OPTS += link=static runtime-link=static
+
+# When detecting features such as lockfree atomic ints, boost does not take into
+# account the configuration provided by the user. Indeed, boost does not specify
+# any value and as a result the default value (shared) is used
+define BOOST_FIX_STATIC_FEATURES_DETECTION
+$(SED) 's/has_atomic_flag_lockfree_test.cpp/has_atomic_flag_lockfree_test.cpp : <link>static <runtime-link>static/g;' \
+		$(@D)/libs/thread/build/Jamfile.v2
+endef
+
+BOOST_PRE_CONFIGURE_HOOKS += BOOST_FIX_STATIC_FEATURES_DETECTION
 else ifeq ($(BR2_SHARED_LIBS),y)
 BOOST_OPTS += link=shared runtime-link=shared
 endif
Signature · 1 line
2.17.1
#1 lore raw

PATCH v2 1/1 boost: fix static detection of lock-free atomic ints

Hello,

Thanks for investigating this issue, I know what it takes to dive into the Boost build system, however, I am a bit confused by what's happening.

Quoted 5 lines · Fabrice Fontaine

When build statically, boost is unable to detect lock-free atomics ints because it tries to link dynamically, see output/build/boost-1.67.0/bin.v2/config.log (with -d5 option):

Using shell: /bin/sh -c

Quoted 8 lines
    argv[0] = '/bin/sh'
    argv[1] = '-c'
    argv[2] = '
    "/home/fabrice/buildroot/output/host/bin/arm-linux-g++"   -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64  -Os   -Wl,-elf2flt -static  -Wl,-elf2flt -static -fPIC -pthread -O0 -fno-inline -Wall -pedantic -g -Wextra -Wno-long-long -Wno-unused-parameter -Wunused-function -pedantic -DBOOST_ALL_NO_LIB=1 -DBOOST_SYSTEM_DYN_LINK=1 -DBOOST_THREAD_BUILD_DLL=1  -I"." -c -o "bin.v2/libs/thread/build/gcc-6.4.0/debug/threading-multi/has_atomic_flag_lockfree_test.o" "libs/thread/src/../build/has_atomic_flag_lockfree_test.cpp"

'
gcc.compile.c++ bin.v2/libs/thread/build/gcc-6.4.0/debug/threading-multi/has_atomic_flag_lockfree_test.o

"/home/fabrice/buildroot/output/host/bin/arm-linux-g++" -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -Os -Wl,-elf2flt -static -Wl,-elf2flt -static -fPIC -pthread -O0 -fno-inline -Wall -pedantic -g -Wextra -Wno-long-long -Wno-unused-parameter -Wunused-function -pedantic -DBOOST_ALL_NO_LIB=1 -DBOOST_SYSTEM_DYN_LINK=1 -DBOOST_THREAD_BUILD_DLL=1 -I"." -c -o "bin.v2/libs/thread/build/gcc-6.4.0/debug/threading-multi/has_atomic_flag_lockfree_test.o" "libs/thread/src/../build/has_atomic_flag_lockfree_test.cpp"

So here, it is compiling (no linking) the has_atomic_flag_lockfree_test.cpp test file into its corresponding object file. It does not use -shared at this point.

Quoted 4 lines

0.033561 sec system; 0.126314 sec user; 288.682473 sec clock gcc.compile.c++ bin.v2/libs/system/build/gcc-6.4.0/debug/threading-multi/error_code.o

"/home/fabrice/buildroot/output/host/bin/arm-linux-g++" -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -Os -Wl,-elf2flt -static -Wl,-elf2flt -static -fPIC -pthread -O0 -fno-inline -Wall -pedantic -g -DBOOST_ALL_NO_LIB=1 -DBOOST_SYSTEM_DYN_LINK=1 -I"." -c -o "bin.v2/libs/system/build/gcc-6.4.0/debug/threading-multi/error_code.o" "libs/system/src/error_code.cpp"

Here, it also compiles error_code.cpp into error_code.o, which I'm not sure how it is related to the atomic stuff.

Quoted 7 lines

0.084060 sec system; 0.644133 sec user; 8.858824 sec clock
SEM: <s>gcc-link-semaphore now used by <pbin.v2/libs/system/build/gcc-6.4.0/debug/threading-multi>libboost_system.so.1.67.0 Using shell: /bin/sh -c

    argv[0] = '/bin/sh'
    argv[1] = '-c'
    argv[2] = '
    "/home/fabrice/buildroot/output/host/bin/arm-linux-g++"    -o "bin.v2/libs/system/build/gcc-6.4.0/debug/threading-multi/libboost_system.so.1.67.0" -Wl,-h -Wl,libboost_system.so.1.67.0 -shared -Wl,--start-group "bin.v2/libs/system/build/gcc-6.4.0/debug/threading-multi/error_code.o"  -Wl,-Bstatic  -Wl,-Bdynamic -lrt -Wl,--end-group -fPIC -pthread -g  -Wl,-elf2flt -static

And here it tries to produce a shared library, libboost_system.so.1.67.0, which just the error_code.o object file in it ?

Questions:

- Why is Boost trying to _produce_ a shared library ?

- How is producing a shared library from the error_code.o object file related to testing the availability of lock-free atomics ?

package/boost/boost.mk endif · line 116

Quoted 7 lines
package/boost/boost.mk +7 -0
+# When detecting features such as lockfree atomic ints, boost does not take into
+# account the configuration provided by the user. Indeed, boost does not specify
+# any value and as a result the default value (shared) is used
+define BOOST_FIX_STATIC_FEATURES_DETECTION
+$(SED) 's/has_atomic_flag_lockfree_test.cpp/has_atomic_flag_lockfree_test.cpp : <link>static <runtime-link>static/g;' \
+		$(@D)/libs/thread/build/Jamfile.v2
+endef

Have you reported the issue upstream to see what they have to say ?

Thanks,

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#2 lore raw

PATCH v2 1/1 boost: fix static detection of lock-free atomic ints

Dear Thomas,
Le mer. 5 sept. 2018 ? 09:19, Thomas Petazzoni <[email protected]> a ?crit :

Quoted 70 lines

Hello,

Thanks for investigating this issue, I know what it takes to dive into the Boost build system, however, I am a bit confused by what's happening.

Quoted 9 lines · Fabrice Fontaine

When build statically, boost is unable to detect lock-free atomics ints because it tries to link dynamically, see output/build/boost-1.67.0/bin.v2/config.log (with -d5 option):

Using shell: /bin/sh -c

    argv[0] = '/bin/sh'
    argv[1] = '-c'
    argv[2] = '
    "/home/fabrice/buildroot/output/host/bin/arm-linux-g++"

-D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -Os -Wl,-elf2flt -static -Wl,-elf2flt -static -fPIC -pthread -O0 -fno-inline

(quoted diff) +0 -3
-Wall -pedantic -g -Wextra -Wno-long-long -Wno-unused-parameter
-Wunused-function -pedantic -DBOOST_ALL_NO_LIB=1 -DBOOST_SYSTEM_DYN_LINK=1
-DBOOST_THREAD_BUILD_DLL=1  -I"." -c -o

"bin.v2/libs/thread/build/gcc-6.4.0/debug/threading-multi/has_atomic_flag_lockfree_test.o" "libs/thread/src/../build/has_atomic_flag_lockfree_test.cpp"

Quoted 2 lines

'
gcc.compile.c++

bin.v2/libs/thread/build/gcc-6.4.0/debug/threading-multi/has_atomic_flag_lockfree_test.o

Quoted 2 lines

"/home/fabrice/buildroot/output/host/bin/arm-linux-g++"

-D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -Os -Wl,-elf2flt -static -Wl,-elf2flt -static -fPIC -pthread -O0 -fno-inline

(quoted diff) +0 -3
-Wall -pedantic -g -Wextra -Wno-long-long -Wno-unused-parameter
-Wunused-function -pedantic -DBOOST_ALL_NO_LIB=1 -DBOOST_SYSTEM_DYN_LINK=1
-DBOOST_THREAD_BUILD_DLL=1  -I"." -c -o

"bin.v2/libs/thread/build/gcc-6.4.0/debug/threading-multi/has_atomic_flag_lockfree_test.o" "libs/thread/src/../build/has_atomic_flag_lockfree_test.cpp"

So here, it is compiling (no linking) the has_atomic_flag_lockfree_test.cpp test file into its corresponding object file. It does not use -shared at this point.

Quoted 2 lines

0.033561 sec system; 0.126314 sec user; 288.682473 sec clock gcc.compile.c++

bin.v2/libs/system/build/gcc-6.4.0/debug/threading-multi/error_code.o

Quoted 2 lines

"/home/fabrice/buildroot/output/host/bin/arm-linux-g++"

-D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -Os -Wl,-elf2flt -static -Wl,-elf2flt -static -fPIC -pthread -O0 -fno-inline

(quoted diff) +0 -2
-Wall -pedantic -g  -DBOOST_ALL_NO_LIB=1 -DBOOST_SYSTEM_DYN_LINK=1  -I"."
-c -o

"bin.v2/libs/system/build/gcc-6.4.0/debug/threading-multi/error_code.o" "libs/system/src/error_code.cpp"

Here, it also compiles error_code.cpp into error_code.o, which I'm not sure how it is related to the atomic stuff.

Quoted 2 lines

0.084060 sec system; 0.644133 sec user; 8.858824 sec clock
SEM: <s>gcc-link-semaphore now used by

<pbin.v2/libs/system/build/gcc-6.4.0/debug/threading-multi>libboost_system.so.1.67.0

Quoted 5 lines

Using shell: /bin/sh -c

    argv[0] = '/bin/sh'
    argv[1] = '-c'
    argv[2] = '
    "/home/fabrice/buildroot/output/host/bin/arm-linux-g++"    -o

"bin.v2/libs/system/build/gcc-6.4.0/debug/threading-multi/libboost_system.so.1.67.0" -Wl,-h -Wl,libboost_system.so.1.67.0 -shared -Wl,--start-group "bin.v2/libs/system/build/gcc-6.4.0/debug/threading-multi/error_code.o"

(quoted diff) +0 -2
-Wl,-Bstatic  -Wl,-Bdynamic -lrt -Wl,--end-group -fPIC -pthread -g
-Wl,-elf2flt -static
Quoted 9 lines

And here it tries to produce a shared library, libboost_system.so.1.67.0, which just the error_code.o object file in it ?

Questions:

- Why is Boost trying to _produce_ a shared library ?

From my understanding and thanks to your helpfull comments, I think I understand better the issue.
The exe directive is after the "project boost/thread" statement in libs/thread/build/Jamfile.v2, so bjam assumes that boost_system is needed and tries to build it even if I don't know why it uses the default shared linking mode (perhaps because the exe statement is executed before boost/thread retrieve the user configuration).
So I tried to move the exe directive in the beginning of the file as boost_system is not needed to check atomics and it seems to work, I will send a v3 and get the feedback from the upstream project.

Quoted 18 lines

- How is producing a shared library from the error_code.o object file related to testing the availability of lock-free atomics ?

Quoted 1 line

+# When detecting features such as lockfree atomic ints, boost does not

take into

Quoted 1 line

+# account the configuration provided by the user. Indeed, boost does

not specify

package/boost/boost.mk endif

Quoted 3 lines
package/boost/boost.mk +3 -0
+# any value and as a result the default value (shared) is used
+define BOOST_FIX_STATIC_FEATURES_DETECTION
+$(SED)

's/has_atomic_flag_lockfree_test.cpp/has_atomic_flag_lockfree_test.cpp :
<link>static <runtime-link>static/g;' \

package/boost/boost.mk endif · line 116

Quoted 2 lines
package/boost/boost.mk +2 -0
+             $(@D)/libs/thread/build/Jamfile.v2
+endef

Have you reported the issue upstream to see what they have to say ?

Quoted 7 lines

Thanks,

Thomas
--
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Best Regards,

Fabrice
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20180905/214695ba/attachment.html>