Skip to contentSkip to conversation

package/numactl: point to target libs via LDFLAGS

PATCH In review Alexey Brodkin via buildroot 1 participant 4 messages Oct 24, 2022 – Oct 1, 2023

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 package/numactl: point to target libs via LDFLAGS

In the commit a699a667bd64 ("package/numactl: add -latomic to numa.pc") we opted-out from unconditional linking with "-latomic" and made it a conditional based on real availability of __atomic_fetch_and_1() in the toolchain. I.e. for many targets "-latomic" was not used any longer.

And in those supposedly rare cases when libatomic was required it was all OK until there's a LD_LIBRARY_PATH set with something where yet another libatomic.so instance exists, especially if that's the one from host (in my case I used a separately built host toolchain). Then we see:
---------------------->8-----------------------
/bin/sh ./libtool --tag=CC --mode=link .../buildroot/output/host/bin/arc32-buildroot-linux-uclibc-gcc -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -Os -g0 -Wl,-z,max-page-size=4096 -fPIC -std=gnu99 -o numastat numastat-numastat.o -latomic

libtool: link: .../buildroot/output/host/bin/arc32-buildroot-linux-uclibc-gcc -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -Os -g0 -Wl,-z -Wl,max-page-size=4096 -fPIC -std=gnu99 -o numastat numastat-numastat.o .../gcc-5.2.0/lib/../lib64/libatomic.so -pthread -Wl,-rpath -Wl,.../gcc-5.2.0/lib/../lib64 -Wl,-rpath -Wl,.../gcc-5.2.0/lib/../lib64 .../buildroot/output/host/lib/gcc/arc32-buildroot-linux-uclibc/12.2.1/../../../../arc32-buildroot-linux-uclibc/bin/ld: .../gcc-5.2.0/lib/../lib64/libatomic.so: error adding symbols: file in wrong format
collect2: error: ld returned 1 exit status
---------------------->8-----------------------

That happens because libtool of numactl adds contents of the aforementioned LD_LIBRARY_PATH to its "$searchdirs" and then the first libatomic.so it finds becomes the one used for linking. What's more our precios cross-built libatomic.so never even gets considered because that first found libatomic.so is used with full path, not as "-latomic" as otherwise it would have been used without helpful libtool.

But with paths pointing to the staging location, these paths get added to "$searchdirs" and so the right libatomic.so gets found, which solves our problem.

---
 package/numactl/numactl.mk | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
package/numactl/numactl.mk +2 -1
index cf9c759693..58b304d6e1 100644
@@ -10,6 +10,7 @@ NUMACTL_LICENSE = LGPL-2.1 (libnuma), GPL-2.0 (programs)
 NUMACTL_LICENSE_FILES = README.md
 NUMACTL_INSTALL_STAGING = YES
 NUMACTL_AUTORECONF = YES
-NUMACTL_CONF_ENV = CFLAGS="$(TARGET_CFLAGS) -fPIC"
+NUMACTL_CONF_ENV = CFLAGS="$(TARGET_CFLAGS) -fPIC" \
+		   LDFLAGS="$(TARGET_LDFLAGS) -L$(STAGING_DIR)/lib"
 
 $(eval $(autotools-package))
Signature · 6 lines
2.16.2

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#1 lore raw

Hello Alexey,

Quoted 18 lines · On Mon, 24 Oct 2022 19:49:11 +0200 Alexey Brodkin via buildroot…

In the commit a699a667bd64 ("package/numactl: add -latomic to numa.pc") we opted-out from unconditional linking with "-latomic" and made it a conditional based on real availability of __atomic_fetch_and_1() in the toolchain. I.e. for many targets "-latomic" was not used any longer.

And in those supposedly rare cases when libatomic was required it was all OK until there's a LD_LIBRARY_PATH set with something where yet another libatomic.so instance exists, especially if that's the one from host (in my case I used a separately built host toolchain). Then we see:
---------------------->8-----------------------
/bin/sh ./libtool --tag=CC --mode=link .../buildroot/output/host/bin/arc32-buildroot-linux-uclibc-gcc -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -Os -g0 -Wl,-z,max-page-size=4096 -fPIC -std=gnu99 -o numastat numastat-numastat.o -latomic

libtool: link: .../buildroot/output/host/bin/arc32-buildroot-linux-uclibc-gcc -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -Os -g0 -Wl,-z -Wl,max-page-size=4096 -fPIC -std=gnu99 -o numastat numastat-numastat.o .../gcc-5.2.0/lib/../lib64/libatomic.so -pthread -Wl,-rpath -Wl,.../gcc-5.2.0/lib/../lib64 -Wl,-rpath -Wl,.../gcc-5.2.0/lib/../lib64 .../buildroot/output/host/lib/gcc/arc32-buildroot-linux-uclibc/12.2.1/../../../../arc32-buildroot-linux-uclibc/bin/ld: .../gcc-5.2.0/lib/../lib64/libatomic.so: error adding symbols: file in wrong format
collect2: error: ld returned 1 exit status
---------------------->8-----------------------

Quoted 6 lines

That happens because libtool of numactl adds contents of the aforementioned LD_LIBRARY_PATH to its "$searchdirs" and then the first libatomic.so it finds becomes the one used for linking. What's more our precios cross-built libatomic.so never even gets considered because that first found libatomic.so is used with full path, not as "-latomic" as otherwise it would have been used without helpful libtool.

Thanks for the patch. However, I don't get it. Where is LD_LIBRARY_PATH defined? Do you have a reproducer for this issue?

package/numactl/numactl.mk NUMACTL_LICENSE = LGPL-2.1 (libnuma), GPL-2.0 (programs) · line 14

Quoted 2 lines
package/numactl/numactl.mk +2 -0
+NUMACTL_CONF_ENV = CFLAGS="$(TARGET_CFLAGS) -fPIC" \
+		   LDFLAGS="$(TARGET_LDFLAGS) -L$(STAGING_DIR)/lib"

This is really a red flag, $(STAGING_DIR)/lib is already in the default search path of the compiler, so that should never be needed.

Thomas

Signature · 7 lines
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#2 lore raw

Hello, Thomas!

Quoted 28 lines
Quoted 18 lines · On Mon, 24 Oct 2022 19:49:11 +0200 Alexey Brodkin via buildroot…

In the commit a699a667bd64 ("package/numactl: add -latomic to numa.pc") we opted-out from unconditional linking with "-latomic" and made it a conditional based on real availability of __atomic_fetch_and_1() in the toolchain. I.e. for many targets "-latomic" was not used any longer.

And in those supposedly rare cases when libatomic was required it was
all OK until there's a LD_LIBRARY_PATH set with something where yet
another libatomic.so instance exists, especially if that's the one from
host (in my case I used a separately built host toolchain). Then we see:
---------------------->8-----------------------
/bin/sh ./libtool  --tag=CC   --mode=link .../buildroot/output/host/bin/arc32-buildroot-linux-uclibc-gcc -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64  -Os -g0  -Wl,-z,max-page-size=4096 -fPIC -std=gnu99    -o numastat numastat-numastat.o  -latomic

libtool: link: .../buildroot/output/host/bin/arc32-buildroot-linux-uclibc-gcc -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -Os -g0 -Wl,-z -Wl,max-page-size=4096 -fPIC -std=gnu99 -o numastat numastat-numastat.o  .../gcc-5.2.0/lib/../lib64/libatomic.so -pthread -Wl,-rpath -Wl,.../gcc-5.2.0/lib/../lib64 -Wl,-rpath -Wl,.../gcc-5.2.0/lib/../lib64 .../buildroot/output/host/lib/gcc/arc32-buildroot-linux-uclibc/12.2.1/../../../../arc32-buildroot-linux-uclibc/bin/ld: .../gcc-5.2.0/lib/../lib64/libatomic.so: error adding symbols: file in wrong format
collect2: error: ld returned 1 exit status
---------------------->8-----------------------

Quoted 6 lines

That happens because libtool of numactl adds contents of the aforementioned LD_LIBRARY_PATH to its "$searchdirs" and then the first libatomic.so it finds becomes the one used for linking. What's more our precios cross-built libatomic.so never even gets considered because that first found libatomic.so is used with full path, not as "-latomic" as otherwise it would have been used without helpful libtool.

Quoted 2 lines

Thanks for the patch. However, I don't get it. Where is LD_LIBRARY_PATH defined?

I meant LD_LIBRARY_PATH is defined before Buildroot's top-level make invocation, i.e. "LD_LIBRARY_PATH=_some_location_of_hosts_libatomic.so_ make". See below.

Quoted 1 line

Do you have a reproducer for this issue?

Yes, I have finally. It took me a bit longer than expected to figure out all the details, thus answering that late. So there're a couple of prerequsites for the reproduction even on vanilla "numactl" sources, I used 2.0.14 with the Buildroot's libatomic patch [1]:

1. For example in my Ubuntu 20.04 on x86_64 I installed ARM cross-tools with:
----------------------------->8----------------------------- apt install gcc-9-arm-linux-gnueabi ----------------------------->8-----------------------------

2. LD_LIBRARY_PATH which points to a location of a foreign (e.g. host or some other architecture) "libatomic.so", plus (that's really important) right there side-by-side with "libatomic.so" it's important to have "libatomic.la" (valid libtool archive, and not an empty file, see below my copy which works for the reproduction). I.e. ----------------------------->8----------------------------- export LD_LIBRARY_PATH=/usr/lib/gcc/x86_64-linux-gnu/9/ ----------------------------->8-----------------------------

3. Configure "numactl" with lt_cv_sys_lib_search_path_spec="", which we exactly do in Buildroot, see [2], i.e.:
----------------------------->8----------------------------- ./configure --host=arm-linux-gnueabi lt_cv_sys_lib_search_path_spec="" ----------------------------->8-----------------------------

4. Try to build now:
----------------------------->8-----------------------------
make V=1
make all-am
make[1]: Entering directory 'numactl'
/bin/bash ./libtool --tag=CC --mode=link arm-linux-gnueabi-gcc -g -O2 -version-info 1:0:0 -Wl,--version-script,./versions.ldscript -Wl,-init,numa_init -Wl,-fini,numa_fini -o libnuma.la -rpath /usr/local/lib libnuma.lo syscall.lo distance.lo affinity.lo sysfs.lo rtnetlink.lo -latomic
libtool: warning: library '/usr/lib/gcc/x86_64-linux-gnu/9//libatomic.la' was moved.
libtool: link: arm-linux-gnueabi-gcc -shared -fPIC -DPIC .libs/libnuma.o .libs/syscall.o .libs/distance.o .libs/affinity.o .libs/sysfs.o .libs/rtnetlink.o -Wl,-rpath -Wl,/usr/lib/gcc/x86_64-linux-gnu/9/ -Wl,-rpath -Wl,/usr/lib/gcc/x86_64-linux-gnu/9/ /usr/lib/gcc/x86_64-linux-gnu/9//libatomic.so -g -O2 -Wl,--version-script -Wl,./versions.ldscript -Wl,-init -Wl,numa_init -Wl,-fini -Wl,numa_fini -pthread -Wl,-soname -Wl,libnuma.so.1 -o .libs/libnuma.so.1.0.0
/usr/lib/gcc/x86_64-linux-gnu/9//libatomic.so: file not recognized: file format not recognized
collect2: error: ld returned 1 exit status
make[1]: *** [Makefile:889: libnuma.la] Error 1
make[1]: Leaving directory 'numactl'
make: *** [Makefile:742: all] Error 2 ----------------------------->8-----------------------------

So, what happens (3) removes compiler's default paths from libtool's "sys_lib_dlsearch_path", while (2) makes libtool think that libatomic.so from LD_LIBRARY_PATH is the right one.

Quoted 5 lines
Quoted 2 lines

+NUMACTL_CONF_ENV = CFLAGS="$(TARGET_CFLAGS) -fPIC" \
+                LDFLAGS="$(TARGET_LDFLAGS) -L$(STAGING_DIR)/lib"

This is really a red flag, $(STAGING_DIR)/lib is already in the default search path of the compiler, so that should never be needed.

Well, if we don't mess with "lt_cv_sys_lib_search_path_spec" in the configure script options, then all works OK, but I guess Peter's change made in 2011 makes some sense and we don't want to revert it, or do we?

Also, I accept the fact that my setup is a bit unusual at least and it might be a good enough reason to disregard all that discussion, but first I'd like to make sure we really don't care about situations like that:
LD_LIBRARY_PATH pointing to a self-built toolchain.

If we assume that situation as one of possible corner-cases, then probably we may want to troubleshoot such funny things which happen due to "libtool" smarts.

-Alexey

[1] https://github.com/numactl/numactl/commit/e0de0d9e981ddb53bdeb4a4b9dc43046c9ff4ff9
[2] https://git.buildroot.net/buildroot/commit/?id=6246704008ef219257f30a824903855d0e00cffb

----------------------------->8-----------------------------
Contents of "/usr/lib/gcc/x86_64-linux-gnu/9/libatomic.la"
----------------------------->8-----------------------------

# libatomic.la - a libtool library file
# Generated by libtool (GNU libtool 1.3134 2009-11-29) 2.2.7a
#
# Please DO NOT delete this file!
# It is necessary for linking the library.

# The name that we can dlopen(3).
dlname='libatomic.so.1'

# Names of this library. library_names='libatomic.so.1.2.0 libatomic.so.1 libatomic.so'

# The name of the static archive.
old_library='libatomic.a'

# Linker flags that can not go in dependency_libs. inherited_linker_flags=' -pthread'

# Libraries that this one depends upon.
dependency_libs=''

# Names of additional weak libraries provided by this library weak_library_names=''

# Version information for libatomic.
current=3
age=2
revision=0

# Is this an already installed library?
installed=yes

# Should we warn about portability when linking against -modules? shouldnotlink=no

# Files to dlopen/dlpreopen
dlopen=''
dlpreopen=''

# Directory that this library needs to be installed in:
libdir='/usr/lib/x86_64-linux-gnu/' ----------------------------->8-----------------------------
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

#3 lore raw
In reply to Alexey Brodkin via buildroot

Hello Alexey,

Quoted 5 lines · On Mon, 24 Oct 2022 19:49:11 +0200 Alexey Brodkin via buildroot…

In the commit a699a667bd64 ("package/numactl: add -latomic to numa.pc") we opted-out from unconditional linking with "-latomic" and made it a conditional based on real availability of __atomic_fetch_and_1() in the toolchain. I.e. for many targets "-latomic" was not used any longer.

We finally looked at this patch today during the on-going Buildroot hackathon. We understand/recognize the problem, but we don't like the proposed solution as you can imagine (especially as the proposed solution is package specific). In addition, and as you have yourself expressed, your situation is quite a corner case.

Here are the options that we see to resolve this:

(1) Patch libtool so that it doesn't look at LD_LIBRARY_PATH. We are already patching libtool to work-around other issues. However, from a quick look at the libtool code base (which of course is not the most pleasant exercise on earth), it's not trivial to see where this logic is taking place.

(2) Unset LD_LIBRARY_PATH entirely in the main Buildroot makefile to avoid interference with LD_LIBRARY_PATH being set. This would of course prevent people who have special/weird setups that require LD_LIBRARY_PATH from using Buildroot.

(3) Warn/error out if LD_LIBRARY_PATH is set, which essentially is a variant of (2), but where we are explicit instead of silently ignoring LD_LIBRARY_PATH defined in the user's environment.

Regarding the setting of lt_cv_sys_lib_search_path_spec="" that you have identified, the motivation for this was explained in 6246704008ef219257f30a824903855d0e00cffb:

package/Makefile.in: ensure libtool doesn't search host dirs for libraries

Commit 7e3e8ec040b (CFLAGS/LDFLAGS: don't add -I / -L args for STAGING_DIR) exposed a lingering libtool problem.

Unless instructed otherwise (using -L) libtool will search its built in system path for libraries, and use those instead if found. The default search path is '/usr/lib, /lib, /usr/local/lib', which is no good for cross compilation.

Fix it by setting the system search path to the empty string, effectively disabling this feature.

So this seems to still be relevant. The reason why when you drop this in your case and it "works" is we believe the following one: due to -latomic, libtool will search for libatomic. It will find it in your system in say /lib or /usr/lib, and because that's a standard search path, libtool will keep -latomic instead of turning into a full absolute path. So the cross linker gets passed -latomic, which it properly resolves to the target/sysroot libatomic. When lt_cv_sys_lib_search_path_spec="" is there, libatomic from /lib or /usr/lib isn't found, so it finds the one in your LD_LIBRARY_PATH directory, which isn't a standard path, so our dear libtool replaces -latomic by the full absolute path to libatomic, and passes that to the cross-linker, which of course blows up.

Best regards,

Thomas

Signature · 7 lines
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot