Solving the musl dynamic linker issues

Hello,

We currently have two outstanding musl dynamic linker issues:

1. We don't handle all musl dynamic linker names for all architectures. I proposed a patch solving this at
https://patchwork.ozlabs.org/patch/780411/, where the approach was to build a dummy program and use readelf to find the dynamic linker name, and therefore the symlink to libc.so that we need to create.

2. We don't support Crosstool-NG musl toolchains, because they put libc.so in /usr/lib and not /lib, and therefore our current creation logic that makes the dynamic linker point to /lib/libc.so incorrect. Ilya Kuzmich <[email protected]> proposed patches
(https://patchwork.ozlabs.org/patch/763977/
https://patchwork.ozlabs.org/patch/748974/) to solve this.

After discussion with Arnout, we realized that all our messy logic for creating the musl dynamic linker symbolic link is useless. The correct symbolic link already exists in the original toolchain sysroot, so we should just copy it from the sysroot, just like we do for glibc/uClibc.

This series of patches implements just that, with the main patch being PATCH 3/4.

The first patch just adds more external toolchain tests to our runtime tests infrastructure, to validate that the changes in this series don't break things.

The second patch prepares the copy_toolchain_lib_root for copying a symlink in /lib that potentially points to a file in /usr/lib (which it currently doesn't handle).

The third patch is the core of the solution, basically putting ld*.so* unconditionally in TOOLCHAIN_EXTERNAL_LIBS.

The fourth patch is a minor refactoring.

Thanks,

Thomas

Thomas Petazzoni (4):
support/testing: add tests for musl and uclibc toolchains toolchain/helpers.mk: re-evaluate DESTDIR in copy_toolchain_lib_root
toolchain-external: copy ld*.so* for all C libraries
toolchain-external: also put libgcc_s.so unconditionally in
TOOLCHAIN_EXTERNAL_LIBS

 support/testing/tests/toolchain/test_external.py   | 75 ++++++++++++++++++++++
 toolchain/helpers.mk                               |  2 +-
 .../toolchain-external/pkg-toolchain-external.mk   | 36 ++---------
 3 files changed, 81 insertions(+), 32 deletions(-)

2.9.4

These tests simply build a system with musl and uclibc toolchains, and boot them under qemu. It allows to minimally validate that our support for musl/uclibc external toolchains is working. We already had some tests covering glibc toolchains, so we can now easily test that all three C libraries are supported.

---
This commit is part of the series, as I've written/used those tests to validate that things are still working correctly with all of glibc, uclibc and musl toolchains.

---
 support/testing/tests/toolchain/test_external.py | 75 ++++++++++++++++++++++++
 1 file changed, 75 insertions(+)

index 0b15d48..f975c56 100644
@@ -154,3 +154,78 @@ BR2_TOOLCHAIN_EXTERNAL_LINARO_ARM=y
                            kernel="builtin",
                            options=["-initrd", img])
         self.emulator.login()
+
+class TestExternalToolchainBuildrootMusl(TestExternalToolchain):
+    config = BASIC_CONFIG + \
+"""
+BR2_arm=y
+BR2_cortex_a9=y
+BR2_ARM_ENABLE_VFP=y
+BR2_TOOLCHAIN_EXTERNAL=y
+BR2_TOOLCHAIN_EXTERNAL_CUSTOM=y
+BR2_TOOLCHAIN_EXTERNAL_DOWNLOAD=y
+BR2_TOOLCHAIN_EXTERNAL_URL="http://autobuild.buildroot.org/toolchains/tarballs/br-arm-cortex-a9-musl-2017.05-444-g6c704ba.tar.bz2"
+BR2_TOOLCHAIN_EXTERNAL_GCC_6=y
+BR2_TOOLCHAIN_EXTERNAL_HEADERS_4_11=y
+BR2_TOOLCHAIN_EXTERNAL_CUSTOM_MUSL=y
+BR2_TOOLCHAIN_EXTERNAL_CXX=y
+"""
+    toolchain_prefix = "arm-linux"
+
+    def test_run(self):
+        TestExternalToolchain.common_check(self)
+        img = os.path.join(self.builddir, "images", "rootfs.cpio")
+        self.emulator.boot(arch="armv7",
+                           kernel="builtin",
+                           options=["-initrd", img])
+        self.emulator.login()
+
+class TestExternalToolchainCtngMusl(TestExternalToolchain):
+    config = BASIC_CONFIG + \
+"""
+BR2_arm=y
+BR2_cortex_a9=y
+BR2_ARM_ENABLE_VFP=y
+BR2_TOOLCHAIN_EXTERNAL=y
+BR2_TOOLCHAIN_EXTERNAL_CUSTOM=y
+BR2_TOOLCHAIN_EXTERNAL_DOWNLOAD=y
+BR2_TOOLCHAIN_EXTERNAL_URL="http://prout.org/arm-ctng-linux-musleabihf.tar.xz"
+BR2_TOOLCHAIN_EXTERNAL_CUSTOM_PREFIX="arm-ctng-linux-musleabihf"
+BR2_TOOLCHAIN_EXTERNAL_GCC_7=y
+BR2_TOOLCHAIN_EXTERNAL_HEADERS_3_10=y
+BR2_TOOLCHAIN_EXTERNAL_CUSTOM_MUSL=y
+BR2_TOOLCHAIN_EXTERNAL_CXX=y
+"""
+    toolchain_prefix = "arm-ctng-linux-musleabihf"
+
+    def test_run(self):
+        TestExternalToolchain.common_check(self)
+        img = os.path.join(self.builddir, "images", "rootfs.cpio")
+        self.emulator.boot(arch="armv7",
+                           kernel="builtin",
+                           options=["-initrd", img])
+        self.emulator.login()
+
+class TestExternalToolchainBuildrootuClibc(TestExternalToolchain):
+    config = BASIC_CONFIG + \
+"""
+BR2_arm=y
+BR2_TOOLCHAIN_EXTERNAL=y
+BR2_TOOLCHAIN_EXTERNAL_CUSTOM=y
+BR2_TOOLCHAIN_EXTERNAL_DOWNLOAD=y
+BR2_TOOLCHAIN_EXTERNAL_URL="http://autobuild.buildroot.org/toolchains/tarballs/br-arm-full-2017.05-444-g6c704ba.tar.bz2"
+BR2_TOOLCHAIN_EXTERNAL_GCC_4_9=y
+BR2_TOOLCHAIN_EXTERNAL_HEADERS_3_10=y
+BR2_TOOLCHAIN_EXTERNAL_LOCALE=y
+# BR2_TOOLCHAIN_EXTERNAL_HAS_THREADS_DEBUG is not set
+BR2_TOOLCHAIN_EXTERNAL_CXX=y
+"""
+    toolchain_prefix = "arm-linux"
+
+    def test_run(self):
+        TestExternalToolchain.common_check(self)
+        img = os.path.join(self.builddir, "images", "rootfs.cpio")
+        self.emulator.boot(arch="armv7",
+                           kernel="builtin",
+                           options=["-initrd", img])
+        self.emulator.login()

2.9.4

Hello,

Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

When we have the in-tree config fragments for the autobuilder toolchains, these
should be used instead of hardcoding. For now however (after your fix):

Regards,
Arnout

Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

copy_toolchain_lib_root copies libraries from staging to target, resolving and copying symbolic links along the way.

The most inner loop, a "while" loop, starts from an initial name, and if it's a symbolic link, gets resolved to the target, and the loop iterates until we reach a real file. However, the destination folder where the symbolic link or real file is created is computed in DESTDIR only once, before this loop starts. Therefore, this loop works fine when all symbolic links in the chain, and the real file all belong to the same directory. But it doesn't do the correct thing when the symbolic link and/or real file are in different folder.

An example is Crosstool-NG musl toolchains, where the dynamic loader is in /lib/ld-musl*.so but points to ../usr/lib/libc.so. With the current logic, we copy /lib/ld-musl*.so to /lib, but we also copy libc.so to /lib instead of the expected /usr/lib.

This currently doesn't cause any problem because the musl dynamic linker is manually created by the TOOLCHAIN_EXTERNAL_MUSL_LD_LINK hook. However, this logic has a number of problems, so in a followup commit, we are going to put the musl dynamic linker in TOOLCHAIN_EXTERNAL_LIBS, which will cause it to be copied by copy_toolchain_lib_root. But we obviously want the link and its target to be copied to the right place, hence this fix.

---
 toolchain/helpers.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

index 90834f4..fe2b4b9 100644
@@ -13,10 +13,10 @@ copy_toolchain_lib_root = \
 \
 	LIBPATHS=`find $(STAGING_DIR)/ -name "$${LIBPATTERN}" 2>/dev/null` ; \
 	for LIBPATH in $${LIBPATHS} ; do \
-		DESTDIR=`echo $${LIBPATH} | sed "s,^$(STAGING_DIR)/,," | xargs dirname` ; \
 		mkdir -p $(TARGET_DIR)/$${DESTDIR}; \
 		while true ; do \
 			LIBNAME=`basename $${LIBPATH}`; \
+			DESTDIR=`echo $${LIBPATH} | sed "s,^$(STAGING_DIR)/,," | xargs dirname` ; \
 			rm -fr $(TARGET_DIR)/$${DESTDIR}/$${LIBNAME}; \
 			if test -h $${LIBPATH} ; then \
 				cp -d $${LIBPATH} $(TARGET_DIR)/$${DESTDIR}/$${LIBNAME}; \

2.9.4

Currently, for the dynamic loader, we're copying ld*.so* for glibc and uClibc, except for glibc/EABIhf where we are explicitly copying ld-linux-armhf.so.*. For musl, we're not copying the dynamic linker because it's simply a symbolic link to libc.so. However, the name of the musl dynamic linker changes from one architecture to the other, and we don't handle all cases.

Since handling the musl dynamic linker symlink creation is becoming more and more annoying to maintain, this commit makes musl use the same mechanism as glibc/uClibc: put the dynamic linker in TOOLCHAIN_EXTERNAL_LIBS.

In addition, the special condition on glibc/EABIhf was added in 11ec38b6950cf3337b52fb97f27c2fd7c776c5c2 ("toolchain-external: fix Linaro ARM toolchain support") because an old Linaro toolchain had two dynamic loaders, and we wanted to copy only one. But 1/ this is old and 2/ having the two dynamic linkers doesn't really matter.

So this commit simply unconditionally adds "ld*.so*" to TOOLCHAIN_EXTERNAL_LIBS, regardless of the C library being chosen. It re-uses the musl dynamic linker symlink from the sysroot, which makes it always correct, and allows us to remove the TOOLCHAIN_EXTERNAL_MUSL_LD_LINK hook, and all the related logic.

This commit therefore solves two problems with the musl dynamic linker symbolic link creation logic:

1 We support all architectures, without having to hardcode in
Buildroot the mapping between the CPU architecture and the corresponding dynamic linker name. For example, our current logic was not handling the mips64+n32 ABI case, where the dynamic linker is named ld-musl-mipsn32el.so.1.

2 We support Crosstool-NG musl toolchains, where the dynamic linker
is in /lib, but libc.so is in /usr/lib.

---
This commit therefore replaces:

- https://patchwork.ozlabs.org/patch/780411/ (was another solution for solving problem 1 above)

- https://patchwork.ozlabs.org/patch/763977/ and
https://patchwork.ozlabs.org/patch/748974/ (was another solution for solving problem 2 above)

---
 .../toolchain-external/pkg-toolchain-external.mk   | 32 ++--------------------
 1 file changed, 3 insertions(+), 29 deletions(-)

index 8460e37..c1c3900 100644
@@ -107,13 +107,11 @@ endif
 #
 # Definitions of the list of libraries that should be copied to the target.
 #
+
+TOOLCHAIN_EXTERNAL_LIBS += ld*.so*
+
 ifeq ($(BR2_TOOLCHAIN_EXTERNAL_GLIBC)$(BR2_TOOLCHAIN_EXTERNAL_UCLIBC),y)
 TOOLCHAIN_EXTERNAL_LIBS += libatomic.so.* libc.so.* libcrypt.so.* libdl.so.* libgcc_s.so.* libm.so.* libnsl.so.* libresolv.so.* librt.so.* libutil.so.*
-ifeq ($(BR2_TOOLCHAIN_EXTERNAL_GLIBC)$(BR2_ARM_EABIHF),yy)
-TOOLCHAIN_EXTERNAL_LIBS += ld-linux-armhf.so.*
-else
-TOOLCHAIN_EXTERNAL_LIBS += ld*.so.*
-endif
 ifeq ($(BR2_TOOLCHAIN_HAS_THREADS),y)
 TOOLCHAIN_EXTERNAL_LIBS += libpthread.so.*
 ifneq ($(BR2_PACKAGE_GDB)$(BR2_TOOLCHAIN_EXTERNAL_GDB_SERVER_COPY),)

@@ -470,30 +468,6 @@ define TOOLCHAIN_EXTERNAL_INSTALL_GDBINIT
 	fi
 endef
 
-# Various utility functions used by the external toolchain based on musl.
-
-# With the musl C library, the libc.so library directly plays the role
-# of the dynamic library loader. We just need to create a symbolic
-# link to libc.so with the appropriate name.
-ifeq ($(BR2_TOOLCHAIN_EXTERNAL_MUSL):$(BR2_STATIC_LIBS),y:)
-ifeq ($(BR2_i386),y)
-MUSL_ARCH = i386
-else ifeq ($(BR2_ARM_EABIHF),y)
-MUSL_ARCH = armhf
-else ifeq ($(BR2_mips):$(BR2_SOFT_FLOAT),y:y)
-MUSL_ARCH = mips-sf
-else ifeq ($(BR2_mipsel):$(BR2_SOFT_FLOAT),y:y)
-MUSL_ARCH = mipsel-sf
-else ifeq ($(BR2_sh),y)
-MUSL_ARCH = sh
-else
-MUSL_ARCH = $(ARCH)
-endif
-define TOOLCHAIN_EXTERNAL_MUSL_LD_LINK
-	ln -sf libc.so $(TARGET_DIR)/lib/ld-musl-$(MUSL_ARCH).so.1
-endef
-endif
-
 # uClibc-ng dynamic loader is called ld-uClibc.so.1, but gcc is not
 # patched specifically for uClibc-ng, so it continues to generate
 # binaries that expect the dynamic loader to be named ld-uClibc.so.0,

2.9.4

Hi Thomas,

     http://baruch.siach.name/blog/                  ~. .~   Tk Open Systems
=}------------------------------------------------ooO--U--Ooo------------{=
   - baruch at tkos.co.il - tel: +972.2.679.5364, http://www.tkos.co.il -

2017-07-02 15:41 GMT+02:00 Thomas Petazzoni
<[email protected]>:

Hello,

I think it's a separate issue, because we're already using ld*.so* today in pkg-toolchain-external.mk for TOOLCHAIN_EXTERNAL_LIBS, I'm just moving it around.

However, I agree we should probably line up things. In which direction do you think we should line up? I guess using ld*.so* in copy_toolchain_sysroot should be fine.

Best regards,

Thomas

Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

libgcc_s.so is now added to TOOLCHAIN_EXTERNAL_LIBS for glibc/uclibc in one place, and for musl in another place. Bottom line: it should be in TOOLCHAIN_EXTERNAL_LIBS unconditionally.

---
 toolchain/toolchain-external/pkg-toolchain-external.mk | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

index c1c3900..f3f0f48 100644
@@ -108,10 +108,10 @@ endif
 # Definitions of the list of libraries that should be copied to the target.
 #
 
-TOOLCHAIN_EXTERNAL_LIBS += ld*.so*
+TOOLCHAIN_EXTERNAL_LIBS += ld*.so* libgcc_s.so.*
 
 ifeq ($(BR2_TOOLCHAIN_EXTERNAL_GLIBC)$(BR2_TOOLCHAIN_EXTERNAL_UCLIBC),y)
-TOOLCHAIN_EXTERNAL_LIBS += libatomic.so.* libc.so.* libcrypt.so.* libdl.so.* libgcc_s.so.* libm.so.* libnsl.so.* libresolv.so.* librt.so.* libutil.so.*
+TOOLCHAIN_EXTERNAL_LIBS += libatomic.so.* libc.so.* libcrypt.so.* libdl.so.* libm.so.* libnsl.so.* libresolv.so.* librt.so.* libutil.so.*
 ifeq ($(BR2_TOOLCHAIN_HAS_THREADS),y)
 TOOLCHAIN_EXTERNAL_LIBS += libpthread.so.*
 ifneq ($(BR2_PACKAGE_GDB)$(BR2_TOOLCHAIN_EXTERNAL_GDB_SERVER_COPY),)

@@ -125,7 +125,7 @@ TOOLCHAIN_EXTERNAL_LIBS += libnss_files.so.* libnss_dns.so.* libmvec.so.* libanl
 endif
 
 ifeq ($(BR2_TOOLCHAIN_EXTERNAL_MUSL),y)
-TOOLCHAIN_EXTERNAL_LIBS += libc.so libgcc_s.so.*
+TOOLCHAIN_EXTERNAL_LIBS += libc.so
 endif
 
 ifeq ($(BR2_INSTALL_LIBSTDCPP),y)

2.9.4

Hello,

I've applied the series, and I've added a 5th patch that removes the ld*.so handling in copy_toolchain_sysroot, keeping only ld*.so.* to be consistent with what TOOLCHAIN_EXTERNAL_LIBS says. This (hopefully) addresses Thomas DS comment :-)

Best regards,

Thomas

Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com