Skip to contentSkip to conversation

Support 32-bit binaries on a 64-bit architecture

RFC New Markus Mayer 4 participants 10 messages Feb 16, 2018

lore
Share & tools
lore
#0 lore raw

cover Support 32-bit binaries on a 64-bit architecture

This series contains a proposal for supporting 32-bit libraries and binaries on a 64-bit platform.

There are some limitations and prerequisites as to the scope this has been tested.

- It has only been tested on ARM/ARM64.
- It has only been tested with an external toolchain
(https://github.com/Broadcom/stbgcc-6.3/releases).
- It requires that the Aarch64 compiler be compiled with "multi-lib" enabled. This ensures that the sysroot doesn't contain a lib64 -> lib symlink. Instead, lib and lib64 are separate directories.
- It has only been tested with a fairly limited number of packages. There might be other packages that don't correctly pass --libdir, similar to bzip2 as mentioned below.

The patches in this series make the following changes, in this order:

- ensure that check-bin-arch allows Aarch32 binaries on Aarch64
- introduce some new BR2 configuration variables (primarily for the 32-bit sysroot location on the host and where system libraries live by default on the target)
- tell the Buildroot core not to create lib32 or lib64 symlink when 32-bit support is enabled
- tell the Buildroot core to copy the 32-bit sysroot when 32-bit support is enabled
- introduce the $(LIBDIR) variable to the bzip2 package, so we can tell it that libbz2.so should go in /usr/lib64 when that is desired
- modify the package core to pass /usr/lib (or /usr/lib64) as the library location for the system

The result is that all 64-bit shared libraries should be installed into /usr/lib64. The 32-bit libraries are copied from the 32-bit sysroot on the host into /usr/lib.

I am quite certain this submission will only be a first step, but it does look like there are others who are faced with the same issue.

Markus Mayer (6):
support/scripts/check-bin-arch: improve architecture check system/Config.in: add configuration options for 32-bit library support
core: system and toolchain: 32-bit run-time support on 64-bit platform core/pkg-toolchain-external: copy external 32-bit libraries to staging
bzip2: introduce make variable $(LIBDIR)
package: use BR2_ROOTFS_LIB_DIR for all libraries we use

 package/bzip2/0003-add-libdir-to-makefile.patch    | 36 ++++++++++++++++++++++
 .../bzip2/0004-add-libdir-to-makefile-libbz2.patch | 31 +++++++++++++++++++
 package/bzip2/bzip2.mk                             |  4 ++-
 package/dmalloc/dmalloc.mk                         |  7 +++--
 package/e2fsprogs/e2fsprogs.mk                     |  2 ++
 package/libzlib/libzlib.mk                         |  3 +-
 package/linux-tools/linux-tool-cpupower.mk.in      |  7 ++++-
 package/pciutils/pciutils.mk                       |  2 ++
 package/pkg-autotools.mk                           |  1 +
 package/readline/readline.mk                       |  2 +-
 support/scripts/check-bin-arch                     | 15 +++++++++
 system/Config.in                                   | 34 ++++++++++++++++++++
 system/system.mk                                   | 11 ++++++-
 .../toolchain-external/pkg-toolchain-external.mk   | 13 +++++++-
 15 files changed, 161 insertions(+), 10 deletions(-)
 create mode 100644 package/bzip2/0003-add-libdir-to-makefile.patch
 create mode 100644 package/bzip2/0004-add-libdir-to-makefile-libbz2.patch
Signature · 1 line
2.7.4
#1 lore raw

RFC 1/6 support/scripts/check-bin-arch: improve architecture check

On 64-bit targets, allow 32-bit binaries of the same architecture (MIPS64 -> MIPS32, AArch64 -> ARM, etc.)

In order for 32-bit binaries to run on the target, the corresponding 32-bit libraries need to be present in the target's root file system and the kernel needs to support execution of 32-bit binaries.

---
 support/scripts/check-bin-arch | 15 +++++++++++++++
 1 file changed, 15 insertions(+)
support/scripts/check-bin-arch +15 -0
index 887b6613cdb8..31711769eabd 100755
@@ -58,6 +58,21 @@ while read f; do
 		continue
 	fi
 
+	# Didn't find a straight match. See if we are on a 64-bit architecture
+	# and our binary is a 32-bit binary of the same architecture.
+	if [[ "${arch_name}" =~ 64$ ]]; then
+		if [[ ${arch_name} = "AArch64" ]]; then
+			arch32='ARM'
+		else
+			arch32=${arch_name/64/32}
+		fi
+
+		if [ "${arch}" = "${arch32}" ]; then
+			echo "Accepting ${arch32} binary on ${arch_name} ($f)"
+			continue
+		fi
+	fi
+
 	printf 'ERROR: architecture for "%s" is "%s", should be "%s"\n' \
 	       "${f}" "${arch}" "${arch_name}"
 
Signature · 1 line
2.7.4
#2 lore raw

RFC 2/6 system/Config.in: add configuration options for 32-bit library support

In reply to Markus Mayer

In order to support 32-bit shared libraries in a 64-bit root file system, we need several configuration options that let us tell buildroot which libraries to put where.

Specifically, we introduce the following:

BR2_ROOTFS_LIB_DIR
------------------

Tells buildroot where to put the main system libraries. On 32-bit setups, this will mean 32-bit libraries. On 64-bit setups, this setting will apply to 64-bit libraries. Typical values for BR2_ROOTFS_LIB_DIR are "lib", "lib32" or "lib64".

The default value for 32-bit setups is "lib". For 64-bit setups it is "lib64".

BR2_ROOTFS_LIB32_DIR
--------------------

This tells buildroot where to put 32-bit libraries on a 64-bit system. Typical values for BR2_ROOTFS_LIB32_DIR are "lib" or "lib32". The default value is "lib".

On 32-bit setups, this variable is not available. On 64-bit systems with 32-bit support disabled, this variable will not be used.

BR2_ROOTFS_RUNTIME32
--------------------

Tells buildroot whether it should install 32-bit shared libraries into a 64-bit root file system. If set to false, the root file system will not be able to support 32-bit binaries.

This variable is not available on 32-bit setups.

BR2_ROOTFS_RUNTIME32_PATH
-------------------------

Points to the sys-root of a 32-bit toolchain for the current 64-bit architecture (i.e. an ARM sys-root on AArch64 or MIPS32 on MIPS64).

This path is used to copy all 32-bit shared libraries from the 32-bit sys-root into the buildroot staging area. From there, they will be included into the root file system. Care is taken that 32-bit libraries will not interfere with 64-bit libraries already in staging.

We need the ability to specify a separate sys-root for 32-bit, because the 64-bit sys-root may not include 32-bit libraries.

This variable is not available on 32-bit setups.

---
 system/Config.in | 34 ++++++++++++++++++++++++++++++++++
 1 file changed, 34 insertions(+)
system/Config.in +34 -0
index d48cf8d8c4ed..1e9415ae6f6d 100644
@@ -24,6 +24,40 @@ config BR2_ROOTFS_SKELETON_CUSTOM
 
 endchoice
 
+config BR2_ROOTFS_LIB_DIR
+	string "Name of system library directories (e.g. lib or lib64)"
+	default "lib64" if BR2_ARCH_IS_64
+	default "lib"
+	help
+	  The name of the directories where system libraries live. This will
+	  most often be "lib", "lib32" or "lib64". This name will be used in
+	  the root directory (e.g. /lib64) as well as the /usr directory (e.g.
+	  /usr/lib64).
+
+if BR2_ARCH_IS_64
+
+config BR2_ROOTFS_LIB32_DIR
+	string "Name of directory for 32-bit libraries (e.g. lib or lib32)"
+	default "lib"
+	help
+	  The name of the directories where 32-bit libraries live. This will
+	  most often be "lib" or "lib32". This name will be used in the root
+	  directory (e.g. /lib) as well as the /usr directory (e.g. /usr/lib).
+
+config BR2_ROOTFS_RUNTIME32
+	bool "Install 32-bit runtime on 64-bit system"
+	help
+	  Install 32-bit libraries and the 32-bit shared library loader on a
+	  64-bit system, so it is possible to execute 32-bit applications.
+
+config BR2_ROOTFS_RUNTIME32_PATH
+	string "Path to the 32-bit run-time"
+	help
+	  Specifies the path to the 32-bit run-time that should be copied into
+	  the root file system.
+
+endif
+
 if BR2_ROOTFS_SKELETON_CUSTOM
 
 config BR2_ROOTFS_SKELETON_CUSTOM_PATH
Signature · 1 line
2.7.4
#3 lore raw

RFC 3/6 core: system and toolchain: 32-bit run-time support on 64-bit platform

In reply to Markus Mayer

Set up library symlinks only if we don't want to support 32-bit libraries on 64-bit platforms.

We cannot set up sym-links such as "lib64" -> "lib" if we want to support 32-bit binaries on a 64-bit platform. Instead, we need to keep 32-bit and 64-bit libraries in separate directories, so we must skip generating sym-links that would prevent this.

---
 system/system.mk                                       | 11 ++++++++++-
 toolchain/toolchain-external/pkg-toolchain-external.mk |  6 ++++++
 2 files changed, 16 insertions(+), 1 deletion(-)
system/system.mk +10 -1
index ca6bf1388f39..e0d45356fe66 100644
@@ -55,6 +55,14 @@ define SYSTEM_RSYNC
 		$(1)/ $(2)/
 endef
 
+ifeq ($(BR2_ROOTFS_RUNTIME32),y)
+
+define SYSTEM_LIB_SYMLINK
+	@echo "SYSTEM_LIB_SYMLINK: nothing to do"
+endef
+
+else
+
 # Make a symlink lib32->lib or lib64->lib as appropriate.
 # MIPS64/n32 requires lib32 even though it's a 64-bit arch.
 # $(1): base dir (either staging or target)
@@ -68,7 +76,8 @@ define SYSTEM_LIB_SYMLINK
 	ln -snf lib $(1)/lib32
 	ln -snf lib $(1)/usr/lib32
 endef
-endif
+endif # BR2_ARCH_IS_64
+endif # BR2_ROOTFS_RUNTIME32
 
 SYSTEM_GETTY_PORT = $(call qstrip,$(BR2_TARGET_GENERIC_GETTY_PORT))
 SYSTEM_GETTY_BAUDRATE = $(call qstrip,$(BR2_TARGET_GENERIC_GETTY_BAUDRATE))
toolchain/toolchain-external/pkg-toolchain-external.mk +6 -0
index 3bf9fac4121c..d7cf97d170ca 100644
@@ -448,6 +448,7 @@ define TOOLCHAIN_EXTERNAL_INSTALL_SYSROOT_LIBS
 	$(call copy_toolchain_sysroot,$${SYSROOT_DIR},$${ARCH_SYSROOT_DIR},$${ARCH_SUBDIR},$${ARCH_LIB_DIR},$${SUPPORT_LIB_DIR})
 endef
 
+ifeq ($(BR2_ROOTFS_RUNTIME32),)
 # Create a symlink from (usr/)$(ARCH_LIB_DIR) to lib.
 # Note: the skeleton package additionally creates lib32->lib or lib64->lib
 # (as appropriate)
@@ -462,6 +463,11 @@ create_lib_symlinks = \
 		ln -snf $${relpath}lib "$${DESTDIR}/usr/$${ARCH_LIB_DIR}" ; \
 	fi
 
+else
+create_lib_symlinks = \
+	@echo "BR2_ROOTFS_RUNTIME32 is set, skipping lib symlinks creation"
+endif
+
 define TOOLCHAIN_EXTERNAL_CREATE_STAGING_LIB_SYMLINK
 	$(call create_lib_symlinks,$(STAGING_DIR))
 endef
Signature · 1 line
2.7.4
#4 lore raw

RFC 4/6 core/pkg-toolchain-external: copy external 32-bit libraries to staging

In reply to Markus Mayer

We use rsync to copy the 32-bit libraries from the 32-bit sysroot into staging.

---
 toolchain/toolchain-external/pkg-toolchain-external.mk | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
toolchain/toolchain-external/pkg-toolchain-external.mk +6 -1
index d7cf97d170ca..0c5cd4b9f6d7 100644
@@ -445,7 +445,12 @@ define TOOLCHAIN_EXTERNAL_INSTALL_SYSROOT_LIBS
 		ARCH_SUBDIR=`echo $${ARCH_SYSROOT_DIR} | sed -r -e "s:^$${SYSROOT_DIR}(.*)/$$:\1:"` ; \
 	fi ; \
 	$(call MESSAGE,"Copying external toolchain sysroot to staging...") ; \
-	$(call copy_toolchain_sysroot,$${SYSROOT_DIR},$${ARCH_SYSROOT_DIR},$${ARCH_SUBDIR},$${ARCH_LIB_DIR},$${SUPPORT_LIB_DIR})
+	$(call copy_toolchain_sysroot,$${SYSROOT_DIR},$${ARCH_SYSROOT_DIR},$${ARCH_SUBDIR},$${ARCH_LIB_DIR},$${SUPPORT_LIB_DIR}); \
+	if [ "$(BR2_ROOTFS_RUNTIME32)" = "y" ]; then \
+		$(call MESSAGE,"Copying external toolchain 32-bit libraries to staging...") ; \
+		mkdir -p "$${STAGING_DIR}/$(BR2_ROOTFS_LIB32_DIR)"; \
+		rsync -a --exclude '*.a' "$(BR2_ROOTFS_RUNTIME32_PATH)/$(BR2_ROOTFS_LIB32_DIR)" "$${STAGING_DIR}"; \
+	fi
 endef
 
 ifeq ($(BR2_ROOTFS_RUNTIME32),)
Signature · 1 line
2.7.4
#5 lore raw

RFC 5/6 bzip2: introduce make variable $(LIBDIR)

In reply to Markus Mayer

In order to allow us to install libz2 into a location other than /usr/lib, we introduce a variable called $(LIBDIR) that can be set by the build system to an alternate location.

$(LIBDIR) defaults to /usr/lib, so the behaviour doesn't change if $(LIBDIR) is never set.

---
 package/bzip2/0003-add-libdir-to-makefile.patch    | 36 ++++++++++++++++++++++
 .../bzip2/0004-add-libdir-to-makefile-libbz2.patch | 31 +++++++++++++++++++
 2 files changed, 67 insertions(+)
 create mode 100644 package/bzip2/0003-add-libdir-to-makefile.patch
 create mode 100644 package/bzip2/0004-add-libdir-to-makefile-libbz2.patch
package/bzip2/0003-add-libdir-to-makefile.patch new file +36 -0
new file mode 100644
index 000000000000..8a724484bd1e
@@ -0,0 +1,36 @@
+Introduce $(LIBDIR) to Makefile
+
+The variable $(LIBDIR) allows us to override where libbz2 is installed.
+
+Index: bzip2-1.0.6/Makefile
+===================================================================
+--- bzip2-1.0.6.orig/Makefile	2010-09-10 15:46:02.000000000 -0700
++++ bzip2-1.0.6/Makefile	2017-11-23 12:51:53.318896836 -0800
+@@ -19,6 +19,7 @@
+ AR=ar
+ RANLIB=ranlib
+ LDFLAGS=
++LIBDIR ?= lib
+ 
+ override CFLAGS += -Wall
+ 
+@@ -71,7 +73,7 @@
+ 
+ install: bzip2 bzip2recover
+ 	if ( test ! -d $(PREFIX)/bin ) ; then mkdir -p $(PREFIX)/bin ; fi
+-	if ( test ! -d $(PREFIX)/lib ) ; then mkdir -p $(PREFIX)/lib ; fi
++	if ( test ! -d $(PREFIX)/$(LIBDIR) ) ; then mkdir -p $(PREFIX)/$(LIBDIR) ; fi
+ 	if ( test ! -d $(PREFIX)/man ) ; then mkdir -p $(PREFIX)/man ; fi
+ 	if ( test ! -d $(PREFIX)/man/man1 ) ; then mkdir -p $(PREFIX)/man/man1 ; fi
+ 	if ( test ! -d $(PREFIX)/include ) ; then mkdir -p $(PREFIX)/include ; fi
+@@ -87,8 +89,8 @@
+ 	chmod a+r $(PREFIX)/man/man1/bzip2.1
+ 	cp -f bzlib.h $(PREFIX)/include
+ 	chmod a+r $(PREFIX)/include/bzlib.h
+-	cp -f libbz2.a $(PREFIX)/lib
+-	chmod a+r $(PREFIX)/lib/libbz2.a
++	cp -f libbz2.a $(PREFIX)/$(LIBDIR)
++	chmod a+r $(PREFIX)/$(LIBDIR)/libbz2.a
+ 	cp -f bzgrep $(PREFIX)/bin/bzgrep
+ 	ln -s -f $(PREFIX)/bin/bzgrep $(PREFIX)/bin/bzegrep
+ 	ln -s -f $(PREFIX)/bin/bzgrep $(PREFIX)/bin/bzfgrep
package/bzip2/0004-add-libdir-to-makefile-libbz2.patch new file +31 -0
new file mode 100644
index 000000000000..9b2f6b114fc6
@@ -0,0 +1,31 @@
+Introduce $(LIBDIR) to Makefile-libbz2_so
+
+The variable $(LIBDIR) allows us to override where libbz2 is installed.
+
+Index: bzip2-1.0.6/Makefile
+===================================================================
+--- bzip2-1.0.6/Makefile-libbz2_so.orig	2017-11-23 11:08:33.067637637 -0800
++++ bzip2-1.0.6/Makefile-libbz2_so	2017-11-23 12:51:38.694728036 -0800
+@@ -23,6 +23,9 @@
+ 
+ SHELL=/bin/sh
+ CC=gcc
++
++LIBDIR ?= lib
++
+ override CFLAGS += -fpic -fPIC -Wall
+ 
+ OBJS= blocksort.sho  \
+@@ -38,9 +41,9 @@
+ 	$(CC) $(CFLAGS) -o bzip2-shared bzip2.c libbz2.so.1.0.6
+ 
+ install:
+-	install -m 0755 -D libbz2.so.1.0.6 $(PREFIX)/lib/libbz2.so.1.0.6
+-	ln -sf libbz2.so.1.0.6 $(PREFIX)/lib/libbz2.so
+-	ln -sf libbz2.so.1.0.6 $(PREFIX)/lib/libbz2.so.1.0
++	install -m 0755 -D libbz2.so.1.0.6 $(PREFIX)/$(LIBDIR)/libbz2.so.1.0.6
++	ln -sf libbz2.so.1.0.6 $(PREFIX)/$(LIBDIR)/libbz2.so
++	ln -sf libbz2.so.1.0.6 $(PREFIX)/$(LIBDIR)/libbz2.so.1.0
+ 
+ clean: 
+ 	rm -f $(OBJS) bzip2.o libbz2.so.1.0.6 libbz2.so.1.0 bzip2-shared
Signature · 1 line
2.7.4
#6 lore raw

RFC 6/6 package: use BR2_ROOTFS_LIB_DIR for all libraries we use

In reply to Markus Mayer

Make sure to pass $(BR2_ROOTFS_LIB_DIR) as the library directory name to all library packages we use.

Autoconf packages are easy. They only require a single added line in pkg-autotools.mk. It is the generic packages not using autoconf that need to be tweaked individually.

---
 package/bzip2/bzip2.mk                        | 4 +++-
 package/dmalloc/dmalloc.mk                    | 7 ++++---
 package/e2fsprogs/e2fsprogs.mk                | 2 ++
 package/libzlib/libzlib.mk                    | 3 ++-
 package/linux-tools/linux-tool-cpupower.mk.in | 7 ++++++-
 package/pciutils/pciutils.mk                  | 2 ++
 package/pkg-autotools.mk                      | 1 +
 package/readline/readline.mk                  | 2 +-
 9 files changed, 23 insertions(+), 8 deletions(-)
package/bzip2/bzip2.mk +3 -1
index e43533072b2d..67f0239944ca 100644
@@ -39,6 +39,7 @@ endef
 ifeq ($(BR2_STATIC_LIBS),)
 define BZIP2_INSTALL_TARGET_SHARED_CMDS
 	$(TARGET_MAKE_ENV) $(MAKE) \
+		LIBDIR=$(TARGET_DIR)/usr/$(BR2_ROOTFS_LIB_DIR) \
 		-f Makefile-libbz2_so PREFIX=$(TARGET_DIR)/usr -C $(@D) install
 endef
 endif
@@ -47,7 +48,8 @@ endif
 define BZIP2_INSTALL_TARGET_CMDS
 	rm -f $(addprefix $(TARGET_DIR)/usr/bin/,bzip2 bunzip2 bzcat)
 	$(TARGET_MAKE_ENV) $(MAKE) \
-		PREFIX=$(TARGET_DIR)/usr -C $(@D) install
+		PREFIX=$(TARGET_DIR)/usr \
+		LIBDIR=$(TARGET_DIR)/usr/$(BR2_ROOTFS_LIB_DIR) -C $(@D) install
 	$(BZIP2_INSTALL_TARGET_SHARED_CMDS)
 endef
 
package/dmalloc/dmalloc.mk +4 -3
index 6ebb44c5c068..d54950fec5a1 100644
@@ -50,15 +50,16 @@ DMALLOC_POST_PATCH_HOOKS += DMALLOC_POST_PATCH
 define DMALLOC_INSTALL_STAGING_CMDS
 	$(TARGET_MAKE_ENV) $(MAKE) includedir="$(STAGING_DIR)/usr/include" \
 		bindir="$(STAGING_DIR)/usr/bin" \
-		libdir="$(STAGING_DIR)/usr/lib" \
-		shlibdir="$(STAGING_DIR)/usr/lib" \
+		libdir="$(STAGING_DIR)/usr/$(BR2_ROOTFS_LIB_DIR)" \
+		shlibdir="$(STAGING_DIR)/usr/$(BR2_ROOTFS_LIB_DIR)" \
 		infodir="$(STAGING_DIR)/usr/share/info/" \
 		-C $(@D) install
 endef
 
 ifeq ($(BR2_STATIC_LIBS),)
 define DMALLOC_INSTALL_SHARED_LIB
-	cp -dpf $(STAGING_DIR)/usr/lib/libdmalloc*.so $(TARGET_DIR)/usr/lib
+	cp -dpf $(STAGING_DIR)/usr/$(BR2_ROOTFS_LIB_DIR)/libdmalloc*.so \
+		$(TARGET_DIR)/usr/$(BR2_ROOTFS_LIB_DIR)
 endef
 endif
 
package/e2fsprogs/e2fsprogs.mk +2 -0
index e77b10f26dae..5cf2445562c9 100644
@@ -65,6 +65,8 @@ ifeq ($(BR2_nios2),y)
 E2FSPROGS_CONF_ENV += ac_cv_func_fallocate=no
 endif
 
+E2FSPROGS_CONF_ENV += PKG_CONFIG_PATH="$(TARGET_DIR)/usr/$(BR2_ROOTFS_LIB_DIR)/pkgconfig"
+
 # Some programs are built for the host, but use definitions guessed by
 # the configure script (i.e with the cross-compiler). Help them by
 # saying that <sys/stat.h> is available on the host, which is needed
package/libzlib/libzlib.mk +2 -1
index eea0c12f2209..7ef79bac21b7 100644
@@ -31,6 +31,7 @@ define LIBZLIB_CONFIGURE_CMDS
 		./configure \
 		$(LIBZLIB_SHARED) \
 		--prefix=/usr \
+		--libdir='$${exec_prefix}/$(BR2_ROOTFS_LIB_DIR)' \
 	)
 endef
 
@@ -65,7 +66,7 @@ endef
 # assembling the filesystem images anyway.
 ifeq ($(BR2_SHARED_LIBS),y)
 define LIBZLIB_RM_STATIC_STAGING
-	rm -f $(STAGING_DIR)/usr/lib/libz.a
+	rm -f $(STAGING_DIR)/usr/$(BR2_ROOTFS_LIB_DIR)/libz.a
 endef
 LIBZLIB_POST_INSTALL_STAGING_HOOKS += LIBZLIB_RM_STATIC_STAGING
 endif
package/linux-tools/linux-tool-cpupower.mk.in +6 -1
index 2a2d3e01dd5a..9a5a32efb2de 100644
@@ -8,10 +8,13 @@ LINUX_TOOLS += cpupower
 
 CPUPOWER_DEPENDENCIES = pciutils $(TARGET_NLS_DEPENDENCIES)
 
+CPUPOWER_LDFLAGS = -L$(TARGET_DIR)/usr/$(BR2_ROOTFS_LIB_DIR)
+CPUPOWER_LDFLAGS += $(TARGET_NLS_LIBS)
+
 CPUPOWER_MAKE_OPTS = CROSS=$(TARGET_CROSS) \
 	CPUFREQ_BENCH=false \
 	NLS=false \
-	LDFLAGS=$(TARGET_NLS_LIBS) \
+	LDFLAGS=$(CPUPOWER_LDFLAGS) \
 	DEBUG=false
 
 define CPUPOWER_BUILD_CMDS
@@ -30,6 +33,7 @@ define CPUPOWER_INSTALL_STAGING_CMDS
 	$(TARGET_MAKE_ENV) $(MAKE) -C $(LINUX_DIR)/tools \
 		$(CPUPOWER_MAKE_OPTS) \
 		DESTDIR=$(STAGING_DIR) \
+		libdir=/usr/$(BR2_ROOTFS_LIB_DIR) \
 		cpupower_install
 endef
 
@@ -37,5 +41,6 @@ define CPUPOWER_INSTALL_TARGET_CMDS
 	$(TARGET_MAKE_ENV) $(MAKE) -C $(LINUX_DIR)/tools \
 		$(CPUPOWER_MAKE_OPTS) \
 		DESTDIR=$(TARGET_DIR) \
+		libdir=/usr/$(BR2_ROOTFS_LIB_DIR) \
 		cpupower_install
 endef
package/pciutils/pciutils.mk +2 -0
index 2dd5771f2193..775425c1d5a4 100644
@@ -67,12 +67,14 @@ endef
 define PCIUTILS_INSTALL_TARGET_CMDS
 	$(TARGET_MAKE_ENV) $(MAKE1) -C $(@D) $(PCIUTILS_MAKE_OPTS) \
 		PREFIX=$(TARGET_DIR)/usr SBINDIR=$(TARGET_DIR)/usr/bin \
+		LIBDIR=$(TARGET_DIR)/usr/$(BR2_ROOTFS_LIB_DIR) \
 		install install-lib install-pcilib
 endef
 
 define PCIUTILS_INSTALL_STAGING_CMDS
 	$(TARGET_MAKE_ENV) $(MAKE1) -C $(@D) $(PCIUTILS_MAKE_OPTS) \
 		PREFIX=$(STAGING_DIR)/usr SBINDIR=$(STAGING_DIR)/usr/bin \
+		LIBDIR=$(TARGET_DIR)/usr/$(BR2_ROOTFS_LIB_DIR) \
 		install install-lib install-pcilib
 endef
 
package/pkg-autotools.mk +1 -0
index 45de99356f5f..95f423737074 100644
@@ -189,6 +189,7 @@ define $(2)_CONFIGURE_CMDS
 		--build=$$(GNU_HOST_NAME) \
 		--prefix=/usr \
 		--exec-prefix=/usr \
+		--libdir='$$$${exec_prefix}/$(BR2_ROOTFS_LIB_DIR)' \
 		--sysconfdir=/etc \
 		--localstatedir=/var \
 		--program-prefix="" \
package/readline/readline.mk +1 -1
index cc5d4f29201e..decb88cfa73c 100644
@@ -27,7 +27,7 @@ READLINE_POST_INSTALL_TARGET_HOOKS += READLINE_INSTALL_INPUTRC
 ifneq ($(BR2_STATIC_LIBS),y)
 # libraries get installed read only, so strip fails
 define READLINE_INSTALL_FIXUPS_SHARED
-	chmod +w $(addprefix $(TARGET_DIR)/usr/lib/,libhistory.so.* libreadline.so.*)
+	chmod +w $(addprefix $(TARGET_DIR)/usr/$(BR2_ROOTFS_LIB_DIR)/,libhistory.so.* libreadline.so.*)
 endef
 READLINE_POST_INSTALL_TARGET_HOOKS += READLINE_INSTALL_FIXUPS_SHARED
 endif
Signature · 1 line
2.7.4
#7 lore raw

cover Support 32-bit binaries on a 64-bit architecture

In reply to Markus Mayer

Hi Markus,

Quoted 6 lines · Markus Mayer

This series contains a proposal for supporting 32-bit libraries and binaries on a 64-bit platform.

There are some limitations and prerequisites as to the scope this has been tested.

Quoted 9 lines

- It has only been tested on ARM/ARM64.
- It has only been tested with an external toolchain
(https://github.com/Broadcom/stbgcc-6.3/releases).
- It requires that the Aarch64 compiler be compiled with "multi-lib" enabled. This ensures that the sysroot doesn't contain a lib64 -> lib symlink. Instead, lib and lib64 are separate directories.
- It has only been tested with a fairly limited number of packages. There might be other packages that don't correctly pass --libdir, similar to bzip2 as mentioned below.

There is one thing I am missing here. AFAICS, this series does not add support for building 32-bit libraries in addition to their 64-bit versions. So the 32-bit binaries that are to run on target must have their dependencies satisfied from the toolchain provided libraries. Is that correct?

baruch

Quoted 47 lines

The patches in this series make the following changes, in this order:

- ensure that check-bin-arch allows Aarch32 binaries on Aarch64
- introduce some new BR2 configuration variables (primarily for the 32-bit sysroot location on the host and where system libraries live by default on the target)
- tell the Buildroot core not to create lib32 or lib64 symlink when 32-bit support is enabled
- tell the Buildroot core to copy the 32-bit sysroot when 32-bit support is enabled
- introduce the $(LIBDIR) variable to the bzip2 package, so we can tell it that libbz2.so should go in /usr/lib64 when that is desired
- modify the package core to pass /usr/lib (or /usr/lib64) as the library location for the system

The result is that all 64-bit shared libraries should be installed into /usr/lib64. The 32-bit libraries are copied from the 32-bit sysroot on the host into /usr/lib.

I am quite certain this submission will only be a first step, but it does look like there are others who are faced with the same issue.

Markus Mayer (6):
support/scripts/check-bin-arch: improve architecture check system/Config.in: add configuration options for 32-bit library support
core: system and toolchain: 32-bit run-time support on 64-bit platform core/pkg-toolchain-external: copy external 32-bit libraries to staging
bzip2: introduce make variable $(LIBDIR)
package: use BR2_ROOTFS_LIB_DIR for all libraries we use

 package/bzip2/0003-add-libdir-to-makefile.patch    | 36 ++++++++++++++++++++++
 .../bzip2/0004-add-libdir-to-makefile-libbz2.patch | 31 +++++++++++++++++++
 package/bzip2/bzip2.mk                             |  4 ++-
 package/dmalloc/dmalloc.mk                         |  7 +++--
 package/e2fsprogs/e2fsprogs.mk                     |  2 ++
 package/libzlib/libzlib.mk                         |  3 +-
 package/linux-tools/linux-tool-cpupower.mk.in      |  7 ++++-
 package/pciutils/pciutils.mk                       |  2 ++
 package/pkg-autotools.mk                           |  1 +
 package/readline/readline.mk                       |  2 +-
 support/scripts/check-bin-arch                     | 15 +++++++++
 system/Config.in                                   | 34 ++++++++++++++++++++
 system/system.mk                                   | 11 ++++++-
 .../toolchain-external/pkg-toolchain-external.mk   | 13 +++++++-
 15 files changed, 161 insertions(+), 10 deletions(-)
 create mode 100644 package/bzip2/0003-add-libdir-to-makefile.patch
 create mode 100644 package/bzip2/0004-add-libdir-to-makefile-libbz2.patch
Signature · 3 lines
     http://baruch.siach.name/blog/                  ~. .~   Tk Open Systems
=}------------------------------------------------ooO--U--Ooo------------{=
   - baruch at tkos.co.il - tel: +972.52.368.4656, http://www.tkos.co.il -
#8 lore raw

cover Support 32-bit binaries on a 64-bit architecture

Hello,

Quoted 17 lines · Baruch Siach
Quoted 6 lines · Markus Mayer

This series contains a proposal for supporting 32-bit libraries and binaries on a 64-bit platform.

There are some limitations and prerequisites as to the scope this has been tested.

Quoted 9 lines

- It has only been tested on ARM/ARM64.
- It has only been tested with an external toolchain
(https://github.com/Broadcom/stbgcc-6.3/releases).
- It requires that the Aarch64 compiler be compiled with "multi-lib" enabled. This ensures that the sysroot doesn't contain a lib64 -> lib symlink. Instead, lib and lib64 are separate directories.
- It has only been tested with a fairly limited number of packages. There might be other packages that don't correctly pass --libdir, similar to bzip2 as mentioned below.

Quoted 4 lines

There is one thing I am missing here. AFAICS, this series does not add support for building 32-bit libraries in addition to their 64-bit versions. So the 32-bit binaries that are to run on target must have their dependencies satisfied from the toolchain provided libraries. Is that correct?

Yes, indeed, I'm interested in understanding that part as well. It seems like the assumption is that the external toolchain has a 32-bit sysroot, and only the libraries from this sysroot are necessary.

This feels very limited/restricted to very specific use cases and therefore not very usable in a broader context.

And extending that to building packages is really difficult with Buildroot:

(1) Kconfig has no way to easily allow to select for each and every package whether it should be built 32-bit, 64-bit or both.

(2) Our package infrastructure is entirely based on the fact that a given package is built only once. So if you assume that you solved problem (1) by adding gazillions of Config.in options to each and every package, there is still no way the zlib package will be built two times, once 32-bit and 64-bit. No without major changes to the package infrastructure.

So supporting multilib has already been discussed several times, but nobody ever came up with a proposal that could reasonably solve problems (1) and (2). Unfortunately, unless there are some proposals to solve those problems, I don't think we want to have some very limited and highly specific multilib support, that only works with external toolchains and that only works for 32-bit libraries provided by the external toolchain.

Best regards,

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering
http://bootlin.com
#9 lore raw

cover Support 32-bit binaries on a 64-bit architecture

Quoted 21 lines · Thomas Petazzoni

Hello,

Quoted 17 lines · Baruch Siach
Quoted 15 lines · Markus Mayer

This series contains a proposal for supporting 32-bit libraries and binaries on a 64-bit platform.

There are some limitations and prerequisites as to the scope this has been tested.

- It has only been tested on ARM/ARM64.
- It has only been tested with an external toolchain
(https://github.com/Broadcom/stbgcc-6.3/releases).
- It requires that the Aarch64 compiler be compiled with "multi-lib" enabled. This ensures that the sysroot doesn't contain a lib64 -> lib symlink. Instead, lib and lib64 are separate directories.
- It has only been tested with a fairly limited number of packages. There might be other packages that don't correctly pass --libdir, similar to bzip2 as mentioned below.

Quoted 8 lines
Quoted 4 lines

There is one thing I am missing here. AFAICS, this series does not add support for building 32-bit libraries in addition to their 64-bit versions. So the 32-bit binaries that are to run on target must have their dependencies satisfied from the toolchain provided libraries. Is that correct?

Yes, indeed, I'm interested in understanding that part as well. It seems like the assumption is that the external toolchain has a 32-bit sysroot, and only the libraries from this sysroot are necessary.

Correct, that is what the stbgcc-6.3 toolchain contains.

Quoted 3 lines

This feels very limited/restricted to very specific use cases and therefore not very usable in a broader context.

I suppose it could be reasonably easy to extend on that and either require a second toolchain to be built or just point to a different location?

Quoted 14 lines

And extending that to building packages is really difficult with Buildroot:

(1) Kconfig has no way to easily allow to select for each and every package whether it should be built 32-bit, 64-bit or both.

(2) Our package infrastructure is entirely based on the fact that a given package is built only once. So if you assume that you solved problem (1) by adding gazillions of Config.in options to each and every package, there is still no way the zlib package will be built two times, once 32-bit and 64-bit. No without major changes to the package infrastructure.

Quoted 7 lines

So supporting multilib has already been discussed several times, but nobody ever came up with a proposal that could reasonably solve problems (1) and (2). Unfortunately, unless there are some proposals to solve those problems, I don't think we want to have some very limited and highly specific multilib support, that only works with external toolchains and that only works for 32-bit libraries provided by the external toolchain.

Would you recommend that this logic be sort of done using a two pass build, by that I mean something like an external top-level Makefile would be controlling the buildroot build process and do something like:

- build all relevant packages for the 32-bit architecture (that could include the toolchain as well)

- build all relevant packages for the 64-bit architecture, and copy the 32-bit binaries and libraries of interest from the previously build 32-bit environment?

I suppose such a thing could be done, it has the advantage of not tweaking buildroot too much, although we still need to teach the 64-bit variant to accept seeing "foreign" binaries.

Signature · 1 line
Florian