Remove /usr component from HOST_DIR

We currently use $(HOST_DIR)/usr as the prefix for host packages. That has a few disadvantages:

- There are some things installed in $(HOST_DIR)/etc and $(HOST_DIR)/sbin, which is inconsistent.

- To pack a buildroot-built toolchain into a tarball for use as an external toolchain, you have to pack output/host/usr instead of the more obvious output/host.

- Because of the above, the internal toolchain wrapper breaks which forces us to work around it (call the actual toolchain executable directly). This is OK for us, but when used in another build system, that's a problem.

- Paths are four characters longer.

This series replaces all occurences of 'host/usr' (in its various forms) with 'host', and updates relative paths accordingly.

To allow for a gradual change, host/usr is replaced with a symlink, so that everything keeps working. Except that it turns out that not everything keeps working when we do that: quite a few things need to be fixed. Therefore that patch is split up into three steps:

- First $(HOST_DIR)/usr/$(GNU_TARGET_NAME) is moved one level up. This can be done safely (tested with both internal and external toolchain and building a few packages). The usr bit is removed from STAGING_DIR in the same patch, though it could have been split off.

- Next, $(HOST_DIR)/usr/lib is moved one level up. This can only be done after the first step, otherwise the internal toolchain fails to find ar, ld, etc. Also, when we do this, check-host-rpath has to be adapted to take into account the new location of the shared libs. The check-host-rpath change could have been partly split off in a separate patch, but then we'd have to check for either lib or usr/lib, and afterwards check only for lib again.

- Then we can move the entire host/usr one level up. At the same time, the ../.. in the external toolchain and toolchain wrapper has to be changed. Doing /bin and /sbin in a separate step still turns out not to be possible, again because of the internal toolchain that doesn't find cc1 etc. And it anyway wouldn't help much, because the ../.. change has to be done in the same patch as moving the bin directory.

I've tested both internal and external toolchains, and a large number of host packages after these changes, including the target packages that depend on them. That turned up one problem: libsepol, which creates a symlink that is no longer valid after the move. Fortunately that could be fixed in an independent patch. So after this series (and at any point in between) everything seems to work fine still.

This is only the first part of the cleanup. It still needs to be followed by about 45 patches to remove the /usr part from all packages. I'm waiting with rebasing and posting that part because it is very fragile w.r.t. merge conflicts.

I also noticed a small issue in pkg-rebar, so I added an additional patch in the very beginning of the series that fixes it. It's totally independent of the rest, however.

Regards,
Arnout

-----------------------------------------------------------------------------
Arnout Vandecappelle (8):
pkg-rebar: remove redundant / in front of $($(PKG)_ERLANG_LIBDIR)
libsepol: rework host installation
Move $(HOST_DIR)/usr/$(GNU_TARGET_NAME) one level up. Move $(HOST_DIR)/usr/lib to $(HOST_DIR)/lib
Eliminate $(HOST_DIR)/usr
check-host-rpath: no longer check $(HOST_DIR)/usr/{bin,sbin}
Makefile: remove $(HOST_DIR)/usr from BR_PATH package/Makefile.in: remove $(HOST_DIR)/usr part from HOST_LDFLAGS

 Makefile                                               | 11 ++++++++---
 package/Makefile.in                                    |  4 ++--
 package/libsepol/libsepol.mk                           | 13 ++++++++-----
 package/pkg-rebar.mk                                   |  6 +++---
 support/scripts/check-host-rpath                       |  8 +++++---
 toolchain/toolchain-external/pkg-toolchain-external.mk |  6 +++---
 toolchain/toolchain-wrapper.c                          |  2 +-
 7 files changed, 30 insertions(+), 20 deletions(-)

2.13.2

Sometimes $(1)/$($(PKG)_ERLANG_LIBDIR) is used, sometimes $(1)$($(PKG)_ERLANG_LIBDIR). Make this consistent, with /. Therefore, remove the leading / from $($(PKG)_ERLANG_LIBDIR).

---
v2: keep the /, remove it from $($(PKG)_ERLANG_LIBDIR)

---
 package/pkg-rebar.mk | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

index 7605a64ade..cc6b775ae5 100644
@@ -44,7 +44,7 @@ REBAR_TARGET_DEPS_ENV = \
 # Install an Erlang application from $(@D).
 #
 # i.e., define a recipe that installs the "bin ebin priv $(2)" directories
-# from $(@D) to $(1)$($(PKG)_ERLANG_LIBDIR).
+# from $(@D) to $(1)/$($(PKG)_ERLANG_LIBDIR).
 #
 #  argument 1 should typically be $(HOST_DIR), $(TARGET_DIR),
 #	      or $(STAGING_DIR).

@@ -59,7 +59,7 @@ define install-erlang-directories
 	$(INSTALL) -d $(1)/$($(PKG)_ERLANG_LIBDIR)
 	for dir in bin ebin priv $(2); do                               \
 		if test -d $(@D)/$$dir; then                            \
-			cp -r $(@D)/$$dir $(1)$($(PKG)_ERLANG_LIBDIR);  \
+			cp -r $(@D)/$$dir $(1)/$($(PKG)_ERLANG_LIBDIR); \
 		fi;                                                     \
 	done
 endef

@@ -122,7 +122,7 @@ $(2)_ERLANG_APP = $(subst -,_,$(patsubst erlang-%,%,$(patsubst host-%,%,$(1))))
 # for host packages, or $(STAGING_DIR) for target packages.
 #
 $(2)_ERLANG_LIBDIR = \
-	/usr/lib/erlang/lib/$$($$(PKG)_ERLANG_APP)-$$($$(PKG)_VERSION)
+	usr/lib/erlang/lib/$$($$(PKG)_ERLANG_APP)-$$($$(PKG)_VERSION)
 
 # If a host package, inherit <pkg>_USE_BUNDLED_REBAR from the target
 # package, if not explicitly defined. Otherwise, default to NO.

2.13.2

Hello,

Applied to master, thanks.

Thomas

Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

libsepol has a pretty peculiar interpretation of DESTDIR and PREFIX. PREFIX is not consistently used: some installation paths are forced to $(DESTDIR)/usr/... . Also, the shared lib symlink is forced to ../../lib (i.e. assumes it's installed under /usr). For these reasons, the host build and install commands are a bit unusual: they use DESTDIR instead of PREFIX, DESTDIR points to $(HOST_DIR) without /usr, and some additional fixup is needed to correct the installation directory of libsepol.so.1 and the symlink to it.

This can be simplified quite a lot by passing both DESTDIR and PREFIX, and including the /usr part in both. The symlink still has to be fixed, but that's much simpler.

Note that a side effect is that the man pages are now installed under $(HOST_DIR)/usr/usr/man. While not very nice, we don't really care about man pages. In addition, this will resolve itself when we later move $(HOST_DIR)/usr to $(HOST_DIR).

---
It's a bit counter-intuitive to have a patch that adds /usr in a series that is meant to remove it, but this is the simplest way to have a bisectable series...

---
 package/libsepol/libsepol.mk | 13 ++++++++-----
 1 file changed, 8 insertions(+), 5 deletions(-)

index 9bdac1e0da..d360bfbcb1 100644
@@ -33,15 +33,18 @@ define LIBSEPOL_INSTALL_TARGET_CMDS
 	$(TARGET_MAKE_ENV) $(MAKE) -C $(@D) install $(LIBSEPOL_MAKE_FLAGS) DESTDIR=$(TARGET_DIR)
 endef
 
+HOST_LIBSEPOL_MAKE_ENV = \
+	$(HOST_MAKE_ENV) \
+	DESTDIR=$(HOST_DIR)/usr \
+	PREFIX=$(HOST_DIR)/usr
+
 define HOST_LIBSEPOL_BUILD_CMDS
-	$(HOST_MAKE_ENV) $(MAKE) -C $(@D) $(HOST_CONFIGURE_OPTS) DESTDIR=$(HOST_DIR)
+	$(HOST_LIBSEPOL_MAKE_ENV) $(MAKE) -C $(@D) $(HOST_CONFIGURE_OPTS)
 endef
 
 define HOST_LIBSEPOL_INSTALL_CMDS
-	$(HOST_MAKE_ENV) $(MAKE) -C $(@D) install $(HOST_CONFIGURE_OPTS) DESTDIR=$(HOST_DIR)
-	mv $(HOST_DIR)/lib/libsepol.so.1 $(HOST_DIR)/usr/lib
-	(cd $(HOST_DIR)/usr/lib; rm -f libsepol.so; ln -s libsepol.so.1 libsepol.so)
-	-rmdir $(HOST_DIR)/lib
+	$(HOST_LIBSEPOL_MAKE_ENV) $(MAKE) -C $(@D) install $(HOST_CONFIGURE_OPTS)
+	ln -sf libsepol.so.1 $(HOST_DIR)/usr/lib/libsepol.so
 endef
 
 $(eval $(generic-package))

2.13.2

Hello,

Applied to master, thanks.

Thomas

Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

This is a step towards eliminating $(HOST_DIR)/usr. It allows us to convert all packages installing things into $(HOST_DIR)/usr/$(GNU_TARGET_NAME) (i.e., binutils and gcc) without affecting the rest.

To allow compatibility with packages that still use $(HOST_DIR)/usr as the prefix, create a symlink from usr/$(GNU_TARGET_NAME) to ../$(GNU_TARGET_NAME).

Note that the symlink creation will break when $(HOST_DIR)/usr/lib already exists as a directory, i.e. when rebuilding in an existing output directory. This is necessary: if we don't break it now, the following commits (which remove the usr part from various variables) _will_ break it.

Effectively, the usr/ part is removed from $(STAGING_SUBDIR) (and therefore from $(STAGING_DIR)), so update the definition of that variable right away.

---
 Makefile            | 2 ++
 package/Makefile.in | 2 +-
 2 files changed, 3 insertions(+), 1 deletion(-)

index 36231fa9d6..fb252f2eb1 100644
@@ -556,6 +556,8 @@ world: target-post-image
 $(STAGING_DIR):
 	@mkdir -p $(STAGING_DIR)
 	@ln -snf $(STAGING_DIR) $(BASE_DIR)/staging
+	@mkdir -p $(HOST_DIR)/usr
+	@ln -snf ../$(GNU_TARGET_NAME) $(HOST_DIR)/usr/$(GNU_TARGET_NAME)
 
 RSYNC_VCS_EXCLUSIONS = \
 	--exclude .svn --exclude .git --exclude .hg --exclude .bzr \

index 8087bde999..6b88ac4feb 100644
@@ -107,7 +107,7 @@ ifeq ($(BR2_arc)$(BR2_ARC_ATOMIC_EXT),yy)
 TARGET_ABI += -matomic
 endif
 
-STAGING_SUBDIR = usr/$(GNU_TARGET_NAME)/sysroot
+STAGING_SUBDIR = $(GNU_TARGET_NAME)/sysroot
 STAGING_DIR    = $(HOST_DIR)/$(STAGING_SUBDIR)
 
 ifeq ($(BR2_OPTIMIZE_0),y)

2.13.2

This is a step towards eliminating $(HOST_DIR)/usr. It allows us to convert all packages installing things into $(HOST_DIR)/usr/lib without affecting the rest.

To allow compatibility with packages that still use $(HOST_DIR)/usr as the prefix, create a symlink from usr/lib to ../lib.

Note that the symlink creation will break when $(HOST_DIR)/usr/lib already exists as a directory, i.e. when rebuilding in an existing output directory. This is necessary: if we don't break it now, the following commits (which remove the usr part from various variables) _will_ break it.

At the same time as creating this symlink, we also have to update the check-host-rpath to accept both $(HOST_DIR)/usr/lib and $(HOST_DIR)/lib, because depending on how the package derives the path, it may be different.

Since there are some dependency chains that involve $(STAGING_DIR), $(STAGING_DIR) may in fact be created before $(HOST_DIR). Since $(STAGING_DIR) is a subdirectory of $(HOST_DIR), it is possible that the newly added rule for $(HOST_DIR) never triggers. To make sure that the rule does trigger, add an order-only dependency from $(STAGING_DIR) to $(HOST_DIR).

---
v2: rebase on modifications of check-host-rpath with $ORIGIN.

---
 Makefile                         | 10 +++++++---
 support/scripts/check-host-rpath |  8 +++++---
 2 files changed, 12 insertions(+), 6 deletions(-)

index fb252f2eb1..dee90b531c 100644
@@ -237,7 +237,7 @@ LEGAL_REPORT = $(LEGAL_INFO_DIR)/README
 # dependencies anywhere else
 #
 ################################################################################
-$(BUILD_DIR) $(TARGET_DIR) $(HOST_DIR) $(BINARIES_DIR) $(LEGAL_INFO_DIR) $(REDIST_SOURCES_DIR_TARGET) $(REDIST_SOURCES_DIR_HOST):
+$(BUILD_DIR) $(TARGET_DIR) $(BINARIES_DIR) $(LEGAL_INFO_DIR) $(REDIST_SOURCES_DIR_TARGET) $(REDIST_SOURCES_DIR_HOST):
 	@mkdir -p $@
 
 BR2_CONFIG = $(CONFIG_DIR)/.config

@@ -552,11 +552,15 @@ prepare: $(BUILD_DIR)/buildroot-config/auto.conf
 .PHONY: world
 world: target-post-image
 
+# When creating HOST_DIR, also symlink usr/lib -> ../lib
+$(HOST_DIR):
+	@mkdir -p $@/usr $@/lib
+	@ln -snf ../lib $@/usr/lib
+
 # Populating the staging with the base directories is handled by the skeleton package
-$(STAGING_DIR):
+$(STAGING_DIR): | $(HOST_DIR)
 	@mkdir -p $(STAGING_DIR)
 	@ln -snf $(STAGING_DIR) $(BASE_DIR)/staging
-	@mkdir -p $(HOST_DIR)/usr
 	@ln -snf ../$(GNU_TARGET_NAME) $(HOST_DIR)/usr/$(GNU_TARGET_NAME)
 
 RSYNC_VCS_EXCLUSIONS = \

index 020c12379f..adabfdf53f 100755
@@ -1,7 +1,7 @@
 #!/usr/bin/env bash
 
-# This script scans $(HOST_DIR)/{bin,sbin} for all ELF files, and checks
-# they have an RPATH to $(HOST_DIR)/usr/lib if they need libraries from
+# This script scans $(HOST_DIR)/{,usr/}{bin,sbin} for all ELF files, and checks
+# they have an RPATH to $(HOST_DIR)/{,usr/}lib if they need libraries from
 # there.
 
 # Override the user's locale so we are sure we can parse the output of

@@ -40,7 +40,7 @@ elf_needs_rpath() {
     local lib
 
     while read lib; do
-        [ -e "${hostdir}/usr/lib/${lib}" ] && return 0
+        [ -e "${hostdir}/lib/${lib}" ] && return 0
     done < <( readelf -d "${file}"                                         \
               |sed -r -e '/^.* \(NEEDED\) .*Shared library: \[(.+)\]$/!d;' \
                      -e 's//\1/;'                                          \

@@ -58,6 +58,8 @@ check_elf_has_rpath() {
         for dir in ${rpath//:/ }; do
             # Remove duplicate and trailing '/' for proper match
             dir="$( sed -r -e 's:/+:/:g; s:/$::;' <<<"${dir}" )"
+            [ "${dir}" = "${hostdir}/lib" -o "${dir}" = "\$ORIGIN/../lib" ] && return 0
+            # For the time being, the rpath is allowed with both usr/lib and lib
             [ "${dir}" = "${hostdir}/usr/lib" -o "${dir}" = "\$ORIGIN/../../usr/lib" ] && return 0
         done
     done < <( readelf -d "${file}"                                              \

2.13.2

We currently use $(HOST_DIR)/usr as the prefix for host packages. That has a few disadvantages:

- There are some things installed in $(HOST_DIR)/etc and $(HOST_DIR)/sbin, which is inconsistent.

- To pack a buildroot-built toolchain into a tarball for use as an external toolchain, you have to pack output/host/usr instead of the more obvious output/host.

- Because of the above, the internal toolchain wrapper breaks which forces us to work around it (call the actual toolchain executable directly). This is OK for us, but when used in another build system, that's a problem.

- Paths are four characters longer.

To allow us to gradually eliminate $(HOST_DIR)/usr while building packages, replace it with a symlink to .

The symlinks from $(HOST_DIR)/usr/$(GNU_TARGET_NAME) and $(HOST_DIR)/usr/lib that were added previously are removed again.

Note that the symlink creation will break when $(HOST_DIR)/usr already exists as a directory, i.e. when rebuilding in an existing output directory. This is necessary: if we don't break it now, the following commits (which remove the usr part from various variables) _will_ break it.

At the same time as creating this symlink, we have to update the external toolchain wrapper and the external toolchain symlinks to go one directory less up. Indeed, $(HOST_DIR) is one level less up than it was before.

---
I intended to also move $(HOST_DIR)/usr/bin in a separate patch, but that turns out to break the internal toolchain: host-gcc-initial uses relative paths to find cc1 etc. and these will be broken.

---
 Makefile                                               | 7 +++----
 toolchain/toolchain-external/pkg-toolchain-external.mk | 6 +++---
 toolchain/toolchain-wrapper.c                          | 2 +-
 3 files changed, 7 insertions(+), 8 deletions(-)

index dee90b531c..c728364ab6 100644
@@ -552,16 +552,15 @@ prepare: $(BUILD_DIR)/buildroot-config/auto.conf
 .PHONY: world
 world: target-post-image
 
-# When creating HOST_DIR, also symlink usr/lib -> ../lib
+# When creating HOST_DIR, also symlink usr -> .
 $(HOST_DIR):
-	@mkdir -p $@/usr $@/lib
-	@ln -snf ../lib $@/usr/lib
+	@mkdir -p $@
+	@ln -snf . $@/usr
 
 # Populating the staging with the base directories is handled by the skeleton package
 $(STAGING_DIR): | $(HOST_DIR)
 	@mkdir -p $(STAGING_DIR)
 	@ln -snf $(STAGING_DIR) $(BASE_DIR)/staging
-	@ln -snf ../$(GNU_TARGET_NAME) $(HOST_DIR)/usr/$(GNU_TARGET_NAME)
 
 RSYNC_VCS_EXCLUSIONS = \
 	--exclude .svn --exclude .git --exclude .hg --exclude .bzr \

index 8460e37d09..adbd3fe67e 100644
@@ -250,18 +250,18 @@ define TOOLCHAIN_EXTERNAL_INSTALL_WRAPPER
 		base=$${i##*/}; \
 		case "$$base" in \
 		*-ar|*-ranlib|*-nm) \
-			ln -sf $$(echo $$i | sed 's%^$(HOST_DIR)%../..%') .; \
+			ln -sf $$(echo $$i | sed 's%^$(HOST_DIR)%..%') .; \
 			;; \
 		*cc|*cc-*|*++|*++-*|*cpp|*-gfortran) \
 			ln -sf toolchain-wrapper $$base; \
 			;; \
 		*gdb|*gdbtui) \
 			if test "$(BR2_PACKAGE_HOST_GDB)" != "y"; then \
-				ln -sf $$(echo $$i | sed 's%^$(HOST_DIR)%../..%') .; \
+				ln -sf $$(echo $$i | sed 's%^$(HOST_DIR)%..%') .; \
 			fi \
 			;; \
 		*) \
-			ln -sf $$(echo $$i | sed 's%^$(HOST_DIR)%../..%') .; \
+			ln -sf $$(echo $$i | sed 's%^$(HOST_DIR)%..%') .; \
 			;; \
 		esac; \
 	done

index 100aa181c6..3a4455ff82 100644
@@ -178,7 +178,7 @@ int main(int argc, char **argv)
 			perror(__FILE__ ": malloc");
 			return 2;
 		}
-		sprintf(relbasedir, "%s/../..", argv[0]);
+		sprintf(relbasedir, "%s/..", argv[0]);
 		absbasedir = realpath(relbasedir, NULL);
 	} else {
 		basename = progpath;

2.13.2

Since $(HOST_DIR)/usr/{bin,sbin} are now symlinks to $(HOST_DIR)/{bin,sbin}, it makes no sense to check them - they are already covered.

---
 support/scripts/check-host-rpath | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

index adabfdf53f..2846d5eb51 100755
@@ -1,6 +1,6 @@
 #!/usr/bin/env bash
 
-# This script scans $(HOST_DIR)/{,usr/}{bin,sbin} for all ELF files, and checks
+# This script scans $(HOST_DIR)/{bin,sbin} for all ELF files, and checks
 # they have an RPATH to $(HOST_DIR)/{,usr/}lib if they need libraries from
 # there.
 

@@ -26,7 +26,7 @@ main() {
             printf "*** ERROR: package %s installs executables without proper RPATH:\n" "${pkg}"
         fi
         printf "***   %s\n" "${file}"
-    done < <( find "${hostdir}"/{,usr/}{bin,sbin} -type f -exec file {} + 2>/dev/null \
+    done < <( find "${hostdir}"/{bin,sbin} -type f -exec file {} + 2>/dev/null \
               |sed -r -e '/^([^:]+):.*\<ELF\>.*\<executable\>.*/!d'                \
                       -e 's//\1/'                                                  \
             )

2.13.2

Now $(HOST_DIR)/usr is a symlink to $(HOST_DIR), it makes no sense to still have it in BR_PATH.

---
 Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

index c728364ab6..257caca21b 100644
@@ -440,7 +440,7 @@ TAR_OPTIONS = $(call qstrip,$(BR2_TAR_OPTIONS)) -xf
 HOST_DIR := $(call qstrip,$(BR2_HOST_DIR))
 
 # Quotes are needed for spaces and all in the original PATH content.
-BR_PATH = "$(HOST_DIR)/bin:$(HOST_DIR)/sbin:$(HOST_DIR)/usr/bin:$(HOST_DIR)/usr/sbin:$(PATH)"
+BR_PATH = "$(HOST_DIR)/bin:$(HOST_DIR)/sbin:$(PATH)"
 
 # Location of a file giving a big fat warning that output/target
 # should not be used as the root filesystem.

2.13.2

Now $(HOST_DIR)/lib and $(HOST_DIR)/usr/lib are the same directory, it doesn't make sense to pass both to LDFLAGS.

Also use $(HOST_DIR)/lib instead of $(HOST_DIR)/usr/lib for the RPATH.

---
 package/Makefile.in | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

index 6b88ac4feb..25f1997fe6 100644
@@ -220,7 +220,7 @@ HOST_CPPFLAGS  = -I$(HOST_DIR)/usr/include
 HOST_CFLAGS   ?= -O2
 HOST_CFLAGS   += $(HOST_CPPFLAGS)
 HOST_CXXFLAGS += $(HOST_CFLAGS)
-HOST_LDFLAGS  += -L$(HOST_DIR)/lib -L$(HOST_DIR)/usr/lib -Wl,-rpath,$(HOST_DIR)/usr/lib
+HOST_LDFLAGS  += -L$(HOST_DIR)/lib -Wl,-rpath,$(HOST_DIR)/lib
 
 # host-intltool should be executed with the system perl, so we save
 # the path to the system perl, before a host-perl built by Buildroot

2.13.2

Arnout, all,

[...]

For the whole series:

Regards,

Samuel