Per-package host/target directory support

As we are going to move to per-package SDK, the location of CCACHE and therefore the definitions of HOSTCC and HOSTCXX need to be evaluated at the time of use and not at the time of assignment. Indeed, the value of HOST_DIR changes from one package to the other.

Therefore, we need to change from := to =.

In addition, while doing A := $(something) $(A) is possible, doing A = $(something) $(A) is not legal. So, instead of defining HOSTCC in terms of the current HOSTCC variable, we re-use HOSTCC_NOCCACHE instead.

Signed-off-by Thomas Petazzoni <[email protected]>

---
Changes since v2:
- New patch

---
 Makefile | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

Makefile +3 -3

index 94adff3406..2819d44124 100644
@@ -471,11 +471,11 @@ BR_PATH = "$(HOST_DIR)/bin:$(HOST_DIR)/sbin:$(PATH)"
 TARGET_DIR_WARNING_FILE = $(BASE_TARGET_DIR)/THIS_IS_NOT_YOUR_ROOT_FILESYSTEM
 
 ifeq ($(BR2_CCACHE),y)
-CCACHE := $(HOST_DIR)/bin/ccache
+CCACHE = $(HOST_DIR)/bin/ccache
 BR_CACHE_DIR ?= $(call qstrip,$(BR2_CCACHE_DIR))
 export BR_CACHE_DIR
-HOSTCC := $(CCACHE) $(HOSTCC)
-HOSTCXX := $(CCACHE) $(HOSTCXX)
+HOSTCC = $(CCACHE) $(HOSTCC_NOCCACHE)
+HOSTCXX = $(CCACHE) $(HOSTCXX_NOCCACHE)
 else
 export BR_NO_CCACHE
 endif

Signature · 1 line

2.19.1

Inside the check_elf_has_rpath(), we check if the host binary has a correct RPATH, which should be either an absolute path to $(HOST_DIR)/lib, or a relative path using $ORIGIN. Those two conditions are checked in a single statements, but as we are going to add a third condition, let's split this up a bit:

- If we have a RPATH to $(HOST_DIR)/lib -> we're good, return 0
- If we have a RPATH to $ORIGIN/../lib -> we're good, return 0
- Otherwise, we will exit the loop, and return 1

Signed-off-by Thomas Petazzoni <[email protected]>

---
 support/scripts/check-host-rpath | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

support/scripts/check-host-rpath +2 -1

index 169628decb..6c5767da05 100755
@@ -63,7 +63,8 @@ check_elf_has_rpath() {
         for dir in ${rpath//:/ }; do
             # Remove duplicate and trailing '/' for proper match
             dir="$( sed -r -e 's:/+:/:g; s:/$::;' <<<"${dir}" )"
-            [ "${dir}" = "${hostdir}/lib" -o "${dir}" = "\$ORIGIN/../lib" ] && return 0
+            [ "${dir}" = "${hostdir}/lib" ] && return 0
+            [ "${dir}" = "\$ORIGIN/../lib" ] && return 0
         done
     done < <( readelf -d "${file}"                                              \
               |sed -r -e '/.* \(R(UN)?PATH\) +Library r(un)?path: \[(.+)\]$/!d' \

Signature · 1 line

2.19.1

In the current code, the creation of the main output folders (BUILD_DIR, STAGING_DIR, HOST_DIR, TARGET_DIR, etc.) is done by a global "dirs" target. While this works fine in the current situation, it doesn't work well in a context where per-package host and target directories are used.

For example, with the current code and per-package host directories, the output/staging symbolic link ends up being created as a link to the per-package package sysroot directory of the first package being built, instead of the global sysroot.

This commit reworks the creation of those directories by having the package/pkg-generic.mk code ensure that the build directory, target directory, host directory, staging directory and binaries directory exist before they are needed.

Two new targets, host-finalize and staging-finalize are added in the main Makefile to create the compatibility symlinks for host and staging directories. They will be extended later with additional logic for per-package directories.

Thanks to those changes, the global "dirs" target is entirely removed.

Signed-off-by Thomas Petazzoni <[email protected]>

---
 Makefile               | 21 +++++++++------------
 fs/common.mk           |  1 +
 package/pkg-generic.mk |  7 ++++++-
 3 files changed, 16 insertions(+), 13 deletions(-)

Makefile +0 -9

index 2819d44124..e675ac26aa 100644
@@ -572,10 +572,6 @@ $(foreach pkg,$(call UPPERCASE,$(PACKAGES)),\
 
 endif
 
-.PHONY: dirs
-dirs: $(BUILD_DIR) $(STAGING_DIR) $(BASE_TARGET_DIR) \
-	$(HOST_DIR) $(HOST_DIR_SYMLINK) $(BINARIES_DIR)
-
 $(BUILD_DIR)/buildroot-config/auto.conf: $(BR2_CONFIG)
 	$(MAKE1) $(EXTRAMAKEARGS) HOSTCC="$(HOSTCC_NOCCACHE)" HOSTCXX="$(HOSTCXX_NOCCACHE)" syncconfig
 

@@ -605,11 +601,6 @@ sdk: prepare-sdk $(BR2_TAR_HOST_DEPENDENCY)
 		--transform='s#^\.#$(BR2_SDK_PREFIX)#' \
 		-C $(HOST_DIR) "."
 
-# Populating the staging with the base directories is handled by the skeleton package
-$(STAGING_DIR):
-	@mkdir -p $(STAGING_DIR)
-	@ln -snf $(STAGING_DIR) $(BASE_DIR)/staging
-
 RSYNC_VCS_EXCLUSIONS = \
 	--exclude .svn --exclude .git --exclude .hg --exclude .bzr \
 	--exclude CVS

@@ -710,8 +701,14 @@ $(TARGETS_ROOTFS): target-finalize
 # Avoid the rootfs name leaking down the dependency chain
 target-finalize: ROOTFS=

Makefile +2 -2

 	@$(call MESSAGE,"Finalizing target directory")
 	# Check files that are touched by more than one package
 	./support/scripts/check-uniq-files -t target $(BUILD_DIR)/packages-file-list.txt

@@ -782,7 +779,7 @@ endif
 	touch $(TARGET_DIR)/usr
 
 .PHONY: target-post-image
-target-post-image: $(TARGETS_ROOTFS) target-finalize
+target-post-image: $(TARGETS_ROOTFS) target-finalize staging-finalize
 	@rm -f $(ROOTFS_COMMON_TAR)
 	@$(foreach s, $(call qstrip,$(BR2_ROOTFS_POST_IMAGE_SCRIPT)), \
 		$(call MESSAGE,"Executing post-image script $(s)"); \

@@ -811,7 +808,7 @@ legal-info-prepare: $(LEGAL_INFO_DIR)
 	@cp $(BR2_CONFIG) $(LEGAL_INFO_DIR)/buildroot.config
 
 .PHONY: legal-info
-legal-info: dirs legal-info-clean legal-info-prepare $(foreach p,$(PACKAGES),$(p)-all-legal-info) \
+legal-info: legal-info-clean legal-info-prepare $(foreach p,$(PACKAGES),$(p)-all-legal-info) \
 		$(REDIST_SOURCES_DIR_TARGET) $(REDIST_SOURCES_DIR_HOST)
 	@cat support/legal-info/README.header >>$(LEGAL_REPORT)
 	@if [ -r $(LEGAL_WARNINGS) ]; then \

fs/common.mk +1 -0

index 2a5a202a89..96658428ba 100644
@@ -145,6 +145,7 @@ $$(BINARIES_DIR)/rootfs.$(1): ROOTFS=$(2)
 $$(BINARIES_DIR)/rootfs.$(1): FAKEROOT_SCRIPT=$$(ROOTFS_$(2)_DIR)/fakeroot
 $$(BINARIES_DIR)/rootfs.$(1): $$(ROOTFS_$(2)_DEPENDENCIES)
 	@$$(call MESSAGE,"Generating root filesystem image rootfs.$(1)")
+	mkdir -p $$(@D)
 	rm -rf $$(ROOTFS_$(2)_DIR)
 	mkdir -p $$(ROOTFS_$(2)_DIR)
 	echo '#!/bin/sh' > $$(FAKEROOT_SCRIPT)

package/pkg-generic.mk +6 -1

index f34f46afc8..309fd8cd48 100644
@@ -173,6 +173,7 @@ $(BUILD_DIR)/%/.stamp_extracted:
 $(BUILD_DIR)/%/.stamp_rsynced:
 	@$(call step_start,rsync)
 	@$(call MESSAGE,"Syncing from source dir $(SRCDIR)")
+	@mkdir -p $(@D)
 	$(foreach hook,$($(PKG)_PRE_RSYNC_HOOKS),$(call $(hook))$(sep))
 	@test -d $(SRCDIR) || (echo "ERROR: $(SRCDIR) does not exist" ; exit 1)
 	rsync -au --chmod=u=rwX,go=rX $(RSYNC_VCS_EXCLUSIONS) $($(PKG)_OVERRIDE_SRCDIR_RSYNC_EXCLUSIONS) $(call qstrip,$(SRCDIR))/ $(@D)

@@ -238,6 +239,7 @@ $(BUILD_DIR)/%/.stamp_built::
 $(BUILD_DIR)/%/.stamp_host_installed:
 	@$(call step_start,install-host)
 	@$(call MESSAGE,"Installing to host directory")
+	@mkdir -p $(HOST_DIR)
 	$(foreach hook,$($(PKG)_PRE_INSTALL_HOOKS),$(call $(hook))$(sep))
 	+$($(PKG)_INSTALL_CMDS)
 	$(foreach hook,$($(PKG)_POST_INSTALL_HOOKS),$(call $(hook))$(sep))

@@ -267,6 +269,7 @@ $(BUILD_DIR)/%/.stamp_host_installed:
 $(BUILD_DIR)/%/.stamp_staging_installed:
 	@$(call step_start,install-staging)
 	@$(call MESSAGE,"Installing to staging directory")
+	@mkdir -p $(STAGING_DIR)
 	$(foreach hook,$($(PKG)_PRE_INSTALL_STAGING_HOOKS),$(call $(hook))$(sep))
 	+$($(PKG)_INSTALL_STAGING_CMDS)
 	$(foreach hook,$($(PKG)_POST_INSTALL_STAGING_HOOKS),$(call $(hook))$(sep))

@@ -298,6 +301,7 @@ $(BUILD_DIR)/%/.stamp_staging_installed:
 # Install to images dir
 $(BUILD_DIR)/%/.stamp_images_installed:
 	@$(call step_start,install-image)
+	@mkdir -p $(BINARIES_DIR)
 	$(foreach hook,$($(PKG)_PRE_INSTALL_IMAGES_HOOKS),$(call $(hook))$(sep))
 	@$(call MESSAGE,"Installing to images directory")
 	+$($(PKG)_INSTALL_IMAGES_CMDS)

@@ -309,6 +313,7 @@ $(BUILD_DIR)/%/.stamp_images_installed:
 $(BUILD_DIR)/%/.stamp_target_installed:
 	@$(call step_start,install-target)
 	@$(call MESSAGE,"Installing to target")
+	@mkdir -p $(TARGET_DIR)
 	$(foreach hook,$($(PKG)_PRE_INSTALL_TARGET_HOOKS),$(call $(hook))$(sep))
 	+$($(PKG)_INSTALL_TARGET_CMDS)
 	$(if $(BR2_INIT_SYSTEMD),\

@@ -735,7 +740,7 @@ $$($(2)_TARGET_BUILD):	$$($(2)_TARGET_CONFIGURE)
 $(1)-configure:			$$($(2)_TARGET_CONFIGURE)
 $$($(2)_TARGET_CONFIGURE):	| $$($(2)_FINAL_DEPENDENCIES)
 
-$$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dirs prepare
+$$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | prepare
 $$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dependencies
 
 ifeq ($$($(2)_OVERRIDE_SRCDIR),)

Signature · 1 line

2.19.1

I don't think we do need anything special to create HOST_DIR, STAGING_DIR and TARGET_DIR:

- host-skeleton is the first package to run, and that one does ensure the existence of $(HOST_DIR) by the mere act of creating it (which it cirrently does not do, but should)

- skeleton is the first package to install in target/ and staging, and currently the skeleton-custom and skeleton-init-common should both create target/ and staging/

Quoted 4 lines

Makefile +4 -0

+host-finalize: $(HOST_DIR_SYMLINK)
+
+.PHONY: staging-finalize
+staging-finalize:

Makefile $(TARGETS_ROOTFS): target-finalize · line 711

Quoted 5 lines

Makefile +3 -1

+	@ln -snf $(STAGING_DIR) $(BASE_DIR)/staging
+
 .PHONY: target-finalize
-target-finalize: $(PACKAGES)
+target-finalize: $(PACKAGES) host-finalize  

Shouldn't we ensure that host-finalize be done before staging-finalize too?

We would also need to ensure that packages are all installed before running staging-finalize.

So,=maybe we need to add:

    staging-finalize: $(PACKAGES) host-finalize

In the current code, the creation of the main output folders (BUILD_DIR, STAGING_DIR, HOST_DIR, TARGET_DIR, etc.) is done by a global "dirs" target. While this works fine in the current situation, it doesn't work well in a context where per-package host and target directories are used.

For example, with the current code and per-package host directories, the output/staging symbolic link ends up being created as a link to the per-package package sysroot directory of the first package being built, instead of the global sysroot.

This commit reworks the creation of those directories by having the package/pkg-generic.mk code ensure that the build directory, target directory, host directory, staging directory and binaries directory exist before they are needed.

Two new targets, host-finalize and staging-finalize are added in the main Makefile to create the compatibility symlinks for host and staging directories. They will be extended later with additional logic for per-package directories.

Thanks to those changes, the global "dirs" target is entirely removed.

Signed-off-by Thomas Petazzoni <[email protected]>

---
 Makefile               | 21 +++++++++------------
 fs/common.mk           |  1 +
 package/pkg-generic.mk |  7 ++++++-
 3 files changed, 16 insertions(+), 13 deletions(-)

Makefile +0 -4

index 2819d44124..e675ac26aa 100644
@@ -572,10 +572,6 @@ $(foreach pkg,$(call UPPERCASE,$(PACKAGES)),\
 
 endif
 
-.PHONY: dirs
-dirs: $(BUILD_DIR) $(STAGING_DIR) $(BASE_TARGET_DIR) \
-	$(HOST_DIR) $(HOST_DIR_SYMLINK) $(BINARIES_DIR)
-
 $(BUILD_DIR)/buildroot-config/auto.conf: $(BR2_CONFIG)
 	$(MAKE1) $(EXTRAMAKEARGS) HOSTCC="$(HOSTCC_NOCCACHE)"

HOSTCXX="$(HOSTCXX_NOCCACHE)" syncconfig

Makefile +0 -1

@@ -605,11 +601,6 @@ sdk: prepare-sdk $(BR2_TAR_HOST_DEPENDENCY)
 		--transform='s#^\.#$(BR2_SDK_PREFIX)#' \
 		-C $(HOST_DIR) "."
 
-# Populating the staging with the base directories is handled by the

skeleton package
-$(STAGING_DIR):

RSYNC_VCS_EXCLUSIONS = \

Makefile +7 -1

 	--exclude .svn --exclude .git --exclude .hg --exclude .bzr \
 	--exclude CVS

@@ -710,8 +701,14 @@ $(TARGETS_ROOTFS): target-finalize
 # Avoid the rootfs name leaking down the dependency chain
 target-finalize: ROOTFS=
 
+host-finalize: $(HOST_DIR_SYMLINK)
+
+.PHONY: staging-finalize
+staging-finalize:
+	@ln -snf $(STAGING_DIR) $(BASE_DIR)/staging
+
 .PHONY: target-finalize
-target-finalize: $(PACKAGES)
+target-finalize: $(PACKAGES) host-finalize
 	@$(call MESSAGE,"Finalizing target directory")
 	# Check files that are touched by more than one package
 	./support/scripts/check-uniq-files -t target

$(BUILD_DIR)/packages-file-list.txt

Makefile +2 -2

@@ -782,7 +779,7 @@ endif
 	touch $(TARGET_DIR)/usr
 
 .PHONY: target-post-image
-target-post-image: $(TARGETS_ROOTFS) target-finalize
+target-post-image: $(TARGETS_ROOTFS) target-finalize staging-finalize
 	@rm -f $(ROOTFS_COMMON_TAR)
 	@$(foreach s, $(call qstrip,$(BR2_ROOTFS_POST_IMAGE_SCRIPT)), \
 		$(call MESSAGE,"Executing post-image script $(s)"); \

@@ -811,7 +808,7 @@ legal-info-prepare: $(LEGAL_INFO_DIR)
 	@cp $(BR2_CONFIG) $(LEGAL_INFO_DIR)/buildroot.config
 
 .PHONY: legal-info
-legal-info: dirs legal-info-clean legal-info-prepare $(foreach
 p,$(PACKAGES),$(p)-all-legal-info) \
+legal-info: legal-info-clean legal-info-prepare $(foreach
 p,$(PACKAGES),$(p)-all-legal-info) \
 		$(REDIST_SOURCES_DIR_TARGET) $(REDIST_SOURCES_DIR_HOST)

Makefile +0 -0

 	@cat support/legal-info/README.header >>$(LEGAL_REPORT)
 	@if [ -r $(LEGAL_WARNINGS) ]; then \

fs/common.mk +1 -0

index 2a5a202a89..96658428ba 100644
@@ -145,6 +145,7 @@ $$(BINARIES_DIR)/rootfs.$(1): ROOTFS=$(2)
 $$(BINARIES_DIR)/rootfs.$(1): FAKEROOT_SCRIPT=$$(ROOTFS_$(2)_DIR)/fakeroot
 $$(BINARIES_DIR)/rootfs.$(1): $$(ROOTFS_$(2)_DEPENDENCIES)
 	@$$(call MESSAGE,"Generating root filesystem image rootfs.$(1)")
+	mkdir -p $$(@D)
 	rm -rf $$(ROOTFS_$(2)_DIR)
 	mkdir -p $$(ROOTFS_$(2)_DIR)
 	echo '#!/bin/sh' > $$(FAKEROOT_SCRIPT)

package/pkg-generic.mk +1 -0

index f34f46afc8..309fd8cd48 100644
@@ -173,6 +173,7 @@ $(BUILD_DIR)/%/.stamp_extracted:
 $(BUILD_DIR)/%/.stamp_rsynced:
 	@$(call step_start,rsync)
 	@$(call MESSAGE,"Syncing from source dir $(SRCDIR)")
+	@mkdir -p $(@D)
 	$(foreach hook,$($(PKG)_PRE_RSYNC_HOOKS),$(call $(hook))$(sep))
 	@test -d $(SRCDIR) || (echo "ERROR: $(SRCDIR) does not exist" ; exit 1)
 	rsync -au --chmod=u=rwX,go=rX $(RSYNC_VCS_EXCLUSIONS)

$($(PKG)_OVERRIDE_SRCDIR_RSYNC_EXCLUSIONS) $(call qstrip,$(SRCDIR))/ $(@D)

package/pkg-generic.mk +2 -0

@@ -238,6 +239,7 @@ $(BUILD_DIR)/%/.stamp_built::
 $(BUILD_DIR)/%/.stamp_host_installed:
 	@$(call step_start,install-host)
 	@$(call MESSAGE,"Installing to host directory")
+	@mkdir -p $(HOST_DIR)
 	$(foreach hook,$($(PKG)_PRE_INSTALL_HOOKS),$(call $(hook))$(sep))
 	+$($(PKG)_INSTALL_CMDS)
 	$(foreach hook,$($(PKG)_POST_INSTALL_HOOKS),$(call $(hook))$(sep))

@@ -267,6 +269,7 @@ $(BUILD_DIR)/%/.stamp_host_installed:
 $(BUILD_DIR)/%/.stamp_staging_installed:
 	@$(call step_start,install-staging)
 	@$(call MESSAGE,"Installing to staging directory")
+	@mkdir -p $(STAGING_DIR)
 	$(foreach hook,$($(PKG)_PRE_INSTALL_STAGING_HOOKS),$(call $(hook))$(sep))
 	+$($(PKG)_INSTALL_STAGING_CMDS)
 	$(foreach hook,$($(PKG)_POST_INSTALL_STAGING_HOOKS),$(call

$(hook))$(sep))

package/pkg-generic.mk +3 -2

@@ -298,6 +301,7 @@ $(BUILD_DIR)/%/.stamp_staging_installed:
 # Install to images dir
 $(BUILD_DIR)/%/.stamp_images_installed:
 	@$(call step_start,install-image)
+	@mkdir -p $(BINARIES_DIR)
 	$(foreach hook,$($(PKG)_PRE_INSTALL_IMAGES_HOOKS),$(call $(hook))$(sep))
 	@$(call MESSAGE,"Installing to images directory")
 	+$($(PKG)_INSTALL_IMAGES_CMDS)

@@ -309,6 +313,7 @@ $(BUILD_DIR)/%/.stamp_images_installed:
 $(BUILD_DIR)/%/.stamp_target_installed:
 	@$(call step_start,install-target)
 	@$(call MESSAGE,"Installing to target")
+	@mkdir -p $(TARGET_DIR)
 	$(foreach hook,$($(PKG)_PRE_INSTALL_TARGET_HOOKS),$(call $(hook))$(sep))
 	+$($(PKG)_INSTALL_TARGET_CMDS)
 	$(if $(BR2_INIT_SYSTEMD),\

@@ -735,7 +740,7 @@ $$($(2)_TARGET_BUILD):	$$($(2)_TARGET_CONFIGURE)
 $(1)-configure:			$$($(2)_TARGET_CONFIGURE)
 $$($(2)_TARGET_CONFIGURE):	| $$($(2)_FINAL_DEPENDENCIES)
 
-$$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dirs prepare
+$$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | prepare
 $$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dependencies
 
 ifeq ($$($(2)_OVERRIDE_SRCDIR),)
--

2.19.1

_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

Makefile sdk: prepare-sdk $(BR2_TAR_HOST_DEPENDENCY) · line 612

Quoted 6 lines

-# Populating the staging with the base directories is handled by the skeleton package
-$(STAGING_DIR):

Makefile +0 -3

-	@mkdir -p $(STAGING_DIR)
-	@ln -snf $(STAGING_DIR) $(BASE_DIR)/staging
-  

With per package disabled, the staging directory never gets created early enough for packages to be able to reference content. Specifically I noticed docker couldn't find -lsystemd. I'm assuming it's because of the new target dependency on target-post-image?

Quoted 11 lines · Matthew Weber

Makefile sdk: prepare-sdk $(BR2_TAR_HOST_DEPENDENCY) · line 612

Quoted 6 lines

-# Populating the staging with the base directories is handled by the skeleton package
-$(STAGING_DIR):

Makefile +0 -3

-	@mkdir -p $(STAGING_DIR)
-	@ln -snf $(STAGING_DIR) $(BASE_DIR)/staging
-    

With per package disabled, the staging directory never gets created early enough for packages to be able to reference content. Specifically I noticed docker couldn't find -lsystemd. I'm assuming it's because of the new target dependency on target-post-image?

The STAGING_DIR itself is definitely created at the beginning of the build, at the first package installing something into it.

However, with this patch, the $(BASE_DIR)/staging symlink indeed gets created at the end of the build rather than at the beginning of the build. However, this symlink is only a compatibility one. The code should use $(STAGING_DIR) everywhere, which should not depend on this "staging" compatibility symlink.

I've started a build here to reproduce the issue, but in general it would be nice when you report issues to include an example defconfig that exhibits the problem, as well as the build log. It would be a lot easier to understand the specifics of the problem, and be able to reproduce it.

Hello,

Quoted 23 lines · Thomas Petazzoni

Quoted 11 lines · Matthew Weber

Quoted 6 lines

-# Populating the staging with the base directories is handled by the skeleton package
-$(STAGING_DIR):
- @mkdir -p $(STAGING_DIR)
- @ln -snf $(STAGING_DIR) $(BASE_DIR)/staging
-

With per package disabled, the staging directory never gets created early enough for packages to be able to reference content. Specifically I noticed docker couldn't find -lsystemd. I'm assuming it's because of the new target dependency on target-post-image?

The STAGING_DIR itself is definitely created at the beginning of the build, at the first package installing something into it.

However, with this patch, the $(BASE_DIR)/staging symlink indeed gets created at the end of the build rather than at the beginning of the build. However, this symlink is only a compatibility one. The code should use $(STAGING_DIR) everywhere, which should not depend on this "staging" compatibility symlink.

Quoted 5 lines

I've started a build here to reproduce the issue, but in general it would be nice when you report issues to include an example defconfig that exhibits the problem, as well as the build log. It would be a lot easier to understand the specifics of the problem, and be able to reproduce it.

FWIW, the following defconfig:

BR2_arm=y
BR2_cortex_a8=y
BR2_TOOLCHAIN_EXTERNAL=y
BR2_TOOLCHAIN_EXTERNAL_LINARO_ARM=y
BR2_INIT_SYSTEMD=y
BR2_SYSTEM_BIN_SH_NONE=y
# BR2_PACKAGE_BUSYBOX is not set
BR2_PACKAGE_DOCKER_ENGINE=
# BR2_TARGET_ROOTFS_TAR is not set

BR2_aarch64=y
BR2_TOOLCHAIN_EXTERNAL=y
BR2_INIT_SYSTEMD=y
BR2_PACKAGE_DOCKER_ENGINE=y
BR2_PACKAGE_DOCKER_ENGINE_STATIC_CLIENT=y
# BR2_TARGET_ROOTFS_TAR is not set

Looking at it a bit further, the "go build" linking fails when the BR2_PACKAGE_DOCKER_ENGINE_STATIC_CLIENT is set. Since there is a pending patch for the docker package, I built with the current "next" that doesn't have [1] and also with it applied. I observed that [1] fixed the static client build. When they broke out the client from the daemon the linking approach must have changed.

In a follow-up commit, we will make the .NOTPARALLEL statement conditional on a Config.in option, so we need to move it further down.

Signed-off-by Thomas Petazzoni <[email protected]>

---
 Makefile | 32 ++++++++++++++++----------------
 1 file changed, 16 insertions(+), 16 deletions(-)

Makefile +16 -16

index e675ac26aa..24c803872d 100644
@@ -105,22 +105,6 @@ ifneq ($(firstword $(sort $(RUNNING_MAKE_VERSION) $(MIN_MAKE_VERSION))),$(MIN_MA
 $(error You have make '$(RUNNING_MAKE_VERSION)' installed. GNU make >= $(MIN_MAKE_VERSION) is required)
 endif
 
-# Parallel execution of this Makefile is disabled because it changes
-# the packages building order, that can be a problem for two reasons:
-# - If a package has an unspecified optional dependency and that
-#   dependency is present when the package is built, it is used,
-#   otherwise it isn't (but compilation happily proceeds) so the end
-#   result will differ if the order is swapped due to parallel
-#   building.
-# - Also changing the building order can be a problem if two packages
-#   manipulate the same file in the target directory.
-#
-# Taking into account the above considerations, if you still want to execute
-# this top-level Makefile in parallel comment the ".NOTPARALLEL" line and
-# use the -j<jobs> option when building, e.g:
-#      make -j$((`getconf _NPROCESSORS_ONLN`+1))
-.NOTPARALLEL:
-
 # absolute path
 TOPDIR := $(CURDIR)
 CONFIG_CONFIG_IN = Config.in

@@ -246,6 +230,22 @@ ifeq ($(filter $(noconfig_targets),$(MAKECMDGOALS)),)
 -include $(BR2_CONFIG)
 endif
 
+# Parallel execution of this Makefile is disabled because it changes
+# the packages building order, that can be a problem for two reasons:
+# - If a package has an unspecified optional dependency and that
+#   dependency is present when the package is built, it is used,
+#   otherwise it isn't (but compilation happily proceeds) so the end
+#   result will differ if the order is swapped due to parallel
+#   building.
+# - Also changing the building order can be a problem if two packages
+#   manipulate the same file in the target directory.
+#
+# Taking into account the above considerations, if you still want to execute
+# this top-level Makefile in parallel comment the ".NOTPARALLEL" line and
+# use the -j<jobs> option when building, e.g:
+#      make -j$((`getconf _NPROCESSORS_ONLN`+1))
+.NOTPARALLEL:
+
 # timezone and locale may affect build output
 ifeq ($(BR2_REPRODUCIBLE),y)
 export TZ = UTC

Signature · 1 line

2.19.1

Ultimately, I don't see why we do need a config option to turn top-level parallel build on/off. It will ultimately be the user's choice to do so when calling 'make -jN' no?

However, what we do need for now, is a config option that enabled per-package directories (PPD, formerly known as PPS, per-package staging). When that is turned off, we must not allow top-level parallel build.

I was thinking that we would not need that option either, because we would enable PPD when we detect that the user is doing top-level parllel build, by inspecting $(MAKEFLAGS) to see if they contained -j.

However, make is our fiend here, because it does not include -j in the MAKFLAGS variable, but the one that is exported:

    $ cat Makefile
    $(info var MAKEFLAGS='$(MAKEFLAGS)')
    all:
        @echo "env MAKEFLAGS='$${MAKEFLAGS}'"

    $ make -j1
    var MAKEFLAGS=''
    env MAKEFLAGS=''

    $ make -j3
    var MAKEFLAGS=''
    env MAKEFLAGS=' -j --jobserver-fds=3,4'

    $ make -j
    var MAKEFLAGS=''
    env MAKEFLAGS=' -j'

And there is no other variable in make that specifies that either...

So yes, this means we have no way from the Makefile to detect whether we're parallel or not. Sigh. :-(

So we do need that option for PPD.

Quoted 1 line

Signed-off-by Thomas Petazzoni <[email protected]>

Reviewed-by "Yann E. MORIN" <[email protected]>

package/pkg-generic.mk +17 -0

+# $1: deps
+ifeq ($(BR2_PER_PACKAGE_FOLDERS),y)
+define prepare-per-package-folder
+	mkdir -p $(HOST_DIR) $(TARGET_DIR)
+	$(foreach pkg,$(1),\
+		rsync -a --link-dest=$(PER_PACKAGE_DIR)/$(pkg)/host/ \
+		$(PER_PACKAGE_DIR)/$(pkg)/host/ \
+		$(HOST_DIR)$(sep))
+	$(foreach pkg,$(1),\
+		rsync -a --link-dest=$(PER_PACKAGE_DIR)/$(pkg)/target/ \
+		$(PER_PACKAGE_DIR)/$(pkg)/target/ \
+		$(TARGET_DIR)$(sep))
+endef
+endif
+
  ################################################################################
  # Implicit targets -- produce a stamp file for each step of a package build
  ################################################################################

@@ -133,6 +148,7 @@ endif
  # Retrieve the archive
  $(BUILD_DIR)/%/.stamp_downloaded:
  	@$(call step_start,download)
+	$(call prepare-per-package-folder,$($(PKG)_FINAL_DOWNLOAD_DEPENDENCIES))
  	$(foreach hook,$($(PKG)_PRE_DOWNLOAD_HOOKS),$(call $(hook))$(sep))
  # Only show the download message if it isn't already downloaded
  	$(Q)for p in $($(PKG)_ALL_DOWNLOADS); do \

@@ -159,6 +175,7 @@ $(BUILD_DIR)/%/.stamp_actual_downloaded:
  $(BUILD_DIR)/%/.stamp_extracted:
  	@$(call step_start,extract)
  	@$(call MESSAGE,"Extracting")
+	$(call prepare-per-package-folder,$($(PKG)_FINAL_EXTRACT_DEPENDENCIES))
  	$(foreach hook,$($(PKG)_PRE_EXTRACT_HOOKS),$(call $(hook))$(sep))
  	$(Q)mkdir -p $(@D)
  	$($(PKG)_EXTRACT_CMDS)

This is much clearer to me than v3, however there is a problem with kconfig packages. Unfortunately, the kconfig infrastructure is running the 'make xxx_defconfig' step before the configure-step is "officially" started. This is due to the kconfig_fixup target being a dependency of -configure. This in turn requires availability of the toolchain and other prerequisites that are part of _FINAL_DEPENDENCIES earlier as prepared in your above patch.

I made a patch which moves the _FINAL_DEPENDENCIES preparation to an additional .stamp-configure-prepare step just before .stamp_configured. That works but is not particularly beautiful.

I guess a more proper solution would be to somehow move the kconfig_fixup code into the configure-step. Maybe use the pre-configure-hook, any suggestions?

Hello,

Yann was already in Cc, I've added Thomas De Schampheleire as well. Yann, Thomas, there is some kconfig related discussion below, we need you :-)

Quoted 11 lines · Andreas Naumann

This is much clearer to me than v3, however there is a problem with kconfig packages. Unfortunately, the kconfig infrastructure is running the 'make xxx_defconfig' step before the configure-step is "officially" started. This is due to the kconfig_fixup target being a dependency of -configure. This in turn requires availability of the toolchain and other prerequisites that are part of _FINAL_DEPENDENCIES earlier as prepared in your above patch.

I made a patch which moves the _FINAL_DEPENDENCIES preparation to an additional .stamp-configure-prepare step just before .stamp_configured. That works but is not particularly beautiful.

That is not too bad actually. Semantically speaking, preparing the per-package folders is not really part of the configuration step. It could be logical to do it in a "prepare" step.

The only thing that I dislike a bit with this is that it would no longer be consistent with what we do for download and extract
dependencies: we prepare the per-package folders with download and extract dependencies respectively in the download and extract steps. So it would be quite logical to also do the same for the "configuration dependencies" (which we name just "dependencies" in Buildroot).

So, this leaves us with 3 options:

- Keep the dependency preparation within the download, extract and configure step, as proposed in this v4. This will require changing the kconfig logic to prepare the configuration file inside the "configure" step and not as a additional step injected before the "configure" step.

- Keep the dependency preparation within the download and extract steps, and make an exception for the configure step, with a separate "prepare" step that comes before. Not nice in terms of consistency, as explained above.

- Introduce "prepare download", "prepare extract" and "prepare configure" steps that would do the dependency preparation.

Quoted 3 lines

I guess a more proper solution would be to somehow move the kconfig_fixup code into the configure-step. Maybe use the pre-configure-hook, any suggestions?

I don't recall why they need to be done before the configuration step, but I'm pretty sure there is a reason for that.

Yann, Thomas, do you remember ?

On Thu, 15 Nov 2018 17:41:35 +0100, Andreas Naumann wrote:

Quoted 3 lines

I made a patch which moves the _FINAL_DEPENDENCIES preparation to an additional .stamp-configure-prepare step just before .stamp_configured. That works but is not particularly beautiful.

That is not too bad actually. Semantically speaking, preparing the per-package folders is not really part of the configuration step. It could be logical to do it in a "prepare" step.

The only thing that I dislike a bit with this is that it would no longer be consistent with what we do for download and extract
dependencies: we prepare the per-package folders with download and extract dependencies respectively in the download and extract steps. So it would be quite logical to also do the same for the "configuration dependencies" (which we name just "dependencies" in Buildroot).

So, this leaves us with 3 options:

?- Keep the dependency preparation within the download, extract and ? ?configure step, as proposed in this v4. This will require changing ? ?the kconfig logic to prepare the configuration file inside the ? ?"configure" step and not as a additional step injected before the ? ?"configure" step.

?- Keep the dependency preparation within the download and extract ? ?steps, and make an exception for the configure step, with a separate ? ?"prepare" step that comes before. Not nice in terms of consistency, ? ?as explained above.

?- Introduce "prepare download", "prepare extract" and "prepare ? ?configure" steps that would do the dependency preparation.

Quoted 3 lines

I guess a more proper solution would be to somehow move the kconfig_fixup code into the configure-step. Maybe use the pre-configure-hook, any suggestions?

I don't recall why they need to be done before the configuration step, but I'm pretty sure there is a reason for that.

Yann, Thomas, do you remember ?

My mind triggers a big red warning light right now, so let's be careful :-)

The goal is that one can run 'make foo-menuconfig' from a clean tree, without first processing (downloading, building, ...) the dependencies of foo first.
If you put this stuff in the configure step of foo, you are bound byi its dependencies. There may be other reasons too, not sure.

Thomas?, All,

On 2018-11-16 16:22 +0100, Thomas De Schampheleire spake thusly:

Quoted 2 lines · On Fri, Nov 16, 2018, 14:47 Thomas Petazzoni < [1][email protected]…

On Thu, 15 Nov 2018 17:41:35 +0100, Andreas Naumann wrote:

[--SNIP--]

Quoted 7 lines

Quoted 3 lines

I made a patch which moves the _FINAL_DEPENDENCIES preparation to an additional .stamp-configure-prepare step just before .stamp_configured. That works but is not particularly beautiful.

That is not too bad actually. Semantically speaking, preparing the per-package folders is not really part of the configuration step. It could be logical to do it in a "prepare" step.

And IIRC, you alreqady proposed such a step, specifically to be able to do the autopreconf for OOT building (which I am still working on, btw).

Maybe this step can be re-used for various pacakges infras, like the kconfig one, to add preparation steps.

Quoted 27 lines

The only thing that I dislike a bit with this is that it would no longer be consistent with what we do for download and extract
dependencies: we prepare the per-package folders with download and extract dependencies respectively in the download and extract steps. So it would be quite logical to also do the same for the "configuration dependencies" (which we name just "dependencies" in Buildroot).

So, this leaves us with 3 options:

?- Keep the dependency preparation within the download, extract and ? ?configure step, as proposed in this v4. This will require changing ? ?the kconfig logic to prepare the configuration file inside the ? ?"configure" step and not as a additional step injected before the ? ?"configure" step.

?- Keep the dependency preparation within the download and extract ? ?steps, and make an exception for the configure step, with a separate ? ?"prepare" step that comes before. Not nice in terms of consistency, ? ?as explained above.

?- Introduce "prepare download", "prepare extract" and "prepare ? ?configure" steps that would do the dependency preparation.

Quoted 3 lines

I guess a more proper solution would be to somehow move the kconfig_fixup code into the configure-step. Maybe use the pre-configure-hook, any suggestions?

Quoted 6 lines

I don't recall why they need to be done before the configuration step, but I'm pretty sure there is a reason for that.

Yann, Thomas, do you remember ?

My mind triggers a big red warning light right now, so let's be careful :-)

Yes, we got bitten pretty hard when preparing the kconfig infra. But what can we remember from an almost-5-year old work initiated after FOSDEM? ;-)

Quoted 5 lines

The goal is that one can run 'make foo-menuconfig' from a clean tree, without first processing (downloading, building, ...) the dependencies of foo first.
If you put this stuff in the configure step of foo, you are bound byi its dependencies. There may be other reasons too, not sure.

I am all in favour of simplifying it if it can be made simpler by adding an official extra 'prepare' step, that exists for all types of package infras. This we'd have 4 levels of dependencies:

1- download dependencies
2- extract dependencies
3- prepare dependencies
4- configure dependencies

Currently, 1, 2, and 4 are implemented in a generic way and thus available to all types of packages, while 3 is implemented only for kconfig-package, in an ad-hoc way, and used only by the linux kernel to ensure the toolchain is available before its kconfig is called.

Regards,
Yann E. MORIN.

Quoted 10 lines

Yes, we got bitten pretty hard when preparing the kconfig infra. But what can we remember from an almost-5-year old work initiated after FOSDEM? ;-)

Quoted 5 lines

The goal is that one can run 'make foo-menuconfig' from a clean tree, without first processing (downloading, building, ...) the dependencies of foo first.
If you put this stuff in the configure step of foo, you are bound byi its dependencies. There may be other reasons too, not sure.

Quoted 8 lines

I am all in favour of simplifying it if it can be made simpler by adding an official extra 'prepare' step, that exists for all types of package infras. This we'd have 4 levels of dependencies:

1- download dependencies
2- extract dependencies
3- prepare dependencies
4- configure dependencies

I think your mixing two things here. The "prepare" that ThomasP is talking
about is really the PPD creation. The additional step for autoreconf or kconfig is something else - it's a step with commands that are defined by the package (infra).

Hello,

Quoted 19 lines · Arnout Vandecappelle

Quoted 10 lines

Yes, we got bitten pretty hard when preparing the kconfig infra. But what can we remember from an almost-5-year old work initiated after FOSDEM? ;-)

Quoted 5 lines

The goal is that one can run 'make foo-menuconfig' from a clean tree, without first processing (downloading, building, ...) the dependencies of foo first.
If you put this stuff in the configure step of foo, you are bound byi its dependencies. There may be other reasons too, not sure.

Quoted 8 lines

I am all in favour of simplifying it if it can be made simpler by adding an official extra 'prepare' step, that exists for all types of package infras. This we'd have 4 levels of dependencies:

1- download dependencies
2- extract dependencies
3- prepare dependencies
4- configure dependencies

Quoted 4 lines

I think your mixing two things here. The "prepare" that ThomasP is talking
about is really the PPD creation. The additional step for autoreconf or kconfig is something else - it's a step with commands that are defined by the package (infra).

I don't think Yann is mixing things here. Indeed we are using the word "prepare" for two different things: preparing the PPD folder, and "doing some stuff before the configure step".

For the PPD folder, some preparation currently occurs at the download step (for download dependencies), some preparation occurs at the extract step (for the extract dependencies) and some preparation occurs at the configuration step (for the normal dependencies).

The issue with the kconfig infrastructure is that it creates a new type of dependency called "kconfig dependency", which should be ready to do all the kconfig-related activities, but that we want separate from the normal "configure dependencies" because we want to be able to do "make linux-menuconfig" or "make busybox-menuconfig" without having to wait for their dependencies to build.

So what Yann is proposing is that instead of having this "kconfig dependency" stuff be shoe-horned into the dependency chain by pkg-kconfig.mk, we should promote it to a new step in the build steps. By the lack of better naming, Yann named that step "prepare", but it has nothing to do with "preparing PPD".

Indeed, what we would do is:

1. Download step
Prepare PPD with download dependencies
Do the normal download stuff

2. Extract step
Prepare PPD with extract dependencies
Do the normal extract stuff

3. Patch step
Prepare PPD with patch dependencies
Do the normal patch stuff

4. Prepare step
Prepare PPD with the prepare dependencies
Do the prepare stuff (i.e only the kconfig stuff for now)

5. Configure step
Prepare PPD with the normal dependencies
Do the normal configure stuff

Does that clarify Yann's proposal ?

I personally find it OK, even though it's a bit annoying to introduce yet another step just for the sake of pkg-kconfig.

Also: I would not move autoreconf into this prepare step, but keep it as it is today, within the configure step. I don't (today at least) see the point/usefulness of moving the autoreconf into this prepare step.

Best regards,

Thomas

Quoted 9 lines

Quoted 8 lines

I am all in favour of simplifying it if it can be made simpler by adding an official extra 'prepare' step, that exists for all types of package infras. This we'd have 4 levels of dependencies:

1- download dependencies
2- extract dependencies
3- prepare dependencies
4- configure dependencies

Quoted 4 lines

I think your mixing two things here. The "prepare" that ThomasP is talking
about is really the PPD creation. The additional step for autoreconf or kconfig is something else - it's a step with commands that are defined by the package (infra).

I don't think Yann is mixing things here. Indeed we are using the word "prepare" for two different things: preparing the PPD folder, and "doing some stuff before the configure step".

For the PPD folder, some preparation currently occurs at the download step (for download dependencies), some preparation occurs at the extract step (for the extract dependencies) and some preparation occurs at the configuration step (for the normal dependencies).

The issue with the kconfig infrastructure is that it creates a new type of dependency called "kconfig dependency", which should be ready to do all the kconfig-related activities, but that we want separate from the normal "configure dependencies" because we want to be able to do "make linux-menuconfig" or "make busybox-menuconfig" without having to wait for their dependencies to build.

So what Yann is proposing is that instead of having this "kconfig dependency" stuff be shoe-horned into the dependency chain by pkg-kconfig.mk, we should promote it to a new step in the build steps.

By the lack of better naming, Yann named that step "prepare", but it has nothing to do with "preparing PPD".

Indeed, what we would do is:

1. Download step
Prepare PPD with download dependencies
Do the normal download stuff

2. Extract step
Prepare PPD with extract dependencies
Do the normal extract stuff

3. Patch step
Prepare PPD with patch dependencies
Do the normal patch stuff

4. Prepare step
Prepare PPD with the prepare dependencies
Do the prepare stuff (i.e only the kconfig stuff for now)

5. Configure step
Prepare PPD with the normal dependencies
Do the normal configure stuff

Does that clarify Yann's proposal ?

I personally find it OK, even though it's a bit annoying to introduce yet another step just for the sake of pkg-kconfig.

Also: I would not move autoreconf into this prepare step, but keep it as it is today, within the configure step. I don't (today at least) see the point/usefulness of moving the autoreconf into this prepare step.

Thomas, All,

On 2018-11-19 11:48 +0100, Thomas Petazzoni spake thusly:

Quoted 18 lines

Indeed, what we would do is:

1. Download step
Prepare PPD with download dependencies
Do the normal download stuff

2. Extract step
Prepare PPD with extract dependencies
Do the normal extract stuff

3. Patch step
Prepare PPD with patch dependencies
Do the normal patch stuff

4. Prepare step
Prepare PPD with the prepare dependencies
Do the prepare stuff (i.e only the kconfig stuff for now)

Quoted 5 lines

5. Configure step
Prepare PPD with the normal dependencies
Do the normal configure stuff

Does that clarify Yann's proposal ?

Yes, that clarifies it perfectly. Thanks! :-)

Quoted 2 lines

I personally find it OK, even though it's a bit annoying to introduce yet another step just for the sake of pkg-kconfig.

Quoted 3 lines

Also: I would not move autoreconf into this prepare step, but keep it as it is today, within the configure step. I don't (today at least) see the point/usefulness of moving the autoreconf into this prepare step.

For the PPD stuff? Definitely not.

But it will be usefull to run autoreconf in a step that is not the configure step, so we are able to share the source tree between host and target build, as well as stop rsyncing from _OVERRIDE_SRCDIR.

But then, I wonder if that is really the same as the 'prepare' step

above, because autoreconf will also require the same dependencies as the configure step.

Quoted 2 lines

Quoted 2 lines

I personally find it OK, even though it's a bit annoying to introduce yet another step just for the sake of pkg-kconfig.

Me too. And I'm not even sure if it really solves the pkg-kconfig problem. I
have a hard time remembering what all the issues were there.

Perhaps, though, we should have a more general strategy of a per-infra
definition of what the steps are, instead of having a fixed sequence of steps. I.e., have a pattern that should be followed by an infra-specific step to make sure things are done correctly. So, for the kconfig case we would have an extra kconfig_fixup step (we already have the stamp file, but it's not a real step with hooks and the KCONFIG_DEPENDENCIES were strapped on as an afterthought), and for the autoreconf case we would have an extra autoreconf step (instead of a hook).

But the problem is, for pkg-kconfig, we actually need the prepare stuff to be
done in the build directory, not the source directory... So it *still* doesn't match.

Quoted 2 lines

above, because autoreconf will also require the same dependencies as the configure step.

... not because of this. Most dependencies are only really needed in the
configure step, it's only some of them (host-automake etc. obviously, but also host-autoconf-archive and host-pkg-config for the .m4 files) that are needed for the autoreconf. So if it is split, we would need to move some dependencies to _PREPARE_DEPENDENCIES.

Quoted 8 lines

Perhaps, though, we should have a more general strategy of a per-infra
definition of what the steps are, instead of having a fixed sequence of steps. I.e., have a pattern that should be followed by an infra-specific step to make sure things are done correctly. So, for the kconfig case we would have an extra kconfig_fixup step (we already have the stamp file, but it's not a real step with hooks and the KCONFIG_DEPENDENCIES were strapped on as an afterthought), and for the autoreconf case we would have an extra autoreconf step (instead of a hook).

And so those "injected" steps should take care of preparing the per-package folder with whatever dependencies they need ?

This is probably something that can be done to fix the pkg-kconfig stuff without having to create a new "prepare" step in the generic infrastructure.

3. Patch step
Prepare PPD with patch dependencies

Here is a fourth iteration of the per-package SDK and target directory implementation.

Hello,

Here is a fourth iteration of the per-package SDK and target directory implementation.

If you're interested in testing it, you can find the patch series at:

http://git.bootlin.com/users/thomas-petazzoni/buildroot/log/?h=ppsh-v4

thank you for the updated version. I tried to run it with my configs.

For that I needed
a) a generic patch for all kconfig packages, see my answer on the final patch

b) linux-specific patch posted two days ago: [PATCH v2 1/1] linux: Make dtc install step more reliable

On my local machine things seemed to work, however final rootfs creation failed in our CI environment due to broken host- binaries, e.g. mkfs.ubifs not finding some required lib (liblzo2.so.2 in that particular case).
It seems some things with rpath manipulation/checking do not yet work properly. Looking into the binary with ldd I see libs used from the host-machine that probably should come from buildroots host/usr/lib...

	linux-vdso.so.1 =>  (0x00007ffc06fcc000)
	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f760428f000)
	liblzo2.so.2 => not found
	libuuid.so.1 => /lib/x86_64-linux-gnu/libuuid.so.1 (0x00007f760408a000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f7603d81000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f76039b7000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f76044a9000)

Hello Andreas,

Thanks for your feedback!

Quoted 1 line · Andreas Naumann

thank you for the updated version. I tried to run it with my configs.

Thanks a lot for testing!

Quoted 3 lines

For that I needed
a) a generic patch for all kconfig packages, see my answer on the final patch

For this one, we need to continue the discussion with Yann and Thomas De Schampheleire to understand what is the right approach.

Quoted 3 lines

b) linux-specific patch posted two days ago: [PATCH v2 1/1] linux: Make dtc install step more reliable

This one is a package-specific fix, which can be merged separately from the core of the PPSH series. I want to keep the PPSH series focused on introducing PPSH support, and not make it grow infinitely with fixup patches for lots of packages.

However, perhaps what we could do is to collect those patches in branch, this way we could have two branches:

- One with the PPSH core stuff

- One with all the PPSH-related fixes, which are being submitted/applied progressively to Buildroot, even before PPSH itself is merged.

What do you think ?

Quoted 8 lines

On my local machine things seemed to work, however final rootfs creation failed in our CI environment due to broken host- binaries, e.g. mkfs.ubifs not finding some required lib (liblzo2.so.2 in that particular case).
It seems some things with rpath manipulation/checking do not yet work properly. Looking into the binary with ldd I see libs used from the host-machine that probably should come from buildroots host/usr/lib...

Quoted 7 lines

	linux-vdso.so.1 =>  (0x00007ffc06fcc000)
	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f760428f000)
	liblzo2.so.2 => not found
	libuuid.so.1 => /lib/x86_64-linux-gnu/libuuid.so.1 (0x00007f760408a000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f7603d81000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f76039b7000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f76044a9000)

This is weird, because it works fine here:

thomas at windsurf:~/projets/outputs/t$ ldd host/sbin/mkfs.ubifs

	linux-vdso.so.1 (0x00007ffe8cd3b000)
	libz.so.1 => /home/thomas/projets/outputs/t/per-package/host-mtd/host/lib/libz.so.1 (0x00007f1a683ff000)
	liblzo2.so.2 => /home/thomas/projets/outputs/t/per-package/host-mtd/host/lib/liblzo2.so.2 (0x00007f1a683db000)
	libuuid.so.1 => /home/thomas/projets/outputs/t/per-package/host-mtd/host/lib/libuuid.so.1 (0x00007f1a683d2000)
	libm.so.6 => /lib64/libm.so.6 (0x00007f1a6821e000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f1a68058000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f1a6841b000)

Could you dump the RPATH of the mkfs.ubifs library, like this:

thomas at windsurf:~/projets/outputs/t$ readelf -d host/sbin/mkfs.ubifs

Dynamic section at offset 0x11dd0 contains 29 entries:
Tag Type Name/Value

  0x0000000000000001 (NEEDED)             Shared library: [libz.so.1]
  0x0000000000000001 (NEEDED)             Shared library: [liblzo2.so.2]
  0x0000000000000001 (NEEDED)             Shared library: [libuuid.so.1]
  0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
  0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
  0x000000000000000f (RPATH)              Library rpath: [/home/thomas/projets/outputs/t/per-package/host-mtd/host/lib:/home/thomas/projets/outputs/t/per-package/host-util-linux/host/lib]
  0x000000000000000c (INIT)               0x402000
  0x000000000000000d (FINI)               0x40bb08
  0x0000000000000019 (INIT_ARRAY)         0x412dc0
  0x000000000000001b (INIT_ARRAYSZ)       8 (bytes)
  0x000000000000001a (FINI_ARRAY)         0x412dc8
  0x000000000000001c (FINI_ARRAYSZ)       8 (bytes)
  0x000000006ffffef5 (GNU_HASH)           0x400308
  0x0000000000000005 (STRTAB)             0x4009b8
  0x0000000000000006 (SYMTAB)             0x400340
  0x000000000000000a (STRSZ)              781 (bytes)
  0x000000000000000b (SYMENT)             24 (bytes)
  0x0000000000000015 (DEBUG)              0x0
  0x0000000000000003 (PLTGOT)             0x413000
  0x0000000000000002 (PLTRELSZ)           1464 (bytes)
  0x0000000000000014 (PLTREL)             RELA

  0x0000000000000017 (JMPREL)             0x400e58
  0x0000000000000007 (RELA)               0x400db0
  0x0000000000000008 (RELASZ)             168 (bytes)
  0x0000000000000009 (RELAENT)            24 (bytes)
  0x000000006ffffffe (VERNEED)            0x400d50
  0x000000006fffffff (VERNEEDNUM)         2
  0x000000006ffffff0 (VERSYM)             0x400cc6
  0x0000000000000000 (NULL)               0x0

Thanks a lot!

Thomas

Hello,

Here is a fourth iteration of the per-package SDK and target directory implementation.

If you're interested in testing it, you can find the patch series at:

http://git.bootlin.com/users/thomas-petazzoni/buildroot/log/?h=ppsh-v4

Changes since v4
================

- Dropped patches that have been merged upstream, mainly the "extract dependency" patches, but also a few other preparation patches.

- Dropped the change of the RPATH handling. As discussed during previous Buildroot Developers meeting, it is not a big problem if during the build, the RPATH of a binary points to the library folder of another package per-package folder. Therefore, instead of fixing up RPATH at the end of each package installation, we keep what Buildroot does today: fix them at the very end of the build.

- Made the support for per-package folders optional. Indeed, we realized there was no way to make sure it would be perfect from day
1. Even if the principle works, there are lots of package-specific details to solve, and solving all of them before merging the base per-package folder support is not reasonable. So for now, an option BR2_PER_PACKAGE_FOLDERS is introduced, which defaults to off. When this option is enabled, the .NOTPARALLEL statement of the main Makefile goes away, which allows to do top-level parallel build.

- Addition of a commit that reworks how the top-level directories are created.

- Introduction of a prepare-per-package-folder function in pkg-generic.mk to encapsulate the logic to create the per-package folders from the dependencies of a package.

- Rebased on top of the latest master.

Changes since v3
================

- Dropped patches that have been merged upstream:
pkgconf: use relative path to STAGING_DIR instead of absolute path
toolchain: post-pone evaluation of TOOLCHAIN_EXTERNAL_BIN

- For now, removed the most controversial/complicated patches (changing the rpath strategy, and switching to per-package host/target directories). The goal for now is to merge only the preparation patches.

Changes since v2
================

- Solved the problem of all DEPENDENCIES_HOST_PREREQ targets:
host-ccache, host-tar, host-lzip, host-xz, host-fakedate.

To achieve this, introduced <pkg>_EXTRACT_DEPENDENCIES and used that to handle the dependencies on host-tar, host-lzip and host-xz.

Used regular dependencies for host-ccache and host-fakedate.

See below for more details.

Changes since v1
================

- Rebased on the latest Buildroot next branch

- The pkg-config changes are reworked according to Arnout's comments:
only @STAGING_SUBDIR@ is replaced in pkg-config.in to generate pkg-config, the rest of the logic is internal to the script.

- New patch to move the "host skeleton" logic into a proper host-skeleton package.

- New patch to have the CMake related files generated as part of the toolchain package.

- New patch to add a new "install" step that depends on the different install steps (target, staging, images, host). This is needed to later add some logic that is executed once all install steps of a package have been done.

- Fix the approach to solve the RPATH logic: instead of using LD_LIBRARY_PATH, we actually fix with patchelf the RPATH of host binaries right after their installation.

- Misc other improvements in the per-package SDK implementation.

What is this all about ?
========================

See the cover letter of v3 for details:
http://lists.busybox.net/pipermail/buildroot/2017-December/208384.html

What remains to be done?
========================

The main problem that remains is fixing packages that don't work well with per-package folders. I have already sent a few fixes (such as adding missing host-pkgconf dependencies, or fixing the freetype package), but I have setup a separate autobuilder instance to test this, and it uncovered a number of other problematic packages:

- apr-util doesn't build, because some hardcoded paths from the "apr" per-package folders leak into the apr-util build.

- Meson packages currently do not build, because pkg-meson.mk doesn't use $$(HOST_DIR) to reference cross-compilation.conf, but more importantly because cross-compilation.conf uses hardcoded paths.

- The qt5 stack is entirely broken, because qmake hardcodes the installation path to be $(STAGING_DIR), so all qt5 packages install to the per-package $(STAGING_DIR) of qt5base. I started working on this topic, and will be sending a separate e-mail about this.

Best regards,

Thomas

Thomas Petazzoni (6):
Makefile: evaluate CCACHE and HOST{CC,CXX} at time of use support/scripts/check-host-rpath: split condition on two statements
Makefile: rework main directory creation logic
Makefile: move .NOTPARALLEL statement after including .config file
Makefile: define TARGET_DIR_WARNING_FILE relative to TARGET_DIR
core: implement per-package SDK and target

 Config.in                        | 18 ++++++
 Makefile                         | 94 ++++++++++++++++++++------------
 fs/common.mk                     |  3 +-
 package/pkg-generic.mk           | 27 ++++++++-
 support/scripts/check-host-rpath |  8 ++-
 5 files changed, 110 insertions(+), 40 deletions(-)