Skip to contentSkip to conversation

Overwritten file detection and fixes, one more step to TLP build

PATCH New Herve Codina 6 participants 83 messages Jun 21, 2021 – Jul 6, 2021 Kbuild

lore
Share & tools
lore
#0 lore raw

cover Overwritten file detection and fixes, one more step to TLP build

Hi,

This series fixes some issues in the TLP build feature.
The starting point was the work done by Thomas Petazzoni on overwrites detection (PATCH 1).

Based on this work, this series fixes some overwrites, introduces and uses <PKG>_PER_PACKAGE_TWEAK_HOOKS. This variable is used to collect specific per-package path tweaks needed by per-package build.
With this tweaks collected in this variable, they can be re-applied when they are needed (<pkg>_{reconfigure,rebuild,reinstall}).

It also reworks the final {HOST,TARGET}_DIR rsync.
For each packages only files generated by the package are rsynced (PATCH 13). This is done using an exclusion list computed in (PATCH 12).
In order to avoid modifications performed in target-finalize step to be seen as overwrites on next build, the final {HOST,TARGET}_DIR do not contain any hardlink anymore and are a full copy of expected files and directories (PATCH 14).

The last patch (PATCH 15) fixes <pkg>_{reconfigure,rebuild,reinstall} recreating per-package {HOST,TARGET}_DIR from scratch.
Indeed, for a given package, re-generating the same file is detected as an overwrite. The simplest way to avoid this is to fully discard previous per-package {HOST,TARGET}_DIR directory and recreate them.

To have a TLP build fully fonctionnal, several things are still missing:
- Be sure that all buildroot packages do not perform any overwrites.
- Undo tweak stuff that have be done for per-package build (fixup-libtool-files, fixup-python-files, <PKG>_PER_PACKAGE_TWEAK_HOOKS) after collecting files in final {HOST,STAGING}_DIR.
This mainly means reverting .../per-package/xxxxx/... path to point to the correct ones in final {HOST,STAGING}_DIR.

I hope this series will help in TLP build feature.

Best regards,
Herv? Codina

Herve Codina (14):
package/e2fsprogs: fix fsck overwrite in HOST_DIR package/pkg-generic.mk: Remove Info documents dir entry package/pkg-generic.mk: Fix .la files overwrite detection package/pkg-generic.mk: Perform .la files fixup in per-package
HOST_DIR
package/pkg-generic.mk: Introduce <PKG>_PER_PACKAGE_TWEAK_HOOKS package/apr-util: Use <PKG>_PER_PACKAGE_TWEAK_HOOKS package/apache: Move APACHE_FIXUP_APR_LIBTOOL to
<PKG>_PER_PACKAGE_TWEAK_HOOKS
package/pkg-python: Remove _sysconfigdata*.pyc files when
_sysconfigdata*.py are changed
package/pkg-generic.mk: Move python fixup to generic package
infrastructure
package/owfs: Remove Python sysconfigdata fixup package/pkg-generic.mk: Generate final rsync exclude file list
Makefile: Rsync global {TARGET,HOST}_DIR using exclusion file list
Makefile: Breaks hardlinks in global {TARGET,HOST}_DIR on per-package
build
package/pkg-generic.mk: Fix per-package

    <pkg>-{reconfigure,rebuild,reinstall}

Thomas Petazzoni (1):
package/pkg-generic.mk: detect files overwritten in TARGET_DIR and
HOST_DIR

 Makefile                       |  24 +++++++-
 package/apache/apache.mk       |   2 +-
 package/apr-util/apr-util.mk   |  12 ++--
 package/e2fsprogs/e2fsprogs.mk |   1 +
 package/owfs/owfs.mk           |   9 ---
 package/pkg-generic.mk         | 103 ++++++++++++++++++++++++++++++++-
 package/pkg-python.mk          |  10 ----
 7 files changed, 134 insertions(+), 27 deletions(-)
Signature · 1 line
2.31.1
#1 lore raw

PATCH 1/15 package/pkg-generic.mk: detect files overwritten in TARGET_DIR and HOST_DIR

From: Thomas Petazzoni <[email protected]>

Without per-package directory support, a package can happily overwrite files installed by other packages. Indeed, because the build order between packages is always guaranteed, Buildroot will always produce the same output.

However, with per-package directory support, it is absolutely critical that a given package does not overwrite files already installed by another package, due to how the final aggregation is done to create the complete target/, staging/ and host/ folders. Unfortunately, we currently don't have anything in Buildroot that detects this situation.

We used to have check-uniq-files, but it was dropped in commit 2496189a4207173e4cd5bbab90256f911175ee57.

This commit is a new implementation of such a detection, which is based on calculating and verifying MD5 hashes of installed files: the calculation is done at the beginning of the configure step, the verification during the newly introduced "install" step that takes place after all installation steps.

Since preventing file overwrites is really only needed when per-package directory support is used, and due to this verification having some overhead, it is only enabled when BR2_PER_PACKAGE_DIRECTORIES=y. This additional verification cost is however not too bad as on average, with per-package directory support, the per-package target/ and host/ directories will contain less files than with a build that doesn't use per-package directory support. This helps a bit in mitigating the additional cost of this verification.

Note that we are not handling separately HOST_DIR and STAGING_DIR, like we're doing with the pkg_size_{before,after} functions. Instead, the verification on HOST_DIR walks down into the STAGING_DIR.

This commit is retreived from Thomas's work.
The first version was discussed
https://patchwork.ozlabs.org/project/buildroot/patch/20200430095249.782597-9-thomas.petazzoni at bootlin.com/ This new version was not already submitted by Thomas or I missed it.

---
 package/pkg-generic.mk | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)
package/pkg-generic.mk +23 -0
index 45589bcbb4..bb9ff4150a 100644
@@ -102,6 +102,25 @@ define fixup-libtool-files
 endef
 endif
 
+# Functions to detect overwritten files
+
+ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y)
+# $(1): base directory to search in
+# $(2): suffix of file (optional)
+define pkg_detect_overwrite_before
+	LC_ALL=C find $(1) -type f -print0 | xargs -0 -r md5sum > $($(PKG)_DIR)/.files$(2).md5;
+endef
+
+# $(1): base directory to search in
+# $(2): suffix of file (optional)
+define pkg_detect_overwrite_after
+	if test -s $($(PKG)_DIR)/.files$(2).md5 ; then \
+		LC_ALL=C md5sum --quiet -c $($(PKG)_DIR)/.files$(2).md5 || \
+		{ echo "ERROR: package $($(PKG)_NAME) has overwritten files installed by a previous package, aborting."; exit 1; } ; \
+	fi
+endef
+endif
+
 # Functions to collect statistics about installed files
 
 # $(1): base directory to search in
@@ -235,6 +254,8 @@ $(BUILD_DIR)/%/.stamp_configured:
 	@$(call pkg_size_before,$(TARGET_DIR))
 	@$(call pkg_size_before,$(STAGING_DIR),-staging)
 	@$(call pkg_size_before,$(HOST_DIR),-host)
+	@$(call pkg_detect_overwrite_before,$(TARGET_DIR))
+	@$(call pkg_detect_overwrite_before,$(HOST_DIR),-host)
 	$(call fixup-libtool-files,$(NAME),$(STAGING_DIR))
 	$(foreach hook,$($(PKG)_PRE_CONFIGURE_HOOKS),$(call $(hook))$(sep))
 	$($(PKG)_CONFIGURE_CMDS)
@@ -360,6 +381,8 @@ $(BUILD_DIR)/%/.stamp_installed:
 	@$(call pkg_size_after,$(STAGING_DIR),-staging)
 	@$(call pkg_size_after,$(HOST_DIR),-host)
 	@$(call check_bin_arch)
+	@$(call pkg_detect_overwrite_after,$(TARGET_DIR))
+	@$(call pkg_detect_overwrite_after,$(HOST_DIR),-host)
 	$(Q)touch $@
 
 # Remove package sources
Signature · 1 line
2.31.1
#9 lore raw

PATCH 2/15 package/e2fsprogs: fix fsck overwrite in HOST_DIR

In reply to Herve Codina

host-e2fsprog overwrites fsck program and some man related files previously installed by host-util-linux.

This patch simply disable fsck in host-e2fsprog.

host-e2fsprog is used to build final ext{2,3,4} images. The missing host-e2fsprog fsck tool (filesystem integrity check tool) in HOST_DIR should not lead to issues.

---
 package/e2fsprogs/e2fsprogs.mk | 1 +
 1 file changed, 1 insertion(+)
package/e2fsprogs/e2fsprogs.mk +1 -0
index 0a69690e2f..bd147e737e 100644
@@ -27,6 +27,7 @@ HOST_E2FSPROGS_CONF_OPTS = \
 	--disable-defrag \
 	--disable-e2initrd-helper \
 	--disable-fuse2fs \
+	--disable-fsck \
 	--disable-libblkid \
 	--disable-libuuid \
 	--disable-testio-debug \
Signature · 1 line
2.31.1
#14 lore raw

PATCH 3/15 package/pkg-generic.mk: Remove Info documents dir entry

In reply to Herve Codina

Some packages (autotools for instance) install documentation files using install-info. This program adds an entry in the Info directory file (share/info/dir) and this causes TARGET_DIR and/or HOST_DIR overwrite.

In order to avoid this overwrite this patch removes the Info directory file right after any installation.

---
 package/pkg-generic.mk | 3 +++
 1 file changed, 3 insertions(+)
package/pkg-generic.mk +3 -0
index bb9ff4150a..2499c94746 100644
@@ -280,6 +280,7 @@ $(BUILD_DIR)/%/.stamp_host_installed:
 	$(foreach hook,$($(PKG)_PRE_INSTALL_HOOKS),$(call $(hook))$(sep))
 	+$($(PKG)_INSTALL_CMDS)
 	$(foreach hook,$($(PKG)_POST_INSTALL_HOOKS),$(call $(hook))$(sep))
+	$(Q) rm -f $(HOST_DIR)/share/info/dir
 	@$(call step_end,install-host)
 	$(Q)touch $@
 
@@ -309,6 +310,7 @@ $(BUILD_DIR)/%/.stamp_staging_installed:
 	$(foreach hook,$($(PKG)_PRE_INSTALL_STAGING_HOOKS),$(call $(hook))$(sep))
 	+$($(PKG)_INSTALL_STAGING_CMDS)
 	$(foreach hook,$($(PKG)_POST_INSTALL_STAGING_HOOKS),$(call $(hook))$(sep))
+	$(Q) rm -f $(STAGING_DIR)/share/info/dir
 	$(Q)if test -n "$($(PKG)_CONFIG_SCRIPTS)" ; then \
 		$(call MESSAGE,"Fixing package configuration files") ;\
 			$(SED)  "s,$(HOST_DIR), at HOST_DIR@,g" \
@@ -368,6 +370,7 @@ $(BUILD_DIR)/%/.stamp_target_installed:
 		$(or $($(PKG)_INSTALL_INIT_OPENRC), \
 			$($(PKG)_INSTALL_INIT_SYSV)))
 	$(foreach hook,$($(PKG)_POST_INSTALL_TARGET_HOOKS),$(call $(hook))$(sep))
+	$(Q) rm -f $(TARGET_DIR)/share/info/dir
 	$(Q)if test -n "$($(PKG)_CONFIG_SCRIPTS)" ; then \
 		$(RM) -f $(addprefix $(TARGET_DIR)/usr/bin/,$($(PKG)_CONFIG_SCRIPTS)) ; \
 	fi
Signature · 1 line
2.31.1
#20 lore raw

PATCH 4/15 package/pkg-generic.mk: Fix .la files overwrite detection

In reply to Herve Codina

During per-package build, original .la files are modified by fixup-libtool-files calls.
But since fixup-libtool-files modifies files using sed --in-place, these modification are done using a temporary file and a call to rename. Rename breaks the hardlink to the original file and leave the temporary file in per-package TARGET dir.
As the original file is not modified, this is no longer considered as an overwrite.

To fix this detection, this patch simply considers the what is done by fixup-libtool-files is part of the original snapshot used to detect overwrites. And so, the original snapshot is taken after fixup-libtool-files call.

---
 package/pkg-generic.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
package/pkg-generic.mk +1 -1
index 2499c94746..f9564831cc 100644
@@ -254,9 +254,9 @@ $(BUILD_DIR)/%/.stamp_configured:
 	@$(call pkg_size_before,$(TARGET_DIR))
 	@$(call pkg_size_before,$(STAGING_DIR),-staging)
 	@$(call pkg_size_before,$(HOST_DIR),-host)
+	$(call fixup-libtool-files,$(NAME),$(STAGING_DIR))
 	@$(call pkg_detect_overwrite_before,$(TARGET_DIR))
 	@$(call pkg_detect_overwrite_before,$(HOST_DIR),-host)
-	$(call fixup-libtool-files,$(NAME),$(STAGING_DIR))
 	$(foreach hook,$($(PKG)_PRE_CONFIGURE_HOOKS),$(call $(hook))$(sep))
 	$($(PKG)_CONFIGURE_CMDS)
 	$(foreach hook,$($(PKG)_POST_CONFIGURE_HOOKS),$(call $(hook))$(sep))
Signature · 1 line
2.31.1
#30 lore raw

PATCH 5/15 package/pkg-generic.mk: Perform .la files fixup in per-package HOST_DIR

In reply to Herve Codina

fixup-libtool-files was called on per-package STAGING_DIR. Some host-xxxx packages have their .la files with directories pointing outside their own per-package directory.

This commit, calling fixup-libtool-files on HOST_DIR, fixes this issue.

---
 package/pkg-generic.mk | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
package/pkg-generic.mk +3 -1
index f9564831cc..e437050175 100644
@@ -97,7 +97,8 @@ endif
 # $2: staging directory of the package
 ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y)
 define fixup-libtool-files
-	$(Q)find $(2)/usr/lib* -name "*.la" | xargs --no-run-if-empty \
+	$(Q)find $(2) \( -path '$(2)/lib*' -o -path '$(2)/usr/lib*' \) \
+		-name "*.la" | xargs --no-run-if-empty \
 		$(SED) "s:$(PER_PACKAGE_DIR)/[^/]\+/:$(PER_PACKAGE_DIR)/$(1)/:g"
 endef
 endif
@@ -254,6 +255,7 @@ $(BUILD_DIR)/%/.stamp_configured:
 	@$(call pkg_size_before,$(TARGET_DIR))
 	@$(call pkg_size_before,$(STAGING_DIR),-staging)
 	@$(call pkg_size_before,$(HOST_DIR),-host)
+	$(call fixup-libtool-files,$(NAME),$(HOST_DIR))
 	$(call fixup-libtool-files,$(NAME),$(STAGING_DIR))
 	@$(call pkg_detect_overwrite_before,$(TARGET_DIR))
 	@$(call pkg_detect_overwrite_before,$(HOST_DIR),-host)
Signature · 1 line
2.31.1
#35 lore raw

PATCH 6/15 package/pkg-generic.mk: Introduce <PKG>_PER_PACKAGE_TWEAK_HOOKS

In reply to Herve Codina

This hook will be used by package infrastucture or some packages to tweak external files (ie files not provides by the package itself) when we build with BR2_PER_PACKAGE_DIRECTORIES set.

This allows to have these tweaks in a specific hook instead of <PKG>_POST_CONFIGURE_HOOK. And so we can call these specific tweaks whenever we need them.

---
 package/pkg-generic.mk | 2 ++
 1 file changed, 2 insertions(+)
package/pkg-generic.mk +2 -0
index e437050175..4069d2cf3c 100644
@@ -257,6 +257,7 @@ $(BUILD_DIR)/%/.stamp_configured:
 	@$(call pkg_size_before,$(HOST_DIR),-host)
 	$(call fixup-libtool-files,$(NAME),$(HOST_DIR))
 	$(call fixup-libtool-files,$(NAME),$(STAGING_DIR))
+	$(foreach hook,$($(PKG)_PER_PACKAGE_TWEAK_HOOKS),$(call $(hook))$(sep))
 	@$(call pkg_detect_overwrite_before,$(TARGET_DIR))
 	@$(call pkg_detect_overwrite_before,$(HOST_DIR),-host)
 	$(foreach hook,$($(PKG)_PRE_CONFIGURE_HOOKS),$(call $(hook))$(sep))
@@ -844,6 +845,7 @@ $(2)_PRE_LEGAL_INFO_HOOKS       ?=
 $(2)_POST_LEGAL_INFO_HOOKS      ?=
 $(2)_TARGET_FINALIZE_HOOKS      ?=
 $(2)_ROOTFS_PRE_CMD_HOOKS       ?=
+$(2)_PER_PACKAGE_TWEAK_HOOKS    ?=
 
 ifeq ($$($(2)_TYPE),target)
 ifneq ($$(HOST_$(2)_KCONFIG_VAR),)
Signature · 1 line
2.31.1
#43 lore raw

PATCH 7/15 package/apr-util: Use <PKG>_PER_PACKAGE_TWEAK_HOOKS

In reply to Herve Codina

Original APR_UTIL_FIX_RULES_MK_LIBTOOL tweaked libtool and rules.mk.

libtool is provided by a dependency (apr). It needs to be tweaked and, as an apr-util external file, this tweak is relevant in <PKG>_PER_PACKAGE_TWEAK_HOOKS.

rules.mk is generated by apr-util configure step and it is private to apr-util. The modification performed needs to be kept in <PKG>_POST_CONFIGURE_HOOKS.

This commit splits original APR_UTIL_FIX_RULES_MK_LIBTOOL and attaches each part to the correct hook.

---
 package/apr-util/apr-util.mk | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)
package/apr-util/apr-util.mk +8 -4
index db4df91564..b26d523265 100644
@@ -18,13 +18,17 @@ APR_UTIL_CONF_OPTS = \
 APR_UTIL_CONFIG_SCRIPTS = apu-1-config
 
 ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y)
-define APR_UTIL_FIX_RULES_MK_LIBTOOL
-	$(SED) 's,$(PER_PACKAGE_DIR)/apr/,$(PER_PACKAGE_DIR)/apr-util/,g' \
-		$(@D)/build/rules.mk
+define APR_UTIL_FIX_LIBTOOL
 	$(SED) 's,$(PER_PACKAGE_DIR)/apr/,$(PER_PACKAGE_DIR)/apr-util/,g' \
 		$(STAGING_DIR)/usr/build-1/libtool
 endef
-APR_UTIL_POST_CONFIGURE_HOOKS += APR_UTIL_FIX_RULES_MK_LIBTOOL
+APR_UTIL_PER_PACKAGE_TWEAK_HOOKS += APR_UTIL_FIX_LIBTOOL
+
+define APR_UTIL_FIX_RULES_MK
+	$(SED) 's,$(PER_PACKAGE_DIR)/apr/,$(PER_PACKAGE_DIR)/apr-util/,g' \
+		$(@D)/build/rules.mk
+endef
+APR_UTIL_POST_CONFIGURE_HOOKS += APR_UTIL_FIX_RULES_MK
 endif
 
 # When iconv is available, then use it to provide charset conversion
Signature · 1 line
2.31.1
#48 lore raw

PATCH 8/15 package/apache: Move APACHE_FIXUP_APR_LIBTOOL to <PKG>_PER_PACKAGE_TWEAK_HOOKS

In reply to Herve Codina

APACHE_FIXUP_APR_LIBTOOL tweaks files for per package directory build. This is typically the kind of operation expected to be in <PKG>_PER_PACKAGE_TWEAK_HOOKS

---
 package/apache/apache.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
package/apache/apache.mk +1 -1
index 7dbd1a4512..f496333174 100644
@@ -23,7 +23,7 @@ define APACHE_FIXUP_APR_LIBTOOL
 	$(SED) "s@$(PER_PACKAGE_DIR)/[^/]\+/@$(PER_PACKAGE_DIR)/apache/@g" \
 		$(STAGING_DIR)/usr/build-1/libtool
 endef
-APACHE_POST_CONFIGURE_HOOKS += APACHE_FIXUP_APR_LIBTOOL
+APACHE_PER_PACKAGE_TWEAK_HOOKS += APACHE_FIXUP_APR_LIBTOOL
 endif
 
 APACHE_CONF_ENV= \
Signature · 1 line
2.31.1
#50 lore raw

PATCH 9/15 package/pkg-python: Remove _sysconfigdata*.pyc files when _sysconfigdata*.py are changed

In reply to Herve Codina

In order to avoid _sysconfigdata*.pyc overwrites when they are generated based on _sysconfigdata*.py changes, this commit simply removes _sysconfigdata*.pyc whenever _sysconfigdata*.py are changed.

As they are removed, overwrite detection will no longer trig and coherency between the two files (.py and .pyc) is ensured.

---
 package/pkg-python.mk | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
package/pkg-python.mk +3 -1
index 59a48e5a87..b3fde77da5 100644
@@ -96,7 +96,9 @@ ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y)
 define PKG_PYTHON_FIXUP_SYSCONFIGDATA
 	find $(HOST_DIR)/lib/python* $(STAGING_DIR)/usr/lib/python* \
 		-name "_sysconfigdata*.py" | xargs --no-run-if-empty \
-		$(SED) "s:$(PER_PACKAGE_DIR)/[^/]\+/:$(PER_PACKAGE_DIR)/$($(PKG)_NAME)/:g"
+		$(SED) "s:$(PER_PACKAGE_DIR)/[^/]\+/:$(PER_PACKAGE_DIR)/$($(PKG)_NAME)/:g" ;\
+	find $(HOST_DIR)/lib/python* $(STAGING_DIR)/usr/lib/python* \
+		-name "_sysconfigdata*.pyc" -print0 | xargs -0 -r rm -f
 endef
 endif
 
Signature · 1 line
2.31.1
#55 lore raw

PATCH 10/15 package/pkg-generic.mk: Move python fixup to generic package infrastructure

In reply to Herve Codina

Fixing _sysconfigdata*.{py,pyc} was previously done by python package infrastructure. Some packages use python stuff without using python package infrastructure.
These packages perform overwrites and need the specific python fixup to fix them.

In order to be sure to fix all of these packages, the python fixup is moved to the generic package infrastructure and applied to all packages.
This follows the same principle as for the .la libtool files fixup.

---
 package/pkg-generic.mk | 17 +++++++++++++++++
 package/pkg-python.mk  | 12 ------------
 2 files changed, 17 insertions(+), 12 deletions(-)
package/pkg-generic.mk +17 -0
index 4069d2cf3c..918e2381af 100644
@@ -103,6 +103,21 @@ define fixup-libtool-files
 endef
 endif
 
+# Make sure python _sysconfigdata*.py files only reference the current
+# per-package directory.
+
+# $1: package name (lower case)
+# $2: staging or host directory of the package
+ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y)
+define fixup-python-files
+	$(Q)find $(2) \( -path '$(2)/lib/python*' -o -path '$(2)/usr/lib/python*' \) \
+		-name "_sysconfigdata*.py" | xargs --no-run-if-empty \
+		$(SED) "s:$(PER_PACKAGE_DIR)/[^/]\+/:$(PER_PACKAGE_DIR)/$(1)/:g" ;\
+	find $(2) \( -path '$(2)/lib/python*' -o -path '$(2)/usr/lib/python*' \) \
+		-name "_sysconfigdata*.pyc" -print0 | xargs -0 -r rm -f
+endef
+endif
+
 # Functions to detect overwritten files
 
 ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y)
@@ -257,6 +272,8 @@ $(BUILD_DIR)/%/.stamp_configured:
 	@$(call pkg_size_before,$(HOST_DIR),-host)
 	$(call fixup-libtool-files,$(NAME),$(HOST_DIR))
 	$(call fixup-libtool-files,$(NAME),$(STAGING_DIR))
+	$(call fixup-python-files,$(NAME),$(HOST_DIR))
+	$(call fixup-python-files,$(NAME),$(STAGING_DIR))
 	$(foreach hook,$($(PKG)_PER_PACKAGE_TWEAK_HOOKS),$(call $(hook))$(sep))
 	@$(call pkg_detect_overwrite_before,$(TARGET_DIR))
 	@$(call pkg_detect_overwrite_before,$(HOST_DIR),-host)
package/pkg-python.mk +0 -12
index b3fde77da5..e6b81bdfd3 100644
@@ -92,16 +92,6 @@ HOST_PKG_PYTHON_SETUPTOOLS_INSTALL_OPTS = \
 	--root=/ \
 	--single-version-externally-managed
 
-ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y)
-define PKG_PYTHON_FIXUP_SYSCONFIGDATA
-	find $(HOST_DIR)/lib/python* $(STAGING_DIR)/usr/lib/python* \
-		-name "_sysconfigdata*.py" | xargs --no-run-if-empty \
-		$(SED) "s:$(PER_PACKAGE_DIR)/[^/]\+/:$(PER_PACKAGE_DIR)/$($(PKG)_NAME)/:g" ;\
-	find $(HOST_DIR)/lib/python* $(STAGING_DIR)/usr/lib/python* \
-		-name "_sysconfigdata*.pyc" -print0 | xargs -0 -r rm -f
-endef
-endif
-
 ################################################################################
 # inner-python-package -- defines how the configuration, compilation
 # and installation of a Python package should be done, implements a
@@ -246,8 +236,6 @@ $(2)_PYTHON_INTERPRETER = $$(HOST_DIR)/bin/$$($(2)_NEEDS_HOST_PYTHON)
 endif
 endif
 
-$(2)_PRE_CONFIGURE_HOOKS += PKG_PYTHON_FIXUP_SYSCONFIGDATA
-
 #
 # Build step. Only define it if not already defined by the package .mk
 # file.
Signature · 1 line
2.31.1
#59 lore raw

PATCH 11/15 package/owfs: Remove Python sysconfigdata fixup

In reply to Herve Codina

This fixup is done at pkg-generic level for all packages. So, it is no more needed in owfs package.

---
 package/owfs/owfs.mk | 9 ---------
 1 file changed, 9 deletions(-)
package/owfs/owfs.mk +0 -9
index ffc0b3098d..10543d3698 100644
@@ -88,15 +88,6 @@ OWFS_DEPENDENCIES += python host-swig
 # override the PYSITEDIR variable in make.
 OWFS_EXTRA_MAKE_OPTS += PYSITEDIR=/usr/lib/python$(PYTHON_VERSION_MAJOR)/site-packages
 
-ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y)
-define OWFS_FIXUP_PYTHON_SYSCONFIGDATA
-	find $(HOST_DIR)/lib/python* $(STAGING_DIR)/usr/lib/python* \
-		-name "_sysconfigdata*.py" | xargs --no-run-if-empty \
-		$(SED) "s:$(PER_PACKAGE_DIR)/[^/]\+/:$(PER_PACKAGE_DIR)/owfs/:g"
-endef
-OWFS_PRE_CONFIGURE_HOOKS += OWFS_FIXUP_PYTHON_SYSCONFIGDATA
-endif
-
 else
 OWFS_CONF_OPTS += --disable-owpython --without-python
 endif
Signature · 1 line
2.31.1
#62 lore raw

PATCH 12/15 package/pkg-generic.mk: Generate final rsync exclude file list

In reply to Herve Codina

The final rsync performed at {host,target}-finalize steps need to be done by rsyncing files generated by each packages without looking at files generated by other packages the current package depends on. This is needed to avoid overwrites in final {HOST,TARGET}_DIR.

In order to prepare the final rsync, an exclusion list is generated. This list lists files that are not generated by the current package and so files that need to be excluded from the final rsync.

Note also that the files list was not based on .files-list.{before,after}. During .file-list.{before,after} built for host directory, staging sub-directory (ie <toolchain>/sysroot) is filtered out.
The final rsync exclusion list needs to take into account the full {host,target} directory to avoid final overwrites.

---
 package/pkg-generic.mk | 28 ++++++++++++++++++++++++++++
 1 file changed, 28 insertions(+)
package/pkg-generic.mk +28 -0
index 918e2381af..546370af7a 100644
@@ -162,6 +162,30 @@ define pkg_size_after
 	rm -f $($(PKG)_DIR)/.files-list$(2).after
 endef
 
+# Functions to collect final rsync exclusion files
+
+# $(1): base directory to search in
+# $(2): suffix of file (optional)
+define pkg_final_rsync_before
+	cd $(1); \
+	LC_ALL=C find . \( -type f -o -type l \) -printf '%T@:%i:%#m:%y:%s,%p\n' \
+		| LC_ALL=C sort > $($(PKG)_DIR)/.files-final-rsync$(2).before
+endef
+
+# $(1): base directory to search in
+# $(2): suffix of file (optional)
+define pkg_final_rsync_after
+	cd $(1); \
+	LC_ALL=C find . \( -type f -o -type l \) -printf '%T@:%i:%#m:%y:%s,%p\n' \
+		| LC_ALL=C sort > $($(PKG)_DIR)/.files-final-rsync$(2).after
+	LC_ALL=C comm -2 \
+		$($(PKG)_DIR)/.files-final-rsync$(2).before \
+		$($(PKG)_DIR)/.files-final-rsync$(2).after \
+		| sed -r -e 's/^[^,]+,./- /' \
+		> $($(PKG)_DIR)/.files-final-rsync$(2).exclude_rsync
+	rm -f $($(PKG)_DIR)/.files-final-rsync$(2).after
+endef
+
 define check_bin_arch
 	support/scripts/check-bin-arch -p $($(PKG)_NAME) \
 		-l $($(PKG)_DIR)/.files-list.txt \
@@ -267,6 +291,8 @@ $(BUILD_DIR)/%/.stamp_configured:
 	@$(call MESSAGE,"Configuring")
 	$(Q)mkdir -p $(HOST_DIR) $(TARGET_DIR) $(STAGING_DIR) $(BINARIES_DIR)
 	$(call prepare-per-package-directory,$($(PKG)_FINAL_DEPENDENCIES))
+	@$(call pkg_final_rsync_before,$(TARGET_DIR))
+	@$(call pkg_final_rsync_before,$(HOST_DIR),-host)
 	@$(call pkg_size_before,$(TARGET_DIR))
 	@$(call pkg_size_before,$(STAGING_DIR),-staging)
 	@$(call pkg_size_before,$(HOST_DIR),-host)
@@ -404,6 +430,8 @@ $(BUILD_DIR)/%/.stamp_installed:
 	@$(call pkg_size_after,$(STAGING_DIR),-staging)
 	@$(call pkg_size_after,$(HOST_DIR),-host)
 	@$(call check_bin_arch)
+	@$(call pkg_final_rsync_after,$(TARGET_DIR))
+	@$(call pkg_final_rsync_after,$(HOST_DIR),-host)
 	@$(call pkg_detect_overwrite_after,$(TARGET_DIR))
 	@$(call pkg_detect_overwrite_after,$(HOST_DIR),-host)
 	$(Q)touch $@
Signature · 1 line
2.31.1
#66 lore raw

PATCH 13/15 Makefile: Rsync global {TARGET, HOST}_DIR using exclusion file list

In reply to Herve Codina

On a per-package build, rsync final {TARGET,HOST}_DIR using exclusion file list computed for each packages.
As we rsync only files generated by each packages, we need to to do this rsync recursively on each dependencies to collect all needed files. This is done based on existing <PKG>_FINAL_RECURSIVE_DEPENDENCIES

---
 Makefile | 24 ++++++++++++++++++++++--
 1 file changed, 22 insertions(+), 2 deletions(-)
Makefile +22 -2
index 472af5a318..50bc6b4503 100644
@@ -740,10 +740,28 @@ TARGET_DIR_FILES_LISTS = $(sort $(wildcard $(BUILD_DIR)/*/.files-list.txt))
 HOST_DIR_FILES_LISTS = $(sort $(wildcard $(BUILD_DIR)/*/.files-list-host.txt))
 STAGING_DIR_FILES_LISTS = $(sort $(wildcard $(BUILD_DIR)/*/.files-list-staging.txt))
 
+ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y)
+# rsync the contents of per-package directories
+# $1: space-separated list of packages to rsync from
+# $2: 'host' or 'target'
+# $3: destination directory
+# $4: exclude file list to use
+define per-package-rsync-delta
+	mkdir -p $(3)
+	$(foreach pkg,$(1),\
+		rsync -a --link-dest=$(PER_PACKAGE_DIR)/$(pkg)/$(strip $(2))/ \
+		--filter='merge $($(call UPPERCASE,$(pkg))_DIR)/$(strip $(4))' \
+		$(PER_PACKAGE_DIR)/$(pkg)/$(strip $(2))/ \
+		$(3)$(sep))
+endef
+endif
+
 .PHONY: host-finalize
 host-finalize: $(PACKAGES) $(HOST_DIR) $(HOST_DIR_SYMLINK)
 	@$(call MESSAGE,"Finalizing host directory")
-	$(call per-package-rsync,$(sort $(PACKAGES)),host,$(HOST_DIR))
+	$(call per-package-rsync-delta, $(sort $(foreach pkg, $(PACKAGES), \
+		$(pkg) $($(call UPPERCASE,$(pkg))_FINAL_RECURSIVE_DEPENDENCIES))), \
+		host,$(HOST_DIR),.files-final-rsync-host.exclude_rsync)
 
 .PHONY: staging-finalize
 staging-finalize: $(STAGING_DIR_SYMLINK)
@@ -751,7 +769,9 @@ staging-finalize: $(STAGING_DIR_SYMLINK)
 .PHONY: target-finalize
 target-finalize: $(PACKAGES) $(TARGET_DIR) host-finalize
 	@$(call MESSAGE,"Finalizing target directory")
-	$(call per-package-rsync,$(sort $(PACKAGES)),target,$(TARGET_DIR))
+	$(call per-package-rsync-delta, $(sort $(foreach pkg, $(PACKAGES), \
+		$(pkg) $($(call UPPERCASE,$(pkg))_FINAL_RECURSIVE_DEPENDENCIES))), \
+		target,$(TARGET_DIR),.files-final-rsync.exclude_rsync)
 	$(foreach hook,$(TARGET_FINALIZE_HOOKS),$($(hook))$(sep))
 	rm -rf $(TARGET_DIR)/usr/include $(TARGET_DIR)/usr/share/aclocal \
 		$(TARGET_DIR)/usr/lib/pkgconfig $(TARGET_DIR)/usr/share/pkgconfig \
Signature · 1 line
2.31.1
#72 lore raw

PATCH 14/15 Makefile: Breaks hardlinks in global {TARGET, HOST}_DIR on per-package build

In reply to Herve Codina

Without this patch, a make <pkg>_rebuild detects overwrites. Indeed, in target_finalize steps some modifications are done on installed files (ie strip or TARGET_FINALIZE_HOOKS for instance).

In order to avoid these modifications seen from per-package {TARGET,HOST}_DIR and so been analyzed as some overwrites, global {TARGET,HOST}_DIR is built using a full copy of the involved per-package files instead of hardlinks.

---
 Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Makefile +1 -1
index 50bc6b4503..fe6e3f95af 100644
@@ -749,7 +749,7 @@ ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y)
 define per-package-rsync-delta
 	mkdir -p $(3)
 	$(foreach pkg,$(1),\
-		rsync -a --link-dest=$(PER_PACKAGE_DIR)/$(pkg)/$(strip $(2))/ \
+		rsync -a \
 		--filter='merge $($(call UPPERCASE,$(pkg))_DIR)/$(strip $(4))' \
 		$(PER_PACKAGE_DIR)/$(pkg)/$(strip $(2))/ \
 		$(3)$(sep))
Signature · 1 line
2.31.1
#74 lore raw

PATCH 15/15 package/pkg-generic.mk: Fix per-package <pkg>-{reconfigure, rebuild, reinstall}

In reply to Herve Codina

Many overwrites are detected on <pkg>-{reconfigure,rebuild,reinstall}. Indeed, files previously installed by a package were detected as overwritten on next reconfigure, rebuild or reinstall.

To avoid this, we recreate per-package host and target dir from scratch as it was done during the first configure step.

---
 package/pkg-generic.mk | 26 ++++++++++++++++++++++++++
 1 file changed, 26 insertions(+)
package/pkg-generic.mk +26 -0
index 546370af7a..0aafd865ad 100644
@@ -470,6 +470,13 @@ define pkg-graph-depends
 		$$(GRAPHS_DIR)/$$(@).dot
 endef
 
+ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y)
+define empty-per-package-directory
+	rm -rf $(HOST_DIR) $(TARGET_DIR)
+	mkdir -p $(HOST_DIR) $(TARGET_DIR)
+endef
+endif
+
 ################################################################################
 # inner-generic-package -- generates the make targets needed to build a
 # generic package
@@ -1072,6 +1079,25 @@ endif
 			rm -f $$($(2)_TARGET_INSTALL_TARGET)
 			rm -f $$($(2)_TARGET_INSTALL_IMAGES)
 			rm -f $$($(2)_TARGET_INSTALL_HOST)
+			$$(call empty-per-package-directory)
+			$$(call prepare-per-package-directory,$$($(2)_FINAL_DOWNLOAD_DEPENDENCIES))
+			$$(call prepare-per-package-directory,$$($(2)_FINAL_EXTRACT_DEPENDENCIES))
+			$$(call prepare-per-package-directory,$$($(2)_FINAL_DEPENDENCIES))
+			@$$(call pkg_final_rsync_before,$$(TARGET_DIR))
+			@$$(call pkg_final_rsync_before,$$(HOST_DIR),-host)
+			@$$(call pkg_size_before,$$(TARGET_DIR))
+			@$$(call pkg_size_before,$$(STAGING_DIR),-staging)
+			@$$(call pkg_size_before,$$(HOST_DIR),-host)
+			$$(call fixup-libtool-files,$$(NAME),$$(HOST_DIR))
+			$$(call fixup-libtool-files,$$(NAME),$$(STAGING_DIR))
+			$$(call fixup-python-files,$$(NAME),$$(HOST_DIR))
+			$$(call fixup-python-files,$$(NAME),$$(STAGING_DIR))
+			$$(foreach hook,$$($(2)_PER_PACKAGE_TWEAK_HOOKS),$$(call $$(hook))$$(sep))
+			@$$(call pkg_detect_overwrite_before,$$(TARGET_DIR))
+			@$$(call pkg_detect_overwrite_before,$$(HOST_DIR),-host)
+
+$(1)-clean-for-reinstall: PKG=$(2)
+$(1)-clean-for-reinstall: NAME=$(1)
 
 $(1)-reinstall:		$(1)-clean-for-reinstall $(1)
 
Signature · 1 line
2.31.1
#36 lore raw

PATCH 6/15 package/pkg-generic.mk: Introduce <PKG>_PER_PACKAGE_TWEAK_HOOKS

In reply to Herve Codina
Quoted 9 lines · On Mon, 21 Jun 2021 16:11:21 +0200 Herve Codina <[email protected]>…

This hook will be used by package infrastucture or some packages to tweak external files (ie files not provides by the package itself) when we build with BR2_PER_PACKAGE_DIRECTORIES set.

This allows to have these tweaks in a specific hook instead of <PKG>_POST_CONFIGURE_HOOK. And so we can call these specific tweaks whenever we need them.

I think this commit log needs an explanation as to why this is necessary, i.e why isn't the current solution of POST_CONFIGURE_HOOKS working.

Thanks!

Thomas

Signature · 3 lines
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#51 lore raw

PATCH 9/15 package/pkg-python: Remove _sysconfigdata*.pyc files when _sysconfigdata*.py are changed

In reply to Herve Codina
Quoted 17 lines · On Mon, 21 Jun 2021 16:11:24 +0200 Herve Codina <[email protected]>…

In order to avoid _sysconfigdata*.pyc overwrites when they are generated based on _sysconfigdata*.py changes, this commit simply removes _sysconfigdata*.pyc whenever _sysconfigdata*.py are changed.

As they are removed, overwrite detection will no longer trig and coherency between the two files (.py and .pyc) is ensured.

---
 package/pkg-python.mk | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
package/pkg-python.mk +0 -0
index 59a48e5a87..b3fde77da5 100644

package/pkg-python.mk ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y) · line 99

Quoted 5 lines
package/pkg-python.mk +1 -1
@@ -96,7 +96,9 @@ ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y)
 define PKG_PYTHON_FIXUP_SYSCONFIGDATA
 	find $(HOST_DIR)/lib/python* $(STAGING_DIR)/usr/lib/python* \
 		-name "_sysconfigdata*.py" | xargs --no-run-if-empty \
-		$(SED) "s:$(PER_PACKAGE_DIR)/[^/]\+/:$(PER_PACKAGE_DIR)/$($(PKG)_NAME)/:g"
+		$(SED) "s:$(PER_PACKAGE_DIR)/[^/]\+/:$(PER_PACKAGE_DIR)/$($(PKG)_NAME)/:g" ;\

The semicolon + backslash is not needed, these two commands can (I believe) be executed as separate shell commands.

package/pkg-python.mk ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y) · line 101

Quoted 2 lines
package/pkg-python.mk +2 -0
+	find $(HOST_DIR)/lib/python* $(STAGING_DIR)/usr/lib/python* \
+		-name "_sysconfigdata*.pyc" -print0 | xargs -0 -r rm -f

Do we need a --no-run-if-empty on the xargs ? Or perhaps it should be using find directly, i.e:

	find $(HOST_DIR)/lib/python* $(STAGING_DIR)/usr/lib/python* \
		-name "_sysconfigdata*.pyc" -exec rm -f {} \;

(or something like that)

Thanks!

Thomas

Signature · 3 lines
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#77 lore raw

cover Overwritten file detection and fixes, one more step to TLP build

In reply to Herve Codina

Hi Herve?,

Great that someone is picking thuis up!

Quoted 5 lines · Herve Codina

Hi,

This series fixes some issues in the TLP build feature.
The starting point was the work done by Thomas Petazzoni on overwrites detection (PATCH 1).

I've marked [1] as Changes Requested.

However, it seems the tests [2][3][4] are missing from this series. Could you
include them next time you post it?

Quoted 5 lines

Based on this work, this series fixes some overwrites, introduces and uses <PKG>_PER_PACKAGE_TWEAK_HOOKS. This variable is used to collect specific per-package path tweaks needed by per-package build.
With this tweaks collected in this variable, they can be re-applied when they are needed (<pkg>_{reconfigure,rebuild,reinstall}).

That's excellent!

Quoted 6 lines

It also reworks the final {HOST,TARGET}_DIR rsync.
For each packages only files generated by the package are rsynced (PATCH 13). This is done using an exclusion list computed in (PATCH 12).
In order to avoid modifications performed in target-finalize step to be seen as overwrites on next build, the final {HOST,TARGET}_DIR do not contain any hardlink anymore and are a full copy of expected files and directories (PATCH 14).

Hm, that will have quite the impact on disk size, no? In non-PPS builds,
HOST_DIR is typically about 1/3 of the total size (the other 2/3 is BUILD_DIR, the rest is negligible). By doing a non-hardlinked copy, the size of HOST_DIR is basically going to double, right?

I still think it's acceptable, but it's something to take into account...

Quoted 5 lines

The last patch (PATCH 15) fixes <pkg>_{reconfigure,rebuild,reinstall} recreating per-package {HOST,TARGET}_DIR from scratch.
Indeed, for a given package, re-generating the same file is detected as an overwrite. The simplest way to avoid this is to fully discard previous per-package {HOST,TARGET}_DIR directory and recreate them.

+1 to that!

Quoted 2 lines

To have a TLP build fully fonctionnal, several things are still missing:
- Be sure that all buildroot packages do not perform any overwrites.

I guess that's easy: increase the chance of PER_PACKAGE enabled in
genrandconfig, and see what happens to the autobuilders. Or is there anything else missing?

Quoted 7 lines

- Undo tweak stuff that have be done for per-package build (fixup-libtool-files, fixup-python-files, <PKG>_PER_PACKAGE_TWEAK_HOOKS) after collecting files in final {HOST,STAGING}_DIR.
This mainly means reverting .../per-package/xxxxx/... path to point to the correct ones in final {HOST,STAGING}_DIR.

I hope this series will help in TLP build feature.

Regards,
Arnout

[1]
https://patchwork.ozlabs.org/project/buildroot/patch/20200430095249.782597-9-thomas.petazzoni at bootlin.com/
[2]
https://patchwork.ozlabs.org/project/buildroot/patch/20200430095249.782597-10-thomas.petazzoni at bootlin.com/
[3]
https://patchwork.ozlabs.org/project/buildroot/patch/20200430095249.782597-11-thomas.petazzoni at bootlin.com/
[4]
https://patchwork.ozlabs.org/project/buildroot/patch/20200430095249.782597-12-thomas.petazzoni at bootlin.com/

Quoted 39 lines

Best regards,
Herv? Codina

Herve Codina (14):
package/e2fsprogs: fix fsck overwrite in HOST_DIR package/pkg-generic.mk: Remove Info documents dir entry package/pkg-generic.mk: Fix .la files overwrite detection package/pkg-generic.mk: Perform .la files fixup in per-package
HOST_DIR
package/pkg-generic.mk: Introduce <PKG>_PER_PACKAGE_TWEAK_HOOKS package/apr-util: Use <PKG>_PER_PACKAGE_TWEAK_HOOKS package/apache: Move APACHE_FIXUP_APR_LIBTOOL to
<PKG>_PER_PACKAGE_TWEAK_HOOKS
package/pkg-python: Remove _sysconfigdata*.pyc files when
_sysconfigdata*.py are changed
package/pkg-generic.mk: Move python fixup to generic package
infrastructure
package/owfs: Remove Python sysconfigdata fixup package/pkg-generic.mk: Generate final rsync exclude file list
Makefile: Rsync global {TARGET,HOST}_DIR using exclusion file list
Makefile: Breaks hardlinks in global {TARGET,HOST}_DIR on per-package
build
package/pkg-generic.mk: Fix per-package

    <pkg>-{reconfigure,rebuild,reinstall}

Thomas Petazzoni (1):
package/pkg-generic.mk: detect files overwritten in TARGET_DIR and
HOST_DIR

 Makefile                       |  24 +++++++-
 package/apache/apache.mk       |   2 +-
 package/apr-util/apr-util.mk   |  12 ++--
 package/e2fsprogs/e2fsprogs.mk |   1 +
 package/owfs/owfs.mk           |   9 ---
 package/pkg-generic.mk         | 103 ++++++++++++++++++++++++++++++++-
 package/pkg-python.mk          |  10 ----
 7 files changed, 134 insertions(+), 27 deletions(-)
#31 lore raw

PATCH 5/15 package/pkg-generic.mk: Perform .la files fixup in per-package HOST_DIR

In reply to Herve Codina
Quoted 3 lines · On Mon, 21 Jun 2021 16:11:20 +0200 Herve Codina <[email protected]>…

fixup-libtool-files was called on per-package STAGING_DIR. Some host-xxxx packages have their .la files with directories pointing outside their own per-package directory.

Was this causing a practical issue, or is being done just to be cleaner?

Thomas

Signature · 3 lines
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#15 lore raw

PATCH 3/15 package/pkg-generic.mk: Remove Info documents dir entry

In reply to Herve Codina
Quoted 18 lines · On Mon, 21 Jun 2021 16:11:18 +0200 Herve Codina <[email protected]>…

Some packages (autotools for instance) install documentation files using install-info. This program adds an entry in the Info directory file (share/info/dir) and this causes TARGET_DIR and/or HOST_DIR overwrite.

In order to avoid this overwrite this patch removes the Info directory file right after any installation.

---
 package/pkg-generic.mk | 3 +++
 1 file changed, 3 insertions(+)
package/pkg-generic.mk +0 -0
index bb9ff4150a..2499c94746 100644

package/pkg-generic.mk $(BUILD_DIR)/%/.stamp_host_installed: · line 283

Quoted 4 lines
package/pkg-generic.mk +1 -0
@@ -280,6 +280,7 @@ $(BUILD_DIR)/%/.stamp_host_installed:
 	$(foreach hook,$($(PKG)_PRE_INSTALL_HOOKS),$(call $(hook))$(sep))
 	+$($(PKG)_INSTALL_CMDS)
 	$(foreach hook,$($(PKG)_POST_INSTALL_HOOKS),$(call $(hook))$(sep))
+	$(Q) rm -f $(HOST_DIR)/share/info/dir

No space between $(Q) and rm. This should perhaps use $(RM) in fact.

However, I'm not a huge fan of having this right in the middle of the infrastructure. It feels like a small detail that gets handled in the middle of super generic infrastructure code.

The issue is that I don't really have a good alternative proposal :-/

Instead of removing that file, ignore it in the overwrite detection, perhaps?

Thomas

Signature · 3 lines
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#10 lore raw

PATCH 2/15 package/e2fsprogs: fix fsck overwrite in HOST_DIR

In reply to Herve Codina
Quoted 2 lines · On Mon, 21 Jun 2021 16:11:17 +0200 Herve Codina <[email protected]>…

host-e2fsprog overwrites fsck program and some man related files previously installed by host-util-linux.

"The host-e2fsprogs package overwrites the fsck program and some manpages previously installed by the host-util-linux package".

Quoted 2 lines

This patch simply disable fsck in host-e2fsprog.

disableS

host-e2fsprogs

Quoted 1 line

host-e2fsprog is used to build final ext{2,3,4} images.

host-e2fsprogs

Quoted 1 line

The missing host-e2fsprog fsck tool (filesystem integrity check

host-e2fsprogs

Quoted 3 lines

tool) in HOST_DIR should not lead to issues.

With that:

Quoted 16 lines
---
 package/e2fsprogs/e2fsprogs.mk | 1 +
 1 file changed, 1 insertion(+)
package/e2fsprogs/e2fsprogs.mk +1 -0
index 0a69690e2f..bd147e737e 100644
@@ -27,6 +27,7 @@ HOST_E2FSPROGS_CONF_OPTS = \
 	--disable-defrag \
 	--disable-e2initrd-helper \
 	--disable-fuse2fs \
+	--disable-fsck \
 	--disable-libblkid \
 	--disable-libuuid \
 	--disable-testio-debug \
Signature · 3 lines
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#21 lore raw

PATCH 4/15 package/pkg-generic.mk: Fix .la files overwrite detection

In reply to Herve Codina
Quoted 10 lines · On Mon, 21 Jun 2021 16:11:19 +0200 Herve Codina <[email protected]>…

During per-package build, original .la files are modified by fixup-libtool-files calls.
But since fixup-libtool-files modifies files using sed --in-place, these modification are done using a temporary file and a call to rename. Rename breaks the hardlink to the original file and leave the temporary file in per-package TARGET dir.
As the original file is not modified, this is no longer considered as an overwrite.

To fix this detection, this patch simply considers the what is done

"the what" -> "that what"

Quoted 5 lines

by fixup-libtool-files is part of the original snapshot used to detect overwrites. And so, the original snapshot is taken after fixup-libtool-files call.

With that:

Thomas

Signature · 3 lines
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#44 lore raw

PATCH 7/15 package/apr-util: Use <PKG>_PER_PACKAGE_TWEAK_HOOKS

In reply to Herve Codina
Quoted 3 lines · On Mon, 21 Jun 2021 16:11:22 +0200 Herve Codina <[email protected]>…

rules.mk is generated by apr-util configure step and it is private to apr-util. The modification performed needs to be kept in <PKG>_POST_CONFIGURE_HOOKS.

Are you sure ? Both the tweak to build-1/libtool and rules.mk were introduced in commit 84b4c19e551288911a230c2b73e96bc6e2ed12f9 to solve per-package issues. So I would strongly suspect that both of them need to be moved to APR_UTIL_PER_PACKAGE_TWEAK_HOOKS.

Thomas

Signature · 3 lines
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#2 lore raw

PATCH 1/15 package/pkg-generic.mk: detect files overwritten in TARGET_DIR and HOST_DIR

In reply to Herve Codina

Herv?, All,

On 2021-06-21 16:11 +0200, Herve Codina spake thusly:

Quoted 32 lines

From: Thomas Petazzoni <[email protected]>

Without per-package directory support, a package can happily overwrite files installed by other packages. Indeed, because the build order between packages is always guaranteed, Buildroot will always produce the same output.

However, with per-package directory support, it is absolutely critical that a given package does not overwrite files already installed by another package, due to how the final aggregation is done to create the complete target/, staging/ and host/ folders. Unfortunately, we currently don't have anything in Buildroot that detects this situation.

We used to have check-uniq-files, but it was dropped in commit 2496189a4207173e4cd5bbab90256f911175ee57.

This commit is a new implementation of such a detection, which is based on calculating and verifying MD5 hashes of installed files: the calculation is done at the beginning of the configure step, the verification during the newly introduced "install" step that takes place after all installation steps.

Since preventing file overwrites is really only needed when per-package directory support is used, and due to this verification having some overhead, it is only enabled when BR2_PER_PACKAGE_DIRECTORIES=y. This additional verification cost is however not too bad as on average, with per-package directory support, the per-package target/ and host/ directories will contain less files than with a build that doesn't use per-package directory support. This helps a bit in mitigating the additional cost of this verification.

Quoted 5 lines

Note that we are not handling separately HOST_DIR and STAGING_DIR, like we're doing with the pkg_size_{before,after} functions. Instead, the verification on HOST_DIR walks down into the STAGING_DIR.

From here...

Quoted 4 lines

This commit is retreived from Thomas's work.
The first version was discussed
https://patchwork.ozlabs.org/project/buildroot/patch/20200430095249.782597-9-thomas.petazzoni at bootlin.com/ This new version was not already submitted by Thomas or I missed it.

... to here: this should have been a post-commit note, after the --- line...

Quoted 2 lines

---

... here.

Additionally, it would have been nice to summarise the changes made between the original submission and htis new one, and how the previous review was handled.

Quoted 19 lines
 package/pkg-generic.mk | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)
package/pkg-generic.mk +8 -0
index 45589bcbb4..bb9ff4150a 100644
@@ -102,6 +102,25 @@ define fixup-libtool-files
 endef
 endif
 
+# Functions to detect overwritten files
+
+ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y)
+# $(1): base directory to search in
+# $(2): suffix of file (optional)
+define pkg_detect_overwrite_before
+	LC_ALL=C find $(1) -type f -print0 | xargs -0 -r md5sum > $($(PKG)_DIR)/.files$(2).md5;
+endef

package/pkg-generic.mk define fixup-libtool-files · line 122

Quoted 10 lines
package/pkg-generic.mk +10 -0
+
+# $(1): base directory to search in
+# $(2): suffix of file (optional)
+define pkg_detect_overwrite_after
+	if test -s $($(PKG)_DIR)/.files$(2).md5 ; then \
+		LC_ALL=C md5sum --quiet -c $($(PKG)_DIR)/.files$(2).md5 || \
+		{ echo "ERROR: package $($(PKG)_NAME) has overwritten files installed by a previous package, aborting."; exit 1; } ; \
+	fi
+endef
+endif

And now I see that only part of the problem is handled; it only tries and detect files which content changes; it does not account for files which type did change. I.e. a symlink that was onverted over to a file, or the other way around.

So I think we could change the pkg_detect_overwrite_before macro thusly;

    LC_ALL=C find -L $(1) -type f -print0 |...

That should cover both cases, with just the esception that a symlink is replaced with a file of the same content (or the other way around). There's just a little quirk, though:

find: File system loop detected; ?host/usr? is part of the same file system loop as ?host?.

Meh, that's starting to be a bit hairy... We can just ignore it explicitly, maybe? For merged-usr in target, we should not have that problem, but there is still the option for a custom skeleton, where people could provide it as well...

So, to summarise: this patch does not cover all cases, but it is still acceptable, and brings in the necessary infra that we can later extend should the need arises.

Regards,
Yann E. MORIN.

Quoted 28 lines
package/pkg-generic.mk +4 -0
 # Functions to collect statistics about installed files
 
 # $(1): base directory to search in
@@ -235,6 +254,8 @@ $(BUILD_DIR)/%/.stamp_configured:
 	@$(call pkg_size_before,$(TARGET_DIR))
 	@$(call pkg_size_before,$(STAGING_DIR),-staging)
 	@$(call pkg_size_before,$(HOST_DIR),-host)
+	@$(call pkg_detect_overwrite_before,$(TARGET_DIR))
+	@$(call pkg_detect_overwrite_before,$(HOST_DIR),-host)
 	$(call fixup-libtool-files,$(NAME),$(STAGING_DIR))
 	$(foreach hook,$($(PKG)_PRE_CONFIGURE_HOOKS),$(call $(hook))$(sep))
 	$($(PKG)_CONFIGURE_CMDS)
@@ -360,6 +381,8 @@ $(BUILD_DIR)/%/.stamp_installed:
 	@$(call pkg_size_after,$(STAGING_DIR),-staging)
 	@$(call pkg_size_after,$(HOST_DIR),-host)
 	@$(call check_bin_arch)
+	@$(call pkg_detect_overwrite_after,$(TARGET_DIR))
+	@$(call pkg_detect_overwrite_after,$(HOST_DIR),-host)
 	$(Q)touch $@
 
 # Remove package sources
Signature · 6 lines
2.31.1

_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#23 lore raw

PATCH 4/15 package/pkg-generic.mk: Fix .la files overwrite detection

In reply to Herve Codina

On 2021-06-21 16:11 +0200, Herve Codina spake thusly:

Quoted 9 lines

During per-package build, original .la files are modified by fixup-libtool-files calls.
But since fixup-libtool-files modifies files using sed --in-place, these modification are done using a temporary file and a call to rename. Rename breaks the hardlink to the original file and leave the temporary file in per-package TARGET dir.
As the original file is not modified, this is no longer considered as an overwrite.

Quoted 4 lines

To fix this detection, this patch simply considers the what is done by fixup-libtool-files is part of the original snapshot used to detect overwrites. And so, the original snapshot is taken after fixup-libtool-files call.

Then this should be squashed together with the first patch, to avoid introducing the issue just to fix it a few patches down the series.

You should however add a note about that in the commit log of the first patch, of course, to explain why the overwrite ifnra is inserted after the .la tweaks.

So, I agree with the explanations, which make sense, but I disagree that it should be a separate patch...

Regards,
Yann E. MORIN.

Quoted 27 lines
---
 package/pkg-generic.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
package/pkg-generic.mk +1 -1
index 2499c94746..f9564831cc 100644
@@ -254,9 +254,9 @@ $(BUILD_DIR)/%/.stamp_configured:
 	@$(call pkg_size_before,$(TARGET_DIR))
 	@$(call pkg_size_before,$(STAGING_DIR),-staging)
 	@$(call pkg_size_before,$(HOST_DIR),-host)
+	$(call fixup-libtool-files,$(NAME),$(STAGING_DIR))
 	@$(call pkg_detect_overwrite_before,$(TARGET_DIR))
 	@$(call pkg_detect_overwrite_before,$(HOST_DIR),-host)
-	$(call fixup-libtool-files,$(NAME),$(STAGING_DIR))
 	$(foreach hook,$($(PKG)_PRE_CONFIGURE_HOOKS),$(call $(hook))$(sep))
 	$($(PKG)_CONFIGURE_CMDS)
 	$(foreach hook,$($(PKG)_POST_CONFIGURE_HOOKS),$(call $(hook))$(sep))
Signature · 6 lines
2.31.1

_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#3 lore raw

PATCH 1/15 package/pkg-generic.mk: detect files overwritten in TARGET_DIR and HOST_DIR

In reply to Yann E. MORIN

Hi,

On Mon, 21 Jun 2021 23:31:40 +0200
"Yann E. MORIN" <[email protected]> wrote:

[...]

Quoted 12 lines
Quoted 1 line

From here...

Quoted 4 lines

This commit is retreived from Thomas's work.
The first version was discussed
https://patchwork.ozlabs.org/project/buildroot/patch/20200430095249.782597-9-thomas.petazzoni at bootlin.com/ This new version was not already submitted by Thomas or I missed it.

... to here: this should have been a post-commit note, after the --- line...

Quoted 8 lines
Quoted 2 lines

---

... here.

Additionally, it would have been nice to summarise the changes made between the original submission and htis new one, and how the previous review was handled.

I moved the note after a '---' line and added:

   Compared to the first version, this patch has an improved commit message and
   generates the md5sum snapshot using
     'find -L $(1) -type f -print0 | xargs -0 -r md5sum > $($(PKG)_DIR)/.files$(2).md5;'
   instead of
     'cd $(1); LC_ALL=C find . -type f -exec md5sum {} \; > $($(PKG)_DIR)/.files$(2).md5'
Quoted 36 lines
Quoted 20 lines
 package/pkg-generic.mk | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)
package/pkg-generic.mk +8 -0
index 45589bcbb4..bb9ff4150a 100644
@@ -102,6 +102,25 @@ define fixup-libtool-files
 endef
 endif
 
+# Functions to detect overwritten files
+
+ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y)
+# $(1): base directory to search in
+# $(2): suffix of file (optional)
+define pkg_detect_overwrite_before
+	LC_ALL=C find $(1) -type f -print0 | xargs -0 -r md5sum > $($(PKG)_DIR)/.files$(2).md5;
+endef

package/pkg-generic.mk define fixup-libtool-files · line 122

Quoted 10 lines
package/pkg-generic.mk +10 -0
+
+# $(1): base directory to search in
+# $(2): suffix of file (optional)
+define pkg_detect_overwrite_after
+	if test -s $($(PKG)_DIR)/.files$(2).md5 ; then \
+		LC_ALL=C md5sum --quiet -c $($(PKG)_DIR)/.files$(2).md5 || \
+		{ echo "ERROR: package $($(PKG)_NAME) has overwritten files installed by a previous package, aborting."; exit 1; } ; \
+	fi
+endef
+endif  

And now I see that only part of the problem is handled; it only tries and detect files which content changes; it does not account for files which type did change. I.e. a symlink that was onverted over to a file, or the other way around.

Quoted 3 lines

So I think we could change the pkg_detect_overwrite_before macro thusly;

    LC_ALL=C find -L $(1) -type f -print0 |...

I added '-L' option.

Quoted 8 lines

That should cover both cases, with just the esception that a symlink is replaced with a file of the same content (or the other way around). There's just a little quirk, though:

find: File system loop detected; ?host/usr? is part of the same file system loop as ?host?.

Quoted 9 lines

Meh, that's starting to be a bit hairy... We can just ignore it explicitly, maybe? For merged-usr in target, we should not have that problem, but there is still the option for a custom skeleton, where people could provide it as well...

So, to summarise: this patch does not cover all cases, but it is still acceptable, and brings in the necessary infra that we can later extend should the need arises.

Ok, v2 with the changes I mentioned will be sent.

Thanks for the review,
Herv? Codina

Signature · 3 lines
Herv? Codina, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#16 lore raw

PATCH 3/15 package/pkg-generic.mk: Remove Info documents dir entry

In reply to Thomas Petazzoni

Hi,

Quoted 16 lines · On Mon, 21 Jun 2021 22:51:20 +0200 Thomas Petazzoni…
Quoted 13 lines · On Mon, 21 Jun 2021 16:11:18 +0200 Herve Codina <[email protected]>…

Some packages (autotools for instance) install documentation files using install-info. This program adds an entry in the Info directory file (share/info/dir) and this causes TARGET_DIR and/or HOST_DIR overwrite.

In order to avoid this overwrite this patch removes the Info directory file right after any installation.

---
 package/pkg-generic.mk | 3 +++
 1 file changed, 3 insertions(+)
Quoted 11 lines

package/pkg-generic.mk $(BUILD_DIR)/%/.stamp_host_installed: · line 283

Quoted 9 lines
package/pkg-generic.mk +1 -0
index bb9ff4150a..2499c94746 100644
@@ -280,6 +280,7 @@ $(BUILD_DIR)/%/.stamp_host_installed:
 	$(foreach hook,$($(PKG)_PRE_INSTALL_HOOKS),$(call $(hook))$(sep))
 	+$($(PKG)_INSTALL_CMDS)
 	$(foreach hook,$($(PKG)_POST_INSTALL_HOOKS),$(call $(hook))$(sep))
+	$(Q) rm -f $(HOST_DIR)/share/info/dir  

No space between $(Q) and rm. This should perhaps use $(RM) in fact.

Space removed

Quoted 6 lines

However, I'm not a huge fan of having this right in the middle of the infrastructure. It feels like a small detail that gets handled in the middle of super generic infrastructure code.

The issue is that I don't really have a good alternative proposal :-/

Maybe using a macro defined closed to fixup-libtool-files and calling this macro here instead of '$(Q)rm ...' will help.
Do you think it will
be better ?

Quoted 3 lines

Instead of removing that file, ignore it in the overwrite detection, perhaps?

This add a little complexity in overwrite detection (filter out) and I prefer having overwrite detection quite stupid. It checks for overwrites without any exception.
Adding exception now in the detection mechanism is opening the door to more and more exceptions.

Herv?

Signature · 3 lines
Herv? Codina, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#4 lore raw

PATCH 1/15 package/pkg-generic.mk: detect files overwritten in TARGET_DIR and HOST_DIR

In reply to Herve Codina
Quoted 3 lines · On Tue, 22 Jun 2021 09:40:52 +0200 Herve Codina <[email protected]>…

Compared to the first version, this patch has an improved commit message and generates the md5sum snapshot using

     'find -L $(1) -type f -print0 | xargs -0 -r md5sum > $($(PKG)_DIR)/.files$(2).md5;'

But is this better ? Due to not cd-ing into the directory, you will have full absolute paths in the .md5 file, and adding LC_ALL=C is quite customary too. Ditto for using | xargs instead of -exec.

Quoted 2 lines
   instead of
     'cd $(1); LC_ALL=C find . -type f -exec md5sum {} \; > $($(PKG)_DIR)/.files$(2).md5'

All in all this version, looked better. Any reason for the change ?

Thomas

Signature · 3 lines
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#24 lore raw

PATCH 4/15 package/pkg-generic.mk: Fix .la files overwrite detection

In reply to Yann E. MORIN

Hi,

Quoted 18 lines · On Mon, 21 Jun 2021 23:42:23 +0200 "Yann E. MORIN" <[email protected]>…

On 2021-06-21 16:11 +0200, Herve Codina spake thusly:

Quoted 9 lines

During per-package build, original .la files are modified by fixup-libtool-files calls.
But since fixup-libtool-files modifies files using sed --in-place, these modification are done using a temporary file and a call to rename. Rename breaks the hardlink to the original file and leave the temporary file in per-package TARGET dir.
As the original file is not modified, this is no longer considered as an overwrite.

Quoted 4 lines

To fix this detection, this patch simply considers the what is done by fixup-libtool-files is part of the original snapshot used to detect overwrites. And so, the original snapshot is taken after fixup-libtool-files call.

Then this should be squashed together with the first patch, to avoid introducing the issue just to fix it a few patches down the series.

Quoted 7 lines

You should however add a note about that in the commit log of the first patch, of course, to explain why the overwrite ifnra is inserted after the .la tweaks.

So, I agree with the explanations, which make sense, but I disagree that it should be a separate patch...

Well, I have seen this when I created the patches.
I kept them separate because on the first patch, I introduced the tool to check the overwrites and i would like it to take its snapshot as soon as possible in the build sequence (ie right after collecting dependencies files and taking snapshots for current package statistics). Then I fixed the issue seen by the overwrites detection and I put at the same level fixing host-e2fsprogs, fixing .la files or a bit later fixing python with one patch per fix to detail (or try to detail) the issue and the way I fixed it.

Squashing the 2 patches leads to one patch that introduces the tool and fixes one of the issues detected by the tool. What about the others issues detected ? Squash also together with the first patch ? I think it will produce a huge patch quite complicate to understand even with all individual commit message squashed.

However, that being said, I can squash this patch (Fix .la files overwrite detection) with the 1st one (detect files overwritten in TARGET_DIR and HOST_DIR) if you still think it will be better.

Regards,
Herv? Codina

Signature · 3 lines
Herv? Codina, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#17 lore raw

PATCH 3/15 package/pkg-generic.mk: Remove Info documents dir entry

In reply to Herve Codina
Quoted 9 lines · On Tue, 22 Jun 2021 10:43:43 +0200 Herve Codina <[email protected]>…
Quoted 5 lines

However, I'm not a huge fan of having this right in the middle of the infrastructure. It feels like a small detail that gets handled in the middle of super generic infrastructure code.

The issue is that I don't really have a good alternative proposal :-/

Maybe using a macro defined closed to fixup-libtool-files and calling this macro here instead of '$(Q)rm ...' will help.
Do you think it will be better ?

Either a macro, or a list of files that are removed, perhaps?

Quoted 5 lines

This add a little complexity in overwrite detection (filter out) and I prefer having overwrite detection quite stupid. It checks for overwrites without any exception.
Adding exception now in the detection mechanism is opening the door to more and more exceptions.

Yes, I understand your argument. As I stated: I'm also not sure which solution to propose here. I was just not a big fan of this removal of one specific file, there, in the middle of a highly generic piece of infrastructure.

Thomas

Signature · 3 lines
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#32 lore raw

PATCH 5/15 package/pkg-generic.mk: Perform .la files fixup in per-package HOST_DIR

In reply to Thomas Petazzoni

Hi,

Quoted 9 lines · On Mon, 21 Jun 2021 22:48:12 +0200 Thomas Petazzoni…
Quoted 3 lines · On Mon, 21 Jun 2021 16:11:20 +0200 Herve Codina <[email protected]>…

fixup-libtool-files was called on per-package STAGING_DIR. Some host-xxxx packages have their .la files with directories pointing outside their own per-package directory.

Was this causing a practical issue, or is being done just to be cleaner?

I didn't see any practical issue on the builds I did and I didn't try to find any use-case that leads to an issue.
So this patch is to be cleaner.

Herv? Codina

Signature · 3 lines
Herv? Codina, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#45 lore raw

PATCH 7/15 package/apr-util: Use <PKG>_PER_PACKAGE_TWEAK_HOOKS

In reply to Thomas Petazzoni

Hi,

Quoted 12 lines · On Mon, 21 Jun 2021 22:56:32 +0200 Thomas Petazzoni…
Quoted 3 lines · On Mon, 21 Jun 2021 16:11:22 +0200 Herve Codina <[email protected]>…

rules.mk is generated by apr-util configure step and it is private to apr-util. The modification performed needs to be kept in <PKG>_POST_CONFIGURE_HOOKS.

Are you sure ? Both the tweak to build-1/libtool and rules.mk were introduced in commit 84b4c19e551288911a230c2b73e96bc6e2ed12f9 to solve per-package issues. So I would strongly suspect that both of them need to be moved to APR_UTIL_PER_PACKAGE_TWEAK_HOOKS.

Yes, both of them need to be done but
rules.mk is generated by apr-util itself at configure step.
This file is not present (not yet generated) when APR_UTIL_PER_PACKAGE_TWEAK_HOOKS are called.

XXX_PER_PACKAGE_TWEAK_HOOKS are for tweaking files external to current package that leads to overwrite -> Ok for build-1/libtool

rules.mk is not correct for per-package build but it is generated by apr-util package for apr-util package without any overwrites -> Keep tweak in <PKG>_POST_CONFIGURE_HOOKS.

Herv?

Signature · 3 lines
Herv? Codina, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#25 lore raw

PATCH 4/15 package/pkg-generic.mk: Fix .la files overwrite detection

In reply to Herve Codina

Herv?, All,

On 2021-06-22 11:31 +0200, Herve Codina spake thusly:

Quoted 26 lines
Quoted 17 lines · On Mon, 21 Jun 2021 23:42:23 +0200 "Yann E. MORIN" <[email protected]>…

On 2021-06-21 16:11 +0200, Herve Codina spake thusly:

Quoted 9 lines

During per-package build, original .la files are modified by fixup-libtool-files calls.
But since fixup-libtool-files modifies files using sed --in-place, these modification are done using a temporary file and a call to rename. Rename breaks the hardlink to the original file and leave the temporary file in per-package TARGET dir.
As the original file is not modified, this is no longer considered as an overwrite.

Quoted 4 lines

To fix this detection, this patch simply considers the what is done by fixup-libtool-files is part of the original snapshot used to detect overwrites. And so, the original snapshot is taken after fixup-libtool-files call.

Then this should be squashed together with the first patch, to avoid introducing the issue just to fix it a few patches down the series.

Quoted 7 lines

You should however add a note about that in the commit log of the first patch, of course, to explain why the overwrite ifnra is inserted after the .la tweaks.

So, I agree with the explanations, which make sense, but I disagree that it should be a separate patch...

Quoted 9 lines

Well, I have seen this when I created the patches.
I kept them separate because on the first patch, I introduced the tool to check the overwrites and i would like it to take its snapshot as soon as possible in the build sequence (ie right after collecting dependencies files and taking snapshots for current package statistics). Then I fixed the issue seen by the overwrites detection and I put at the same level fixing host-e2fsprogs, fixing .la files or a bit later fixing python with one patch per fix to detail (or try to detail) the issue and the way I fixed it.

But really, it *is* the first patch that introduces the issue, so it should be fixed from the onset, rather than after-the-fact.

Quoted 2 lines

Squashing the 2 patches leads to one patch that introduces the tool and fixes one of the issues detected by the tool.

Sorry, but this patch (4/15) is fixing an issue introduced by the first patch, with:

        @$(call pkg_size_before,$(TARGET_DIR))
        @$(call pkg_size_before,$(STAGING_DIR),-staging)
        @$(call pkg_size_before,$(HOST_DIR),-host)
    +   @$(call pkg_detect_overwrite_before,$(TARGET_DIR))
    +   @$(call pkg_detect_overwrite_before,$(HOST_DIR),-host)
        $(call fixup-libtool-files,$(NAME),$(STAGING_DIR))
        $(foreach hook,$($(PKG)_PRE_CONFIGURE_HOOKS),$(call $(hook))$(sep))
        $($(PKG)_CONFIGURE_CMDS)

... when it should directly be:

        @$(call pkg_size_before,$(STAGING_DIR),-staging)
        @$(call pkg_size_before,$(HOST_DIR),-host)
        $(call fixup-libtool-files,$(NAME),$(STAGING_DIR))
    +   @$(call pkg_detect_overwrite_before,$(TARGET_DIR))
    +   @$(call pkg_detect_overwrite_before,$(HOST_DIR),-host)
        $(foreach hook,$($(PKG)_PRE_CONFIGURE_HOOKS),$(call $(hook))$(sep))
        $($(PKG)_CONFIGURE_CMDS)
        $(foreach hook,$($(PKG)_POST_CONFIGURE_HOOKS),$(call $(hook))$(sep))

And as I said, the commit log should explain why the 'overwrite' calls are inserted after the .la fixup.

Quoted 4 lines

What about the others issues
detected ? Squash also together with the first patch ? I think it will produce a huge patch quite complicate to understand even with all individual commit message squashed.

Ideally, I would say a series should first fix the issues, then introduce the tooling.

Otherwise, if only the first patch(es) are applied, the tree is broken:
indeed the tooling has been applied, and thus is used, but the issues are still there.

Also, it is usually easier and less controversial to get fixes applied, than new tooling.

Quoted 3 lines

However, that being said, I can squash this patch (Fix .la files overwrite detection) with the 1st one (detect files overwritten in TARGET_DIR and HOST_DIR) if you still think it will be better.

Yes, I still think that it is better.

Regards,
Yann E. MORIN.

Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#5 lore raw

PATCH 1/15 package/pkg-generic.mk: detect files overwritten in TARGET_DIR and HOST_DIR

In reply to Thomas Petazzoni
Quoted 10 lines · Thomas Petazzoni
Quoted 3 lines · On Tue, 22 Jun 2021 09:40:52 +0200 Herve Codina <[email protected]>…

Compared to the first version, this patch has an improved commit message and generates the md5sum snapshot using
'find -L $(1) -type f -print0 | xargs -0 -r md5sum > $($(PKG)_DIR)/.files$(2).md5;'

But is this better ? Due to not cd-ing into the directory, you will have full absolute paths in the .md5 file, and adding LC_ALL=C is quite customary too. Ditto for using | xargs instead of -exec.

xargs will run only start one md5sum process if the command line size permits it, whereas "-exec md5sum {} ;" will start one md5sum per file.

An alternative is "-exec md5sum {} +", which is essentially xargs-like.

#33 lore raw

PATCH 5/15 package/pkg-generic.mk: Perform .la files fixup in per-package HOST_DIR

In reply to Herve Codina

Herv?, All,

On 2021-06-21 16:11 +0200, Herve Codina spake thusly:

Quoted 6 lines

fixup-libtool-files was called on per-package STAGING_DIR. Some host-xxxx packages have their .la files with directories pointing outside their own per-package directory.

This commit, calling fixup-libtool-files on HOST_DIR, fixes this issue.

Usually, I don't much like that we have a fix for some theoretical bug, but in this case I think it just does make sense.

And because a .la file is only used to find the libraries at link time, not runtime, this is has most probably gone unnoticed because, at build time, the libraries are present in the out-of-package tree, and are present at runtime in the per-package tree...

Quoted 10 lines
---
 package/pkg-generic.mk | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
package/pkg-generic.mk +0 -0
index f9564831cc..e437050175 100644

package/pkg-generic.mk endif · line 102

Quoted 7 lines
package/pkg-generic.mk +2 -1
@@ -97,7 +97,8 @@ endif
 # $2: staging directory of the package
 ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y)
 define fixup-libtool-files
-	$(Q)find $(2)/usr/lib* -name "*.la" | xargs --no-run-if-empty \
+	$(Q)find $(2) \( -path '$(2)/lib*' -o -path '$(2)/usr/lib*' \) \
+		-name "*.la" | xargs --no-run-if-empty \
 		$(SED) "s:$(PER_PACKAGE_DIR)/[^/]\+/:$(PER_PACKAGE_DIR)/$(1)/:g"

While at it, can you switch over to using find's -print0 and xarg's -0 ?

package/pkg-generic.mk $(BUILD_DIR)/%/.stamp_configured: · line 259

Quoted 8 lines
package/pkg-generic.mk +1 -0
 endef
 endif
@@ -254,6 +255,7 @@ $(BUILD_DIR)/%/.stamp_configured:
 	@$(call pkg_size_before,$(TARGET_DIR))
 	@$(call pkg_size_before,$(STAGING_DIR),-staging)
 	@$(call pkg_size_before,$(HOST_DIR),-host)
+	$(call fixup-libtool-files,$(NAME),$(HOST_DIR))
 	$(call fixup-libtool-files,$(NAME),$(STAGING_DIR))

Regards,
Yann E. MORIN.

Quoted 9 lines
package/pkg-generic.mk +0 -0
 	@$(call pkg_detect_overwrite_before,$(TARGET_DIR))
 	@$(call pkg_detect_overwrite_before,$(HOST_DIR),-host)
Signature · 6 lines
2.31.1

_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#26 lore raw

PATCH 4/15 package/pkg-generic.mk: Fix .la files overwrite detection

In reply to Yann E. MORIN
Quoted 2 lines · On Tue, 22 Jun 2021 11:56:09 +0200 "Yann E. MORIN" <[email protected]>…

Sorry, but this patch (4/15) is fixing an issue introduced by the first patch, with:

I agree in principle. But here, it's really a new requirement that we are adding on packages, and we *know* that even with Herv? patches there will be some other packages that will need fixing.

So to me it's not really like introducing a regression in a patch, and fixing it later.

Quoted 7 lines
Quoted 4 lines

What about the others issues
detected ? Squash also together with the first patch ? I think it will produce a huge patch quite complicate to understand even with all individual commit message squashed.

Ideally, I would say a series should first fix the issues, then introduce the tooling.

No, please keep one series, but having the fixes *before* we introduce the check. And as stated above, the fixes will not fix all problems, they will only fix the problems we know about. More problems will be detected by the autobuilders thanks to the overwrite check.

Quoted 5 lines
Quoted 3 lines

However, that being said, I can squash this patch (Fix .la files overwrite detection) with the 1st one (detect files overwritten in TARGET_DIR and HOST_DIR) if you still think it will be better.

Yes, I still think that it is better.

No, please don't squash, have fixes added before the overwrite check instead.

Thomas

Signature · 3 lines
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#6 lore raw

PATCH 1/15 package/pkg-generic.mk: detect files overwritten in TARGET_DIR and HOST_DIR

In reply to Nicolas Cavallari

Nicolas, All,

On 2021-06-22 11:57 +0200, Nicolas Cavallari spake thusly:

Quoted 10 lines · Thomas Petazzoni
Quoted 9 lines
Quoted 3 lines · On Tue, 22 Jun 2021 09:40:52 +0200 Herve Codina <[email protected]>…

Compared to the first version, this patch has an improved commit message and generates the md5sum snapshot using

     'find -L $(1) -type f -print0 | xargs -0 -r md5sum > $($(PKG)_DIR)/.files$(2).md5;'

But is this better ? Due to not cd-ing into the directory, you will have full absolute paths in the .md5 file,

But is that even an issue? Those paths are only used for one per-package directory; they do not follow files around into another package's ppd.

So, relative or absolute, we don't much care.

Quoted 2 lines
Quoted 2 lines

and adding LC_ALL=C is quite
customary too.

In this case, do we also even care or need it? I pretty sure the output of find is not sorted at all; it most probably depend on the order of dentries in the directory, and those have absolutely no ordering guarantee...

Note that if you really want to use LC_ALL, passing it in front of 'find' will not make LC_ALL available to anythin after the pipeline, so you'd still have xargs (and md5sum) run without it...

Quoted 3 lines
Quoted 1 line

Ditto for using | xargs instead of -exec.

xargs will run only start one md5sum process if the command line size permits it, whereas "-exec md5sum {} ;" will start one md5sum per file.

Agreed. But I think this is exactly what Thomas suggested to use, right?

Quoted 1 line

An alternative is "-exec md5sum {} +", which is essentially xargs-like.

I once used that in the past, and it breaks on non-GNU find, because '+' is a GNUism. So I think using xargs is still better.

Regards,
Yann E. MORIN.

Quoted 4 lines

_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#27 lore raw

PATCH 4/15 package/pkg-generic.mk: Fix .la files overwrite detection

In reply to Thomas Petazzoni

Thomas, All,

On 2021-06-22 12:12 +0200, Thomas Petazzoni spake thusly:

Quoted 7 lines
Quoted 2 lines · On Tue, 22 Jun 2021 11:56:09 +0200 "Yann E. MORIN" <[email protected]>…

Sorry, but this patch (4/15) is fixing an issue introduced by the first patch, with:

I agree in principle. But here, it's really a new requirement that we are adding on packages, and we *know* that even with Herv? patches there will be some other packages that will need fixing.

Sorry, but this patch is fixing the infra, not the packages.

Quoted 2 lines

So to me it's not really like introducing a regression in a patch, and fixing it later.

I never said it was a regression. ;-)

Quoted 10 lines
Quoted 7 lines
Quoted 4 lines

What about the others issues
detected ? Squash also together with the first patch ? I think it will produce a huge patch quite complicate to understand even with all individual commit message squashed.

Ideally, I would say a series should first fix the issues, then introduce the tooling.

No, please keep one series, but having the fixes *before* we introduce the check.

Exactly; I never said to split the series into two series. It should still be one.

Quoted 3 lines

And as stated above, the fixes will not fix all problems, they will only fix the problems we know about. More problems will be detected by the autobuilders thanks to the overwrite check.

And this is totally OK. [0]

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

However, that being said, I can squash this patch (Fix .la files overwrite detection) with the 1st one (detect files overwritten in TARGET_DIR and HOST_DIR) if you still think it will be better.

Yes, I still think that it is better.

No, please don't squash, have fixes added before the overwrite check instead.

Well, I still disagree, because this patch really fixes an issue introduced *in the infra* by the first patch.

Regards,
Yann E. MORIN.

[0] but brace for impact! ;-]

Quoted 5 lines

Thomas

Signature · 3 lines
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#11 lore raw

PATCH 2/15 package/e2fsprogs: fix fsck overwrite in HOST_DIR

In reply to Thomas Petazzoni

Hi,

Quoted 9 lines · On Mon, 21 Jun 2021 22:52:54 +0200 Thomas Petazzoni…
Quoted 2 lines · On Mon, 21 Jun 2021 16:11:17 +0200 Herve Codina <[email protected]>…

host-e2fsprog overwrites fsck program and some man related files previously installed by host-util-linux.

"The host-e2fsprogs package overwrites the fsck program and some manpages previously installed by the host-util-linux package".

Ok, changed

Quoted 6 lines
Quoted 2 lines

This patch simply disable fsck in host-e2fsprog.

disableS

host-e2fsprogs

Changed and s/host-e2fsprog/host-e2fsprogs/g

Quoted 9 lines
Quoted 2 lines

host-e2fsprog is used to build final ext{2,3,4} images.

host-e2fsprogs

Quoted 1 line

The missing host-e2fsprog fsck tool (filesystem integrity check

host-e2fsprogs

Quoted 7 lines
Quoted 3 lines

tool) in HOST_DIR should not lead to issues.

With that:

Added.
I will remove this Reviewed-by in my v2 patch if more modifications are needed for this patch.

Thanks for the review,
Herv? Codina

Signature · 3 lines
Herv? Codina, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#52 lore raw

PATCH 9/15 package/pkg-python: Remove _sysconfigdata*.pyc files when _sysconfigdata*.py are changed

In reply to Thomas Petazzoni
Quoted 15 lines · On Mon, 21 Jun 2021 17:12:02 +0200 Thomas Petazzoni…
Quoted 12 lines · On Mon, 21 Jun 2021 16:11:24 +0200 Herve Codina <[email protected]>…

In order to avoid _sysconfigdata*.pyc overwrites when they are generated based on _sysconfigdata*.py changes, this commit simply removes _sysconfigdata*.pyc whenever _sysconfigdata*.py are changed.

As they are removed, overwrite detection will no longer trig and coherency between the two files (.py and .pyc) is ensured.

---
 package/pkg-python.mk | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
Quoted 13 lines

package/pkg-python.mk ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y) · line 99

Quoted 10 lines
package/pkg-python.mk +1 -1
index 59a48e5a87..b3fde77da5 100644
@@ -96,7 +96,9 @@ ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y)
 define PKG_PYTHON_FIXUP_SYSCONFIGDATA
 	find $(HOST_DIR)/lib/python* $(STAGING_DIR)/usr/lib/python* \
 		-name "_sysconfigdata*.py" | xargs --no-run-if-empty \
-		$(SED) "s:$(PER_PACKAGE_DIR)/[^/]\+/:$(PER_PACKAGE_DIR)/$($(PKG)_NAME)/:g"
+		$(SED) "s:$(PER_PACKAGE_DIR)/[^/]\+/:$(PER_PACKAGE_DIR)/$($(PKG)_NAME)/:g" ;\  

The semicolon + backslash is not needed, these two commands can (I believe) be executed as separate shell commands.

Yes, they can be executed in separate shell commands -> I removed the semicolon and backslash.

Quoted 5 lines

package/pkg-python.mk ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y) · line 101

Quoted 3 lines
package/pkg-python.mk +2 -0
+	find $(HOST_DIR)/lib/python* $(STAGING_DIR)/usr/lib/python* \
+		-name "_sysconfigdata*.pyc" -print0 | xargs -0 -r rm -f  

Do we need a --no-run-if-empty on the xargs ? Or perhaps it should be

It is cleaner to avoid running rm is there is nothing to remove. So, I prefer keeping the xargs -r option.

Quoted 6 lines

using find directly, i.e:

	find $(HOST_DIR)/lib/python* $(STAGING_DIR)/usr/lib/python* \
		-name "_sysconfigdata*.pyc" -exec rm -f {} \;

(or something like that)

Using "-exec rm -f {} \;" will lead to start as many rm as there are files to remove whereas xargs will run only one rm.

Herv?

Signature · 3 lines
Herv? Codina, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#22 lore raw

PATCH 4/15 package/pkg-generic.mk: Fix .la files overwrite detection

In reply to Thomas Petazzoni
Quoted 12 lines · On Mon, 21 Jun 2021 22:54:11 +0200 Thomas Petazzoni…
Quoted 9 lines · On Mon, 21 Jun 2021 16:11:19 +0200 Herve Codina <[email protected]>…

During per-package build, original .la files are modified by fixup-libtool-files calls.
But since fixup-libtool-files modifies files using sed --in-place, these modification are done using a temporary file and a call to rename. Rename breaks the hardlink to the original file and leave the temporary file in per-package TARGET dir.
As the original file is not modified, this is no longer considered as an overwrite.

Quoted 3 lines
Quoted 1 line

To fix this detection, this patch simply considers the what is done

"the what" -> "that what"

Fixed.

Quoted 10 lines
Quoted 6 lines

by fixup-libtool-files is part of the original snapshot used to detect overwrites. And so, the original snapshot is taken after fixup-libtool-files call.

With that:

Thanks for this first review.

Herv?

Signature · 3 lines
Herv? Codina, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#12 lore raw

PATCH 2/15 package/e2fsprogs: fix fsck overwrite in HOST_DIR

In reply to Herve Codina

Herv?, All,

On 2021-06-21 16:11 +0200, Herve Codina spake thusly:

Quoted 8 lines

host-e2fsprog overwrites fsck program and some man related files previously installed by host-util-linux.

This patch simply disable fsck in host-e2fsprog.

host-e2fsprog is used to build final ext{2,3,4} images. The missing host-e2fsprog fsck tool (filesystem integrity check tool) in HOST_DIR should not lead to issues.

And the target variant is not susceptible to that issue, because e2fsprogs' fsck depends on !BR2_PACKAGE_UTIL_LINUX_FSCK. So we're clean there.

Except for the comments by Thomas, you can carry my:

when you respin this patch.

Regards,
Yann E. MORIN.

Quoted 24 lines
---
 package/e2fsprogs/e2fsprogs.mk | 1 +
 1 file changed, 1 insertion(+)
package/e2fsprogs/e2fsprogs.mk +1 -0
index 0a69690e2f..bd147e737e 100644
@@ -27,6 +27,7 @@ HOST_E2FSPROGS_CONF_OPTS = \
 	--disable-defrag \
 	--disable-e2initrd-helper \
 	--disable-fuse2fs \
+	--disable-fsck \
 	--disable-libblkid \
 	--disable-libuuid \
 	--disable-testio-debug \
Signature · 6 lines
2.31.1

_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#18 lore raw

PATCH 3/15 package/pkg-generic.mk: Remove Info documents dir entry

In reply to Thomas Petazzoni

Thomas, Herv?, All,

On 2021-06-22 11:34 +0200, Thomas Petazzoni spake thusly:

Quoted 7 lines
Quoted 5 lines · On Tue, 22 Jun 2021 10:43:43 +0200 Herve Codina <[email protected]>…
Quoted 5 lines

However, I'm not a huge fan of having this right in the middle of the infrastructure. It feels like a small detail that gets handled in the middle of super generic infrastructure code.

The issue is that I don't really have a good alternative proposal :-/

I too am not too fond of this, but I don't have an obvious better solution in mind...

Quoted 5 lines
Quoted 3 lines

Maybe using a macro defined closed to fixup-libtool-files and calling this macro here instead of '$(Q)rm ...' will help.
Do you think it will be better ?

Either a macro, or a list of files that are removed, perhaps?

I think a list+macro would be a better solution.

Alternatively, we could append to post-install hooks, like we do for pre-configure hooks in the autotools infra for autoreconf et al., for example...

    # Outside generic-package-inner:

    # $1: base directory (target, staging, host)
    define remove-conflicting-useless-files
        $(Q)$(RM) -rf $(patsubst %, $(1)/%, $($(PKG)_DROP_FILES_OR_DIRS))
    endef
    define REMOVE_CONFLICTING_USELESS_FILES_IN_HOST
        $(call remove-conflicting-useless-files,$(HOST_DIR))
    endef
    define REMOVE_CONFLICTING_USELESS_FILES_IN_STAGING
        $(call remove-conflicting-useless-files,$(STAGING_DIR))
    endef
    define REMOVE_CONFLICTING_USELESS_FILES_IN_TARGET
        $(call remove-conflicting-useless-files,$(TARGET_DIR))
    endef


    # In generic-package-inner:

    $(2)_DROP_FILES_OR_DIRS += /share/info/dir

    # For host packages:
    $(2)_POST_INSTALL_HOOKS += REMOVE_CONFLICTING_USELESS_FILES_IN_HOST

    # For target packages:
    $(2)_POST_INSTALL_STAGING_HOOKS += REMOVE_CONFLICTING_USELESS_FILES_IN_STAGING
    $(2)_POST_INSTALL_TARGET_HOOKS += REMOVE_CONFLICTING_USELESS_FILES_IN_TARGET

This way, it also paves the way to add other hooks to actually fix some files instead of removing them...

For example, assume that packages (e.g. 'wonders') can register themselves against another package (e.g. 'manager') by appending a line to a text file, like so (over-simplified, of course):

echo "name=wonders path=/usr/lib/wonders/v42" >>/usr/share/manager/registry

Then we could have a hook that detects that, extracts the delta installed by the package, stash that somewhere in staging, and have a target-finalise hook that aggregates all the packages in the end.

Quoted 10 lines
Quoted 5 lines

This add a little complexity in overwrite detection (filter out) and I prefer having overwrite detection quite stupid. It checks for overwrites without any exception.
Adding exception now in the detection mechanism is opening the door to more and more exceptions.

Yes, I understand your argument. As I stated: I'm also not sure which solution to propose here. I was just not a big fan of this removal of one specific file, there, in the middle of a highly generic piece of infrastructure.

Yes, this bothered me too...

Regards,
Yann E. MORIN.

Quoted 9 lines

Thomas

Signature · 7 lines
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#37 lore raw

PATCH 6/15 package/pkg-generic.mk: Introduce <PKG>_PER_PACKAGE_TWEAK_HOOKS

In reply to Herve Codina

Herv?, All,

On 2021-06-21 16:11 +0200, Herve Codina spake thusly:

Quoted 7 lines

This hook will be used by package infrastucture or some packages to tweak external files (ie files not provides by the package itself) when we build with BR2_PER_PACKAGE_DIRECTORIES set.

This allows to have these tweaks in a specific hook instead of <PKG>_POST_CONFIGURE_HOOK. And so we can call these specific tweaks whenever we need them.

I'm a bit confused, because those new hooks are even before _PRE_CONFIGURE_HOOKS, so smeantically, they are different from _POST_CONFIGURE_HOOKS...

So I fail to see how this articulates with POST_CONFIGURE_HOOKS. Looking at the next patch, about apr-utils, does not explain why the existing apr-utils fixups need to change, with this new hooks...

Ah, yes, yes, now I see. OK, so here's a suggestion for a commit log:

package/pkg-generic: add early pre-configure hooks

Currently, when a package needs to modify files it inherits from its dependencies, because they contain paths, we can only do that in a pre- or post-configure hook.

However, whatever is done as part of those hooks, will be accounted to the package itself, and thus will trigger file-overwrite detection.

So, we need a way to be able to actually modify files before we start monitoring for them.

We introduce a new set of hooks that an individual package can set, or that a package infra can set, and that are called right before we snapshot the state of target, and host (to which staging belongs),

Quoted 10 lines
---
 package/pkg-generic.mk | 2 ++
 1 file changed, 2 insertions(+)
package/pkg-generic.mk +0 -0
index e437050175..4069d2cf3c 100644

package/pkg-generic.mk $(BUILD_DIR)/%/.stamp_configured: · line 260

Quoted 4 lines
package/pkg-generic.mk +1 -0
@@ -257,6 +257,7 @@ $(BUILD_DIR)/%/.stamp_configured:
 	@$(call pkg_size_before,$(HOST_DIR),-host)
 	$(call fixup-libtool-files,$(NAME),$(HOST_DIR))
 	$(call fixup-libtool-files,$(NAME),$(STAGING_DIR))
+	$(foreach hook,$($(PKG)_PER_PACKAGE_TWEAK_HOOKS),$(call $(hook))$(sep))

I am going to bikeshed on the naming of this variable (which is a good sing that I am pretty much OK with the feature).

I think $(2)_EARLY_PRE_CONFIGURE_HOOKS is more sensible.

package/pkg-generic.mk $(2)_PRE_LEGAL_INFO_HOOKS ?= · line 848

Quoted 8 lines
package/pkg-generic.mk +1 -0
 	@$(call pkg_detect_overwrite_before,$(TARGET_DIR))
 	@$(call pkg_detect_overwrite_before,$(HOST_DIR),-host)
 	$(foreach hook,$($(PKG)_PRE_CONFIGURE_HOOKS),$(call $(hook))$(sep))
@@ -844,6 +845,7 @@ $(2)_PRE_LEGAL_INFO_HOOKS       ?=
 $(2)_POST_LEGAL_INFO_HOOKS      ?=
 $(2)_TARGET_FINALIZE_HOOKS      ?=
 $(2)_ROOTFS_PRE_CMD_HOOKS       ?=
+$(2)_PER_PACKAGE_TWEAK_HOOKS    ?=

Actually, this is totally useless: if the variable is unset, this would set it to empty, which for 'make' is totally identical (an unset variable can be expanded, and the expansion is the emopty string).

I know you are just adhering to existing code, and so you should indeed keep it for the next series, but a cleanup in the infra would be welcome too! ;-)

Regards,
Yann E. MORIN.

Quoted 9 lines
package/pkg-generic.mk +0 -0
 ifeq ($$($(2)_TYPE),target)
 ifneq ($$(HOST_$(2)_KCONFIG_VAR),)
Signature · 6 lines
2.31.1

_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#46 lore raw

PATCH 7/15 package/apr-util: Use <PKG>_PER_PACKAGE_TWEAK_HOOKS

In reply to Herve Codina

Herv?, All,

On 2021-06-22 11:47 +0200, Herve Codina spake thusly:

Quoted 20 lines
Quoted 12 lines · On Mon, 21 Jun 2021 22:56:32 +0200 Thomas Petazzoni…
Quoted 3 lines · On Mon, 21 Jun 2021 16:11:22 +0200 Herve Codina <[email protected]>…

rules.mk is generated by apr-util configure step and it is private to apr-util. The modification performed needs to be kept in <PKG>_POST_CONFIGURE_HOOKS.

Are you sure ? Both the tweak to build-1/libtool and rules.mk were introduced in commit 84b4c19e551288911a230c2b73e96bc6e2ed12f9 to solve per-package issues. So I would strongly suspect that both of them need to be moved to APR_UTIL_PER_PACKAGE_TWEAK_HOOKS.

Yes, both of them need to be done but
rules.mk is generated by apr-util itself at configure step.
This file is not present (not yet generated) when APR_UTIL_PER_PACKAGE_TWEAK_HOOKS are called.

Quoted 6 lines

XXX_PER_PACKAGE_TWEAK_HOOKS are for tweaking files external to current package that leads to overwrite -> Ok for build-1/libtool

rules.mk is not correct for per-package build but it is generated by apr-util package for apr-util package without any overwrites -> Keep tweak in <PKG>_POST_CONFIGURE_HOOKS.

Yes, this all makes sense now.

Also, for apr and apr-utils, we are fortunate enough that the tweak is not required at configure time, but at build time, which so far allowed us to use POST_CONFIGURE hooks.

However, for some other packages, we might not be so lucky, and we would have needed pre- and -postconfigure hooks. The pre-hooks can't be used anymore for such tweaks, because of file-overwrite detection, so we need those earlier hook.

Makes sense, makes sense.

So, except for the naming of the variable, when you respin, you can carry my:

Regards,
Yann E. MORIN.

Quoted 10 lines

Herv?

Signature · 7 lines
Herv? Codina, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#49 lore raw

PATCH 8/15 package/apache: Move APACHE_FIXUP_APR_LIBTOOL to <PKG>_PER_PACKAGE_TWEAK_HOOKS

In reply to Herve Codina

Herv?, All,

On 2021-06-21 16:11 +0200, Herve Codina spake thusly:

Quoted 14 lines

APACHE_FIXUP_APR_LIBTOOL tweaks files for per package directory build. This is typically the kind of operation expected to be in <PKG>_PER_PACKAGE_TWEAK_HOOKS

---
 package/apache/apache.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
package/apache/apache.mk +0 -0
index 7dbd1a4512..f496333174 100644

package/apache/apache.mk define APACHE_FIXUP_APR_LIBTOOL · line 26

Quoted 5 lines
package/apache/apache.mk +1 -1
@@ -23,7 +23,7 @@ define APACHE_FIXUP_APR_LIBTOOL
 	$(SED) "s@$(PER_PACKAGE_DIR)/[^/]\+/@$(PER_PACKAGE_DIR)/apache/@g" \
 		$(STAGING_DIR)/usr/build-1/libtool
 endef
-APACHE_POST_CONFIGURE_HOOKS += APACHE_FIXUP_APR_LIBTOOL
+APACHE_PER_PACKAGE_TWEAK_HOOKS += APACHE_FIXUP_APR_LIBTOOL

Same, with the variable name:

Regards,
Yann E. MORIN.

Quoted 10 lines
package/apache/apache.mk +0 -0
 endif
 
 APACHE_CONF_ENV= \
Signature · 6 lines
2.31.1

_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#53 lore raw

PATCH 9/15 package/pkg-python: Remove _sysconfigdata*.pyc files when _sysconfigdata*.py are changed

In reply to Herve Codina

Herv?, All,

On 2021-06-22 19:52 +0200, Herve Codina spake thusly:

Quoted 16 lines
Quoted 14 lines · On Mon, 21 Jun 2021 17:12:02 +0200 Thomas Petazzoni…
Quoted 12 lines · On Mon, 21 Jun 2021 16:11:24 +0200 Herve Codina <[email protected]>…

In order to avoid _sysconfigdata*.pyc overwrites when they are generated based on _sysconfigdata*.py changes, this commit simply removes _sysconfigdata*.pyc whenever _sysconfigdata*.py are changed.

As they are removed, overwrite detection will no longer trig and coherency between the two files (.py and .pyc) is ensured.

---
 package/pkg-python.mk | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
Quoted 10 lines
Quoted 10 lines

package/pkg-python.mk ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y) · line 99

Quoted 10 lines
package/pkg-python.mk +1 -1
index 59a48e5a87..b3fde77da5 100644
@@ -96,7 +96,9 @@ ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y)
 define PKG_PYTHON_FIXUP_SYSCONFIGDATA
 	find $(HOST_DIR)/lib/python* $(STAGING_DIR)/usr/lib/python* \
 		-name "_sysconfigdata*.py" | xargs --no-run-if-empty \
-		$(SED) "s:$(PER_PACKAGE_DIR)/[^/]\+/:$(PER_PACKAGE_DIR)/$($(PKG)_NAME)/:g"
+		$(SED) "s:$(PER_PACKAGE_DIR)/[^/]\+/:$(PER_PACKAGE_DIR)/$($(PKG)_NAME)/:g" ;\  

[--SNIP--]

Quoted 8 lines
Quoted 3 lines

package/pkg-python.mk ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y) · line 101

Quoted 2 lines
package/pkg-python.mk +2 -0
+	find $(HOST_DIR)/lib/python* $(STAGING_DIR)/usr/lib/python* \
+		-name "_sysconfigdata*.pyc" -print0 | xargs -0 -r rm -f  

Do we need a --no-run-if-empty on the xargs ? Or perhaps it should be

It is cleaner to avoid running rm is there is nothing to remove. So, I prefer keeping the xargs -r option.

Quoted 2 lines

using find directly, i.e:

Quoted 7 lines
Quoted 4 lines
	find $(HOST_DIR)/lib/python* $(STAGING_DIR)/usr/lib/python* \
		-name "_sysconfigdata*.pyc" -exec rm -f {} \;

(or something like that)

Using "-exec rm -f {} \;" will lead to start as many rm as there are files to remove whereas xargs will run only one rm.

So, 1: wrong. xargs may have to exec more than one rm if there are too many arguments, or the command line is too long. But of course, this is inconsequential, because even on a huge build, this would be just a few exec instead of tens of thousands.

And 2: why not using find's -delete option to begin with:

find $(HOST_DIR)/lib/python* $(STAGING_DIR)/usr/lib/python* \
-name "_sysconfigdata*.pyc" -delete

There, not even a pipe, or an exec of anything else...

Regards,
Yann E. MORIN.

Quoted 11 lines

Herv?

Signature · 7 lines
Herv? Codina, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#56 lore raw

PATCH 10/15 package/pkg-generic.mk: Move python fixup to generic package infrastructure

In reply to Herve Codina

Herv?, All,

On 2021-06-21 16:11 +0200, Herve Codina spake thusly:

Quoted 44 lines

Fixing _sysconfigdata*.{py,pyc} was previously done by python package infrastructure. Some packages use python stuff without using python package infrastructure.
These packages perform overwrites and need the specific python fixup to fix them.

In order to be sure to fix all of these packages, the python fixup is moved to the generic package infrastructure and applied to all packages.
This follows the same principle as for the .la libtool files fixup.

---
 package/pkg-generic.mk | 17 +++++++++++++++++
 package/pkg-python.mk  | 12 ------------
 2 files changed, 17 insertions(+), 12 deletions(-)
package/pkg-generic.mk +15 -0
index 4069d2cf3c..918e2381af 100644
@@ -103,6 +103,21 @@ define fixup-libtool-files
 endef
 endif
 
+# Make sure python _sysconfigdata*.py files only reference the current
+# per-package directory.
+
+# $1: package name (lower case)
+# $2: staging or host directory of the package
+ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y)
+define fixup-python-files
+	$(Q)find $(2) \( -path '$(2)/lib/python*' -o -path '$(2)/usr/lib/python*' \) \
+		-name "_sysconfigdata*.py" | xargs --no-run-if-empty \
+		$(SED) "s:$(PER_PACKAGE_DIR)/[^/]\+/:$(PER_PACKAGE_DIR)/$(1)/:g" ;\
+	find $(2) \( -path '$(2)/lib/python*' -o -path '$(2)/usr/lib/python*' \) \
+		-name "_sysconfigdata*.pyc" -print0 | xargs -0 -r rm -f
+endef
+endif
+
 # Functions to detect overwritten files
 
 ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y)

package/pkg-generic.mk $(BUILD_DIR)/%/.stamp_configured: · line 277

Quoted 6 lines
package/pkg-generic.mk +2 -0
@@ -257,6 +272,8 @@ $(BUILD_DIR)/%/.stamp_configured:
 	@$(call pkg_size_before,$(HOST_DIR),-host)
 	$(call fixup-libtool-files,$(NAME),$(HOST_DIR))
 	$(call fixup-libtool-files,$(NAME),$(STAGING_DIR))
+	$(call fixup-python-files,$(NAME),$(HOST_DIR))
+	$(call fixup-python-files,$(NAME),$(STAGING_DIR))
 	$(foreach hook,$($(PKG)_PER_PACKAGE_TWEAK_HOOKS),$(call $(hook))$(sep))

So, earlier in the series, you introduced _PER_PACKAGE_TWEAK_HOOKS (or whatever the name it will eventually bear), stating that infras could set it, but here you are not taking the chance to demonstrate this, and instead you are inserting the python fixups in an ad-hoc way.

Which also illustrates that the libtool fixups should be converted to the _PER_PACKAGE_TWEAK_HOOKS, probably...

Otherwise, this is a good change.

Regards,
Yann E. MORIN.

Quoted 39 lines
package/pkg-generic.mk +0 -0
 	@$(call pkg_detect_overwrite_before,$(TARGET_DIR))
 	@$(call pkg_detect_overwrite_before,$(HOST_DIR),-host)
package/pkg-python.mk +0 -12
index b3fde77da5..e6b81bdfd3 100644
@@ -92,16 +92,6 @@ HOST_PKG_PYTHON_SETUPTOOLS_INSTALL_OPTS = \
 	--root=/ \
 	--single-version-externally-managed
 
-ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y)
-define PKG_PYTHON_FIXUP_SYSCONFIGDATA
-	find $(HOST_DIR)/lib/python* $(STAGING_DIR)/usr/lib/python* \
-		-name "_sysconfigdata*.py" | xargs --no-run-if-empty \
-		$(SED) "s:$(PER_PACKAGE_DIR)/[^/]\+/:$(PER_PACKAGE_DIR)/$($(PKG)_NAME)/:g" ;\
-	find $(HOST_DIR)/lib/python* $(STAGING_DIR)/usr/lib/python* \
-		-name "_sysconfigdata*.pyc" -print0 | xargs -0 -r rm -f
-endef
-endif
-
 ################################################################################
 # inner-python-package -- defines how the configuration, compilation
 # and installation of a Python package should be done, implements a
@@ -246,8 +236,6 @@ $(2)_PYTHON_INTERPRETER = $$(HOST_DIR)/bin/$$($(2)_NEEDS_HOST_PYTHON)
 endif
 endif
 
-$(2)_PRE_CONFIGURE_HOOKS += PKG_PYTHON_FIXUP_SYSCONFIGDATA
-
 #
 # Build step. Only define it if not already defined by the package .mk
 # file.
Signature · 6 lines
2.31.1

_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#60 lore raw

PATCH 11/15 package/owfs: Remove Python sysconfigdata fixup

In reply to Herve Codina

Herv?, All,

On 2021-06-21 16:11 +0200, Herve Codina spake thusly:

Quoted 4 lines

This fixup is done at pkg-generic level for all packages. So, it is no more needed in owfs package.

When you respin the series, you can carry my:

Regards,
Yann E. MORIN.

Quoted 31 lines
---
 package/owfs/owfs.mk | 9 ---------
 1 file changed, 9 deletions(-)
package/owfs/owfs.mk +0 -9
index ffc0b3098d..10543d3698 100644
@@ -88,15 +88,6 @@ OWFS_DEPENDENCIES += python host-swig
 # override the PYSITEDIR variable in make.
 OWFS_EXTRA_MAKE_OPTS += PYSITEDIR=/usr/lib/python$(PYTHON_VERSION_MAJOR)/site-packages
 
-ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y)
-define OWFS_FIXUP_PYTHON_SYSCONFIGDATA
-	find $(HOST_DIR)/lib/python* $(STAGING_DIR)/usr/lib/python* \
-		-name "_sysconfigdata*.py" | xargs --no-run-if-empty \
-		$(SED) "s:$(PER_PACKAGE_DIR)/[^/]\+/:$(PER_PACKAGE_DIR)/owfs/:g"
-endef
-OWFS_PRE_CONFIGURE_HOOKS += OWFS_FIXUP_PYTHON_SYSCONFIGDATA
-endif
-
 else
 OWFS_CONF_OPTS += --disable-owpython --without-python
 endif
Signature · 6 lines
2.31.1

_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#63 lore raw

PATCH 12/15 package/pkg-generic.mk: Generate final rsync exclude file list

In reply to Herve Codina

Herv?, All,

On 2021-06-21 16:11 +0200, Herve Codina spake thusly:

Quoted 10 lines

The final rsync performed at {host,target}-finalize steps need to be done by rsyncing files generated by each packages without looking at files generated by other packages the current package depends on. This is needed to avoid overwrites in final {HOST,TARGET}_DIR.

In order to prepare the final rsync, an exclusion list is generated. This list lists files that are not generated by the current package and so files that need to be excluded from the final rsync.

Quoted 5 lines

Note also that the files list was not based on .files-list.{before,after}. During .file-list.{before,after} built for host directory, staging sub-directory (ie <toolchain>/sysroot) is filtered out.
The final rsync exclusion list needs to take into account the full {host,target} directory to avoid final overwrites.

Ideally, we would in fact redirect the package installation to an emoty directory, so that we can actually find what it installs.

However, as has been discussed in the past, this is fraught with unworkable issues. For example, some paths may be hard-coded at configure time and/or build time, and thus the package would still install in the original stagin we presented it with (for target/, this is not an isue, because target/ is never looked at during configure or build, only at install time). Or a package may try to modify an existing file (bad, but still). Or a file just assumes that tdirectory structure exists (bad, but eh...)

So, yes, the rsync exclusion list is a good workaround.

I see you have provided detailed comit logs, that is great. Still, for such core stuff, I think they still miss the bigger picture, like I explained above for example, and which should have been part of the commit log to explain why we resort to an exclusion list rather than the more obvious and simple empty-DESTDIR.

Quoted 25 lines
---
 package/pkg-generic.mk | 28 ++++++++++++++++++++++++++++
 1 file changed, 28 insertions(+)
package/pkg-generic.mk +12 -0
index 918e2381af..546370af7a 100644
@@ -162,6 +162,30 @@ define pkg_size_after
 	rm -f $($(PKG)_DIR)/.files-list$(2).after
 endef
 
+# Functions to collect final rsync exclusion files
+
+# $(1): base directory to search in
+# $(2): suffix of file (optional)
+define pkg_final_rsync_before
+	cd $(1); \
+	LC_ALL=C find . \( -type f -o -type l \) -printf '%T@:%i:%#m:%y:%s,%p\n' \
+		| LC_ALL=C sort > $($(PKG)_DIR)/.files-final-rsync$(2).before
+endef
+
+# $(1): base directory to search in
+# $(2): suffix of file (optional)

package/pkg-generic.mk define pkg_size_after · line 186

Quoted 10 lines
package/pkg-generic.mk +10 -0
+define pkg_final_rsync_after
+	cd $(1); \
+	LC_ALL=C find . \( -type f -o -type l \) -printf '%T@:%i:%#m:%y:%s,%p\n' \
+		| LC_ALL=C sort > $($(PKG)_DIR)/.files-final-rsync$(2).after
+	LC_ALL=C comm -2 \
+		$($(PKG)_DIR)/.files-final-rsync$(2).before \
+		$($(PKG)_DIR)/.files-final-rsync$(2).after \
+		| sed -r -e 's/^[^,]+,./- /' \
+		> $($(PKG)_DIR)/.files-final-rsync$(2).exclude_rsync
+	rm -f $($(PKG)_DIR)/.files-final-rsync$(2).after

You forgot to remove .files-final-rsync$(2).before

package/pkg-generic.mk $(BUILD_DIR)/%/.stamp_configured: · line 295

Quoted 11 lines
package/pkg-generic.mk +4 -0
+endef
+
 define check_bin_arch
 	support/scripts/check-bin-arch -p $($(PKG)_NAME) \
 		-l $($(PKG)_DIR)/.files-list.txt \
@@ -267,6 +291,8 @@ $(BUILD_DIR)/%/.stamp_configured:
 	@$(call MESSAGE,"Configuring")
 	$(Q)mkdir -p $(HOST_DIR) $(TARGET_DIR) $(STAGING_DIR) $(BINARIES_DIR)
 	$(call prepare-per-package-directory,$($(PKG)_FINAL_DEPENDENCIES))
+	@$(call pkg_final_rsync_before,$(TARGET_DIR))
+	@$(call pkg_final_rsync_before,$(HOST_DIR),-host)

The more I look at this, the more I believe we should resurrect the idea of a 'prepare' step that goes in-between 'patch' and 'configure'...

Regards,
Yann E. MORIN.

Quoted 19 lines
package/pkg-generic.mk +2 -0
 	@$(call pkg_size_before,$(TARGET_DIR))
 	@$(call pkg_size_before,$(STAGING_DIR),-staging)
 	@$(call pkg_size_before,$(HOST_DIR),-host)
@@ -404,6 +430,8 @@ $(BUILD_DIR)/%/.stamp_installed:
 	@$(call pkg_size_after,$(STAGING_DIR),-staging)
 	@$(call pkg_size_after,$(HOST_DIR),-host)
 	@$(call check_bin_arch)
+	@$(call pkg_final_rsync_after,$(TARGET_DIR))
+	@$(call pkg_final_rsync_after,$(HOST_DIR),-host)
 	@$(call pkg_detect_overwrite_after,$(TARGET_DIR))
 	@$(call pkg_detect_overwrite_after,$(HOST_DIR),-host)
 	$(Q)touch $@
Signature · 6 lines
2.31.1

_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#38 lore raw

PATCH 6/15 package/pkg-generic.mk: Introduce <PKG>_PER_PACKAGE_TWEAK_HOOKS

In reply to Yann E. MORIN
Quoted 8 lines · On Tue, 22 Jun 2021 22:39:09 +0200 "Yann E. MORIN" <[email protected]>…

Ah, yes, yes, now I see. OK, so here's a suggestion for a commit log:

package/pkg-generic: add early pre-configure hooks

Currently, when a package needs to modify files it inherits from its dependencies, because they contain paths, we can only do that in a pre- or post-configure hook.

Quoted 5 lines

However, whatever is done as part of those hooks, will be accounted to the package itself, and thus will trigger file-overwrite detection.

So, we need a way to be able to actually modify files before we start monitoring for them.

"before we start monitoring changes in those files".

Quoted 3 lines

We introduce a new set of hooks that an individual package can set, or that a package infra can set, and that are called right before we snapshot the state of target, and host (to which staging belongs),

But yes, you've perfectly understood the problem being solved here!

Thomas

Signature · 3 lines
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#7 lore raw

PATCH 1/15 package/pkg-generic.mk: detect files overwritten in TARGET_DIR and HOST_DIR

In reply to Yann E. MORIN

Hi,

Quoted 35 lines · On Tue, 22 Jun 2021 12:24:33 +0200 "Yann E. MORIN" <[email protected]>…

Nicolas, All,

On 2021-06-22 11:57 +0200, Nicolas Cavallari spake thusly:

Quoted 10 lines · Thomas Petazzoni
Quoted 9 lines
Quoted 3 lines · On Tue, 22 Jun 2021 09:40:52 +0200 Herve Codina <[email protected]>…

Compared to the first version, this patch has an improved commit message and generates the md5sum snapshot using

     'find -L $(1) -type f -print0 | xargs -0 -r md5sum > $($(PKG)_DIR)/.files$(2).md5;'  

But is this better ? Due to not cd-ing into the directory, you will have full absolute paths in the .md5 file,

But is that even an issue? Those paths are only used for one per-package directory; they do not follow files around into another package's ppd.

So, relative or absolute, we don't much care.

Quoted 2 lines
Quoted 2 lines

and adding LC_ALL=C is quite
customary too.

In this case, do we also even care or need it? I pretty sure the output of find is not sorted at all; it most probably depend on the order of dentries in the directory, and those have absolutely no ordering guarantee...

Note that if you really want to use LC_ALL, passing it in front of 'find' will not make LC_ALL available to anythin after the pipeline, so you'd still have xargs (and md5sum) run without it...

Quoted 3 lines
Quoted 1 line

Ditto for using | xargs instead of -exec.

xargs will run only start one md5sum process if the command line size permits it, whereas "-exec md5sum {} ;" will start one md5sum per file.

Quoted 7 lines

Agreed. But I think this is exactly what Thomas suggested to use, right?

Quoted 1 line

An alternative is "-exec md5sum {} +", which is essentially xargs-like.

I once used that in the past, and it breaks on non-GNU find, because '+' is a GNUism. So I think using xargs is still better.

Well, according to the different exchanges,

I think I can remove LC_ALL (we don't care about any order), and I keep '... | xargs -0 -r md5sum'.
'cd' is not needed (we do not care about absolute or relative path).

So, the snapshot is taken using:
find -L $(1) -type f -print0 | xargs -0 -r md5sum > $($(PKG)_DIR)/.files$(2).md5;

and the detection is done using:
md5sum --quiet -c $($(PKG)_DIR)/.files$(2).md5

Is that ok for everyone ?

Regards,
Herv?

Signature · 3 lines
Herv? Codina, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#13 lore raw

PATCH 2/15 package/e2fsprogs: fix fsck overwrite in HOST_DIR

In reply to Yann E. MORIN
Quoted 16 lines · On Tue, 22 Jun 2021 21:40:24 +0200 "Yann E. MORIN" <[email protected]>…

Herv?, All,

On 2021-06-21 16:11 +0200, Herve Codina spake thusly:

Quoted 8 lines

host-e2fsprog overwrites fsck program and some man related files previously installed by host-util-linux.

This patch simply disable fsck in host-e2fsprog.

host-e2fsprog is used to build final ext{2,3,4} images. The missing host-e2fsprog fsck tool (filesystem integrity check tool) in HOST_DIR should not lead to issues.

And the target variant is not susceptible to that issue, because e2fsprogs' fsck depends on !BR2_PACKAGE_UTIL_LINUX_FSCK. So we're clean there.

Quoted 5 lines

Except for the comments by Thomas, you can carry my:

when you respin this patch.

Ok,
Reviewed-by added.

Thanks for the review,
Herv?

Signature · 3 lines
Herv? Codina, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#19 lore raw

PATCH 3/15 package/pkg-generic.mk: Remove Info documents dir entry

In reply to Yann E. MORIN

Hi,

Quoted 41 lines · On Tue, 22 Jun 2021 22:18:18 +0200 "Yann E. MORIN" <[email protected]>…

I think a list+macro would be a better solution.

Alternatively, we could append to post-install hooks, like we do for pre-configure hooks in the autotools infra for autoreconf et al., for example...

    # Outside generic-package-inner:

    # $1: base directory (target, staging, host)
    define remove-conflicting-useless-files
        $(Q)$(RM) -rf $(patsubst %, $(1)/%, $($(PKG)_DROP_FILES_OR_DIRS))
    endef
    define REMOVE_CONFLICTING_USELESS_FILES_IN_HOST
        $(call remove-conflicting-useless-files,$(HOST_DIR))
    endef
    define REMOVE_CONFLICTING_USELESS_FILES_IN_STAGING
        $(call remove-conflicting-useless-files,$(STAGING_DIR))
    endef
    define REMOVE_CONFLICTING_USELESS_FILES_IN_TARGET
        $(call remove-conflicting-useless-files,$(TARGET_DIR))
    endef


    # In generic-package-inner:

    $(2)_DROP_FILES_OR_DIRS += /share/info/dir

    # For host packages:
    $(2)_POST_INSTALL_HOOKS += REMOVE_CONFLICTING_USELESS_FILES_IN_HOST

    # For target packages:
    $(2)_POST_INSTALL_STAGING_HOOKS += REMOVE_CONFLICTING_USELESS_FILES_IN_STAGING
    $(2)_POST_INSTALL_TARGET_HOOKS += REMOVE_CONFLICTING_USELESS_FILES_IN_TARGET

This way, it also paves the way to add other hooks to actually fix some files instead of removing them...

For example, assume that packages (e.g. 'wonders') can register themselves against another package (e.g. 'manager') by appending a line to a text file, like so (over-simplified, of course):

Quoted 6 lines

echo "name=wonders path=/usr/lib/wonders/v42" >>/usr/share/manager/registry

Then we could have a hook that detects that, extracts the delta installed by the package, stash that somewhere in staging, and have a target-finalise hook that aggregates all the packages in the end.

I like this solution.

If it is okay for everyone, I will rework my patch accordingly.

Thomas, your opinion ?

Regards,
Herv?

Signature · 3 lines
Herv? Codina, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#28 lore raw

PATCH 4/15 package/pkg-generic.mk: Fix .la files overwrite detection

In reply to Yann E. MORIN

Hi,

Quoted 40 lines · On Tue, 22 Jun 2021 12:30:39 +0200 "Yann E. MORIN" <[email protected]>…

On 2021-06-22 12:12 +0200, Thomas Petazzoni spake thusly:

Quoted 7 lines
Quoted 2 lines · On Tue, 22 Jun 2021 11:56:09 +0200 "Yann E. MORIN" <[email protected]>…

Sorry, but this patch (4/15) is fixing an issue introduced by the first patch, with:

I agree in principle. But here, it's really a new requirement that we are adding on packages, and we *know* that even with Herv? patches there will be some other packages that will need fixing.

Sorry, but this patch is fixing the infra, not the packages.

Quoted 2 lines

So to me it's not really like introducing a regression in a patch, and fixing it later.

I never said it was a regression. ;-)

Quoted 10 lines
Quoted 7 lines
Quoted 4 lines

What about the others issues
detected ? Squash also together with the first patch ? I think it will produce a huge patch quite complicate to understand even with all individual commit message squashed.

Ideally, I would say a series should first fix the issues, then introduce the tooling.

No, please keep one series, but having the fixes *before* we introduce the check.

Exactly; I never said to split the series into two series. It should still be one.

Quoted 3 lines

And as stated above, the fixes will not fix all problems, they will only fix the problems we know about. More problems will be detected by the autobuilders thanks to the overwrite check.

And this is totally OK. [0]

Quoted 4 lines
Quoted 4 lines
Quoted 3 lines

However, that being said, I can squash this patch (Fix .la files overwrite detection) with the 1st one (detect files overwritten in TARGET_DIR and HOST_DIR) if you still think it will be better.

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

Yes, I still think that it is better.

No, please don't squash, have fixes added before the overwrite check instead.

Well, I still disagree, because this patch really fixes an issue introduced *in the infra* by the first patch.

Well, my bad.

I think that the better thing to do is to fully rework the history. First fixing overwrites and then introducing the overwrite detection tooling (ie the actual PATCH 1 will be moved just before actual PATCH 12 and so actual PATCH 4 simply disappears).

The patch introducing the tooling will explain in its commit message why the calls to fixup-libtool-files and fixup-python-files are performed before taking the overwrite snapshot and why it is safe (sed --in-place).

The variable <PKG>_PER_PACKAGE_TWEAK_HOOKS (or other name) will be introduced before the tooling. The patches changing some packages to use this variable (move tweaks from <PKG>_POST_CONFIGURE_HOOKS to <PKG>_PER_PACKAGE_TWEAK_HOOKS) will also be introduced before the tooling even if these changes really make sense after the tooling introduction.
Indeed, the notion of action done before taking the overwrite detection snapshot (<PKG>_PER_PACKAGE_TWEAK_HOOKS) and action done after (<PKG>_PRE_CONFIGURE_HOOKS, configure and <PKG>_POST_CONFIGURE_HOOKS) makes sense only with the overwrite detection tool.

Is that seem better for everyone ?

Regards,
Herv?

Signature · 3 lines
Herv? Codina, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#8 lore raw

PATCH 1/15 package/pkg-generic.mk: detect files overwritten in TARGET_DIR and HOST_DIR

In reply to Herve Codina

Herv?, All,

On 2021-06-24 16:09 +0200, Herve Codina spake thusly:

Quoted 2 lines

On Tue, 22 Jun 2021 12:24:33 +0200
"Yann E. MORIN" <[email protected]> wrote:

[--SNIP--]

Quoted 8 lines

Well, according to the different exchanges,

I think I can remove LC_ALL (we don't care about any order), and I keep '... | xargs -0 -r md5sum'.
'cd' is not needed (we do not care about absolute or relative path).

So, the snapshot is taken using:
find -L $(1) -type f -print0 | xargs -0 -r md5sum > $($(PKG)_DIR)/.files$(2).md5;

I spoke about -L as a suggestion to follow and monitor symlinks. However, as I said, that does not work, at least for host/, because it will emit a warning about a directory loop. Also, there is no telling what users will install i their target/ and we may have a similar situation for target/.

This warning is not nice as it will happen for each and every packages.

So, drop -L. At the very least, we have the infrastructure in place, and we can extend it later should we need additional details.

But now that I think about it, the entry-changed-to-omething-else case is already covered by the existing instrumenttion, as part of the pkg_size_before/after hooks (find's %y printf directive)

So, yes, just drop the -L.

Quoted 4 lines

and the detection is done using:
md5sum --quiet -c $($(PKG)_DIR)/.files$(2).md5

Is that ok for everyone ?

Except for -L, yes, this is a proper summary from my point if view.

Thanks! :-)

Regards,
Yann E. MORIN.

Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#34 lore raw

PATCH 5/15 package/pkg-generic.mk: Perform .la files fixup in per-package HOST_DIR

In reply to Yann E. MORIN

Hi,

Quoted 9 lines · On Tue, 22 Jun 2021 12:12:02 +0200 "Yann E. MORIN" <[email protected]>…

package/pkg-generic.mk endif · line 102

Quoted 7 lines

# $2: staging directory of the package
ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y)
define fixup-libtool-files

package/pkg-generic.mk +2 -1
-	$(Q)find $(2)/usr/lib* -name "*.la" | xargs --no-run-if-empty \
+	$(Q)find $(2) \( -path '$(2)/lib*' -o -path '$(2)/usr/lib*' \) \
+		-name "*.la" | xargs --no-run-if-empty \
 		$(SED) "s:$(PER_PACKAGE_DIR)/[^/]\+/:$(PER_PACKAGE_DIR)/$(1)/:g"  

While at it, can you switch over to using find's -print0 and xarg's -0 ?

Yes, changed.

I will also change PATCH 10 of this series. Same construction in fixup-python-files same sanction.

Regards,
Herv?

Signature · 3 lines
Herv? Codina, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#29 lore raw

PATCH 4/15 package/pkg-generic.mk: Fix .la files overwrite detection

In reply to Herve Codina

Herv?, All,

On 2021-06-24 17:44 +0200, Herve Codina spake thusly:

Quoted 2 lines

On Tue, 22 Jun 2021 12:30:39 +0200
"Yann E. MORIN" <[email protected]> wrote:

[--SNIP--]

Quoted 3 lines
Quoted 2 lines

Well, I still disagree, because this patch really fixes an issue introduced *in the infra* by the first patch.

Well, my bad.

No worries, really! This series is quite the complex one. ;-)

Quoted 8 lines

I think that the better thing to do is to fully rework the history. First fixing overwrites and then introducing the overwrite detection tooling (ie the actual PATCH 1 will be moved just before actual PATCH 12 and so actual PATCH 4 simply disappears).

The patch introducing the tooling will explain in its commit message why the calls to fixup-libtool-files and fixup-python-files are performed before taking the overwrite snapshot and why it is safe (sed --in-place).

Exactly! :-)

Quoted 11 lines

The variable <PKG>_PER_PACKAGE_TWEAK_HOOKS (or other name) will be introduced before the tooling. The patches changing some packages to use this variable (move tweaks from <PKG>_POST_CONFIGURE_HOOKS to <PKG>_PER_PACKAGE_TWEAK_HOOKS) will also be introduced before the tooling even if these changes really make sense after the tooling introduction.
Indeed, the notion of action done before taking the overwrite detection snapshot (<PKG>_PER_PACKAGE_TWEAK_HOOKS) and action done after (<PKG>_PRE_CONFIGURE_HOOKS, configure and <PKG>_POST_CONFIGURE_HOOKS) makes sense only with the overwrite detection tool.

Is that seem better for everyone ?

It looks a sane ordering to me, yes. THanks for working on this!

Regards,
Yann E. MORIN.

Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#67 lore raw

PATCH 13/15 Makefile: Rsync global {TARGET, HOST}_DIR using exclusion file list

In reply to Herve Codina

Herv?, All,

Nitpick, applicable to most of your patches in this series: there should not be an uppercase in the title, after the colon sign, and the tense should be imperative, e.g:

Makefile: rsync global {TARGET,HOST}_DIR using exclusion file list
Makefile: break hardlinks in global {TARGET,HOST}_DIR on per-package build package/pkg-generic.mk: fix per-package <pkg>-{reconfigure, rebuild, reinstall}

On 2021-06-21 16:11 +0200, Herve Codina spake thusly:

Quoted 6 lines

On a per-package build, rsync final {TARGET,HOST}_DIR using exclusion file list computed for each packages.
As we rsync only files generated by each packages, we need to to do this rsync recursively on each dependencies to collect all needed files. This is done based on existing <PKG>_FINAL_RECURSIVE_DEPENDENCIES

I am not sure I understand why this is needed...

One thing that comes to mind is speedup. Indeed, now that we can ensure that packages only install new files and don;t change existing files, we can speedup the final aggregation by just handling the new files.

But since we are using hardlinks, the aggregation is rather fast...

Can you expand on the rationale in the commit log when you respin, please?

Quoted 11 lines
---
 Makefile | 24 ++++++++++++++++++++++--
 1 file changed, 22 insertions(+), 2 deletions(-)
Makefile +0 -0
index 472af5a318..50bc6b4503 100644
@@ -740,10 +740,28 @@ TARGET_DIR_FILES_LISTS = $(sort $(wildcard $(BUILD_DIR)/*/.files-list.txt))
 HOST_DIR_FILES_LISTS = $(sort $(wildcard $(BUILD_DIR)/*/.files-list-host.txt))

Makefile TARGET_DIR_FILES_LISTS = $(sort $(wildcard $(BUILD_DIR)/*/.files-list.txt)) · line 750

Quoted 10 lines
Makefile +8 -0
 STAGING_DIR_FILES_LISTS = $(sort $(wildcard $(BUILD_DIR)/*/.files-list-staging.txt))
 
+ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y)
+# rsync the contents of per-package directories
+# $1: space-separated list of packages to rsync from
+# $2: 'host' or 'target'
+# $3: destination directory
+# $4: exclude file list to use
+define per-package-rsync-delta
+	mkdir -p $(3)
    $(Q)mkdir -p $(3)

Makefile TARGET_DIR_FILES_LISTS = $(sort $(wildcard $(BUILD_DIR)/*/.files-list.txt)) · line 755

Quoted 5 lines
Makefile +5 -0
+	$(foreach pkg,$(1),\
+		rsync -a --link-dest=$(PER_PACKAGE_DIR)/$(pkg)/$(strip $(2))/ \
+		--filter='merge $($(call UPPERCASE,$(pkg))_DIR)/$(strip $(4))' \
+		$(PER_PACKAGE_DIR)/$(pkg)/$(strip $(2))/ \
+		$(3)$(sep))

I would read it more easily if the lines after rsync are indented furhter, to represent the fact that they are stilll rsync arguments. Also, proably it would be nice to have this quiet by default. Indeed, with rather-deep dependency chains, there can be quite a lot of rsync lines...

    $(foreach pkg,$(1),\
        $(Q)rsync -a --link-dest=$(PER_PACKAGE_DIR)/$(pkg)/$(strip $(2))/ \
            --filter='merge $($(call UPPERCASE,$(pkg))_DIR)/$(strip $(4))' \
            $(PER_PACKAGE_DIR)/$(pkg)/$(strip $(2))/ \
            $(3)$(sep))

Furthermore, this rsync filter is non-obvious. It would be nice to explain the commit log that the list is already a proper filer, not just a list.

Additionally, in the commit that generates that list, the commit log should also mention that a list of filters is created, not a list of filenames.

Makefile TARGET_DIR_FILES_LISTS = $(sort $(wildcard $(BUILD_DIR)/*/.files-list.txt)) · line 764

Quoted 10 lines
Makefile +6 -1
+endef
+endif
+
 .PHONY: host-finalize
 host-finalize: $(PACKAGES) $(HOST_DIR) $(HOST_DIR_SYMLINK)
 	@$(call MESSAGE,"Finalizing host directory")
-	$(call per-package-rsync,$(sort $(PACKAGES)),host,$(HOST_DIR))
+	$(call per-package-rsync-delta, $(sort $(foreach pkg, $(PACKAGES), \
+		$(pkg) $($(call UPPERCASE,$(pkg))_FINAL_RECURSIVE_DEPENDENCIES))), \
+		host,$(HOST_DIR),.files-final-rsync-host.exclude_rsync)

This is unparseable by a human... :-(

    $(call per-package-rsync-delta, \
        $(sort $(foreach pkg, $(PACKAGES), \
                    $(pkg) $($(call UPPERCASE,$(pkg))_FINAL_RECURSIVE_DEPENDENCIES)) \
        ), \
        host, \
        $(HOST_DIR), \
        .files-final-rsync-host.exclude_rsync \
    )

Makefile staging-finalize: $(STAGING_DIR_SYMLINK) · line 774

Quoted 10 lines
Makefile +3 -1
 .PHONY: staging-finalize
 staging-finalize: $(STAGING_DIR_SYMLINK)
@@ -751,7 +769,9 @@ staging-finalize: $(STAGING_DIR_SYMLINK)
 .PHONY: target-finalize
 target-finalize: $(PACKAGES) $(TARGET_DIR) host-finalize
 	@$(call MESSAGE,"Finalizing target directory")
-	$(call per-package-rsync,$(sort $(PACKAGES)),target,$(TARGET_DIR))
+	$(call per-package-rsync-delta, $(sort $(foreach pkg, $(PACKAGES), \
+		$(pkg) $($(call UPPERCASE,$(pkg))_FINAL_RECURSIVE_DEPENDENCIES))), \
+		target,$(TARGET_DIR),.files-final-rsync.exclude_rsync)

Ditto.

But now that I review this patch, I notice that there is still a hole in our file-overwrite detection. It only handles files modified by a package, that comes from it dependencies.

However, if two packages in two different depednency chains, install the same file, then we're still toast. For example:

    A --> B --> C
      `-> D

If C and D install the same file, then we don't detect the conflict, and we can't easil predict which the final file will come from...

Of course, such a detection will have to be *in addition* to what this series already does, so this new issue does not invalidate this series.

Regards,
Yann E. MORIN.

Quoted 10 lines
Makefile +0 -0
 	$(foreach hook,$(TARGET_FINALIZE_HOOKS),$($(hook))$(sep))
 	rm -rf $(TARGET_DIR)/usr/include $(TARGET_DIR)/usr/share/aclocal \
 		$(TARGET_DIR)/usr/lib/pkgconfig $(TARGET_DIR)/usr/share/pkgconfig \
Signature · 6 lines
2.31.1

_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#73 lore raw

PATCH 14/15 Makefile: Breaks hardlinks in global {TARGET, HOST}_DIR on per-package build

In reply to Herve Codina

Herv?, All,

On 2021-06-21 16:11 +0200, Herve Codina spake thusly:

Quoted 7 lines

Without this patch, a make <pkg>_rebuild detects overwrites. Indeed, in target_finalize steps some modifications are done on installed files (ie strip or TARGET_FINALIZE_HOOKS for instance).

In order to avoid these modifications seen from per-package {TARGET,HOST}_DIR and so been analyzed as some overwrites, global {TARGET,HOST}_DIR is built using a full copy of the involved per-package files instead of hardlinks.

OK, now it makes sense to do a partial rsync with just the needed files.

When you respin:

Regards,
Yann E. MORIN.

Quoted 25 lines
---
 Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Makefile +1 -1
index 50bc6b4503..fe6e3f95af 100644
@@ -749,7 +749,7 @@ ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y)
 define per-package-rsync-delta
 	mkdir -p $(3)
 	$(foreach pkg,$(1),\
-		rsync -a --link-dest=$(PER_PACKAGE_DIR)/$(pkg)/$(strip $(2))/ \
+		rsync -a \
 		--filter='merge $($(call UPPERCASE,$(pkg))_DIR)/$(strip $(4))' \
 		$(PER_PACKAGE_DIR)/$(pkg)/$(strip $(2))/ \
 		$(3)$(sep))
Signature · 6 lines
2.31.1

_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#68 lore raw

PATCH 13/15 Makefile: Rsync global {TARGET, HOST}_DIR using exclusion file list

In reply to Yann E. MORIN

Herv?, All,

Additional revview I forgot previously...

On 2021-06-24 22:20 +0200, Yann E. MORIN spake thusly:

Quoted 9 lines

On 2021-06-21 16:11 +0200, Herve Codina spake thusly:

Quoted 6 lines

On a per-package build, rsync final {TARGET,HOST}_DIR using exclusion file list computed for each packages.
As we rsync only files generated by each packages, we need to to do this rsync recursively on each dependencies to collect all needed files. This is done based on existing <PKG>_FINAL_RECURSIVE_DEPENDENCIES

I am not sure I understand why this is needed...

Quoted 8 lines

One thing that comes to mind is speedup. Indeed, now that we can ensure that packages only install new files and don;t change existing files, we can speedup the final aggregation by just handling the new files.

But since we are using hardlinks, the aggregation is rather fast...

Can you expand on the rationale in the commit log when you respin, please?

Now I have seen the next patch, it makes sense. Yet, it should be explained here nonetheless. ;-)

[--SNIP--]

Quoted 5 lines

Makefile TARGET_DIR_FILES_LISTS = $(sort $(wildcard $(BUILD_DIR)/*/.files-list.txt)) · line 755

Quoted 5 lines
Makefile +5 -0
+	$(foreach pkg,$(1),\
+		rsync -a --link-dest=$(PER_PACKAGE_DIR)/$(pkg)/$(strip $(2))/ \
+		--filter='merge $($(call UPPERCASE,$(pkg))_DIR)/$(strip $(4))' \
+		$(PER_PACKAGE_DIR)/$(pkg)/$(strip $(2))/ \
+		$(3)$(sep))

[--SNIP--]

Quoted 6 lines

Furthermore, this rsync filter is non-obvious. It would be nice to explain the commit log that the list is already a proper filer, not just a list.
Additionally, in the commit that generates that list, the commit log should also mention that a list of filters is created, not a list of filenames.

And to begin with, why are we generating an _exclude_ list, rather than an _include_ list?

I.e. rather than rsync everything except what we don't want, why not just rsync just what we want, and generate a list of filters about exactly just the files installed by a package?

I.e. in previous patch, crea create '+' filters rather than '-' filters. Also, I think using the full-length filter 'include' is nicer than the shorter '+'.

Probably something like:

    LC_ALL=C comm -1 \
        $($(PKG)_DIR)/.files-final-rsync$(2).before \
        $($(PKG)_DIR)/.files-final-rsync$(2).after \
        | sed -r -e 's/^[^,]+,./include /' \
        > $($(PKG)_DIR)/.files-final-rsync$(2).exclude_rsync

(Totally untested, slippery code, excersise caution.)

Regards,
Yann E. MORIN.

Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#75 lore raw

PATCH 15/15 package/pkg-generic.mk: Fix per-package <pkg>-{reconfigure, rebuild, reinstall}

In reply to Herve Codina

Herv?, All,

On 2021-06-21 16:11 +0200, Herve Codina spake thusly:

Quoted 40 lines

Many overwrites are detected on <pkg>-{reconfigure,rebuild,reinstall}. Indeed, files previously installed by a package were detected as overwritten on next reconfigure, rebuild or reinstall.

To avoid this, we recreate per-package host and target dir from scratch as it was done during the first configure step.

---
 package/pkg-generic.mk | 26 ++++++++++++++++++++++++++
 1 file changed, 26 insertions(+)
package/pkg-generic.mk +13 -0
index 546370af7a..0aafd865ad 100644
@@ -470,6 +470,13 @@ define pkg-graph-depends
 		$$(GRAPHS_DIR)/$$(@).dot
 endef
 
+ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y)
+define empty-per-package-directory
+	rm -rf $(HOST_DIR) $(TARGET_DIR)
+	mkdir -p $(HOST_DIR) $(TARGET_DIR)
+endef
+endif
+
 ################################################################################
 # inner-generic-package -- generates the make targets needed to build a
 # generic package
@@ -1072,6 +1079,25 @@ endif
 			rm -f $$($(2)_TARGET_INSTALL_TARGET)
 			rm -f $$($(2)_TARGET_INSTALL_IMAGES)
 			rm -f $$($(2)_TARGET_INSTALL_HOST)
+			$$(call empty-per-package-directory)
+			$$(call prepare-per-package-directory,$$($(2)_FINAL_DOWNLOAD_DEPENDENCIES))
+			$$(call prepare-per-package-directory,$$($(2)_FINAL_EXTRACT_DEPENDENCIES))
+			$$(call prepare-per-package-directory,$$($(2)_FINAL_DEPENDENCIES))
+			@$$(call pkg_final_rsync_before,$$(TARGET_DIR))
+			@$$(call pkg_final_rsync_before,$$(HOST_DIR),-host)

package/pkg-generic.mk endif · line 1097

Quoted 10 lines
package/pkg-generic.mk +10 -0
+			@$$(call pkg_size_before,$$(TARGET_DIR))
+			@$$(call pkg_size_before,$$(STAGING_DIR),-staging)
+			@$$(call pkg_size_before,$$(HOST_DIR),-host)
+			$$(call fixup-libtool-files,$$(NAME),$$(HOST_DIR))
+			$$(call fixup-libtool-files,$$(NAME),$$(STAGING_DIR))
+			$$(call fixup-python-files,$$(NAME),$$(HOST_DIR))
+			$$(call fixup-python-files,$$(NAME),$$(STAGING_DIR))
+			$$(foreach hook,$$($(2)_PER_PACKAGE_TWEAK_HOOKS),$$(call $$(hook))$$(sep))
+			@$$(call pkg_detect_overwrite_before,$$(TARGET_DIR))
+			@$$(call pkg_detect_overwrite_before,$$(HOST_DIR),-host)

I don;t like this duplication, because it will inevitably diverge. And already, it is different: rather than rely on $(PKG)_FINAL_DEPENDENCIES) and a single call to prepare-per-package-directory, there are now three calls to prepare-per-package-directory with three types of depenencies.

Why can't we comonalise this into a big macro that gets expanded twise, once in the original $(BUILD_DIR)/%/.stamp_configured rule, and once here?

This would also termendously help into factorising this away and into its own $(1)-prepare step, eventually.

package/pkg-generic.mk endif · line 1100

Quoted 2 lines
package/pkg-generic.mk +2 -0
+$(1)-clean-for-reinstall: PKG=$(2)
+$(1)-clean-for-reinstall: NAME=$(1)

Please move those two right before the actual rule (i.e. just before line 991 in mastr as of today.

Regards,
Yann E. MORIN.

Quoted 9 lines
package/pkg-generic.mk +0 -0
 $(1)-reinstall:		$(1)-clean-for-reinstall $(1)
 
Signature · 6 lines
2.31.1

_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#79 lore raw

cover Overwritten file detection and fixes, one more step to TLP build

In reply to Herve Codina

Herv?, All,

On 2021-06-21 16:11 +0200, Herve Codina spake thusly:

Quoted 3 lines

This series fixes some issues in the TLP build feature.
The starting point was the work done by Thomas Petazzoni on overwrites detection (PATCH 1).

[--SNIP--]

Thanks for working on this complex topic. :-)

I'm now done with my review; I may have missed details of lesser importance, so we'll deal with those later; Thomas also provided some feedback. I see that you have already taken initial reviews into account, and that you seem nicely on track for a v2.

As such, I've marked this entire series as "changes requested" in patchwork.

Can you add me in Cc of the full series when you later respin, please?

Regards,
Yann E. MORIN.

Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#39 lore raw

PATCH 6/15 package/pkg-generic.mk: Introduce <PKG>_PER_PACKAGE_TWEAK_HOOKS

In reply to Thomas Petazzoni

Hi,

Quoted 17 lines · On Wed, 23 Jun 2021 14:40:20 +0200 Thomas Petazzoni…
Quoted 8 lines · On Tue, 22 Jun 2021 22:39:09 +0200 "Yann E. MORIN" <[email protected]>…

Ah, yes, yes, now I see. OK, so here's a suggestion for a commit log:

package/pkg-generic: add early pre-configure hooks

Currently, when a package needs to modify files it inherits from its dependencies, because they contain paths, we can only do that in a pre- or post-configure hook.

Quoted 5 lines

However, whatever is done as part of those hooks, will be accounted to the package itself, and thus will trigger file-overwrite detection.

So, we need a way to be able to actually modify files before we start monitoring for them.

Quoted 8 lines

"before we start monitoring changes in those files".

Quoted 3 lines

We introduce a new set of hooks that an individual package can set, or that a package infra can set, and that are called right before we snapshot the state of target, and host (to which staging belongs),

But yes, you've perfectly understood the problem being solved here!

This will be the new commit log.

Herv?

Signature · 3 lines
Herv? Codina, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#40 lore raw

PATCH 6/15 package/pkg-generic.mk: Introduce <PKG>_PER_PACKAGE_TWEAK_HOOKS

In reply to Yann E. MORIN

Hi,

Quoted 7 lines · On Tue, 22 Jun 2021 22:39:09 +0200 "Yann E. MORIN" <[email protected]>…

package/pkg-generic.mk $(BUILD_DIR)/%/.stamp_configured: · line 260

Quoted 2 lines
package/pkg-generic.mk +1 -0
 	$(call fixup-libtool-files,$(NAME),$(STAGING_DIR))
+	$(foreach hook,$($(PKG)_PER_PACKAGE_TWEAK_HOOKS),$(call $(hook))$(sep))  

I am going to bikeshed on the naming of this variable (which is a good sing that I am pretty much OK with the feature).

I think $(2)_EARLY_PRE_CONFIGURE_HOOKS is more sensible.

That's good for me.

Okay for everyone ?

Regards,
Herve

Signature · 3 lines
Herv? Codina, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#54 lore raw

PATCH 9/15 package/pkg-python: Remove _sysconfigdata*.pyc files when _sysconfigdata*.py are changed

In reply to Yann E. MORIN

Hi,

Quoted 7 lines · On Tue, 22 Jun 2021 22:50:40 +0200 "Yann E. MORIN" <[email protected]>…

And 2: why not using find's -delete option to begin with:

find $(HOST_DIR)/lib/python* $(STAGING_DIR)/usr/lib/python* \
-name "_sysconfigdata*.pyc" -delete

There, not even a pipe, or an exec of anything else...

Yes, indeed.
I just didn't think about '-delete' when I wrote the "find ...".

Changed to '-delete' option.

Herv?

Signature · 3 lines
Herv? Codina, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#57 lore raw

PATCH 10/15 package/pkg-generic.mk: Move python fixup to generic package infrastructure

In reply to Yann E. MORIN

Hi,

Quoted 8 lines · On Tue, 22 Jun 2021 23:01:03 +0200 "Yann E. MORIN" <[email protected]>…

So, earlier in the series, you introduced _PER_PACKAGE_TWEAK_HOOKS (or whatever the name it will eventually bear), stating that infras could set it, but here you are not taking the chance to demonstrate this, and instead you are inserting the python fixups in an ad-hoc way.

Which also illustrates that the libtool fixups should be converted to the _PER_PACKAGE_TWEAK_HOOKS, probably...

Yes that's perfectly true.

But do you think we need it right now in this patch series. I really would prefer to keep it as it is for this patch series.

The series is already quite complex and I think that converting libtool and python fixups to PER_PACKAGE_TWEAK_HOOKS (or other name) can be done afterward.

Herve

Signature · 3 lines
Herv? Codina, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#61 lore raw

PATCH 11/15 package/owfs: Remove Python sysconfigdata fixup

In reply to Yann E. MORIN
Quoted 12 lines · On Tue, 22 Jun 2021 23:02:26 +0200 "Yann E. MORIN" <[email protected]>…

Herv?, All,

On 2021-06-21 16:11 +0200, Herve Codina spake thusly:

Quoted 4 lines

This fixup is done at pkg-generic level for all packages. So, it is no more needed in owfs package.

When you respin the series, you can carry my:

This will be done.

Herv?

Signature · 3 lines
Herv? Codina, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#64 lore raw

PATCH 12/15 package/pkg-generic.mk: Generate final rsync exclude file list

In reply to Yann E. MORIN

Hi,

Quoted 11 lines · On Tue, 22 Jun 2021 23:15:10 +0200 "Yann E. MORIN" <[email protected]>…

Ideally, we would in fact redirect the package installation to an emoty directory, so that we can actually find what it installs.

However, as has been discussed in the past, this is fraught with unworkable issues. For example, some paths may be hard-coded at configure time and/or build time, and thus the package would still install in the original stagin we presented it with (for target/, this is not an isue, because target/ is never looked at during configure or build, only at install time). Or a package may try to modify an existing file (bad, but still). Or a file just assumes that tdirectory structure exists (bad, but eh...)

Modifying existing files is no more allowed -> Overwrites detection.

Quoted 9 lines

So, yes, the rsync exclusion list is a good workaround.

I see you have provided detailed comit logs, that is great. Still, for such core stuff, I think they still miss the bigger picture, like I explained above for example, and which should have been part of the commit log to explain why we resort to an exclusion list rather than the more obvious and simple empty-DESTDIR.

I can add at the end of commit log :
--- 8< ---
Using an empty directory for per-package installation directory would be the simplest way to find what a package installs.
However, as it has been discussed in the past, this is fraught with unworkable issues. For example, some paths may be hard-coded at configure time and/or build time, and thus the package would still install in the original stagin we presented it with (for target/, this is not an isue, because target/ is never looked at during configure or build, only@install time). Or a package installation process just assumes that the directory structure exists (bad, but eh...). --- 8< ---

Is that ok for you ?

Quoted 4 lines
Quoted 4 lines
package/pkg-generic.mk +4 -0
+define pkg_final_rsync_after
+	cd $(1); \
+	LC_ALL=C find . \( -type f -o -type l \) -printf '%T@:%i:%#m:%y:%s,%p\n' \
+		| LC_ALL=C sort > $($(PKG)_DIR)/.files-final-rsync$(2).after
Quoted 8 lines

package/pkg-generic.mk define pkg_size_after · line 186

Quoted 6 lines
package/pkg-generic.mk +6 -0
+	LC_ALL=C comm -2 \
+		$($(PKG)_DIR)/.files-final-rsync$(2).before \
+		$($(PKG)_DIR)/.files-final-rsync$(2).after \
+		| sed -r -e 's/^[^,]+,./- /' \
+		> $($(PKG)_DIR)/.files-final-rsync$(2).exclude_rsync
+	rm -f $($(PKG)_DIR)/.files-final-rsync$(2).after  

You forgot to remove .files-final-rsync$(2).before

I cannot remove it in this patch.

.files-final-rsync$(2).before is generated by .stamp_configured rule calling pkg_final_rsync_before call and if we run 'make && make foo-rebuild' the file generated by first make invocation will be no more present for second make invocation.

We can remove .files-final-rsync$(2).before only after patch "package/pkg-generic.mk: Fix per-package <pkg>-{reconfigure,rebuild,reinstall}" Indeed, in this patch, pkg_final_rsync_after is called after the per-package host and target dir are recreated from scratch.

I will remove .files-final-rsync$(2).before in a dedicated patch after "package/pkg-generic.mk: Fix per-package <pkg>-{reconfigure,rebuild,reinstall}"

Is that ok for you ?

Herv?

Signature · 3 lines
Herv? Codina, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#80 lore raw

cover Overwritten file detection and fixes, one more step to TLP build

In reply to Herve Codina

Hi Herve, all,

Quoted 5 lines · Herve Codina

Hi,

This series fixes some issues in the TLP build feature.
The starting point was the work done by Thomas Petazzoni on overwrites detection (PATCH 1).

after a long time I have recently picked up playing with TLP build again. As part of that I found a problem in qt5 where I sent a patch two days ago "qt5: Fix sporadic build failure during top-level parallel build" which is caused by manipulation of the hard-linked qt.conf in HOST_DIR from different qt5 packages.

Now I see your promising set, but the matter is quite complex and so I wonder if the "overwritten file detection" would a) uncover the qt.conf problem and b)if your patch series somehow fixes it in a generic way?

regards,
Andreas

Quoted 68 lines

Based on this work, this series fixes some overwrites, introduces and uses <PKG>_PER_PACKAGE_TWEAK_HOOKS. This variable is used to collect specific per-package path tweaks needed by per-package build.
With this tweaks collected in this variable, they can be re-applied when they are needed (<pkg>_{reconfigure,rebuild,reinstall}).

It also reworks the final {HOST,TARGET}_DIR rsync.
For each packages only files generated by the package are rsynced (PATCH 13). This is done using an exclusion list computed in (PATCH 12).
In order to avoid modifications performed in target-finalize step to be seen as overwrites on next build, the final {HOST,TARGET}_DIR do not contain any hardlink anymore and are a full copy of expected files and directories (PATCH 14).

The last patch (PATCH 15) fixes <pkg>_{reconfigure,rebuild,reinstall} recreating per-package {HOST,TARGET}_DIR from scratch.
Indeed, for a given package, re-generating the same file is detected as an overwrite. The simplest way to avoid this is to fully discard previous per-package {HOST,TARGET}_DIR directory and recreate them.

To have a TLP build fully fonctionnal, several things are still missing:
- Be sure that all buildroot packages do not perform any overwrites.
- Undo tweak stuff that have be done for per-package build (fixup-libtool-files, fixup-python-files, <PKG>_PER_PACKAGE_TWEAK_HOOKS) after collecting files in final {HOST,STAGING}_DIR.
This mainly means reverting .../per-package/xxxxx/... path to point to the correct ones in final {HOST,STAGING}_DIR.

I hope this series will help in TLP build feature.

Best regards,
Herv? Codina

Herve Codina (14):
package/e2fsprogs: fix fsck overwrite in HOST_DIR package/pkg-generic.mk: Remove Info documents dir entry package/pkg-generic.mk: Fix .la files overwrite detection package/pkg-generic.mk: Perform .la files fixup in per-package
HOST_DIR
package/pkg-generic.mk: Introduce <PKG>_PER_PACKAGE_TWEAK_HOOKS package/apr-util: Use <PKG>_PER_PACKAGE_TWEAK_HOOKS package/apache: Move APACHE_FIXUP_APR_LIBTOOL to
<PKG>_PER_PACKAGE_TWEAK_HOOKS
package/pkg-python: Remove _sysconfigdata*.pyc files when
_sysconfigdata*.py are changed
package/pkg-generic.mk: Move python fixup to generic package
infrastructure
package/owfs: Remove Python sysconfigdata fixup package/pkg-generic.mk: Generate final rsync exclude file list
Makefile: Rsync global {TARGET,HOST}_DIR using exclusion file list
Makefile: Breaks hardlinks in global {TARGET,HOST}_DIR on per-package
build
package/pkg-generic.mk: Fix per-package

     <pkg>-{reconfigure,rebuild,reinstall}

Thomas Petazzoni (1):
package/pkg-generic.mk: detect files overwritten in TARGET_DIR and
HOST_DIR

  Makefile                       |  24 +++++++-
  package/apache/apache.mk       |   2 +-
  package/apr-util/apr-util.mk   |  12 ++--
  package/e2fsprogs/e2fsprogs.mk |   1 +
  package/owfs/owfs.mk           |   9 ---
  package/pkg-generic.mk         | 103 ++++++++++++++++++++++++++++++++-
  package/pkg-python.mk          |  10 ----
  7 files changed, 134 insertions(+), 27 deletions(-)
#58 lore raw

PATCH 10/15 package/pkg-generic.mk: Move python fixup to generic package infrastructure

In reply to Herve Codina

Herv?, All,

On 2021-06-25 10:22 +0200, Herve Codina spake thusly:

Quoted 9 lines
Quoted 7 lines · On Tue, 22 Jun 2021 23:01:03 +0200 "Yann E. MORIN" <[email protected]>…

So, earlier in the series, you introduced _PER_PACKAGE_TWEAK_HOOKS (or whatever the name it will eventually bear), stating that infras could set it, but here you are not taking the chance to demonstrate this, and instead you are inserting the python fixups in an ad-hoc way.

Which also illustrates that the libtool fixups should be converted to the _PER_PACKAGE_TWEAK_HOOKS, probably...

Quoted 8 lines

Yes that's perfectly true.

But do you think we need it right now in this patch series. I really would prefer to keep it as it is for this patch series.

The series is already quite complex and I think that converting libtool and python fixups to PER_PACKAGE_TWEAK_HOOKS (or other name) can be done afterward.

I am always a bit wary about the "cleanup can be done in a later pass" excuse, because that later pass almost never manifests.

But still, the issue is real, and we better have a solution rather than nothing, and we better have it earlier rather than later. So yes, keep things as they are.

Regards,
Yann E. MORIN, first-shoted in the morning!

Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#65 lore raw

PATCH 12/15 package/pkg-generic.mk: Generate final rsync exclude file list

In reply to Herve Codina

Herv?, All,

On 2021-06-25 11:05 +0200, Herve Codina spake thusly:

Quoted 13 lines
Quoted 11 lines · On Tue, 22 Jun 2021 23:15:10 +0200 "Yann E. MORIN" <[email protected]>…

Ideally, we would in fact redirect the package installation to an emoty directory, so that we can actually find what it installs.

However, as has been discussed in the past, this is fraught with unworkable issues. For example, some paths may be hard-coded at configure time and/or build time, and thus the package would still install in the original stagin we presented it with (for target/, this is not an isue, because target/ is never looked at during configure or build, only at install time). Or a package may try to modify an existing file (bad, but still). Or a file just assumes that tdirectory structure exists (bad, but eh...)

Quoted 1 line

Modifying existing files is no more allowed -> Overwrites detection.

Exactly. But what I mean is that, for packages that really need to modify an existing file (e.g. to register themselves against another package's registry), providing an empty target/ is not viable. Yes, that will trigger the overwrite-detection, and we will have to find a proper solution for those. Just, installing in an empty target would not work.

So yes, the list of installed files is still the best solution.

Quoted 12 lines

I can add at the end of commit log :
--- 8< ---
Using an empty directory for per-package installation directory would be the simplest way to find what a package installs.
However, as it has been discussed in the past, this is fraught with unworkable issues. For example, some paths may be hard-coded at configure time and/or build time, and thus the package would still install in the original stagin we presented it with (for target/, this is not an isue, because target/ is never looked at during configure or build, only at install time). Or a package installation process just assumes that the directory structure exists (bad, but eh...). --- 8< ---

Quoted 1 line

Is that ok for you ?

Yes. The registration thing is only just a corner case, so needs not be in the commit log.

Quoted 13 lines
Quoted 4 lines
Quoted 4 lines
package/pkg-generic.mk +4 -0
+define pkg_final_rsync_after
+	cd $(1); \
+	LC_ALL=C find . \( -type f -o -type l \) -printf '%T@:%i:%#m:%y:%s,%p\n' \
+		| LC_ALL=C sort > $($(PKG)_DIR)/.files-final-rsync$(2).after
Quoted 8 lines

package/pkg-generic.mk define pkg_size_after · line 186

Quoted 6 lines
package/pkg-generic.mk +6 -0
+	LC_ALL=C comm -2 \
+		$($(PKG)_DIR)/.files-final-rsync$(2).before \
+		$($(PKG)_DIR)/.files-final-rsync$(2).after \
+		| sed -r -e 's/^[^,]+,./- /' \
+		> $($(PKG)_DIR)/.files-final-rsync$(2).exclude_rsync
+	rm -f $($(PKG)_DIR)/.files-final-rsync$(2).after  

You forgot to remove .files-final-rsync$(2).before

Quoted 6 lines

I cannot remove it in this patch.

.files-final-rsync$(2).before is generated by .stamp_configured rule calling pkg_final_rsync_before call and if we run 'make && make foo-rebuild' the file generated by first make invocation will be no more present for second make invocation.

Ah, yes, I see now. Thanks.

Quoted 9 lines

We can remove .files-final-rsync$(2).before only after patch "package/pkg-generic.mk: Fix per-package <pkg>-{reconfigure,rebuild,reinstall}" Indeed, in this patch, pkg_final_rsync_after is called after the per-package host and target dir are recreated from scratch.

I will remove .files-final-rsync$(2).before in a dedicated patch after "package/pkg-generic.mk: Fix per-package <pkg>-{reconfigure,rebuild,reinstall}"

Is that ok for you ?

Yes, it is. ?

Regards,
Yann E. MORIN.

Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#69 lore raw

PATCH 13/15 Makefile: Rsync global {TARGET, HOST}_DIR using exclusion file list

In reply to Yann E. MORIN

Hi,

Quoted 18 lines · On Thu, 24 Jun 2021 22:34:10 +0200 "Yann E. MORIN" <[email protected]>…

Herv?, All,

Additional revview I forgot previously...

On 2021-06-24 22:20 +0200, Yann E. MORIN spake thusly:

Quoted 13 lines

On 2021-06-21 16:11 +0200, Herve Codina spake thusly:

Quoted 6 lines

On a per-package build, rsync final {TARGET,HOST}_DIR using exclusion file list computed for each packages.
As we rsync only files generated by each packages, we need to to do this rsync recursively on each dependencies to collect all needed files. This is done based on existing <PKG>_FINAL_RECURSIVE_DEPENDENCIES

I am not sure I understand why this is needed...

One thing that comes to mind is speedup. Indeed, now that we can ensure that packages only install new files and don;t change existing files, we can speedup the final aggregation by just handling the new files.

Quoted 7 lines
Quoted 4 lines

But since we are using hardlinks, the aggregation is rather fast...

Can you expand on the rationale in the commit log when you respin, please?

Now I have seen the next patch, it makes sense. Yet, it should be explained here nonetheless. ;-)

The next patch just breaks the hardlinks had nothing to do with the use of <PKG>_FINAL_RECURSIVE_DEPENDENCIES here.

Previously files were collected using a simple rsync and iterating on $(PACKAGES). The rsync collected all files from current package host or target so it collected files generated by the current package and the packages it depends on.

Now, we collect only files generated by the current package and we discard files generated by the packages it depends on.
Iterating on $(PACKAGES) is no longer enough.

Indeed $(PACKAGES) is the list of packages that are enabled in .config file (ie selected during 'make menuconfig' or defconfig file)
But some packages (sure for host packages) can have some "hidden" dependencies by hidden I means not seen or selected by config files.
This "hidden" dependencies are created at Makefile level.

HOST_FOO_DEPENDENCIES = host-bar

We need to collect files from this kind on dependency to generate the final HOST_DIR.

<PKG>_FINAL_RECURSIVE_DEPENDENCIES contains all makefile level dependencies a given package depends on.

Finally, we do:
for pkg selected in .config ($(PACKAGES))
# Add current pkg and packages it depends on at makefile level FULL_PACKAGES_AND_DEPENDS += pkg + $(<PKG>_FINAL_RECURSIVE_DEPENDENCIES)
And then we call sort $(FULL_PACKAGES_AND_DEPENDS) to remove duplicates

I can add in the commit log message:
---- 8< ----
Using only packages present in $(PACKAGES) is no longer enough. Indeed, we collect only files generated for a given package
and $(PACKAGES) contains the list of packages selected in .config file Some dependencies exists at Makefile level that are no [email protected] file level (at least for host packages).
HOST_FOO_DEPENDENCIES += host-bar
To take into account these dependencies, <PKG>_FINAL_RECURSIVE_DEPENDENCIES is used in addition to $(PACKAGES).
---- 8< ----

Is that ok for you ?

Quoted 28 lines

[--SNIP--]

Quoted 5 lines

Makefile TARGET_DIR_FILES_LISTS = $(sort $(wildcard $(BUILD_DIR)/*/.files-list.txt)) · line 755

Quoted 5 lines
Makefile +5 -0
+	$(foreach pkg,$(1),\
+		rsync -a --link-dest=$(PER_PACKAGE_DIR)/$(pkg)/$(strip $(2))/ \
+		--filter='merge $($(call UPPERCASE,$(pkg))_DIR)/$(strip $(4))' \
+		$(PER_PACKAGE_DIR)/$(pkg)/$(strip $(2))/ \
+		$(3)$(sep))  

[--SNIP--]

Quoted 6 lines

Furthermore, this rsync filter is non-obvious. It would be nice to explain the commit log that the list is already a proper filer, not just a list.
Additionally, in the commit that generates that list, the commit log should also mention that a list of filters is created, not a list of filenames.

And to begin with, why are we generating an _exclude_ list, rather than an _include_ list?

I.e. rather than rsync everything except what we don't want, why not just rsync just what we want, and generate a list of filters about exactly just the files installed by a package?

I.e. in previous patch, crea create '+' filters rather than '-' filters. Also, I think using the full-length filter 'include' is nicer than the shorter '+'.

Probably something like:

Quoted 8 lines
    LC_ALL=C comm -1 \
        $($(PKG)_DIR)/.files-final-rsync$(2).before \
        $($(PKG)_DIR)/.files-final-rsync$(2).after \
        | sed -r -e 's/^[^,]+,./include /' \
        > $($(PKG)_DIR)/.files-final-rsync$(2).exclude_rsync  

(Totally untested, slippery code, excersise caution.)

Inclusion filter is not so simple ...

In rsync man, we can read:
---- 8< ----
Note that, when using the --recursive (-r) option (which is implied by -a), every subdir component of every path is visited left to right, with each directory having a chance for exclusion before its content. In this way include/exclude patterns are applied recursively to the pathname of each node in the filesystem's tree (those inside the transfer).
The exclude patterns short-circuit the directory traversal
stage as rsync finds the files to send.

For instance, to include "/foo/bar/baz", the directories "/foo" and "/foo/bar" must not be excluded. Excluding one of those parent directories prevents the examination of its content, cutting off rsync's recursion into those paths and rendering the include for "/foo/bar/baz" ineffectual (since rsync can't match something it never sees in the cut-off section of the directory hierarchy).

The concept path exclusion is particularly important when using a trailing '*' rule. For instance, this won't work:

+ /some/path/this-file-will-not-be-found
+ /file-is-included
- *

This fails because the parent directory "some" is excluded by the '*' rule, so rsync never visits any of the files in the "some" or "some/path" directories. One solution is to ask for all directories in the hierarchy to be included by using a single rule: "+ */" (put it somewhere before the "- *" rule), and perhaps use the --prune-empty-dirs option.
Another solution is to add specific include rules for all the parent dirs that need to be visited. For instance, this set of rules works fine:

+ /some/
+ /some/path/
+ /some/path/this-file-is-found
+ /file-also-included
- *

---- 8< ----

I feel tricky and error prone to use include filter and really simpler to take the problem on the other side.
Instead of including what we want, exclude what we don't want.

The filter file passed to rsync is as simple as:
- /foo/bar/not_thisone
- /foo/not_thisone_too
- /not_thisone

With this filter file, rsync will include all files that are not mentioned without any more needs.

Regards,
Herv?

Signature · 3 lines
Herv? Codina, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#71 lore raw

PATCH 13/15 Makefile: Rsync global {TARGET, HOST}_DIR using exclusion file list

In reply to Yann E. MORIN
Quoted 6 lines · On Thu, 24 Jun 2021 22:20:10 +0200 "Yann E. MORIN" <[email protected]>…

Herv?, All,

Nitpick, applicable to most of your patches in this series: there should not be an uppercase in the title, after the colon sign, and the tense should be imperative, e.g:

Ok, thanks.

Quoted 33 lines

On 2021-06-21 16:11 +0200, Herve Codina spake thusly:

Quoted 6 lines

On a per-package build, rsync final {TARGET,HOST}_DIR using exclusion file list computed for each packages.
As we rsync only files generated by each packages, we need to to do this rsync recursively on each dependencies to collect all needed files. This is done based on existing <PKG>_FINAL_RECURSIVE_DEPENDENCIES

I am not sure I understand why this is needed...

One thing that comes to mind is speedup. Indeed, now that we can ensure that packages only install new files and don;t change existing files, we can speedup the final aggregation by just handling the new files.

But since we are using hardlinks, the aggregation is rather fast...

Can you expand on the rationale in the commit log when you respin, please?

Quoted 13 lines
---
 Makefile | 24 ++++++++++++++++++++++--
 1 file changed, 22 insertions(+), 2 deletions(-)
Makefile +0 -0
index 472af5a318..50bc6b4503 100644
@@ -740,10 +740,28 @@ TARGET_DIR_FILES_LISTS = $(sort $(wildcard $(BUILD_DIR)/*/.files-list.txt))
 HOST_DIR_FILES_LISTS = $(sort $(wildcard $(BUILD_DIR)/*/.files-list-host.txt))
 STAGING_DIR_FILES_LISTS = $(sort $(wildcard $(BUILD_DIR)/*/.files-list-staging.txt))
 
Quoted 10 lines

Makefile TARGET_DIR_FILES_LISTS = $(sort $(wildcard $(BUILD_DIR)/*/.files-list.txt)) · line 750

Quoted 8 lines
Makefile +8 -0
+ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y)
+# rsync the contents of per-package directories
+# $1: space-separated list of packages to rsync from
+# $2: 'host' or 'target'
+# $3: destination directory
+# $4: exclude file list to use
+define per-package-rsync-delta
+	mkdir -p $(3)  
    $(Q)mkdir -p $(3)

Changed.

Quoted 13 lines

Makefile TARGET_DIR_FILES_LISTS = $(sort $(wildcard $(BUILD_DIR)/*/.files-list.txt)) · line 755

Quoted 6 lines
Makefile +5 -0
+	$(foreach pkg,$(1),\
+		rsync -a --link-dest=$(PER_PACKAGE_DIR)/$(pkg)/$(strip $(2))/ \
+		--filter='merge $($(call UPPERCASE,$(pkg))_DIR)/$(strip $(4))' \
+		$(PER_PACKAGE_DIR)/$(pkg)/$(strip $(2))/ \
+		$(3)$(sep))  

I would read it more easily if the lines after rsync are indented furhter, to represent the fact that they are stilll rsync arguments. Also, proably it would be nice to have this quiet by default. Indeed, with rather-deep dependency chains, there can be quite a lot of rsync lines...

Quoted 5 lines
    $(foreach pkg,$(1),\
        $(Q)rsync -a --link-dest=$(PER_PACKAGE_DIR)/$(pkg)/$(strip $(2))/ \
            --filter='merge $($(call UPPERCASE,$(pkg))_DIR)/$(strip $(4))' \
            $(PER_PACKAGE_DIR)/$(pkg)/$(strip $(2))/ \
            $(3)$(sep))

Changed

Quoted 22 lines

Furthermore, this rsync filter is non-obvious. It would be nice to explain the commit log that the list is already a proper filer, not just a list.

Additionally, in the commit that generates that list, the commit log should also mention that a list of filters is created, not a list of filenames.

Makefile TARGET_DIR_FILES_LISTS = $(sort $(wildcard $(BUILD_DIR)/*/.files-list.txt)) · line 764

Quoted 10 lines
Makefile +6 -1
+endef
+endif
+
 .PHONY: host-finalize
 host-finalize: $(PACKAGES) $(HOST_DIR) $(HOST_DIR_SYMLINK)
 	@$(call MESSAGE,"Finalizing host directory")
-	$(call per-package-rsync,$(sort $(PACKAGES)),host,$(HOST_DIR))
+	$(call per-package-rsync-delta, $(sort $(foreach pkg, $(PACKAGES), \
+		$(pkg) $($(call UPPERCASE,$(pkg))_FINAL_RECURSIVE_DEPENDENCIES))), \
+		host,$(HOST_DIR),.files-final-rsync-host.exclude_rsync)  

This is unparseable by a human... :-(

Quoted 8 lines
    $(call per-package-rsync-delta, \
        $(sort $(foreach pkg, $(PACKAGES), \
                    $(pkg) $($(call UPPERCASE,$(pkg))_FINAL_RECURSIVE_DEPENDENCIES)) \
        ), \
        host, \
        $(HOST_DIR), \
        .files-final-rsync-host.exclude_rsync \
    )

Changed.

Quoted 13 lines

Makefile staging-finalize: $(STAGING_DIR_SYMLINK) · line 774

Quoted 11 lines
Makefile +3 -1
 .PHONY: staging-finalize
 staging-finalize: $(STAGING_DIR_SYMLINK)
@@ -751,7 +769,9 @@ staging-finalize: $(STAGING_DIR_SYMLINK)
 .PHONY: target-finalize
 target-finalize: $(PACKAGES) $(TARGET_DIR) host-finalize
 	@$(call MESSAGE,"Finalizing target directory")
-	$(call per-package-rsync,$(sort $(PACKAGES)),target,$(TARGET_DIR))
+	$(call per-package-rsync-delta, $(sort $(foreach pkg, $(PACKAGES), \
+		$(pkg) $($(call UPPERCASE,$(pkg))_FINAL_RECURSIVE_DEPENDENCIES))), \
+		target,$(TARGET_DIR),.files-final-rsync.exclude_rsync)  

Ditto.

Changed

Quoted 8 lines

But now that I review this patch, I notice that there is still a hole in our file-overwrite detection. It only handles files modified by a package, that comes from it dependencies.

However, if two packages in two different depednency chains, install the same file, then we're still toast. For example:

Quoted 9 lines
    A --> B --> C
      `-> D

If C and D install the same file, then we don't detect the conflict, and we can't easil predict which the final file will come from...

Of course, such a detection will have to be *in addition* to what this series already does, so this new issue does not invalidate this series.

I missed this use case.

You you think it can help to keep in mind if
(1) I add it in my v2 my cover letter
or
(2) I add it in this patch commit log

Herv?

Signature · 3 lines
Herv? Codina, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#70 lore raw

PATCH 13/15 Makefile: Rsync global {TARGET, HOST}_DIR using exclusion file list

In reply to Herve Codina

Herv?, All,

On 2021-06-25 13:59 +0200, Herve Codina spake thusly:

Quoted 18 lines
Quoted 16 lines · On Thu, 24 Jun 2021 22:34:10 +0200 "Yann E. MORIN" <[email protected]>…

On 2021-06-24 22:20 +0200, Yann E. MORIN spake thusly:

Quoted 15 lines

On 2021-06-21 16:11 +0200, Herve Codina spake thusly:

Quoted 6 lines

On a per-package build, rsync final {TARGET,HOST}_DIR using exclusion file list computed for each packages.
As we rsync only files generated by each packages, we need to to do this rsync recursively on each dependencies to collect all needed files. This is done based on existing <PKG>_FINAL_RECURSIVE_DEPENDENCIES

I am not sure I understand why this is needed...

One thing that comes to mind is speedup. Indeed, now that we can ensure that packages only install new files and don;t change existing files, we can speedup the final aggregation by just handling the new files.

But since we are using hardlinks, the aggregation is rather fast...

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

Can you expand on the rationale in the commit log when you respin, please?

Now I have seen the next patch, it makes sense. Yet, it should be explained here nonetheless. ;-)

The next patch just breaks the hardlinks had nothing to do with the use of <PKG>_FINAL_RECURSIVE_DEPENDENCIES here.

Oh, I was not commenting about the _FINAL_RECURSIVE_DEPENDENCIES, but about the need for an exclusion list to begin with. I totally agree witht the rationale for using _FINAL_RECURSIVE_DEPENDENCIES.

[--SNIP--]

Quoted 2 lines
Quoted 2 lines

And to begin with, why are we generating an _exclude_ list, rather than an _include_ list?

[--SNIP--]

Quoted 4 lines
Quoted 1 line

(Totally untested, slippery code, excersise caution.)

Inclusion filter is not so simple ...

In rsync man, we can read:

[--SNIP--]

Indeed, I see now. The rsync filter is not about listing files; it is deciding what should be or should not be transferred. Anything that is not filtered-out is transferred.

So OK, using an exclude list toally makes sense.

Regards,
Yann E. MORIN.

Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#81 lore raw

cover Overwritten file detection and fixes, one more step to TLP build

In reply to Andreas Naumann

Hi,

Quoted 11 lines · On Fri, 25 Jun 2021 11:08:43 +0200 Andreas Naumann <[email protected]> wrote:

after a long time I have recently picked up playing with TLP build again. As part of that I found a problem in qt5 where I sent a patch two days ago "qt5: Fix sporadic build failure during top-level parallel build" which is caused by manipulation of the hard-linked qt.conf in HOST_DIR from different qt5 packages.

Now I see your promising set, but the matter is quite complex and so I wonder if the "overwritten file detection" would a) uncover the qt.conf problem and b)if your patch series somehow fixes it in a generic way?

I quickly looked at your patch and it looks like an overwrite issue.

Your patch:
---- 8< ----

 # compiled into the Qt library. We need it to make "qmake" relocatable and
 # tweak the per-package install pathes
 define QT5_INSTALL_QT_CONF
+	rm -f $(HOST_DIR)/bin/qt.conf
 	sed -e "s|@@HOST_DIR@@|$(HOST_DIR)|" -e "s|@@STAGING_DIR@@|$(STAGING_DIR)|" \
 		$(QT5BASE_PKGDIR)/qt.conf.in > $(HOST_DIR)/bin/qt.conf
 endef
---- 8< ----

This overwrite is done by sed whose output is redirected $(HOST_DIR)/bin/qt.conf

With TLP this overwite leads to package A seeing modification done by package B

Without your patch and with overwrite detection present in my series the overwrite should be detected and the build stopped.

My series do not contains any fixes for this specific Qt5 overwrite. This is a concrete example of what I said still missing ...

Your 'rm' breaks the hardlink and so the following sed does not perform overwrite anymore.
That's the correct fix.

To go further with interactions with my current series still ongoing and not ready to be merged, the entire macro (QT5_INSTALL_QT_CONF) should be moved to <PKG>_PER_PACKAGE_TWEAK_HOOKS and QT5_QT_CONF_FIXUP is no more needed.

Herv?

Signature · 3 lines
Herv? Codina, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#76 lore raw

PATCH 15/15 package/pkg-generic.mk: Fix per-package <pkg>-{reconfigure, rebuild, reinstall}

In reply to Yann E. MORIN
Quoted 50 lines · On Thu, 24 Jun 2021 22:44:42 +0200 "Yann E. MORIN" <[email protected]>…

Herv?, All,

On 2021-06-21 16:11 +0200, Herve Codina spake thusly:

Quoted 47 lines

Many overwrites are detected on <pkg>-{reconfigure,rebuild,reinstall}. Indeed, files previously installed by a package were detected as overwritten on next reconfigure, rebuild or reinstall.

To avoid this, we recreate per-package host and target dir from scratch as it was done during the first configure step.

---
 package/pkg-generic.mk | 26 ++++++++++++++++++++++++++
 1 file changed, 26 insertions(+)
package/pkg-generic.mk +20 -0
index 546370af7a..0aafd865ad 100644
@@ -470,6 +470,13 @@ define pkg-graph-depends
 		$$(GRAPHS_DIR)/$$(@).dot
 endef
 
+ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y)
+define empty-per-package-directory
+	rm -rf $(HOST_DIR) $(TARGET_DIR)
+	mkdir -p $(HOST_DIR) $(TARGET_DIR)
+endef
+endif
+
 ################################################################################
 # inner-generic-package -- generates the make targets needed to build a
 # generic package
@@ -1072,6 +1079,25 @@ endif
 			rm -f $$($(2)_TARGET_INSTALL_TARGET)
 			rm -f $$($(2)_TARGET_INSTALL_IMAGES)
 			rm -f $$($(2)_TARGET_INSTALL_HOST)
+			$$(call empty-per-package-directory)
+			$$(call prepare-per-package-directory,$$($(2)_FINAL_DOWNLOAD_DEPENDENCIES))
+			$$(call prepare-per-package-directory,$$($(2)_FINAL_EXTRACT_DEPENDENCIES))
+			$$(call prepare-per-package-directory,$$($(2)_FINAL_DEPENDENCIES))
+			@$$(call pkg_final_rsync_before,$$(TARGET_DIR))
+			@$$(call pkg_final_rsync_before,$$(HOST_DIR),-host)
+			@$$(call pkg_size_before,$$(TARGET_DIR))
+			@$$(call pkg_size_before,$$(STAGING_DIR),-staging)
+			@$$(call pkg_size_before,$$(HOST_DIR),-host)
+			$$(call fixup-libtool-files,$$(NAME),$$(HOST_DIR))
+			$$(call fixup-libtool-files,$$(NAME),$$(STAGING_DIR))
+			$$(call fixup-python-files,$$(NAME),$$(HOST_DIR))
+			$$(call fixup-python-files,$$(NAME),$$(STAGING_DIR))
Quoted 8 lines

package/pkg-generic.mk endif · line 1097

Quoted 3 lines
package/pkg-generic.mk +3 -0
+			$$(foreach hook,$$($(2)_PER_PACKAGE_TWEAK_HOOKS),$$(call $$(hook))$$(sep))
+			@$$(call pkg_detect_overwrite_before,$$(TARGET_DIR))
+			@$$(call pkg_detect_overwrite_before,$$(HOST_DIR),-host)  

I don;t like this duplication, because it will inevitably diverge. And already, it is different: rather than rely on $(PKG)_FINAL_DEPENDENCIES) and a single call to prepare-per-package-directory, there are now three calls to prepare-per-package-directory with three types of depenencies.

The basic sequence (out of <pkg>_{reconfigure, rebuild, reinstall} is:
.stamp_downloaded
-> call prepare-per-package-directory with <PKG>_FINAL_DOWNLOAD_DEPENDENCIES
.stamp_extracted
-> call prepare-per-package-directory with <PKG>_FINAL_EXTRACT_DEPENDENCIES
.stamp_configured
-> call prepare-per-package-directory with <PKG>_FINAL_FINAL_DEPENDENCIES

Here, I wanted to recreate host and target dir with all dependencies as they were at previous .stamp_configure and so did the 3 calls.

Quoted 4 lines

Why can't we comonalise this into a big macro that gets expanded twise, once in the original $(BUILD_DIR)/%/.stamp_configured rule, and once here?

Excluding call to prepare-per-package-directory as they are not the same.

What name for this macro ? PREPARE_PRE_CONFIGURE ?

Quoted 9 lines

This would also termendously help into factorising this away and into its own $(1)-prepare step, eventually.

package/pkg-generic.mk endif · line 1100

Quoted 2 lines
package/pkg-generic.mk +2 -0
+$(1)-clean-for-reinstall: PKG=$(2)
+$(1)-clean-for-reinstall: NAME=$(1)  

Please move those two right before the actual rule (i.e. just before line 991 in mastr as of today.

Done

Quoted 14 lines

Regards,
Yann E. MORIN.

Quoted 9 lines
package/pkg-generic.mk +0 -0
 $(1)-reinstall:		$(1)-clean-for-reinstall $(1)
 
Signature · 6 lines
2.31.1

_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot  
Signature · 3 lines
Herv? Codina, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#82 lore raw

cover Overwritten file detection and fixes, one more step to TLP build

In reply to Herve Codina

Hi Herve,

Quoted 32 lines · Herve Codina

Hi,

Quoted 11 lines · On Fri, 25 Jun 2021 11:08:43 +0200 Andreas Naumann <[email protected]> wrote:

after a long time I have recently picked up playing with TLP build again. As part of that I found a problem in qt5 where I sent a patch two days ago "qt5: Fix sporadic build failure during top-level parallel build" which is caused by manipulation of the hard-linked qt.conf in HOST_DIR from different qt5 packages.

Now I see your promising set, but the matter is quite complex and so I wonder if the "overwritten file detection" would a) uncover the qt.conf problem and b)if your patch series somehow fixes it in a generic way?

I quickly looked at your patch and it looks like an overwrite issue.

Your patch:
---- 8< ----

  # compiled into the Qt library. We need it to make "qmake" relocatable and
  # tweak the per-package install pathes
  define QT5_INSTALL_QT_CONF
(quoted diff) +1 -1
+	rm -f $(HOST_DIR)/bin/qt.conf
  	sed -e "s|@@HOST_DIR@@|$(HOST_DIR)|" -e "s|@@STAGING_DIR@@|$(STAGING_DIR)|" \
  		$(QT5BASE_PKGDIR)/qt.conf.in > $(HOST_DIR)/bin/qt.conf
  endef
---- 8< ----

This overwrite is done by sed whose output is redirected $(HOST_DIR)/bin/qt.conf

Quoted 7 lines

With TLP this overwite leads to package A seeing modification done by package B

Without your patch and with overwrite detection present in my series the overwrite should be detected and the build stopped.

My series do not contains any fixes for this specific Qt5 overwrite. This is a concrete example of what I said still missing ...

Thank you for the confirmation.

Quoted 8 lines

Your 'rm' breaks the hardlink and so the following sed does not perform overwrite anymore.
That's the correct fix.

To go further with interactions with my current series still ongoing and not ready to be merged, the entire macro (QT5_INSTALL_QT_CONF) should be moved to <PKG>_PER_PACKAGE_TWEAK_HOOKS and QT5_QT_CONF_FIXUP is no more needed.

Ok, I'll try to catch the point when your set is accepted and the repost an updated version.

have a nice weekend,
Andreas

Quoted 3 lines

Herv?

#41 lore raw

PATCH 6/15 package/pkg-generic.mk: Introduce <PKG>_PER_PACKAGE_TWEAK_HOOKS

In reply to Herve Codina

Hi Yann, All

Quoted 11 lines · On Fri, 25 Jun 2021 09:21:20 +0200 Herve Codina <[email protected]>…

Hi,

Quoted 6 lines · On Tue, 22 Jun 2021 22:39:09 +0200 "Yann E. MORIN" <[email protected]>…

package/pkg-generic.mk $(BUILD_DIR)/%/.stamp_configured: · line 260

Quoted 2 lines
package/pkg-generic.mk +1 -0
 	$(call fixup-libtool-files,$(NAME),$(STAGING_DIR))
+	$(foreach hook,$($(PKG)_PER_PACKAGE_TWEAK_HOOKS),$(call $(hook))$(sep))    

I am going to bikeshed on the naming of this variable (which is a good sing that I am pretty much OK with the feature).

Quoted 6 lines
Quoted 1 line

I think $(2)_EARLY_PRE_CONFIGURE_HOOKS is more sensible.

That's good for me.

Okay for everyone ?

The more I read the code, the more I think $(2)_EARLY_PRE_CONFIGURE_HOOKS is not a good name.

Actions done in this hook are for package preparation.
Yann talks about $(2)-prepare step and even if this step is not implemented as a buildroot step (.stamp_xxxxxx) this is really a package preparation.

Indeed, we prepare the package for the next step (retrieving dependencies files for per-package host and target dir) and finally we do the fixups. Once done, the package is ready and so we take the snapshot for overwrite detection.

What do you think about $(2)_POST_PREPARE_HOOKS ?

Regards,
Herv?

Signature · 3 lines
Herv? Codina, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#42 lore raw

PATCH 6/15 package/pkg-generic.mk: Introduce <PKG>_PER_PACKAGE_TWEAK_HOOKS

Herv?, All,

On 2021-07-02 09:18 +0200, Herve Codina spake thusly:

Quoted 15 lines
Quoted 8 lines · On Fri, 25 Jun 2021 09:21:20 +0200 Herve Codina <[email protected]>…
Quoted 5 lines · On Tue, 22 Jun 2021 22:39:09 +0200 "Yann E. MORIN" <[email protected]>…

package/pkg-generic.mk $(BUILD_DIR)/%/.stamp_configured: · line 260

Quoted 2 lines
package/pkg-generic.mk +1 -0
 	$(call fixup-libtool-files,$(NAME),$(STAGING_DIR))
+	$(foreach hook,$($(PKG)_PER_PACKAGE_TWEAK_HOOKS),$(call $(hook))$(sep))    

I am going to bikeshed on the naming of this variable (which is a good sing that I am pretty much OK with the feature).
I think $(2)_EARLY_PRE_CONFIGURE_HOOKS is more sensible.

That's good for me.

Actions done in this hook are for package preparation.
Yann talks about $(2)-prepare step and even if this step is not implemented as a buildroot step (.stamp_xxxxxx) this is really a package preparation.

Quoted 6 lines

Indeed, we prepare the package for the next step (retrieving dependencies files for per-package host and target dir) and finally we do the fixups. Once done, the package is ready and so we take the snapshot for overwrite detection.

What do you think about $(2)_POST_PREPARE_HOOKS ?

Works for me.

Regards,
Yann E. MORIN.

Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#78 lore raw

cover Overwritten file detection and fixes, one more step to TLP build

In reply to Arnout Vandecappelle

Hi,

Quoted 2 lines · On Mon, 21 Jun 2021 22:42:18 +0200 Arnout Vandecappelle <[email protected]> wrote:

However, it seems the tests [2][3][4] are missing from this series. Could you
include them next time you post it?

Added in v2 series.

Signature · 3 lines
Herv? Codina, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com