Skip to contentSkip to conversation

Overwritten file detection and fixes, one more step to TLP build

PATCH Applied Herve Codina 6 participants 35 messages Aug 17, 2021 – Oct 13, 2023 Kbuild

lore
Share & tools
lore
#0 lore raw

v3 cover Overwritten file detection and fixes, one more step to TLP build

Hi,

This v3 series adds a new patch (patch 1) to fix a find failure already present in PKG_PYTHON_FIXUP_SYSCONFIG that was highlighted in the previous version of this series.
It also reworks the python fixup according to previous review (patches 2 and 3) and fixes the test given in patch 12.

Two patches are added related to Qt5. The first one (patch 5) was previously sent by Andreas Naumann and fixes a Qt5 overwrite.
https://lore.kernel.org/buildroot/[email protected]/
https://lore.kernel.org/buildroot/[email protected]/
The second one (patch 6) moves the fixup function to the post-prepare hooks.

One more patch was added in this serie (patch 14) to move fixup-libtool-file to the post-prepare hooks. This was mentionned during the review of the first version of this series (https://lore.kernel.org/buildroot/20210625092704.GJ2852@scaer/).

More changes details are given in each individual patches.

Best regards,
Hervé Codina

Andreas Naumann (1):
qt5: Fix sporadic build failure during top-level parallel build

Herve Codina (11):
package/pkg-python: fix PKG_PYTHON_FIXUP_SYSCONFIGDATA package/pkg-python: invalidate precompiled _sysconfigdata*.pyc package/pkg-generic.mk: move python fixup to generic package
infrastructure
package/owfs: remove Python sysconfigdata fixup package/pkg-qmake.mk: Move QT5_QT_CONF_FIXUP to post-prepare hook package/pkg-generic.mk: generate final rsync exclude file list
Makefile: rsync global {TARGET,HOST}_DIR using exclusion file list
Makefile: breaks hardlinks in global {TARGET,HOST}_DIR on per-package
build
package/pkg-generic.mk: fix per-package

    <pkg>-{reconfigure,rebuild,reinstall}

package/pkg-generic.mk: remove .files-final-rsync.before temporary
file
package/pkg-generic.mk: move fixup-libtool-files to post-prepare hook

Thomas Petazzoni (4):
package/pkg-generic.mk: detect files overwritten in TARGET_DIR and
HOST_DIR
support/testing/infra: add log_file_path() function support/testing/tests: add test for check_bin_arch support/testing/tests: add test for file overwrite detection

 Makefile                                      |  34 ++++-
 package/owfs/owfs.mk                          |   9 --
 package/pkg-generic.mk                        | 125 ++++++++++++++++--
 package/pkg-python.mk                         |  10 --
 package/pkg-qmake.mk                          |   3 +-
 package/qt5/qt5.mk                            |   1 +
 support/testing/infra/__init__.py             |  11 +-
 .../br2-external/detect-bad-arch/Config.in    |   1 +
 .../detect-bad-arch/external.desc             |   1 +
 .../br2-external/detect-bad-arch/external.mk  |   1 +
 .../package/detect-bad-arch/Config.in         |   4 +
 .../detect-bad-arch/detect-bad-arch.mk        |  15 +++
 .../br2-external/detect-overwrite/Config.in   |   1 +
 .../detect-overwrite/external.desc            |   1 +
 .../br2-external/detect-overwrite/external.mk |   1 +
 .../package/detect-overwrite/Config.in        |   5 +
 .../detect-overwrite/detect-overwrite.mk      |  19 +++
 support/testing/tests/core/test_bad_arch.py   |  19 +++
 .../testing/tests/core/test_file_overwrite.py |  48 +++++++
 19 files changed, 269 insertions(+), 40 deletions(-)
 create mode 100644 support/testing/tests/core/br2-external/detect-bad-arch/Config.in
 create mode 100644 support/testing/tests/core/br2-external/detect-bad-arch/external.desc
 create mode 100644 support/testing/tests/core/br2-external/detect-bad-arch/external.mk
 create mode 100644 support/testing/tests/core/br2-external/detect-bad-arch/package/detect-bad-arch/Config.in
 create mode 100644 support/testing/tests/core/br2-external/detect-bad-arch/package/detect-bad-arch/detect-bad-arch.mk
 create mode 100644 support/testing/tests/core/br2-external/detect-overwrite/Config.in
 create mode 100644 support/testing/tests/core/br2-external/detect-overwrite/external.desc
 create mode 100644 support/testing/tests/core/br2-external/detect-overwrite/external.mk
 create mode 100644 support/testing/tests/core/br2-external/detect-overwrite/package/detect-overwrite/Config.in
 create mode 100644 support/testing/tests/core/br2-external/detect-overwrite/package/detect-overwrite/detect-overwrite.mk
 create mode 100644 support/testing/tests/core/test_bad_arch.py
 create mode 100644 support/testing/tests/core/test_file_overwrite.py
Signature · 6 lines
2.31.1

_______________________________________________
buildroot mailing list
[email protected]
http://lists.busybox.net/mailman/listinfo/buildroot
#1 lore raw

PATCH v3 1/16 package/pkg-python: fix PKG_PYTHON_FIXUP_SYSCONFIGDATA

Find used by PKG_PYTHON_FIXUP_SYSCONFIGDATA can fail if directories are not present. This failure is silently ignored because find is on the LHS of a pipe.

This commit fixes the find failure. HOST_DIR is used as the starting point and the search is filtered on the expected directories.

This commit also adds -print0 and the $(Q) verbosity flag as minor changes.

---

New patch in this v3 series.

The find failure was highlighted in 836528f03eb89fdc64432f7a7470145ddf553b8c and reverted in b928074e2d83f5b873bf1ba381b212dfad9bf207.

 package/pkg-python.mk | 17 ++++++++++++++---
 1 file changed, 14 insertions(+), 3 deletions(-)
package/pkg-python.mk +14 -3
index 59a48e5a87..cd48402e02 100644
@@ -92,11 +92,22 @@ HOST_PKG_PYTHON_SETUPTOOLS_INSTALL_OPTS = \
 	--root=/ \
 	--single-version-externally-managed
 
+# Make sure python _sysconfigdata*.py files only reference the current
+# per-package directory.
+#
+# Can't use $(foreach d, $(HOST_DIR)/lib/python* $(STAGING_DIR)/usr/lib/python*, ...)
+# because those directories may be created in the same recipe this macro will
+# be expanded in.
+# Additionally, either or both may be missing, which would make find whine and
+# fail.
+# So we just use HOST_DIR as a starting point, and filter on the two directories
+# of interest.
 ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y)
 define PKG_PYTHON_FIXUP_SYSCONFIGDATA
-	find $(HOST_DIR)/lib/python* $(STAGING_DIR)/usr/lib/python* \
-		-name "_sysconfigdata*.py" | xargs --no-run-if-empty \
-		$(SED) "s:$(PER_PACKAGE_DIR)/[^/]\+/:$(PER_PACKAGE_DIR)/$($(PKG)_NAME)/:g"
+	$(Q)find $(HOST_DIR) \
+		\( -path '$(HOST_DIR)/lib/python*' -o -path '$(STAGING_DIR)/usr/lib/python*' \) \
+		-name "_sysconfigdata*.py" -print0 | xargs -0 --no-run-if-empty \
+		$(SED) 's:$(PER_PACKAGE_DIR)/[^/]\+/:$(PER_PACKAGE_DIR)/$($(PKG)_NAME)/:g'
 endef
 endif
 
Signature · 6 lines
2.31.1

_______________________________________________
buildroot mailing list
[email protected]
http://lists.busybox.net/mailman/listinfo/buildroot
#2 lore raw

PATCH v3 2/16 package/pkg-python: invalidate precompiled _sysconfigdata*.pyc

In reply to Herve Codina

For per-package directories, we fixup the _sysconfigdata*.py files, so that they get proper path pointing to the current package's direcotry structure.

However, the corresponding, pre-compiled blobs _sysconfigdata*.pyc were left around, and thus are inconsistent with their source. They might also be regenerated when a package would install a python module; this regeneration would trigger the soon-to-be-introduced overwrite detection.

This commit simply removes _sysconfigdata*.pyc files; they will anyway be regenerated by the PYTHON{,3}_CREATE_PYC_FILES target finalize hooks. This is an efficient way to guarantee the consistency between the source and precompiled versions, and to not trigger the overwrite detection.

---
Changes from v1 to v2:
- Used separate shells for the two 'find' commands.
- Used '-deleted' to remove files

Changes v2 to v3:
- Reworked the find command to avoid failure highlighted in 836528f03eb89fdc64432f7a7470145ddf553b8c and reverted in b928074e2d83f5b873bf1ba381b212dfad9bf207.
- Used find syntax discussed in the previous review.
https://lore.kernel.org/buildroot/[email protected]/
- Kept Yann's commit log.

 package/pkg-python.mk | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
package/pkg-python.mk +3 -1
index cd48402e02..0f65bd0f75 100644
@@ -106,7 +106,9 @@ ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y)
 define PKG_PYTHON_FIXUP_SYSCONFIGDATA
 	$(Q)find $(HOST_DIR) \
 		\( -path '$(HOST_DIR)/lib/python*' -o -path '$(STAGING_DIR)/usr/lib/python*' \) \
-		-name "_sysconfigdata*.py" -print0 | xargs -0 --no-run-if-empty \
+		\(    \( -name "_sysconfigdata*.pyc" -delete \) \
+		   -o \( -name "_sysconfigdata*.py" -print0 \) \) \
+		| xargs -0 --no-run-if-empty \
 		$(SED) 's:$(PER_PACKAGE_DIR)/[^/]\+/:$(PER_PACKAGE_DIR)/$($(PKG)_NAME)/:g'
 endef
 endif
Signature · 6 lines
2.31.1

_______________________________________________
buildroot mailing list
[email protected]
http://lists.busybox.net/mailman/listinfo/buildroot
#3 lore raw

PATCH v3 3/16 package/pkg-generic.mk: move python fixup to generic package infrastructure

In reply to Herve Codina

Fixing _sysconfigdata*.{py,pyc} was previously done by python package infrastructure. Some packages use python stuff without using python package infrastructure.
These packages perform overwrites and need the specific python fixup to fix them.

In order to be sure to fix all of these packages, the python fixup is moved to the generic package infrastructure and applied to all packages.
This follows the same principle as for the .la libtool files fixup.

---
Changes v1 to v2:
- Used '... -print0 | xargs -0 -r ...' construction.
- Used '-deleted' to remove files.

Changes v2 to v3:
- Used find syntax discussed in the previous review.
https://lore.kernel.org/buildroot/[email protected]/

 package/pkg-generic.mk | 24 +++++++++++++++++++++++-
 package/pkg-python.mk  | 23 -----------------------
 2 files changed, 23 insertions(+), 24 deletions(-)
package/pkg-generic.mk +23 -1
index 97ee204b80..38381d93b5 100644
@@ -103,6 +103,27 @@ define fixup-libtool-files
 endef
 endif
 
+# Make sure python _sysconfigdata*.py files only reference the current
+# per-package directory.
+#
+# Can't use $(foreach d, $(HOST_DIR)/lib/python* $(STAGING_DIR)/usr/lib/python*, ...)
+# because those directories may be created in the same recipe this macro will
+# be expanded in.
+# Additionally, either or both may be missing, which would make find whine and
+# fail.
+# So we just use HOST_DIR as a starting point, and filter on the two directories
+# of interest.
+ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y)
+define FIXUP_PYTHON_SYSCONFIGDATA
+	$(Q)find $(HOST_DIR) \
+		\( -path '$(HOST_DIR)/lib/python*' -o -path '$(STAGING_DIR)/usr/lib/python*' \) \
+		\(    \( -name "_sysconfigdata*.pyc" -delete \) \
+		   -o \( -name "_sysconfigdata*.py" -print0 \) \) \
+	| xargs -0 --no-run-if-empty \
+	$(SED) 's:$(PER_PACKAGE_DIR)/[^/]\+/:$(PER_PACKAGE_DIR)/$($(PKG)_NAME)/:g'
+endef
+endif
+
 # Functions to collect statistics about installed files
 
 # $(1): base directory to search in
@@ -836,7 +857,8 @@ $(2)_PRE_LEGAL_INFO_HOOKS       ?=
 $(2)_POST_LEGAL_INFO_HOOKS      ?=
 $(2)_TARGET_FINALIZE_HOOKS      ?=
 $(2)_ROOTFS_PRE_CMD_HOOKS       ?=
-$(2)_POST_PREPARE_HOOKS         ?=
+
+$(2)_POST_PREPARE_HOOKS += FIXUP_PYTHON_SYSCONFIGDATA
 
 ifeq ($$($(2)_TYPE),target)
 ifneq ($$(HOST_$(2)_KCONFIG_VAR),)
package/pkg-python.mk +0 -23
index 0f65bd0f75..e6b81bdfd3 100644
@@ -92,27 +92,6 @@ HOST_PKG_PYTHON_SETUPTOOLS_INSTALL_OPTS = \
 	--root=/ \
 	--single-version-externally-managed
 
-# Make sure python _sysconfigdata*.py files only reference the current
-# per-package directory.
-#
-# Can't use $(foreach d, $(HOST_DIR)/lib/python* $(STAGING_DIR)/usr/lib/python*, ...)
-# because those directories may be created in the same recipe this macro will
-# be expanded in.
-# Additionally, either or both may be missing, which would make find whine and
-# fail.
-# So we just use HOST_DIR as a starting point, and filter on the two directories
-# of interest.
-ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y)
-define PKG_PYTHON_FIXUP_SYSCONFIGDATA
-	$(Q)find $(HOST_DIR) \
-		\( -path '$(HOST_DIR)/lib/python*' -o -path '$(STAGING_DIR)/usr/lib/python*' \) \
-		\(    \( -name "_sysconfigdata*.pyc" -delete \) \
-		   -o \( -name "_sysconfigdata*.py" -print0 \) \) \
-		| xargs -0 --no-run-if-empty \
-		$(SED) 's:$(PER_PACKAGE_DIR)/[^/]\+/:$(PER_PACKAGE_DIR)/$($(PKG)_NAME)/:g'
-endef
-endif
-
 ################################################################################
 # inner-python-package -- defines how the configuration, compilation
 # and installation of a Python package should be done, implements a
@@ -257,8 +236,6 @@ $(2)_PYTHON_INTERPRETER = $$(HOST_DIR)/bin/$$($(2)_NEEDS_HOST_PYTHON)
 endif
 endif
 
-$(2)_PRE_CONFIGURE_HOOKS += PKG_PYTHON_FIXUP_SYSCONFIGDATA
-
 #
 # Build step. Only define it if not already defined by the package .mk
 # file.
Signature · 6 lines
2.31.1

_______________________________________________
buildroot mailing list
[email protected]
http://lists.busybox.net/mailman/listinfo/buildroot
#4 lore raw

PATCH v3 4/16 package/owfs: remove Python sysconfigdata fixup

In reply to Herve Codina

This fixup is done at pkg-generic level for all packages. So, it is no more needed in owfs package.

---
Changes v1 to v2:
- Added 'Reviewed-by: Yann E. MORIN'

Changes v2 to v3:
None

 package/owfs/owfs.mk | 9 ---------
 1 file changed, 9 deletions(-)
package/owfs/owfs.mk +0 -9
index ffc0b3098d..10543d3698 100644
@@ -88,15 +88,6 @@ OWFS_DEPENDENCIES += python host-swig
 # override the PYSITEDIR variable in make.
 OWFS_EXTRA_MAKE_OPTS += PYSITEDIR=/usr/lib/python$(PYTHON_VERSION_MAJOR)/site-packages
 
-ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y)
-define OWFS_FIXUP_PYTHON_SYSCONFIGDATA
-	find $(HOST_DIR)/lib/python* $(STAGING_DIR)/usr/lib/python* \
-		-name "_sysconfigdata*.py" | xargs --no-run-if-empty \
-		$(SED) "s:$(PER_PACKAGE_DIR)/[^/]\+/:$(PER_PACKAGE_DIR)/owfs/:g"
-endef
-OWFS_PRE_CONFIGURE_HOOKS += OWFS_FIXUP_PYTHON_SYSCONFIGDATA
-endif
-
 else
 OWFS_CONF_OPTS += --disable-owpython --without-python
 endif
Signature · 6 lines
2.31.1

_______________________________________________
buildroot mailing list
[email protected]
http://lists.busybox.net/mailman/listinfo/buildroot
#5 lore raw

PATCH v3 5/16 qt5: Fix sporadic build failure during top-level parallel build

In reply to Herve Codina

From: Andreas Naumann <[email protected]>

When using top level parallel build, independent qt5 packages may be built in parallel. Because of their staging dirs being hardlinked, they all use the same qt.conf file to manipulate during configure, while another qt5 package might already use it. This leads to weird build failures because the folders qmake is using are diverted in erratic ways. Fix this by actually recreating a non-shared qt.conf file for every package.

---
New patch in this v3 series

This patch was previously send by Andreas Naumann.
https://lore.kernel.org/buildroot/[email protected]/
https://lore.kernel.org/buildroot/[email protected]/
As this patch makes sense in this series, it was integrated in this v3 version.

 package/qt5/qt5.mk | 1 +
 1 file changed, 1 insertion(+)
package/qt5/qt5.mk +1 -0
index db6ccd2b42..3ffb7b0063 100644
@@ -15,6 +15,7 @@ include $(sort $(wildcard package/qt5/*/*.mk))
 # compiled into the Qt library. We need it to make "qmake" relocatable and
 # tweak the per-package install pathes
 define QT5_INSTALL_QT_CONF
+	rm -f $(HOST_DIR)/bin/qt.conf
 	sed -e "s|@@HOST_DIR@@|$(HOST_DIR)|" -e "s|@@STAGING_DIR@@|$(STAGING_DIR)|" \
 		$(QT5BASE_PKGDIR)/qt.conf.in > $(HOST_DIR)/bin/qt.conf
 endef
Signature · 6 lines
2.31.1

_______________________________________________
buildroot mailing list
[email protected]
http://lists.busybox.net/mailman/listinfo/buildroot
#6 lore raw

Re: [Buildroot] [PATCH v3 05/16] qt5: Fix sporadic build failure during top-level parallel build

Hervé, All,

On 2021-08-17 10:39 +0200, Herve Codina spake thusly:

Quoted 8 lines

From: Andreas Naumann <[email protected]>

When using top level parallel build, independent qt5 packages may be built in parallel. Because of their staging dirs being hardlinked, they all use the same qt.conf file to manipulate during configure, while another qt5 package might already use it. This leads to weird build failures because the folders qmake is using are diverted in erratic ways. Fix this by actually recreating a non-shared qt.conf file for every package.

But with patch 10 (Makefile: breaks hardlinks in global {TARGET, HOST}_DIR on per-package build), is this patch 5 still needed?

IIUC, patch 10 makes the per-package directories actual copies rather than hardlinks, so the per-package host directory is a complete copy, thus this qt.conf is no longer a hardlink...

Did I miss something?

Regards,
Yann E. MORIN.

Quoted 32 lines

---
New patch in this v3 series

This patch was previously send by Andreas Naumann.
https://lore.kernel.org/buildroot/[email protected]/
https://lore.kernel.org/buildroot/[email protected]/
As this patch makes sense in this series, it was integrated in this v3 version.

 package/qt5/qt5.mk | 1 +
 1 file changed, 1 insertion(+)
package/qt5/qt5.mk +1 -0
index db6ccd2b42..3ffb7b0063 100644
@@ -15,6 +15,7 @@ include $(sort $(wildcard package/qt5/*/*.mk))
 # compiled into the Qt library. We need it to make "qmake" relocatable and
 # tweak the per-package install pathes
 define QT5_INSTALL_QT_CONF
+	rm -f $(HOST_DIR)/bin/qt.conf
 	sed -e "s|@@HOST_DIR@@|$(HOST_DIR)|" -e "s|@@STAGING_DIR@@|$(STAGING_DIR)|" \
 		$(QT5BASE_PKGDIR)/qt.conf.in > $(HOST_DIR)/bin/qt.conf
 endef
Signature · 6 lines
2.31.1

_______________________________________________
buildroot mailing list
[email protected]
http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 10 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
[email protected]
http://lists.busybox.net/mailman/listinfo/buildroot
#7 lore raw

Re: [Buildroot] [PATCH v3 05/16] qt5: Fix sporadic build failure during top-level parallel build

Hervé, All,

On 2021-08-28 18:38 +0200, Yann E. MORIN spake thusly:

Quoted 10 lines

On 2021-08-17 10:39 +0200, Herve Codina spake thusly:

Quoted 8 lines

From: Andreas Naumann <[email protected]>

When using top level parallel build, independent qt5 packages may be built in parallel. Because of their staging dirs being hardlinked, they all use the same qt.conf file to manipulate during configure, while another qt5 package might already use it. This leads to weird build failures because the folders qmake is using are diverted in erratic ways. Fix this by actually recreating a non-shared qt.conf file for every package.

Quoted 8 lines

But with patch 10 (Makefile: breaks hardlinks in global {TARGET, HOST}_DIR on per-package build), is this patch 5 still needed?

IIUC, patch 10 makes the per-package directories actual copies rather than hardlinks, so the per-package host directory is a complete copy, thus this qt.conf is no longer a hardlink...

Did I miss something?

Yes, I missed that patch 10 is not about assembling the per-package directories from their dependencies, but to gather those per-package directories to assemble the final target/ and host/.

So, yes, this qt5.conf patch is still needed.

Regards,
Yann E. MORIN.

Quoted 47 lines

Regards,
Yann E. MORIN.

Quoted 32 lines

---
New patch in this v3 series

This patch was previously send by Andreas Naumann.
https://lore.kernel.org/buildroot/[email protected]/
https://lore.kernel.org/buildroot/[email protected]/
As this patch makes sense in this series, it was integrated in this v3 version.

 package/qt5/qt5.mk | 1 +
 1 file changed, 1 insertion(+)
package/qt5/qt5.mk +1 -0
index db6ccd2b42..3ffb7b0063 100644
@@ -15,6 +15,7 @@ include $(sort $(wildcard package/qt5/*/*.mk))
 # compiled into the Qt library. We need it to make "qmake" relocatable and
 # tweak the per-package install pathes
 define QT5_INSTALL_QT_CONF
+	rm -f $(HOST_DIR)/bin/qt.conf
 	sed -e "s|@@HOST_DIR@@|$(HOST_DIR)|" -e "s|@@STAGING_DIR@@|$(STAGING_DIR)|" \
 		$(QT5BASE_PKGDIR)/qt.conf.in > $(HOST_DIR)/bin/qt.conf
 endef
Signature · 6 lines
2.31.1

_______________________________________________
buildroot mailing list
[email protected]
http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 10 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
[email protected]
http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 10 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
[email protected]
http://lists.busybox.net/mailman/listinfo/buildroot
#8 lore raw

Re: [Buildroot] [PATCH v3 05/16] qt5: Fix sporadic build failure during top-level parallel build

In reply to Herve Codina

Hervé, Andreas, All,

On 2021-08-17 10:39 +0200, Herve Codina spake thusly:

Quoted 11 lines

From: Andreas Naumann <[email protected]>

When using top level parallel build, independent qt5 packages may be built in parallel. Because of their staging dirs being hardlinked, they all use the same qt.conf file to manipulate during configure, while another qt5 package might already use it. This leads to weird build failures because the folders qmake is using are diverted in erratic ways. Fix this by actually recreating a non-shared qt.conf file for every package.

Applied to next, thanks! :-)

Regards,
Yann E. MORIN.

Quoted 30 lines

---
New patch in this v3 series

This patch was previously send by Andreas Naumann.
https://lore.kernel.org/buildroot/[email protected]/
https://lore.kernel.org/buildroot/[email protected]/
As this patch makes sense in this series, it was integrated in this v3 version.

 package/qt5/qt5.mk | 1 +
 1 file changed, 1 insertion(+)
package/qt5/qt5.mk +1 -0
index db6ccd2b42..3ffb7b0063 100644
@@ -15,6 +15,7 @@ include $(sort $(wildcard package/qt5/*/*.mk))
 # compiled into the Qt library. We need it to make "qmake" relocatable and
 # tweak the per-package install pathes
 define QT5_INSTALL_QT_CONF
+	rm -f $(HOST_DIR)/bin/qt.conf
 	sed -e "s|@@HOST_DIR@@|$(HOST_DIR)|" -e "s|@@STAGING_DIR@@|$(STAGING_DIR)|" \
 		$(QT5BASE_PKGDIR)/qt.conf.in > $(HOST_DIR)/bin/qt.conf
 endef
Signature · 6 lines
2.31.1

_______________________________________________
buildroot mailing list
[email protected]
http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 10 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
[email protected]
http://lists.busybox.net/mailman/listinfo/buildroot
#9 lore raw

PATCH v3 6/16 package/pkg-qmake.mk: Move QT5_QT_CONF_FIXUP to post-prepare hook

In reply to Herve Codina

QT5_QT_CONF_FIXUP tweaks files for per-package directory build. This is typically the kind of operation expected to be in post-prepare hook.

---
New patch in this v3 series.
As said previously in
https://lore.kernel.org/buildroot/[email protected]/
This patch moves Qt5 fixup to <PKG>_POST_PREPARE_HOOKS (old <PKG>_PER_PACKAGE_TWEAK_HOOKS mentioned in discussion).

 package/pkg-qmake.mk | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
package/pkg-qmake.mk +2 -1
index a77286fe3a..deb033c1d6 100644
@@ -56,13 +56,14 @@ $(2)_DEPENDENCIES 		+= host-perl
 $(2)_PRE_CONFIGURE_HOOKS        += QT_HEADERS_SYNC_HOOK
 endif
 
+$(2)_POST_PREPARE_HOOKS += QT5_QT_CONF_FIXUP
+
 #
 # Configure step. Only define it if not already defined by the package
 # .mk file.
 #
 ifndef $(2)_CONFIGURE_CMDS
 define $(2)_CONFIGURE_CMDS
-	$$(QT5_QT_CONF_FIXUP)
 	cd $$($(2)_BUILDDIR) && \
 	$$(TARGET_MAKE_ENV) $$($(2)_CONF_ENV) $$(QT5_QMAKE) $$($(2)_CONF_OPTS)
 endef
Signature · 6 lines
2.31.1

_______________________________________________
buildroot mailing list
[email protected]
http://lists.busybox.net/mailman/listinfo/buildroot
#10 lore raw

Re: [Buildroot] [PATCH v3 06/16] package/pkg-qmake.mk: Move QT5_QT_CONF_FIXUP to post-prepare hook

Hervé, All,

On 2021-08-17 10:39 +0200, Herve Codina spake thusly:

Quoted 5 lines

QT5_QT_CONF_FIXUP tweaks files for per-package directory build. This is typically the kind of operation expected to be in post-prepare hook.

Applied to next, thanks.

Regards,
Yann E. MORIN.

Quoted 37 lines

---
New patch in this v3 series.
As said previously in
https://lore.kernel.org/buildroot/[email protected]/
This patch moves Qt5 fixup to <PKG>_POST_PREPARE_HOOKS (old <PKG>_PER_PACKAGE_TWEAK_HOOKS mentioned in discussion).

 package/pkg-qmake.mk | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
package/pkg-qmake.mk +2 -1
index a77286fe3a..deb033c1d6 100644
@@ -56,13 +56,14 @@ $(2)_DEPENDENCIES 		+= host-perl
 $(2)_PRE_CONFIGURE_HOOKS        += QT_HEADERS_SYNC_HOOK
 endif
 
+$(2)_POST_PREPARE_HOOKS += QT5_QT_CONF_FIXUP
+
 #
 # Configure step. Only define it if not already defined by the package
 # .mk file.
 #
 ifndef $(2)_CONFIGURE_CMDS
 define $(2)_CONFIGURE_CMDS
-	$$(QT5_QT_CONF_FIXUP)
 	cd $$($(2)_BUILDDIR) && \
 	$$(TARGET_MAKE_ENV) $$($(2)_CONF_ENV) $$(QT5_QMAKE) $$($(2)_CONF_OPTS)
 endef
Signature · 6 lines
2.31.1

_______________________________________________
buildroot mailing list
[email protected]
http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 10 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
[email protected]
http://lists.busybox.net/mailman/listinfo/buildroot
#11 lore raw

PATCH v3 7/16 package/pkg-generic.mk: detect files overwritten in TARGET_DIR and HOST_DIR

In reply to Herve Codina

From: Thomas Petazzoni <[email protected]>

Without per-package directory support, a package can happily overwrite files installed by other packages. Indeed, because the build order between packages is always guaranteed, Buildroot will always produce the same output.

However, with per-package directory support, it is absolutely critical that a given package does not overwrite files already installed by another package, due to how the final aggregation is done to create the complete target/, staging/ and host/ folders. Unfortunately, we currently don't have anything in Buildroot that detects this situation.

We used to have check-uniq-files, but it was dropped in commit 2496189a4207173e4cd5bbab90256f911175ee57.

This commit is a new implementation of such a detection, which is based on calculating and verifying MD5 hashes of installed files: the calculation is done at the beginning of the configure step, the verification during the newly introduced "install" step that takes place after all installation steps.

Since preventing file overwrites is really only needed when per-package directory support is used, and due to this verification having some overhead, it is only enabled when BR2_PER_PACKAGE_DIRECTORIES=y. This additional verification cost is however not too bad as on average, with per-package directory support, the per-package target/ and host/ directories will contain less files than with a build that doesn't use per-package directory support. This helps a bit in mitigating the additional cost of this verification.

Note that we are not handling separately HOST_DIR and STAGING_DIR, like we're doing with the pkg_size_{before,after} functions. Instead, the verification on HOST_DIR walks down into the STAGING_DIR.

During per-package build, original files are modified by fixup-libtool-files and fixup-python-files calls.
But since these fixups modify files using sed --in-place, these modifications are done using a temporary file and a call to rename. Rename breaks the hardlink to the original file and leave the temporary file in per-package TARGET dir.
As the original file is not modified, this is no longer considered as an overwrite. This patch simply considers that what is done by fixup-libtool-files and fixup-python-files is part of the original snapshot used to detect overwrites. And so, the original snapshot is taken after fixup-libtool-files and fixup-python-files calls.

---
This commit is retreived from Thomas's work.
The first version was discussed
https://patchwork.ozlabs.org/project/buildroot/patch/[email protected]/
This new version was not already submitted by Thomas or I missed it. Compared to the first version, this patch has an improved commit message and generates the md5sum snapshot using
'LC_ALL=C find $(1) -type f -print0 | xargs -0 -r md5sum > $($(PKG)_DIR)/.files$(2).md5;'
instead of
'cd $(1); LC_ALL=C find . -type f -exec md5sum {} \; > $($(PKG)_DIR)/.files$(2).md5'

Changes v1 to v2:
- Added note about why fixup-{libtool,python}-files are not considered overwrites and so take the overwrite snapshot after fixup-{libtool,python}-files call.
- Removed 'LC_ALL=C'

Changes v2 to v3:
None

 package/pkg-generic.mk | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)
package/pkg-generic.mk +23 -0
index 38381d93b5..414862e267 100644
@@ -124,6 +124,25 @@ define FIXUP_PYTHON_SYSCONFIGDATA
 endef
 endif
 
+# Functions to detect overwritten files
+
+ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y)
+# $(1): base directory to search in
+# $(2): suffix of file (optional)
+define pkg_detect_overwrite_before
+	find $(1) -type f -print0 | xargs -0 -r md5sum > $($(PKG)_DIR)/.files$(2).md5;
+endef
+
+# $(1): base directory to search in
+# $(2): suffix of file (optional)
+define pkg_detect_overwrite_after
+	if test -s $($(PKG)_DIR)/.files$(2).md5 ; then \
+		md5sum --quiet -c $($(PKG)_DIR)/.files$(2).md5 || \
+		{ echo "ERROR: package $($(PKG)_NAME) has overwritten files installed by a previous package, aborting."; exit 1; } ; \
+	fi
+endef
+endif
+
 # Functions to collect statistics about installed files
 
 # $(1): base directory to search in
@@ -277,6 +296,8 @@ $(BUILD_DIR)/%/.stamp_configured:
 	$(call fixup-libtool-files,$(NAME),$(HOST_DIR))
 	$(call fixup-libtool-files,$(NAME),$(STAGING_DIR))
 	$(foreach hook,$($(PKG)_POST_PREPARE_HOOKS),$(call $(hook))$(sep))
+	@$(call pkg_detect_overwrite_before,$(TARGET_DIR))
+	@$(call pkg_detect_overwrite_before,$(HOST_DIR),-host)
 	$(foreach hook,$($(PKG)_PRE_CONFIGURE_HOOKS),$(call $(hook))$(sep))
 	$($(PKG)_CONFIGURE_CMDS)
 	$(foreach hook,$($(PKG)_POST_CONFIGURE_HOOKS),$(call $(hook))$(sep))
@@ -401,6 +422,8 @@ $(BUILD_DIR)/%/.stamp_installed:
 	@$(call pkg_size_after,$(STAGING_DIR),-staging)
 	@$(call pkg_size_after,$(HOST_DIR),-host)
 	@$(call check_bin_arch)
+	@$(call pkg_detect_overwrite_after,$(TARGET_DIR))
+	@$(call pkg_detect_overwrite_after,$(HOST_DIR),-host)
 	$(Q)touch $@
 
 # Remove package sources
Signature · 6 lines
2.31.1

_______________________________________________
buildroot mailing list
[email protected]
http://lists.busybox.net/mailman/listinfo/buildroot
#12 lore raw

Re: [Buildroot] [PATCH v3 07/16] package/pkg-generic.mk: detect files overwritten in TARGET_DIR and HOST_DIR

Hervé, All,

On 2021-08-17 10:39 +0200, Herve Codina spake thusly:

Quoted 36 lines

From: Thomas Petazzoni <[email protected]>

Without per-package directory support, a package can happily overwrite files installed by other packages. Indeed, because the build order between packages is always guaranteed, Buildroot will always produce the same output.

However, with per-package directory support, it is absolutely critical that a given package does not overwrite files already installed by another package, due to how the final aggregation is done to create the complete target/, staging/ and host/ folders. Unfortunately, we currently don't have anything in Buildroot that detects this situation.

We used to have check-uniq-files, but it was dropped in commit 2496189a4207173e4cd5bbab90256f911175ee57.

This commit is a new implementation of such a detection, which is based on calculating and verifying MD5 hashes of installed files: the calculation is done at the beginning of the configure step, the verification during the newly introduced "install" step that takes place after all installation steps.

Since preventing file overwrites is really only needed when per-package directory support is used, and due to this verification having some overhead, it is only enabled when BR2_PER_PACKAGE_DIRECTORIES=y. This additional verification cost is however not too bad as on average, with per-package directory support, the per-package target/ and host/ directories will contain less files than with a build that doesn't use per-package directory support. This helps a bit in mitigating the additional cost of this verification.

Note that we are not handling separately HOST_DIR and STAGING_DIR, like we're doing with the pkg_size_{before,after} functions. Instead, the verification on HOST_DIR walks down into the STAGING_DIR.

Quoted 11 lines

During per-package build, original files are modified by fixup-libtool-files and fixup-python-files calls.
But since these fixups modify files using sed --in-place, these modifications are done using a temporary file and a call to rename. Rename breaks the hardlink to the original file and leave the temporary file in per-package TARGET dir.
As the original file is not modified, this is no longer considered as an overwrite. This patch simply considers that what is done by fixup-libtool-files and fixup-python-files is part of the original snapshot used to detect overwrites. And so, the original snapshot is taken after fixup-libtool-files and fixup-python-files calls.

This paragraph should take into consideration the fact that we also now have those _POST_PREPARE_HOOKS (and first moving the libtool fixups to be hooks would have simplified this paragraph).

However, what prompted me from applying for now, is that this new detection is a hard error.

Previously, check-uniq-files was just emitting warnings, but would not prevent the build from failing. Now, with this patch, even an innocuous overwrite (e.g. because a post-build script deletes the file, or the content of the file really does not matter at runtime), the build will fail.

I.e. configurations that are currently working with PPD, despite the overwrite, will suddenly no longer build.

OTOH, if we do not make that a hard-error, we will never detect most issues, because users will never spot those warnings and wil enver report issues, and the autobuilders will not fail and we will not notice either...

One solution is to add a configuration knob to make that a hard-error, like we have the paranoid libs/headers check:

config BR2_PPD_OVERWRITE_STRICT
bool "Strict file overwrite detection" depends on BR2_PER_PACKAGE_DIRECTORIES help
Say 'y' here to turn the file overwrite detection to a hard error. By default, only warnings will be printed.

And use that to exit in error only if set:

    [...]
    { \
        echo "ERROR: [...]"; \
        $(if $(BR2_PPD_OVERWRITE_STRICT),exit 1;) \
    } ; \
    [...]

And then, ensure it is always enabled in our autobuilders, so in utils/genrandconfig:

    diff --git a/utils/genrandconfig b/utils/genrandconfig index 622cfd4891..a891f93b6d 100755
    --- a/utils/genrandconfig
    +++ b/utils/genrandconfig
    @@ -353,6 +353,7 @@ def gen_config(args):
         # Per-package folder
         if randint(0, 15) == 0:
             configlines.append("BR2_PER_PACKAGE_DIRECTORIES=y\n")
    +        configlines.append("BR2_PPD_OVERWRITE_STRICT=y\n")
     
         # Amend the configuration with a few things.
         if randint(0, 20) == 0:

I am not totally sold to the new config knob either, but I'd still rather have it than unconditionally break the build...

After all, we know that there are some people using PPD (and TLPB) and are happy with how it works for them, so breaking the build on purpose would probably not be seen as an improvement...

Regards,
Yann E. MORIN.

Quoted 79 lines

---
This commit is retreived from Thomas's work.
The first version was discussed
https://patchwork.ozlabs.org/project/buildroot/patch/[email protected]/
This new version was not already submitted by Thomas or I missed it. Compared to the first version, this patch has an improved commit message and generates the md5sum snapshot using
'LC_ALL=C find $(1) -type f -print0 | xargs -0 -r md5sum > $($(PKG)_DIR)/.files$(2).md5;'
instead of
'cd $(1); LC_ALL=C find . -type f -exec md5sum {} \; > $($(PKG)_DIR)/.files$(2).md5'

Changes v1 to v2:
- Added note about why fixup-{libtool,python}-files are not considered overwrites and so take the overwrite snapshot after fixup-{libtool,python}-files call.
- Removed 'LC_ALL=C'

Changes v2 to v3:
None

 package/pkg-generic.mk | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)
package/pkg-generic.mk +23 -0
index 38381d93b5..414862e267 100644
@@ -124,6 +124,25 @@ define FIXUP_PYTHON_SYSCONFIGDATA
 endef
 endif
 
+# Functions to detect overwritten files
+
+ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y)
+# $(1): base directory to search in
+# $(2): suffix of file (optional)
+define pkg_detect_overwrite_before
+	find $(1) -type f -print0 | xargs -0 -r md5sum > $($(PKG)_DIR)/.files$(2).md5;
+endef
+
+# $(1): base directory to search in
+# $(2): suffix of file (optional)
+define pkg_detect_overwrite_after
+	if test -s $($(PKG)_DIR)/.files$(2).md5 ; then \
+		md5sum --quiet -c $($(PKG)_DIR)/.files$(2).md5 || \
+		{ echo "ERROR: package $($(PKG)_NAME) has overwritten files installed by a previous package, aborting."; exit 1; } ; \
+	fi
+endef
+endif
+
 # Functions to collect statistics about installed files
 
 # $(1): base directory to search in
@@ -277,6 +296,8 @@ $(BUILD_DIR)/%/.stamp_configured:
 	$(call fixup-libtool-files,$(NAME),$(HOST_DIR))
 	$(call fixup-libtool-files,$(NAME),$(STAGING_DIR))
 	$(foreach hook,$($(PKG)_POST_PREPARE_HOOKS),$(call $(hook))$(sep))
+	@$(call pkg_detect_overwrite_before,$(TARGET_DIR))
+	@$(call pkg_detect_overwrite_before,$(HOST_DIR),-host)
 	$(foreach hook,$($(PKG)_PRE_CONFIGURE_HOOKS),$(call $(hook))$(sep))
 	$($(PKG)_CONFIGURE_CMDS)
 	$(foreach hook,$($(PKG)_POST_CONFIGURE_HOOKS),$(call $(hook))$(sep))
@@ -401,6 +422,8 @@ $(BUILD_DIR)/%/.stamp_installed:
 	@$(call pkg_size_after,$(STAGING_DIR),-staging)
 	@$(call pkg_size_after,$(HOST_DIR),-host)
 	@$(call check_bin_arch)
+	@$(call pkg_detect_overwrite_after,$(TARGET_DIR))
+	@$(call pkg_detect_overwrite_after,$(HOST_DIR),-host)
 	$(Q)touch $@
 
 # Remove package sources
Signature · 6 lines
2.31.1

_______________________________________________
buildroot mailing list
[email protected]
http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 10 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
[email protected]
http://lists.busybox.net/mailman/listinfo/buildroot
#13 lore raw

Re: [Buildroot] [PATCH v3 07/16] package/pkg-generic.mk: detect files overwritten in TARGET_DIR and HOST_DIR

Hello Yann,

First of all, thanks a lot for reviewing and merging bits of this patch series, I'm glad to see we're making progress with the TLP stuff.

Quoted 20 lines · On Sun, 29 Aug 2021 00:47:40 +0200 "Yann E. MORIN" <[email protected]>…

However, what prompted me from applying for now, is that this new detection is a hard error.

Previously, check-uniq-files was just emitting warnings, but would not prevent the build from failing. Now, with this patch, even an innocuous overwrite (e.g. because a post-build script deletes the file, or the content of the file really does not matter at runtime), the build will fail.

I.e. configurations that are currently working with PPD, despite the overwrite, will suddenly no longer build.

OTOH, if we do not make that a hard-error, we will never detect most issues, because users will never spot those warnings and wil enver report issues, and the autobuilders will not fail and we will not notice either...

One solution is to add a configuration knob to make that a hard-error, like we have the paranoid libs/headers check:

Quoted 7 lines

config BR2_PPD_OVERWRITE_STRICT
bool "Strict file overwrite detection" depends on BR2_PER_PACKAGE_DIRECTORIES help
Say 'y' here to turn the file overwrite detection to a hard error. By default, only warnings will be printed.

We had some discussion with Hervé back when he worked on this, and I disagreed with adding an option. When BR2_PER_PACKAGE_DIRECTORIES=y, a file overwrite must be a hard error, as the result of the build is incorrect if there is an overwrite. It's not the "latest" package that wins in an overwrite situation, like it does in a non-PPD case.

So I really think this must be a hard error for PPD builds, and just a warning for non-PPD builds.

Yes, for PPD builds, it means users will get failures, but those failures are pointing to real problems.

So, my preference would be to merge as an unconditional check, and see how it goes. Perhaps the situation will be so bad that we will have to make it conditional, but I would prefer to have it unconditional first and see the impact.

Best regards,

Thomas

Signature · 7 lines
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
_______________________________________________
buildroot mailing list
[email protected]
http://lists.busybox.net/mailman/listinfo/buildroot
#14 lore raw

Re: [Buildroot] [PATCH v3 07/16] package/pkg-generic.mk: detect files overwritten in TARGET_DIR and HOST_DIR

Thomas, Hervé, All,

On 2021-08-29 13:39 +0200, Thomas Petazzoni spake thusly:

Quoted 4 lines
Quoted 2 lines · On Sun, 29 Aug 2021 00:47:40 +0200 "Yann E. MORIN" <[email protected]>…

However, what prompted me from applying for now, is that this new detection is a hard error.

[--SNIP--]

Quoted 5 lines

We had some discussion with Hervé back when he worked on this, and I disagreed with adding an option. When BR2_PER_PACKAGE_DIRECTORIES=y, a file overwrite must be a hard error, as the result of the build is incorrect if there is an overwrite. It's not the "latest" package that wins in an overwrite situation, like it does in a non-PPD case.

So, existing builds that succeed with PPD, and for which an overwrite is not an issue even at runtime, will now be broken at build time.

After all, we _know_ there are users that already use PPD (for TLPB), and those users have working systems, but I highly doubt these builds have no file overwrite whatsoever...

Quoted 2 lines

So I really think this must be a hard error for PPD builds, and just a warning for non-PPD builds.

For non-PPD, that code is not even used, because the macros are only defined if BR2_PER_PACKAGE_DIRECTORIES=y; so for non-PPD, we will not even get a warning.

Quoted 2 lines

Yes, for PPD builds, it means users will get failures, but those failures are pointing to real problems.

I fully agree, but in practice, some of those overwrites may be totally innocusous when running the system. Think the gnu info db: we 100% don't care about it at runtime, but this is an overwrite (yes, we already fixed that one, but it was in this cycle, so users of previous releases who use PPD do have overwrites that they don't care about).

Quoted 2 lines

So, my preference would be to merge as an unconditional check, and see how it goes.

OK, I am aligning with that position. Unless someone else beats me to it, I'll resume work on this series later in the WE, or early next week...

Quoted 3 lines

Perhaps the situation will be so bad that we will have to make it conditional, but I would prefer to have it unconditional first and see the impact.

Note that my proposal would have unconditionally enabled it in the autobuilders, so we would have noticed as well, so we can still do that if we later decide to make the hard error conditional.

Finally, as a side note: there is still a case of file overwrite that we do not detect, even with this series: if two packages that are not part of the same dependency chain (i.e. they do not depend one on the other), and they both install the same file, then the file-overwrite will only happen when we eventually assemble the global target/ and host/ from the individual PPD target/ and host/ and we still have to add a detection for that case (which is not a pre-requisite before we apply the current series, of course).

Regards,
Yann E. MORIN.

Quoted 11 lines

Best regards,

Thomas

Signature · 7 lines
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
_______________________________________________
buildroot mailing list
[email protected]
http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 10 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
[email protected]
http://lists.busybox.net/mailman/listinfo/buildroot
#15 lore raw

Re: [Buildroot] [PATCH v3 07/16] package/pkg-generic.mk: detect files overwritten in TARGET_DIR and HOST_DIR

Thomas, All,

On 2021-08-29 14:51 +0200, Yann E. MORIN spake thusly:
[--SNIP--]

Quoted 8 lines

Finally, as a side note: there is still a case of file overwrite that we do not detect, even with this series: if two packages that are not part of the same dependency chain (i.e. they do not depend one on the other), and they both install the same file, then the file-overwrite will only happen when we eventually assemble the global target/ and host/ from the individual PPD target/ and host/ and we still have to add a detection for that case (which is not a pre-requisite before we apply the current series, of course).

And of course, this will also cause an overwrite when two such packages are also dependencies of a third patckage, and so when the PPD of that third package is prepared, we would have to detect file overwrite.

Note: let's assume the following hypotetical dependency graph (where the notation A --<-- B means that B depends on A):

      .-<-- A --<-.                 S: skeleton
     /             \                F: target-finalize
    S --<-- B --<-- C
     \               \
      `-<-- D --<-----<-- F

If A and B both install the same file, then we do not have a sane PPD to build C.

If D and any package in the dependency chain of C (i.e. S, A, B, or C itself) install the same file, then we can only detect it in target-finalize.

So, let's take a step back, and have a wider look. I wonder if the end of a package installation step is the best location to detected file overwrite...

Indeed, let's assume that we do file overwrite detection:

1. in PPD preparation (when we rsync the dependencies, in the macro per-package-rsync): we can also detect that a package overwrites one or more files of another package, report those files, *and* prevent building a package with unsound dependencies;

2. in {target,host}-finalize: we can detect that a package overwrites one or more files provided by another package, and report those files.

So, the above (still theoretical implementation) is all we would need, in the end, and we would not need to do the detection at the end of a package installation step of each package, because it would not provide any information that we couldn't have with the above.

However, a benefit of also doing it at the end of the installation step, of a package, is that we can cut short on the build time. However, I am not sure that benefit is that important or major...

So, I'd like we think a bit harder before we apply this patch...

Regards,
Yann E. MORIN.

Signature · 10 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
[email protected]
http://lists.busybox.net/mailman/listinfo/buildroot
#16 lore raw

Re: [Buildroot] [PATCH v3 07/16] package/pkg-generic.mk: detect files overwritten in TARGET_DIR and HOST_DIR

Quoted 44 lines · Yann E. MORIN

Thomas, All,

On 2021-08-29 14:51 +0200, Yann E. MORIN spake thusly:
[--SNIP--]

Quoted 8 lines

Finally, as a side note: there is still a case of file overwrite that we do not detect, even with this series: if two packages that are not part of the same dependency chain (i.e. they do not depend one on the other), and they both install the same file, then the file-overwrite will only happen when we eventually assemble the global target/ and host/ from the individual PPD target/ and host/ and we still have to add a detection for that case (which is not a pre-requisite before we apply the current series, of course).

And of course, this will also cause an overwrite when two such packages are also dependencies of a third patckage, and so when the PPD of that third package is prepared, we would have to detect file overwrite.

Note: let's assume the following hypotetical dependency graph (where the notation A --<-- B means that B depends on A):

      .-<-- A --<-.                 S: skeleton
     /             \                F: target-finalize
    S --<-- B --<-- C
     \               \
      `-<-- D --<-----<-- F

If A and B both install the same file, then we do not have a sane PPD to build C.

If D and any package in the dependency chain of C (i.e. S, A, B, or C itself) install the same file, then we can only detect it in target-finalize.

So, let's take a step back, and have a wider look. I wonder if the end of a package installation step is the best location to detected file overwrite...

Indeed, let's assume that we do file overwrite detection:

1. in PPD preparation (when we rsync the dependencies, in the macro per-package-rsync): we can also detect that a package overwrites one or more files of another package, report those files, *and* prevent building a package with unsound dependencies;

Quoted 8 lines

2. in {target,host}-finalize: we can detect that a package overwrites one or more files provided by another package, and report those files.

So, the above (still theoretical implementation) is all we would need, in the end, and we would not need to do the detection at the end of a package installation step of each package, because it would not provide any information that we couldn't have with the above.

Fabrice's patches have shown that the overwritten file detection is too
aggressive in the sense to that it *will* cause (many?) failures in the current situation.

So, taking a step further back, what really is the problem we're trying to
solve here? Why can't we overwrite files from other packages?

I think there are three distinct problems. Let's take Yann's example from above.

1. A file written in A and overwritten in C is only a problem because the finalize step doesn't take into account dependencies when combining the trees. We could solve that by taking into the account the dependencies.

2. A file written by both A and B is a problem for package C, because it is not predictable which version it will see. This problem can be solved by adding a dependency between A and B.

3. A file written by S and overwritten by A - this, I think, is the big problem. At first sight it's like case 1 - by taking into account the dependencies, the finalize stage will see the version from A. However, the problem is for stuff installed into staging, where B sees the version from S while C sees the version from A or from S, depending on order of A/B installation. This has two problems:
3a. C's version depends on the order of A/B installation;
3b. B sees a different version of the file than A - this could be a problem for e.g. header files that affect compilation.

The original scheme we discussed three or four years ago already took into
account situations 1 and 2, I think, but not yet situation 3. Note that back then, the overwritten file detection was assumed to happen only in finalize and accept overwrites in case there's a dependency.

The curses and crypt issues that Fabrice detected fall in category 3. And
they're extra nasty, because one of the packages involved is the toolchain that everybody depends on, so you can't just add a dependency to break it.

I haven't given sufficient thought to it yet, but one idea is to allow packages
to add a post-install hook to another package. That way, ncurses could remove curses.h from the toolchain, and libxcrypt could remove crypt.h. Still tricky though because it would mean that any package that uses crypt.h would need to optionally depend on libxcrypt - so that particular case would probably have to be solved by making libxcrypt be part of the toolchain... Or maybe making sure it doesn't overwrite crypt.h and teach its dependencies to include the libxcrypt-specific crypt.h.

One thing is certain: it's complicated :-)

Regards,
Arnout

Quoted 10 lines

However, a benefit of also doing it at the end of the installation step, of a package, is that we can cut short on the build time. However, I am not sure that benefit is that important or major...

So, I'd like we think a bit harder before we apply this patch...

Regards,
Yann E. MORIN.

_______________________________________________
buildroot mailing list
[email protected]
http://lists.busybox.net/mailman/listinfo/buildroot

#17 lore raw

Re: [Buildroot] [PATCH v3 07/16] package/pkg-generic.mk: detect files overwritten in TARGET_DIR and HOST_DIR

In reply to Thomas Petazzoni
Quoted 45 lines · Thomas Petazzoni

Hello Yann,

First of all, thanks a lot for reviewing and merging bits of this patch series, I'm glad to see we're making progress with the TLP stuff.

Quoted 20 lines · On Sun, 29 Aug 2021 00:47:40 +0200 "Yann E. MORIN" <[email protected]>…

However, what prompted me from applying for now, is that this new detection is a hard error.

Previously, check-uniq-files was just emitting warnings, but would not prevent the build from failing. Now, with this patch, even an innocuous overwrite (e.g. because a post-build script deletes the file, or the content of the file really does not matter at runtime), the build will fail.

I.e. configurations that are currently working with PPD, despite the overwrite, will suddenly no longer build.

OTOH, if we do not make that a hard-error, we will never detect most issues, because users will never spot those warnings and wil enver report issues, and the autobuilders will not fail and we will not notice either...

One solution is to add a configuration knob to make that a hard-error, like we have the paranoid libs/headers check:

Quoted 7 lines

config BR2_PPD_OVERWRITE_STRICT
bool "Strict file overwrite detection" depends on BR2_PER_PACKAGE_DIRECTORIES help
Say 'y' here to turn the file overwrite detection to a hard error. By default, only warnings will be printed.

We had some discussion with Hervé back when he worked on this, and I disagreed with adding an option. When BR2_PER_PACKAGE_DIRECTORIES=y, a file overwrite must be a hard error, as the result of the build is incorrect if there is an overwrite. It's not the "latest" package that wins in an overwrite situation, like it does in a non-PPD case.

So I really think this must be a hard error for PPD builds, and just a warning for non-PPD builds.

Quoted 7 lines

Yes, for PPD builds, it means users will get failures, but those failures are pointing to real problems.

So, my preference would be to merge as an unconditional check, and see how it goes. Perhaps the situation will be so bad that we will have to make it conditional, but I would prefer to have it unconditional first and see the impact.

I was originally with Yann, but these arguments convinced that it is indeed
better to not have the option (for now).

Regards,
Arnout

_______________________________________________
buildroot mailing list
[email protected]
http://lists.busybox.net/mailman/listinfo/buildroot

#18 lore raw

Re: [Buildroot] [PATCH v3 07/16] package/pkg-generic.mk: detect files overwritten in TARGET_DIR and HOST_DIR

Hi all,

Quoted 49 lines · Arnout Vandecappelle
Quoted 46 lines · Thomas Petazzoni

Hello Yann,

First of all, thanks a lot for reviewing and merging bits of this patch series, I'm glad to see we're making progress with the TLP stuff.

Quoted 17 lines · On Sun, 29 Aug 2021 00:47:40 +0200 "Yann E. MORIN" <[email protected]>…

However, what prompted me from applying for now, is that this new detection is a hard error.

Previously, check-uniq-files was just emitting warnings, but would not prevent the build from failing. Now, with this patch, even an innocuous overwrite (e.g. because a post-build script deletes the file, or the content of the file really does not matter at runtime), the build will fail.

I.e. configurations that are currently working with PPD, despite the overwrite, will suddenly no longer build.

OTOH, if we do not make that a hard-error, we will never detect most issues, because users will never spot those warnings and wil enver report issues, and the autobuilders will not fail and we will not notice either...

Quoted 7 lines

One solution is to add a configuration knob to make that a hard-error, like we have the paranoid libs/headers check:

config BR2_PPD_OVERWRITE_STRICT bool "Strict file overwrite detection" depends on BR2_PER_PACKAGE_DIRECTORIES help Say 'y' here to turn the file overwrite detection to a hard error. By default, only warnings will be printed.

We had some discussion with Hervé back when he worked on this, and I disagreed with adding an option. When BR2_PER_PACKAGE_DIRECTORIES=y, a file overwrite must be a hard error, as the result of the build is incorrect if there is an overwrite. It's not the "latest" package that wins in an overwrite situation, like it does in a non-PPD case.

So I really think this must be a hard error for PPD builds, and just a warning for non-PPD builds.

Yes, for PPD builds, it means users will get failures, but those failures are pointing to real problems.

Quoted 7 lines
Quoted 4 lines

So, my preference would be to merge as an unconditional check, and see how it goes. Perhaps the situation will be so bad that we will have to make it conditional, but I would prefer to have it unconditional first and see the impact.

I was originally with Yann, but these arguments convinced that it is indeed better to not have the option (for now).

So I'm one of those actively using TLPD and I'm happy to have this producing hard errors to find possible issues in my configs as well as helping to fix them.

So I gave this a spin and immediately found util-linux recompiling some libs which util-linux-libs already installed. Looking for the reason I came upon Carlos Santos Commit 8bafc6dc "package/util-linux: build programs and libraries in separate packages" where the commit msg says ".... Installing util-linux overrides files installed by util-linux-libs but this is not a problem: it's allowed for a package to overwrite files from another package, as long as there is a dependency between the two."

So if this is still true, the logic should be changed to take dependencies into account like suggested in the other discussion.

However, the check also found a conflict where GL/glext.h of mesa3d is overwritten by gst1-plugins-base. It is a dependency, thus it's deterministic, but still we probably end up with an unexpected version glext.h in the SDK, so it's nice that things like this are brought to attention.

Now the build of our standard qt-demo config is almost done and another
conflict pops up when some kernel modules of Wifi-Sticks are installed.
Of course modules.dep, modules.alias and so on are modified. That will have to be taken care of.

However, only 3 problems up to just before the target-finalize step. So far that's not too aggressive for me.

regards,
Andreas

Quoted 6 lines

Regards, Arnout

_______________________________________________ buildroot mailing list [email protected] http://lists.busybox.net/mailman/listinfo/buildroot

_______________________________________________
buildroot mailing list
[email protected]
http://lists.busybox.net/mailman/listinfo/buildroot

#19 lore raw

Re: [Buildroot] [PATCH v3 07/16] package/pkg-generic.mk: detect files overwritten in TARGET_DIR and HOST_DIR

In reply to Herve Codina

Hervé, Thomas, All,

On 2021-08-17 10:39 +0200, Herve Codina spake thusly:

Quoted 5 lines

From: Thomas Petazzoni <[email protected]> Without per-package directory support, a package can happily overwrite files installed by other packages. Indeed, because the build order between packages is always guaranteed, Buildroot will always produce the same output.

[--SNIP--]

Quoted 5 lines

This commit is a new implementation of such a detection, which is based on calculating and verifying MD5 hashes of installed files: the calculation is done at the beginning of the configure step, the verification during the newly introduced "install" step that takes place after all installation steps.

Following the various mails in the thread, and the discussion we had on Tuesday, I'm markign this patch as Changes Requested.

I'll try to scavenge all I can from the remaining patches, and mark the rest as Changes Requested as well.

So, to summarise the discussion, taking my previous example as a basis, and with Arnout's comment in mind (Aha, pun!):

      .-<-- A --<-.                 S: skeleton
     /             \                F: target-finalize
    S --<-- B --<-- C
     \               \
      `-<-- D --<-----<-- F

1. If C overwrites a file from (e.g.) A: we don't know the order in which the target-finalize will copy files.

- we can guarantee that ordering by taking the dependency chain into account when doing target-finalize.

2. If A and B install the same package: we don't know what to provide to package C:

- adding an artificial dependency between A and B is not acceptable, or we could end up with a lot of such unmanageable dependencies, not even accounting for packages in a br32-external tree;

- if the file is a DB (like the GNU info DB), stuff is "added" to that DB; only providing an arbitrary version of the file (from A or from B) is incorrect.

3. If a package in the C dependency chain (C included), and package D install the same file, then we're in the same boat as the above: we do not know what version to use in tergaet-finalize.

If I missed something, then feel free to complete. ;-)

Regards,
Yann E. MORIN.

Signature · 10 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#20 lore raw

PATCH v3 8/16 package/pkg-generic.mk: generate final rsync exclude file list

In reply to Herve Codina

The final rsync performed at {host,target}-finalize steps need to be done by rsyncing files generated by each packages without looking at files generated by other packages the current package depends on. This is needed to avoid overwrites in final {HOST,TARGET}_DIR.

In order to prepare the final rsync, an exclusion list is generated. This list lists files that are not generated by the current package and so files that need to be excluded from the final rsync.

Note also that the files list was not based on .files-list.{before,after}. During .file-list.{before,after} built for host directory, staging sub-directory (ie <toolchain>/sysroot) is filtered out.
The final rsync exclusion list needs to take into account the full {host,target} directory to avoid final overwrites.

Using an empty directory for per-package installation directory would be the simplest way to find what a package installs.
However, as it has been discussed in the past, this is fraught with unworkable issues. For example, some paths may be hard-coded at configure time and/or build time, and thus the package would still install in the original stagin we presented it with (for target/, this is not an isue, because target/ is never looked at during configure or build, only at install time). Or a package installation process just assumes that the directory structure exists (bad, but eh...).

---
Changes v1 to v2:
- Added a note in the commit message related to why an empty directory is not used to find what a package installs.

Changes v2 to v3:
None

 package/pkg-generic.mk | 28 ++++++++++++++++++++++++++++
 1 file changed, 28 insertions(+)
package/pkg-generic.mk +28 -0
index 414862e267..4226cfe0dc 100644
@@ -168,6 +168,30 @@ define pkg_size_after
 	rm -f $($(PKG)_DIR)/.files-list$(2).after
 endef
 
+# Functions to collect final rsync exclusion files
+
+# $(1): base directory to search in
+# $(2): suffix of file (optional)
+define pkg_final_rsync_before
+	cd $(1); \
+	LC_ALL=C find . \( -type f -o -type l \) -printf '%T@:%i:%#m:%y:%s,%p\n' \
+		| LC_ALL=C sort > $($(PKG)_DIR)/.files-final-rsync$(2).before
+endef
+
+# $(1): base directory to search in
+# $(2): suffix of file (optional)
+define pkg_final_rsync_after
+	cd $(1); \
+	LC_ALL=C find . \( -type f -o -type l \) -printf '%T@:%i:%#m:%y:%s,%p\n' \
+		| LC_ALL=C sort > $($(PKG)_DIR)/.files-final-rsync$(2).after
+	LC_ALL=C comm -2 \
+		$($(PKG)_DIR)/.files-final-rsync$(2).before \
+		$($(PKG)_DIR)/.files-final-rsync$(2).after \
+		| sed -r -e 's/^[^,]+,./- /' \
+		> $($(PKG)_DIR)/.files-final-rsync$(2).exclude_rsync
+	rm -f $($(PKG)_DIR)/.files-final-rsync$(2).after
+endef
+
 define check_bin_arch
 	support/scripts/check-bin-arch -p $($(PKG)_NAME) \
 		-l $($(PKG)_DIR)/.files-list.txt \
@@ -290,6 +314,8 @@ $(BUILD_DIR)/%/.stamp_configured:
 	@$(call MESSAGE,"Configuring")
 	$(Q)mkdir -p $(HOST_DIR) $(TARGET_DIR) $(STAGING_DIR) $(BINARIES_DIR)
 	$(call prepare-per-package-directory,$($(PKG)_FINAL_DEPENDENCIES))
+	@$(call pkg_final_rsync_before,$(TARGET_DIR))
+	@$(call pkg_final_rsync_before,$(HOST_DIR),-host)
 	@$(call pkg_size_before,$(TARGET_DIR))
 	@$(call pkg_size_before,$(STAGING_DIR),-staging)
 	@$(call pkg_size_before,$(HOST_DIR),-host)
@@ -422,6 +448,8 @@ $(BUILD_DIR)/%/.stamp_installed:
 	@$(call pkg_size_after,$(STAGING_DIR),-staging)
 	@$(call pkg_size_after,$(HOST_DIR),-host)
 	@$(call check_bin_arch)
+	@$(call pkg_final_rsync_after,$(TARGET_DIR))
+	@$(call pkg_final_rsync_after,$(HOST_DIR),-host)
 	@$(call pkg_detect_overwrite_after,$(TARGET_DIR))
 	@$(call pkg_detect_overwrite_after,$(HOST_DIR),-host)
 	$(Q)touch $@
Signature · 6 lines
2.31.1

_______________________________________________
buildroot mailing list
[email protected]
http://lists.busybox.net/mailman/listinfo/buildroot
#21 lore raw

PATCH v3 9/16 Makefile: rsync global {TARGET, HOST}_DIR using exclusion file list

In reply to Herve Codina

On a per-package build, rsync final {TARGET,HOST}_DIR using exclusion file list computed for each packages.

This exclusion list allows to rsync only files generated by a given package. This is needed to avoid any overwrites in final {TARGET,HOST}_DIR.

As we rsync only files generated by each packages, we need to to do this rsync recursively on each dependencies to collect all needed files. This is done based on existing <PKG>_FINAL_RECURSIVE_DEPENDENCIES

---
Changes v1 to v2:
- Added a note in the commit message related to why this exclusion list is needed.
- Added $(Q) to mkdir and rsync
- Changed indentations to make calls more readable

Changes v2 to v3:
None

 Makefile | 34 ++++++++++++++++++++++++++++++++--
 1 file changed, 32 insertions(+), 2 deletions(-)
Makefile +32 -2
index 3923e5875f..32fad004fe 100644
@@ -725,10 +725,33 @@ TARGET_DIR_FILES_LISTS = $(sort $(wildcard $(BUILD_DIR)/*/.files-list.txt))
 HOST_DIR_FILES_LISTS = $(sort $(wildcard $(BUILD_DIR)/*/.files-list-host.txt))
 STAGING_DIR_FILES_LISTS = $(sort $(wildcard $(BUILD_DIR)/*/.files-list-staging.txt))
 
+ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y)
+# rsync the contents of per-package directories
+# $1: space-separated list of packages to rsync from
+# $2: 'host' or 'target'
+# $3: destination directory
+# $4: exclude file list to use
+define per-package-rsync-delta
+	$(Q)mkdir -p $(3)
+	$(foreach pkg,$(1),\
+		$(Q)rsync -a --link-dest=$(PER_PACKAGE_DIR)/$(pkg)/$(strip $(2))/ \
+			--filter='merge $($(call UPPERCASE,$(pkg))_DIR)/$(strip $(4))' \
+			$(PER_PACKAGE_DIR)/$(pkg)/$(strip $(2))/ \
+			$(3)$(sep))
+endef
+endif
+
 .PHONY: host-finalize
 host-finalize: $(PACKAGES) $(HOST_DIR) $(HOST_DIR_SYMLINK)
 	@$(call MESSAGE,"Finalizing host directory")
-	$(call per-package-rsync,$(sort $(PACKAGES)),host,$(HOST_DIR))
+	$(call per-package-rsync-delta, \
+		$(sort $(foreach pkg, $(PACKAGES), \
+			$(pkg) $($(call UPPERCASE,$(pkg))_FINAL_RECURSIVE_DEPENDENCIES)) \
+		), \
+		host, \
+		$(HOST_DIR), \
+		.files-final-rsync-host.exclude_rsync \
+	)
 
 .PHONY: staging-finalize
 staging-finalize: $(STAGING_DIR_SYMLINK)
@@ -736,7 +759,14 @@ staging-finalize: $(STAGING_DIR_SYMLINK)
 .PHONY: target-finalize
 target-finalize: $(PACKAGES) $(TARGET_DIR) host-finalize
 	@$(call MESSAGE,"Finalizing target directory")
-	$(call per-package-rsync,$(sort $(PACKAGES)),target,$(TARGET_DIR))
+	$(call per-package-rsync-delta, \
+		$(sort $(foreach pkg, $(PACKAGES), \
+			$(pkg) $($(call UPPERCASE,$(pkg))_FINAL_RECURSIVE_DEPENDENCIES)) \
+		), \
+		target, \
+		$(TARGET_DIR), \
+		.files-final-rsync.exclude_rsync \
+	)
 	$(foreach hook,$(TARGET_FINALIZE_HOOKS),$($(hook))$(sep))
 	rm -rf $(TARGET_DIR)/usr/include $(TARGET_DIR)/usr/share/aclocal \
 		$(TARGET_DIR)/usr/lib/pkgconfig $(TARGET_DIR)/usr/share/pkgconfig \
Signature · 6 lines
2.31.1

_______________________________________________
buildroot mailing list
[email protected]
http://lists.busybox.net/mailman/listinfo/buildroot
#22 lore raw

PATCH v3 10/16 Makefile: breaks hardlinks in global {TARGET, HOST}_DIR on per-package build

In reply to Herve Codina

Without this patch, a make <pkg>_rebuild detects overwrites. Indeed, in target_finalize steps some modifications are done on installed files (ie strip or TARGET_FINALIZE_HOOKS for instance).

In order to avoid these modifications seen from per-package {TARGET,HOST}_DIR and so been analyzed as some overwrites, global {TARGET,HOST}_DIR is built using a full copy of the involved per-package files instead of hardlinks.

---
Changes v1 to v2:
- Added 'Reviewed-by: Yann E. MORIN'

Changes v2 to v2:
None

 Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Makefile +1 -1
index 32fad004fe..ccf2020565 100644
@@ -734,7 +734,7 @@ ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y)
 define per-package-rsync-delta
 	$(Q)mkdir -p $(3)
 	$(foreach pkg,$(1),\
-		$(Q)rsync -a --link-dest=$(PER_PACKAGE_DIR)/$(pkg)/$(strip $(2))/ \
+		$(Q)rsync -a \
 			--filter='merge $($(call UPPERCASE,$(pkg))_DIR)/$(strip $(4))' \
 			$(PER_PACKAGE_DIR)/$(pkg)/$(strip $(2))/ \
 			$(3)$(sep))
Signature · 6 lines
2.31.1

_______________________________________________
buildroot mailing list
[email protected]
http://lists.busybox.net/mailman/listinfo/buildroot
#23 lore raw

Re: [Buildroot] [PATCH v3 10/16] Makefile: breaks hardlinks in global {TARGET, HOST}_DIR on per-package build

Hervé, All,

On 2021-08-17 10:39 +0200, Herve Codina spake thusly:

Quoted 25 lines

Without this patch, a make <pkg>_rebuild detects overwrites. Indeed, in target_finalize steps some modifications are done on installed files (ie strip or TARGET_FINALIZE_HOOKS for instance).

In order to avoid these modifications seen from per-package {TARGET,HOST}_DIR and so been analyzed as some overwrites, global {TARGET,HOST}_DIR is built using a full copy of the involved per-package files instead of hardlinks.

---
Changes v1 to v2:
- Added 'Reviewed-by: Yann E. MORIN'

Changes v2 to v2:
None

 Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Makefile +0 -0
index 32fad004fe..ccf2020565 100644

Makefile ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y) · line 737

Quoted 5 lines
Makefile +1 -1
@@ -734,7 +734,7 @@ ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y)
 define per-package-rsync-delta
 	$(Q)mkdir -p $(3)
 	$(foreach pkg,$(1),\
-		$(Q)rsync -a --link-dest=$(PER_PACKAGE_DIR)/$(pkg)/$(strip $(2))/ \
+		$(Q)rsync -a \

I was about to apply that one, because it does not look like it (semantically) depends on the previous patches, but of course it does not apply cleanly... I found it too dangerous to apply and tweak locally without further consideration. So it follows the fate of the rest of the
series: marked Changes Requested.

Regards,
Yann E. MORIN.

Quoted 10 lines
Makefile +0 -0
 			--filter='merge $($(call UPPERCASE,$(pkg))_DIR)/$(strip $(4))' \
 			$(PER_PACKAGE_DIR)/$(pkg)/$(strip $(2))/ \
 			$(3)$(sep))
Signature · 6 lines
2.31.1

_______________________________________________
buildroot mailing list
[email protected]
http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 10 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#24 lore raw

Re: [Buildroot] [PATCH v3 10/16] Makefile: breaks hardlinks in global {TARGET, HOST}_DIR on per-package build

In reply to Herve Codina

On 17/08/2021 10.39, Herve Codina wrote:

Hi,

Sorry, very old patch.

Quoted 8 lines

Without this patch, a make <pkg>_rebuild detects overwrites. Indeed, in target_finalize steps some modifications are done on installed files (ie strip or TARGET_FINALIZE_HOOKS for instance).

In order to avoid these modifications seen from per-package {TARGET,HOST}_DIR and so been analyzed as some overwrites, global {TARGET,HOST}_DIR is built using a full copy of the involved per-package files instead of hardlinks.

Quoted 5 lines

---
Changes v1 to v2:
- Added 'Reviewed-by: Yann E. MORIN'

We have similar issues if you have a post-build script making changes with sed or similar, so committed to master after fixing it up (the logic was in the mean time moved to package/pkg-utils.mk), thanks.

Quoted 20 lines

Changes v2 to v2:
None

  Makefile | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
Makefile +1 -1
index 32fad004fe..ccf2020565 100644
@@ -734,7 +734,7 @@ ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y)
  define per-package-rsync-delta
  	$(Q)mkdir -p $(3)
  	$(foreach pkg,$(1),\
-		$(Q)rsync -a --link-dest=$(PER_PACKAGE_DIR)/$(pkg)/$(strip $(2))/ \
+		$(Q)rsync -a \
  			--filter='merge $($(call UPPERCASE,$(pkg))_DIR)/$(strip $(4))' \
  			$(PER_PACKAGE_DIR)/$(pkg)/$(strip $(2))/ \
  			$(3)$(sep))
Signature · 6 lines
Bye, Peter Korsgaard

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#25 lore raw

Re: [Buildroot] [PATCH v3 10/16] Makefile: breaks hardlinks in global {TARGET, HOST}_DIR on per-package build

Quoted 21 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Peter" == Peter Korsgaard <[email protected]> writes:

On 17/08/2021 10.39, Herve Codina wrote:
Hi,

Sorry, very old patch.

Quoted 7 lines

Without this patch, a make <pkg>_rebuild detects overwrites. Indeed, in target_finalize steps some modifications are done on installed files (ie strip or TARGET_FINALIZE_HOOKS for instance).
In order to avoid these modifications seen from per-package {TARGET,HOST}_DIR
and so been analyzed as some overwrites, global {TARGET,HOST}_DIR is built using a full copy of the involved per-package files instead of hardlinks.

Quoted 5 lines

---
Changes v1 to v2:
- Added 'Reviewed-by: Yann E. MORIN'

Quoted 3 lines

We have similar issues if you have a post-build script making changes with sed or similar, so committed to master after fixing it up (the logic was in the mean time moved to package/pkg-utils.mk), thanks.

Committed to 2023.02.x and 2023.08.x, thanks.

Signature · 5 lines
Bye, Peter Korsgaard
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#26 lore raw

PATCH v3 11/16 package/pkg-generic.mk: fix per-package <pkg>-{reconfigure, rebuild, reinstall}

In reply to Herve Codina

Many overwrites are detected on <pkg>-{reconfigure,rebuild,reinstall}. Indeed, files previously installed by a package were detected as overwritten on next reconfigure, rebuild or reinstall.

To avoid this, we recreate per-package host and target dir from scratch as it was done during the first configure step.

In order not to duplicate the code, this commit shares common code in prepare-pre-configure macro.

---
Changes v1 to v2:
- Avoid code duplication using prepare-pre-configure macro

Changes v2 to v3:
None

 package/pkg-generic.mk | 40 ++++++++++++++++++++++++++++++----------
 1 file changed, 30 insertions(+), 10 deletions(-)
package/pkg-generic.mk +30 -10
index 4226cfe0dc..fb219b8ec7 100644
@@ -217,6 +217,21 @@ define REMOVE_CONFLICTING_USELESS_FILES_IN_TARGET
 	$(call remove-conflicting-useless-files,$(TARGET_DIR))
 endef
 
+# Function to prepare package
+
+define prepare-pre-configure
+	@$(call pkg_final_rsync_before,$(TARGET_DIR))
+	@$(call pkg_final_rsync_before,$(HOST_DIR),-host)
+	@$(call pkg_size_before,$(TARGET_DIR))
+	@$(call pkg_size_before,$(STAGING_DIR),-staging)
+	@$(call pkg_size_before,$(HOST_DIR),-host)
+	$(call fixup-libtool-files,$(NAME),$(HOST_DIR))
+	$(call fixup-libtool-files,$(NAME),$(STAGING_DIR))
+	$(foreach hook,$($(PKG)_POST_PREPARE_HOOKS),$(call $(hook))$(sep))
+	@$(call pkg_detect_overwrite_before,$(TARGET_DIR))
+	@$(call pkg_detect_overwrite_before,$(HOST_DIR),-host)
+endef
+
 ################################################################################
 # Implicit targets -- produce a stamp file for each step of a package build
 ################################################################################
@@ -314,16 +329,7 @@ $(BUILD_DIR)/%/.stamp_configured:
 	@$(call MESSAGE,"Configuring")
 	$(Q)mkdir -p $(HOST_DIR) $(TARGET_DIR) $(STAGING_DIR) $(BINARIES_DIR)
 	$(call prepare-per-package-directory,$($(PKG)_FINAL_DEPENDENCIES))
-	@$(call pkg_final_rsync_before,$(TARGET_DIR))
-	@$(call pkg_final_rsync_before,$(HOST_DIR),-host)
-	@$(call pkg_size_before,$(TARGET_DIR))
-	@$(call pkg_size_before,$(STAGING_DIR),-staging)
-	@$(call pkg_size_before,$(HOST_DIR),-host)
-	$(call fixup-libtool-files,$(NAME),$(HOST_DIR))
-	$(call fixup-libtool-files,$(NAME),$(STAGING_DIR))
-	$(foreach hook,$($(PKG)_POST_PREPARE_HOOKS),$(call $(hook))$(sep))
-	@$(call pkg_detect_overwrite_before,$(TARGET_DIR))
-	@$(call pkg_detect_overwrite_before,$(HOST_DIR),-host)
+	$(call prepare-pre-configure)
 	$(foreach hook,$($(PKG)_PRE_CONFIGURE_HOOKS),$(call $(hook))$(sep))
 	$($(PKG)_CONFIGURE_CMDS)
 	$(foreach hook,$($(PKG)_POST_CONFIGURE_HOOKS),$(call $(hook))$(sep))
@@ -488,6 +494,13 @@ define pkg-graph-depends
 		$$(GRAPHS_DIR)/$$(@).dot
 endef
 
+ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y)
+define empty-per-package-directory
+	rm -rf $(HOST_DIR) $(TARGET_DIR)
+	mkdir -p $(HOST_DIR) $(TARGET_DIR)
+endef
+endif
+
 ################################################################################
 # inner-generic-package -- generates the make targets needed to build a
 # generic package
@@ -1092,6 +1105,8 @@ $(1)-all-legal-info:	$$(foreach p,$$($(2)_FINAL_ALL_DEPENDENCIES),$$(p)-all-lega
 
 $(1)-dirclean:		$$($(2)_TARGET_DIRCLEAN)
 
+$(1)-clean-for-reinstall: PKG=$(2)
+$(1)-clean-for-reinstall: NAME=$(1)
 $(1)-clean-for-reinstall:
 ifneq ($$($(2)_OVERRIDE_SRCDIR),)
 			rm -f $$($(2)_TARGET_RSYNC)
@@ -1101,6 +1116,11 @@ endif
 			rm -f $$($(2)_TARGET_INSTALL_TARGET)
 			rm -f $$($(2)_TARGET_INSTALL_IMAGES)
 			rm -f $$($(2)_TARGET_INSTALL_HOST)
+			$$(call empty-per-package-directory)
+			$$(call prepare-per-package-directory,$$($(2)_FINAL_DOWNLOAD_DEPENDENCIES))
+			$$(call prepare-per-package-directory,$$($(2)_FINAL_EXTRACT_DEPENDENCIES))
+			$$(call prepare-per-package-directory,$$($(2)_FINAL_DEPENDENCIES))
+			$$(call prepare-pre-configure)
 
 $(1)-reinstall:		$(1)-clean-for-reinstall $(1)
 
Signature · 6 lines
2.31.1

_______________________________________________
buildroot mailing list
[email protected]
http://lists.busybox.net/mailman/listinfo/buildroot
#27 lore raw

PATCH v3 12/16 package/pkg-generic.mk: remove .files-final-rsync.before temporary file

In reply to Herve Codina

Keeping the .files-final-rsync.before file between make invocations was needed to ensure that this file will be present in case of 'make <pkg>_{reconfigure,rebuild,reinstall}' invocation.

The .files-final-rsync.before is now recreated on <pkg>_{reconfigure,rebuild,reinstall} and we don't need to keep it between make invocation.

This patch simply removes this file as soon as it is no longer needed.

---
New patch in this v2 series

Changes v2 to v3:
None

 package/pkg-generic.mk | 1 +
 1 file changed, 1 insertion(+)
package/pkg-generic.mk +1 -0
index fb219b8ec7..bb9f395f27 100644
@@ -189,6 +189,7 @@ define pkg_final_rsync_after
 		$($(PKG)_DIR)/.files-final-rsync$(2).after \
 		| sed -r -e 's/^[^,]+,./- /' \
 		> $($(PKG)_DIR)/.files-final-rsync$(2).exclude_rsync
+	rm -f $($(PKG)_DIR)/.files-final-rsync$(2).before
 	rm -f $($(PKG)_DIR)/.files-final-rsync$(2).after
 endef
 
Signature · 6 lines
2.31.1

_______________________________________________
buildroot mailing list
[email protected]
http://lists.busybox.net/mailman/listinfo/buildroot
#28 lore raw

PATCH v3 13/16 support/testing/infra: add log_file_path() function

In reply to Herve Codina

From: Thomas Petazzoni <[email protected]>

Some tests will need to grep through the build log to verify that some features are working are expected. In order to allow them to open the build log, we provide a new function called log_file_path(), which returns the path to the log file if available.

We also use this function in open_log_file().

Note that open_log_file() cannot be used directly to grep through the log file at the end of a build: because it opens in "a+" mode, it greps starting from the end of the file.

---
New patch in this v2 series

This patch is retrieved from Thomas's work.
The first version was discussed
https://patchwork.ozlabs.org/project/buildroot/patch/[email protected]/

This new version was not already submitted by Thomas or I missed it. Compared to the first version, this patch use a more python-like syntax as proposed by Yann in the previous review.

Changes v2 to v3:
None

 support/testing/infra/__init__.py | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)
support/testing/infra/__init__.py +6 -5
index b10a7601a3..1f003f24c6 100644
@@ -10,17 +10,18 @@ ARTIFACTS_URL = "http://autobuild.buildroot.net/artefacts/"
 BASE_DIR = os.path.realpath(os.path.join(os.path.dirname(__file__), "../../.."))
 
 
+def log_file_path(builddir, stage, logtofile=True):
+    """Return path to log file"""
+    return "{}-{}.log".format(builddir, stage) if logtofile else None
+
+
 def open_log_file(builddir, stage, logtofile=True):
     """
     Open a file for logging and return its handler.
     If logtofile is True, returns sys.stdout. Otherwise opens a file
     with a suitable name in the build directory.
     """
-    if logtofile:
-        fhandle = open("{}-{}.log".format(builddir, stage), 'a+')
-    else:
-        fhandle = sys.stdout
-    return fhandle
+    return open(log_file_path(builddir, stage, logtofile), 'a+') if logtofile else sys.stdout
 
 
 def basepath(relpath=""):
Signature · 6 lines
2.31.1

_______________________________________________
buildroot mailing list
[email protected]
http://lists.busybox.net/mailman/listinfo/buildroot
#29 lore raw

Re: [Buildroot] [PATCH v3 13/16] support/testing/infra: add log_file_path() function

Hervé, All,

On 2021-08-17 10:39 +0200, Herve Codina spake thusly:

Quoted 7 lines

From: Thomas Petazzoni <[email protected]>

Some tests will need to grep through the build log to verify that some features are working are expected. In order to allow them to open the build log, we provide a new function called log_file_path(), which returns the path to the log file if available.

Quoted 8 lines

We also use this function in open_log_file().

Note that open_log_file() cannot be used directly to grep through the log file at the end of a build: because it opens in "a+" mode, it greps starting from the end of the file.

This was not really part of the PPD/TLPB stuff, and did not seem to depend on it in any way, so: applied to next, thanks.

Regards,
Yann E. MORIN.

Quoted 52 lines

---
New patch in this v2 series

This patch is retrieved from Thomas's work.
The first version was discussed
https://patchwork.ozlabs.org/project/buildroot/patch/[email protected]/

This new version was not already submitted by Thomas or I missed it. Compared to the first version, this patch use a more python-like syntax as proposed by Yann in the previous review.

Changes v2 to v3:
None

 support/testing/infra/__init__.py | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)
support/testing/infra/__init__.py +6 -5
index b10a7601a3..1f003f24c6 100644
@@ -10,17 +10,18 @@ ARTIFACTS_URL = "http://autobuild.buildroot.net/artefacts/"
 BASE_DIR = os.path.realpath(os.path.join(os.path.dirname(__file__), "../../.."))
 
 
+def log_file_path(builddir, stage, logtofile=True):
+    """Return path to log file"""
+    return "{}-{}.log".format(builddir, stage) if logtofile else None
+
+
 def open_log_file(builddir, stage, logtofile=True):
     """
     Open a file for logging and return its handler.
     If logtofile is True, returns sys.stdout. Otherwise opens a file
     with a suitable name in the build directory.
     """
-    if logtofile:
-        fhandle = open("{}-{}.log".format(builddir, stage), 'a+')
-    else:
-        fhandle = sys.stdout
-    return fhandle
+    return open(log_file_path(builddir, stage, logtofile), 'a+') if logtofile else sys.stdout
 
 
 def basepath(relpath=""):
Signature · 6 lines
2.31.1

_______________________________________________
buildroot mailing list
[email protected]
http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 10 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
[email protected]
http://lists.busybox.net/mailman/listinfo/buildroot
#30 lore raw

PATCH v3 14/16 support/testing/tests: add test for check_bin_arch

In reply to Herve Codina

From: Thomas Petazzoni <[email protected]>

This tests build a bogus package that installs a binary built for the host architecture into $(TARGET_DIR), which should cause a build failure, at least as long as the host architecture isn't ARM.

---
New patch in this v2 series

This patch is retrieved from Thomas's work.
The first version was discussed
https://patchwork.ozlabs.org/project/buildroot/patch/[email protected]/

Compared to the first version, as proposed by Yann in the previous review, this patch does not launch a subprocess (grep) to search the string.

Note also that the .gitlab-ci.yml modification present in the previous version is not present in this patch.

Changes v2 to v3:
- Fixed the test (https://lore.kernel.org/buildroot/[email protected]/)

 .../br2-external/detect-bad-arch/Config.in    |  1 +
 .../detect-bad-arch/external.desc             |  1 +
 .../br2-external/detect-bad-arch/external.mk  |  1 +
 .../package/detect-bad-arch/Config.in         |  4 ++++
 .../detect-bad-arch/detect-bad-arch.mk        | 15 +++++++++++++++
 support/testing/tests/core/test_bad_arch.py   | 19 +++++++++++++++++++
 6 files changed, 41 insertions(+)
 create mode 100644 support/testing/tests/core/br2-external/detect-bad-arch/Config.in
 create mode 100644 support/testing/tests/core/br2-external/detect-bad-arch/external.desc
 create mode 100644 support/testing/tests/core/br2-external/detect-bad-arch/external.mk
 create mode 100644 support/testing/tests/core/br2-external/detect-bad-arch/package/detect-bad-arch/Config.in
 create mode 100644 support/testing/tests/core/br2-external/detect-bad-arch/package/detect-bad-arch/detect-bad-arch.mk
 create mode 100644 support/testing/tests/core/test_bad_arch.py
support/testing/tests/core/br2-external/detect-bad-arch/Config.in new file +1 -0
new file mode 100644
index 0000000000..530c077bbe
@@ -0,0 +1 @@
+source "$BR2_EXTERNAL_DETECT_BAD_ARCH_PATH/package/detect-bad-arch/Config.in"
support/testing/tests/core/br2-external/detect-bad-arch/external.desc new file +1 -0
new file mode 100644
index 0000000000..3c4232c90d
@@ -0,0 +1 @@
+name: DETECT_BAD_ARCH
support/testing/tests/core/br2-external/detect-bad-arch/external.mk new file +1 -0
new file mode 100644
index 0000000000..71b9821ddc
@@ -0,0 +1 @@
+include $(sort $(wildcard $(BR2_EXTERNAL_DETECT_BAD_ARCH_PATH)/package/*/*.mk))
support/testing/tests/core/br2-external/detect-bad-arch/package/detect-bad-arch/Config.in new file +4 -0
new file mode 100644
index 0000000000..9893e9afc1
@@ -0,0 +1,4 @@
+config BR2_PACKAGE_DETECT_BAD_ARCH
+	bool
+	default y
+
support/testing/tests/core/br2-external/detect-bad-arch/package/detect-bad-arch/detect-bad-arch.mk new file +15 -0
new file mode 100644
index 0000000000..5e78c55f1f
@@ -0,0 +1,15 @@
+################################################################################
+#
+# detect-bad-arch
+#
+################################################################################
+
+define DETECT_BAD_ARCH_BUILD_CMDS
+	echo "int main(void) { return 0; }" | $(HOSTCC) -x c -o $(@D)/foo -
+endef
+
+define DETECT_BAD_ARCH_INSTALL_TARGET_CMDS
+	$(INSTALL) -D -m 0755 $(@D)/foo $(TARGET_DIR)/usr/bin/foo
+endef
+
+$(eval $(generic-package))
support/testing/tests/core/test_bad_arch.py new file +19 -0
new file mode 100644
index 0000000000..76acba3711
@@ -0,0 +1,19 @@
+import infra
+import infra.basetest
+import subprocess
+
+
+class DetectBadArchTest(infra.basetest.BRConfigTest):
+    config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + infra.basetest.MINIMAL_CONFIG
+    br2_external = [infra.filepath("tests/core/br2-external/detect-bad-arch")]
+
+    def test_run(self):
+        with self.assertRaises(SystemError):
+            self.b.build()
+        logf_path = infra.log_file_path(self.b.builddir, "build",
+                                        infra.basetest.BRConfigTest.logtofile)
+        if logf_path:
+            s = 'ERROR: architecture for "/usr/bin/foo" is'
+            with open(logf_path, "r") as f:
+                lines = [l for l in f.readlines() if l.startswith(s)]
+            self.assertEqual(len(lines), 1)
Signature · 6 lines
2.31.1

_______________________________________________
buildroot mailing list
[email protected]
http://lists.busybox.net/mailman/listinfo/buildroot
#31 lore raw

Re: [Buildroot] [PATCH v3 14/16] support/testing/tests: add test for check_bin_arch

On 2021-08-17 10:39 +0200, Herve Codina spake thusly:

Quoted 8 lines

From: Thomas Petazzoni <[email protected]>

This tests build a bogus package that installs a binary built for the host architecture into $(TARGET_DIR), which should cause a build failure, at least as long as the host architecture isn't ARM.

This was not really part of the PPD/TLPB stuff, and did not seem to depend on it in any way, so: applied to next, thanks.

Seem a small nit below, though...

Quoted 1 line

---

[--SNIP--]

support/testing/tests/core/test_bad_arch.py line 3

Quoted 9 lines
support/testing/tests/core/test_bad_arch.py new file +3 -0
new file mode 100644
index 0000000000..76acba3711
@@ -0,0 +1,19 @@
+import infra
+import infra.basetest
+import subprocess
    $ make check-flake8
    support/testing/tests/core/test_bad_arch.py:3:1: F401 'subprocess' imported but unused
    1     F401 'subprocess' imported but unused

I fixed that when applying.

Regards,
Yann E. MORIN.

Signature · 10 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
[email protected]
http://lists.busybox.net/mailman/listinfo/buildroot
#32 lore raw

PATCH v3 15/16 support/testing/tests: add test for file overwrite detection

In reply to Herve Codina

From: Thomas Petazzoni <[email protected]>

---
New patch in this v2 series

This patch is retrieved from Thomas's work.
The first version was discussed
https://patchwork.ozlabs.org/project/buildroot/patch/[email protected]/

Compared to the first version, as proposed by Yann in the previous review, this patch does not launch a subprocess (grep) to find the string.
Additionally, the strings searched were changed from a specific file ('path/file: FAILED') to the "ERROR: package ..." overwrite detection message.

Note also that the .gitlab-ci.yml modification present in the previous version is not present in this patch.

Changes v2 to v3:
None

 .../br2-external/detect-overwrite/Config.in   |  1 +
 .../detect-overwrite/external.desc            |  1 +
 .../br2-external/detect-overwrite/external.mk |  1 +
 .../package/detect-overwrite/Config.in        |  5 ++
 .../detect-overwrite/detect-overwrite.mk      | 19 ++++++++
 .../testing/tests/core/test_file_overwrite.py | 48 +++++++++++++++++++
 6 files changed, 75 insertions(+)
 create mode 100644 support/testing/tests/core/br2-external/detect-overwrite/Config.in
 create mode 100644 support/testing/tests/core/br2-external/detect-overwrite/external.desc
 create mode 100644 support/testing/tests/core/br2-external/detect-overwrite/external.mk
 create mode 100644 support/testing/tests/core/br2-external/detect-overwrite/package/detect-overwrite/Config.in
 create mode 100644 support/testing/tests/core/br2-external/detect-overwrite/package/detect-overwrite/detect-overwrite.mk
 create mode 100644 support/testing/tests/core/test_file_overwrite.py
support/testing/tests/core/br2-external/detect-overwrite/Config.in new file +1 -0
new file mode 100644
index 0000000000..b5514510bd
@@ -0,0 +1 @@
+source "$BR2_EXTERNAL_DETECT_OVERWRITE_PATH/package/detect-overwrite/Config.in"
support/testing/tests/core/br2-external/detect-overwrite/external.desc new file +1 -0
new file mode 100644
index 0000000000..6fedc276e8
@@ -0,0 +1 @@
+name: DETECT_OVERWRITE
support/testing/tests/core/br2-external/detect-overwrite/external.mk new file +1 -0
new file mode 100644
index 0000000000..90927b33ef
@@ -0,0 +1 @@
+include $(sort $(wildcard $(BR2_EXTERNAL_DETECT_OVERWRITE_PATH)/package/*/*.mk))
support/testing/tests/core/br2-external/detect-overwrite/package/detect-overwrite/Config.in new file +5 -0
new file mode 100644
index 0000000000..fff8b0320f
@@ -0,0 +1,5 @@
+config BR2_PACKAGE_DETECT_OVERWRITE
+	bool "detect-overwrite"
+
+config BR2_PACKAGE_HOST_DETECT_OVERWRITE
+	bool "host-detect-overwrite"
support/testing/tests/core/br2-external/detect-overwrite/package/detect-overwrite/detect-overwrite.mk new file +19 -0
new file mode 100644
index 0000000000..c6df2a339d
@@ -0,0 +1,19 @@
+################################################################################
+#
+# detect-overwrite
+#
+################################################################################
+
+define DETECT_OVERWRITE_INSTALL_TARGET_CMDS
+	grep -q "^foo" $(TARGET_DIR)/etc/passwd || \
+		echo "foo" >> $(TARGET_DIR)/etc/passwd
+endef
+
+HOST_DETECT_OVERWRITE_DEPENDENCIES = host-pkgconf
+
+define HOST_DETECT_OVERWRITE_INSTALL_CMDS
+	$(SED) 's/manipulating/tweaking/' $(HOST_DIR)/lib/pkgconfig/libpkgconf.pc
+endef
+
+$(eval $(generic-package))
+$(eval $(host-generic-package))
support/testing/tests/core/test_file_overwrite.py new file +48 -0

48 lines not on this page · raw message

Signature · 6 lines
2.31.1

_______________________________________________
buildroot mailing list
[email protected]
http://lists.busybox.net/mailman/listinfo/buildroot
#33 lore raw

PATCH v3 16/16 package/pkg-generic.mk: move fixup-libtool-files to post-prepare hook

In reply to Herve Codina

This commit moves fixup-libtool-files to the post-prepare hook infrastructure.

---
New patch in this v3 series.

This modification was mentionned in the previous review
https://lore.kernel.org/buildroot/20210625092704.GJ2852@scaer/

 package/pkg-generic.mk | 20 ++++++++++++--------
 1 file changed, 12 insertions(+), 8 deletions(-)
package/pkg-generic.mk +12 -8
index bb9f395f27..d8ca8aaf31 100644
@@ -92,14 +92,19 @@ endif
 
 # Make sure .la files only reference the current per-package
 # directory.
-
-# $1: package name (lower case)
-# $2: staging directory of the package
+#
+# Can't use $(foreach d, $(HOST_DIR)/lib* $(STAGING_DIR)/usr/lib*, ...)
+# because those directories may be created in the same recipe this macro will
+# be expanded in.
+# Additionally, either or both may be missing, which would make find whine and
+# fail.
+# So we just use HOST_DIR as a starting point, and filter on the two directories
+# of interest.
 ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y)
-define fixup-libtool-files
-	$(Q)find $(2) \( -path '$(2)/lib*' -o -path '$(2)/usr/lib*' \) \
+define FIXUP_LIBTOOL_FILES
+	$(Q)find $(HOST_DIR) \( -path '$(HOST_DIR)/lib*' -o -path '$(STAGING_DIR)/usr/lib*' \) \
 		-name "*.la" -print0 | xargs -0 --no-run-if-empty \
-		$(SED) "s:$(PER_PACKAGE_DIR)/[^/]\+/:$(PER_PACKAGE_DIR)/$(1)/:g"
+		$(SED) "s:$(PER_PACKAGE_DIR)/[^/]\+/:$(PER_PACKAGE_DIR)/$($(PKG)_NAME)/:g"
 endef
 endif
 
@@ -226,8 +231,6 @@ define prepare-pre-configure
 	@$(call pkg_size_before,$(TARGET_DIR))
 	@$(call pkg_size_before,$(STAGING_DIR),-staging)
 	@$(call pkg_size_before,$(HOST_DIR),-host)
-	$(call fixup-libtool-files,$(NAME),$(HOST_DIR))
-	$(call fixup-libtool-files,$(NAME),$(STAGING_DIR))
 	$(foreach hook,$($(PKG)_POST_PREPARE_HOOKS),$(call $(hook))$(sep))
 	@$(call pkg_detect_overwrite_before,$(TARGET_DIR))
 	@$(call pkg_detect_overwrite_before,$(HOST_DIR),-host)
@@ -923,6 +926,7 @@ $(2)_POST_LEGAL_INFO_HOOKS      ?=
 $(2)_TARGET_FINALIZE_HOOKS      ?=
 $(2)_ROOTFS_PRE_CMD_HOOKS       ?=
 
+$(2)_POST_PREPARE_HOOKS += FIXUP_LIBTOOL_FILES
 $(2)_POST_PREPARE_HOOKS += FIXUP_PYTHON_SYSCONFIGDATA
 
 ifeq ($$($(2)_TYPE),target)
Signature · 6 lines
2.31.1

_______________________________________________
buildroot mailing list
[email protected]
http://lists.busybox.net/mailman/listinfo/buildroot
#34 lore raw
In reply to Herve Codina

Hervé, All,

On 2021-08-17 10:39 +0200, Herve Codina spake thusly:

Quoted 24 lines

This v3 series adds a new patch (patch 1) to fix a find failure already present in PKG_PYTHON_FIXUP_SYSCONFIG that was highlighted in the previous version of this series.
It also reworks the python fixup according to previous review (patches 2 and 3) and fixes the test given in patch 12.

Two patches are added related to Qt5. The first one (patch 5) was previously sent by Andreas Naumann and fixes a Qt5 overwrite.
https://lore.kernel.org/buildroot/[email protected]/
https://lore.kernel.org/buildroot/[email protected]/
The second one (patch 6) moves the fixup function to the post-prepare hooks.

One more patch was added in this serie (patch 14) to move fixup-libtool-file to the post-prepare hooks. This was mentionned during the review of the first version of this series (https://lore.kernel.org/buildroot/20210625092704.GJ2852@scaer/).

More changes details are given in each individual patches.

Best regards,
Hervé Codina

Andreas Naumann (1):
qt5: Fix sporadic build failure during top-level parallel build

Quoted 6 lines

Herve Codina (11):
package/pkg-python: fix PKG_PYTHON_FIXUP_SYSCONFIGDATA package/pkg-python: invalidate precompiled _sysconfigdata*.pyc package/pkg-generic.mk: move python fixup to generic package
infrastructure
package/owfs: remove Python sysconfigdata fixup

I've applied those four python-related fixes to next, thanks!

I'll resume the rest of the series later in the WE; no need to respin!

Regards,
Yann E. MORIN.

Quoted 54 lines

package/pkg-qmake.mk: Move QT5_QT_CONF_FIXUP to post-prepare hook package/pkg-generic.mk: generate final rsync exclude file list
Makefile: rsync global {TARGET,HOST}_DIR using exclusion file list
Makefile: breaks hardlinks in global {TARGET,HOST}_DIR on per-package
build
package/pkg-generic.mk: fix per-package

    <pkg>-{reconfigure,rebuild,reinstall}

package/pkg-generic.mk: remove .files-final-rsync.before temporary
file
package/pkg-generic.mk: move fixup-libtool-files to post-prepare hook

Thomas Petazzoni (4):
package/pkg-generic.mk: detect files overwritten in TARGET_DIR and
HOST_DIR
support/testing/infra: add log_file_path() function support/testing/tests: add test for check_bin_arch support/testing/tests: add test for file overwrite detection

 Makefile                                      |  34 ++++-
 package/owfs/owfs.mk                          |   9 --
 package/pkg-generic.mk                        | 125 ++++++++++++++++--
 package/pkg-python.mk                         |  10 --
 package/pkg-qmake.mk                          |   3 +-
 package/qt5/qt5.mk                            |   1 +
 support/testing/infra/__init__.py             |  11 +-
 .../br2-external/detect-bad-arch/Config.in    |   1 +
 .../detect-bad-arch/external.desc             |   1 +
 .../br2-external/detect-bad-arch/external.mk  |   1 +
 .../package/detect-bad-arch/Config.in         |   4 +
 .../detect-bad-arch/detect-bad-arch.mk        |  15 +++
 .../br2-external/detect-overwrite/Config.in   |   1 +
 .../detect-overwrite/external.desc            |   1 +
 .../br2-external/detect-overwrite/external.mk |   1 +
 .../package/detect-overwrite/Config.in        |   5 +
 .../detect-overwrite/detect-overwrite.mk      |  19 +++
 support/testing/tests/core/test_bad_arch.py   |  19 +++
 .../testing/tests/core/test_file_overwrite.py |  48 +++++++
 19 files changed, 269 insertions(+), 40 deletions(-)
 create mode 100644 support/testing/tests/core/br2-external/detect-bad-arch/Config.in
 create mode 100644 support/testing/tests/core/br2-external/detect-bad-arch/external.desc
 create mode 100644 support/testing/tests/core/br2-external/detect-bad-arch/external.mk
 create mode 100644 support/testing/tests/core/br2-external/detect-bad-arch/package/detect-bad-arch/Config.in
 create mode 100644 support/testing/tests/core/br2-external/detect-bad-arch/package/detect-bad-arch/detect-bad-arch.mk
 create mode 100644 support/testing/tests/core/br2-external/detect-overwrite/Config.in
 create mode 100644 support/testing/tests/core/br2-external/detect-overwrite/external.desc
 create mode 100644 support/testing/tests/core/br2-external/detect-overwrite/external.mk
 create mode 100644 support/testing/tests/core/br2-external/detect-overwrite/package/detect-overwrite/Config.in
 create mode 100644 support/testing/tests/core/br2-external/detect-overwrite/package/detect-overwrite/detect-overwrite.mk
 create mode 100644 support/testing/tests/core/test_bad_arch.py
 create mode 100644 support/testing/tests/core/test_file_overwrite.py
Signature · 1 line
2.31.1
Signature · 10 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
[email protected]
http://lists.busybox.net/mailman/listinfo/buildroot