Overwritten file detection and fixes, one more step to TLP build
v3 cover Overwritten file detection and fixes, one more step to TLP build
Hi,
This v3 series adds a new patch (patch 1) to fix a find failure already present in PKG_PYTHON_FIXUP_SYSCONFIG that was highlighted in the previous version of this series.
It also reworks the python fixup according to previous review (patches 2 and 3) and fixes the test given in patch 12.
Two patches are added related to Qt5. The first one (patch 5) was previously sent by Andreas Naumann and fixes a Qt5 overwrite.
https://lore.kernel.org/buildroot/[email protected]/
https://lore.kernel.org/buildroot/[email protected]/
The second one (patch 6) moves the fixup function to the post-prepare hooks.
One more patch was added in this serie (patch 14) to move fixup-libtool-file to the post-prepare hooks. This was mentionned during the review of the first version of this series (https://lore.kernel.org/buildroot/20210625092704.GJ2852@scaer/).
More changes details are given in each individual patches.
Best regards,
Hervé Codina
Andreas Naumann (1):
qt5: Fix sporadic build failure during top-level parallel build
Herve Codina (11):
package/pkg-python: fix PKG_PYTHON_FIXUP_SYSCONFIGDATA package/pkg-python: invalidate precompiled _sysconfigdata*.pyc package/pkg-generic.mk: move python fixup to generic package
infrastructure
package/owfs: remove Python sysconfigdata fixup package/pkg-qmake.mk: Move QT5_QT_CONF_FIXUP to post-prepare hook package/pkg-generic.mk: generate final rsync exclude file list
Makefile: rsync global {TARGET,HOST}_DIR using exclusion file list
Makefile: breaks hardlinks in global {TARGET,HOST}_DIR on per-package
build
package/pkg-generic.mk: fix per-package
<pkg>-{reconfigure,rebuild,reinstall} package/pkg-generic.mk: remove .files-final-rsync.before temporary
file
package/pkg-generic.mk: move fixup-libtool-files to post-prepare hook
Thomas Petazzoni (4):
package/pkg-generic.mk: detect files overwritten in TARGET_DIR and
HOST_DIR
support/testing/infra: add log_file_path() function support/testing/tests: add test for check_bin_arch support/testing/tests: add test for file overwrite detection
Makefile | 34 ++++- package/owfs/owfs.mk | 9 -- package/pkg-generic.mk | 125 ++++++++++++++++-- package/pkg-python.mk | 10 -- package/pkg-qmake.mk | 3 +- package/qt5/qt5.mk | 1 + support/testing/infra/__init__.py | 11 +- .../br2-external/detect-bad-arch/Config.in | 1 + .../detect-bad-arch/external.desc | 1 + .../br2-external/detect-bad-arch/external.mk | 1 + .../package/detect-bad-arch/Config.in | 4 + .../detect-bad-arch/detect-bad-arch.mk | 15 +++ .../br2-external/detect-overwrite/Config.in | 1 + .../detect-overwrite/external.desc | 1 + .../br2-external/detect-overwrite/external.mk | 1 + .../package/detect-overwrite/Config.in | 5 + .../detect-overwrite/detect-overwrite.mk | 19 +++ support/testing/tests/core/test_bad_arch.py | 19 +++ .../testing/tests/core/test_file_overwrite.py | 48 +++++++ 19 files changed, 269 insertions(+), 40 deletions(-) create mode 100644 support/testing/tests/core/br2-external/detect-bad-arch/Config.in create mode 100644 support/testing/tests/core/br2-external/detect-bad-arch/external.desc create mode 100644 support/testing/tests/core/br2-external/detect-bad-arch/external.mk create mode 100644 support/testing/tests/core/br2-external/detect-bad-arch/package/detect-bad-arch/Config.in create mode 100644 support/testing/tests/core/br2-external/detect-bad-arch/package/detect-bad-arch/detect-bad-arch.mk create mode 100644 support/testing/tests/core/br2-external/detect-overwrite/Config.in create mode 100644 support/testing/tests/core/br2-external/detect-overwrite/external.desc create mode 100644 support/testing/tests/core/br2-external/detect-overwrite/external.mk create mode 100644 support/testing/tests/core/br2-external/detect-overwrite/package/detect-overwrite/Config.in create mode 100644 support/testing/tests/core/br2-external/detect-overwrite/package/detect-overwrite/detect-overwrite.mk create mode 100644 support/testing/tests/core/test_bad_arch.py create mode 100644 support/testing/tests/core/test_file_overwrite.py
Signature · 6 lines
2.31.1 _______________________________________________ buildroot mailing list [email protected] http://lists.busybox.net/mailman/listinfo/buildroot
PATCH v3 1/16 package/pkg-python: fix PKG_PYTHON_FIXUP_SYSCONFIGDATA
Find used by PKG_PYTHON_FIXUP_SYSCONFIGDATA can fail if directories are not present. This failure is silently ignored because find is on the LHS of a pipe.
This commit fixes the find failure. HOST_DIR is used as the starting point and the search is filtered on the expected directories.
This commit also adds -print0 and the $(Q) verbosity flag as minor changes.
---
New patch in this v3 series.
The find failure was highlighted in 836528f03eb89fdc64432f7a7470145ddf553b8c and reverted in b928074e2d83f5b873bf1ba381b212dfad9bf207.
package/pkg-python.mk | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-)
package/pkg-python.mk +14 -3
@@ -92,11 +92,22 @@ HOST_PKG_PYTHON_SETUPTOOLS_INSTALL_OPTS = \ --root=/ \ --single-version-externally-managed +# Make sure python _sysconfigdata*.py files only reference the current +# per-package directory. +# +# Can't use $(foreach d, $(HOST_DIR)/lib/python* $(STAGING_DIR)/usr/lib/python*, ...) +# because those directories may be created in the same recipe this macro will +# be expanded in. +# Additionally, either or both may be missing, which would make find whine and +# fail. +# So we just use HOST_DIR as a starting point, and filter on the two directories +# of interest. ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y) define PKG_PYTHON_FIXUP_SYSCONFIGDATA - find $(HOST_DIR)/lib/python* $(STAGING_DIR)/usr/lib/python* \ - -name "_sysconfigdata*.py" | xargs --no-run-if-empty \ - $(SED) "s:$(PER_PACKAGE_DIR)/[^/]\+/:$(PER_PACKAGE_DIR)/$($(PKG)_NAME)/:g" + $(Q)find $(HOST_DIR) \ + \( -path '$(HOST_DIR)/lib/python*' -o -path '$(STAGING_DIR)/usr/lib/python*' \) \ + -name "_sysconfigdata*.py" -print0 | xargs -0 --no-run-if-empty \ + $(SED) 's:$(PER_PACKAGE_DIR)/[^/]\+/:$(PER_PACKAGE_DIR)/$($(PKG)_NAME)/:g' endef endif
Signature · 6 lines
2.31.1 _______________________________________________ buildroot mailing list [email protected] http://lists.busybox.net/mailman/listinfo/buildroot
PATCH v3 2/16 package/pkg-python: invalidate precompiled _sysconfigdata*.pyc
In reply to Herve CodinaFor per-package directories, we fixup the _sysconfigdata*.py files, so that they get proper path pointing to the current package's direcotry structure.
However, the corresponding, pre-compiled blobs _sysconfigdata*.pyc were left around, and thus are inconsistent with their source. They might also be regenerated when a package would install a python module; this regeneration would trigger the soon-to-be-introduced overwrite detection.
This commit simply removes _sysconfigdata*.pyc files; they will anyway be regenerated by the PYTHON{,3}_CREATE_PYC_FILES target finalize hooks. This is an efficient way to guarantee the consistency between the source and precompiled versions, and to not trigger the overwrite detection.
---
Changes from v1 to v2:
- Used separate shells for the two 'find' commands.
- Used '-deleted' to remove files
Changes v2 to v3:
- Reworked the find command to avoid failure highlighted in 836528f03eb89fdc64432f7a7470145ddf553b8c and reverted in b928074e2d83f5b873bf1ba381b212dfad9bf207.
- Used find syntax discussed in the previous review.
https://lore.kernel.org/buildroot/[email protected]/
- Kept Yann's commit log.
package/pkg-python.mk | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
package/pkg-python.mk +3 -1
@@ -106,7 +106,9 @@ ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y) define PKG_PYTHON_FIXUP_SYSCONFIGDATA $(Q)find $(HOST_DIR) \ \( -path '$(HOST_DIR)/lib/python*' -o -path '$(STAGING_DIR)/usr/lib/python*' \) \ - -name "_sysconfigdata*.py" -print0 | xargs -0 --no-run-if-empty \ + \( \( -name "_sysconfigdata*.pyc" -delete \) \ + -o \( -name "_sysconfigdata*.py" -print0 \) \) \ + | xargs -0 --no-run-if-empty \ $(SED) 's:$(PER_PACKAGE_DIR)/[^/]\+/:$(PER_PACKAGE_DIR)/$($(PKG)_NAME)/:g' endef endif
Signature · 6 lines
2.31.1 _______________________________________________ buildroot mailing list [email protected] http://lists.busybox.net/mailman/listinfo/buildroot
PATCH v3 3/16 package/pkg-generic.mk: move python fixup to generic package infrastructure
In reply to Herve CodinaFixing _sysconfigdata*.{py,pyc} was previously done by python package infrastructure. Some packages use python stuff without using python package infrastructure.
These packages perform overwrites and need the specific python fixup to fix them.
In order to be sure to fix all of these packages, the python fixup is moved to the generic package infrastructure and applied to all packages.
This follows the same principle as for the .la libtool files fixup.
---
Changes v1 to v2:
- Used '... -print0 | xargs -0 -r ...' construction.
- Used '-deleted' to remove files.
Changes v2 to v3:
- Used find syntax discussed in the previous review.
https://lore.kernel.org/buildroot/[email protected]/
package/pkg-generic.mk | 24 +++++++++++++++++++++++- package/pkg-python.mk | 23 ----------------------- 2 files changed, 23 insertions(+), 24 deletions(-)
package/pkg-generic.mk +23 -1
@@ -103,6 +103,27 @@ define fixup-libtool-files endef endif +# Make sure python _sysconfigdata*.py files only reference the current +# per-package directory. +# +# Can't use $(foreach d, $(HOST_DIR)/lib/python* $(STAGING_DIR)/usr/lib/python*, ...) +# because those directories may be created in the same recipe this macro will +# be expanded in. +# Additionally, either or both may be missing, which would make find whine and +# fail. +# So we just use HOST_DIR as a starting point, and filter on the two directories +# of interest. +ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y) +define FIXUP_PYTHON_SYSCONFIGDATA + $(Q)find $(HOST_DIR) \ + \( -path '$(HOST_DIR)/lib/python*' -o -path '$(STAGING_DIR)/usr/lib/python*' \) \ + \( \( -name "_sysconfigdata*.pyc" -delete \) \ + -o \( -name "_sysconfigdata*.py" -print0 \) \) \ + | xargs -0 --no-run-if-empty \ + $(SED) 's:$(PER_PACKAGE_DIR)/[^/]\+/:$(PER_PACKAGE_DIR)/$($(PKG)_NAME)/:g' +endef +endif + # Functions to collect statistics about installed files # $(1): base directory to search in
@@ -836,7 +857,8 @@ $(2)_PRE_LEGAL_INFO_HOOKS ?= $(2)_POST_LEGAL_INFO_HOOKS ?= $(2)_TARGET_FINALIZE_HOOKS ?= $(2)_ROOTFS_PRE_CMD_HOOKS ?= -$(2)_POST_PREPARE_HOOKS ?= + +$(2)_POST_PREPARE_HOOKS += FIXUP_PYTHON_SYSCONFIGDATA ifeq ($$($(2)_TYPE),target) ifneq ($$(HOST_$(2)_KCONFIG_VAR),)
package/pkg-python.mk +0 -23
@@ -92,27 +92,6 @@ HOST_PKG_PYTHON_SETUPTOOLS_INSTALL_OPTS = \ --root=/ \ --single-version-externally-managed -# Make sure python _sysconfigdata*.py files only reference the current -# per-package directory. -# -# Can't use $(foreach d, $(HOST_DIR)/lib/python* $(STAGING_DIR)/usr/lib/python*, ...) -# because those directories may be created in the same recipe this macro will -# be expanded in. -# Additionally, either or both may be missing, which would make find whine and -# fail. -# So we just use HOST_DIR as a starting point, and filter on the two directories -# of interest. -ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y) -define PKG_PYTHON_FIXUP_SYSCONFIGDATA - $(Q)find $(HOST_DIR) \ - \( -path '$(HOST_DIR)/lib/python*' -o -path '$(STAGING_DIR)/usr/lib/python*' \) \ - \( \( -name "_sysconfigdata*.pyc" -delete \) \ - -o \( -name "_sysconfigdata*.py" -print0 \) \) \ - | xargs -0 --no-run-if-empty \ - $(SED) 's:$(PER_PACKAGE_DIR)/[^/]\+/:$(PER_PACKAGE_DIR)/$($(PKG)_NAME)/:g' -endef -endif - ################################################################################ # inner-python-package -- defines how the configuration, compilation # and installation of a Python package should be done, implements a
@@ -257,8 +236,6 @@ $(2)_PYTHON_INTERPRETER = $$(HOST_DIR)/bin/$$($(2)_NEEDS_HOST_PYTHON) endif endif -$(2)_PRE_CONFIGURE_HOOKS += PKG_PYTHON_FIXUP_SYSCONFIGDATA - # # Build step. Only define it if not already defined by the package .mk # file.
Signature · 6 lines
2.31.1 _______________________________________________ buildroot mailing list [email protected] http://lists.busybox.net/mailman/listinfo/buildroot
PATCH v3 4/16 package/owfs: remove Python sysconfigdata fixup
In reply to Herve CodinaThis fixup is done at pkg-generic level for all packages. So, it is no more needed in owfs package.
---
Changes v1 to v2:
- Added 'Reviewed-by: Yann E. MORIN'
Changes v2 to v3:
None
package/owfs/owfs.mk | 9 --------- 1 file changed, 9 deletions(-)
package/owfs/owfs.mk +0 -9
@@ -88,15 +88,6 @@ OWFS_DEPENDENCIES += python host-swig # override the PYSITEDIR variable in make. OWFS_EXTRA_MAKE_OPTS += PYSITEDIR=/usr/lib/python$(PYTHON_VERSION_MAJOR)/site-packages -ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y) -define OWFS_FIXUP_PYTHON_SYSCONFIGDATA - find $(HOST_DIR)/lib/python* $(STAGING_DIR)/usr/lib/python* \ - -name "_sysconfigdata*.py" | xargs --no-run-if-empty \ - $(SED) "s:$(PER_PACKAGE_DIR)/[^/]\+/:$(PER_PACKAGE_DIR)/owfs/:g" -endef -OWFS_PRE_CONFIGURE_HOOKS += OWFS_FIXUP_PYTHON_SYSCONFIGDATA -endif - else OWFS_CONF_OPTS += --disable-owpython --without-python endif
Signature · 6 lines
2.31.1 _______________________________________________ buildroot mailing list [email protected] http://lists.busybox.net/mailman/listinfo/buildroot
PATCH v3 5/16 qt5: Fix sporadic build failure during top-level parallel build
In reply to Herve CodinaFrom: Andreas Naumann <[email protected]>
When using top level parallel build, independent qt5 packages may be built in parallel. Because of their staging dirs being hardlinked, they all use the same qt.conf file to manipulate during configure, while another qt5 package might already use it. This leads to weird build failures because the folders qmake is using are diverted in erratic ways. Fix this by actually recreating a non-shared qt.conf file for every package.
---
New patch in this v3 series
This patch was previously send by Andreas Naumann.
https://lore.kernel.org/buildroot/[email protected]/
https://lore.kernel.org/buildroot/[email protected]/
As this patch makes sense in this series, it was integrated in this v3 version.
package/qt5/qt5.mk | 1 + 1 file changed, 1 insertion(+)
package/qt5/qt5.mk +1 -0
@@ -15,6 +15,7 @@ include $(sort $(wildcard package/qt5/*/*.mk)) # compiled into the Qt library. We need it to make "qmake" relocatable and # tweak the per-package install pathes define QT5_INSTALL_QT_CONF + rm -f $(HOST_DIR)/bin/qt.conf sed -e "s|@@HOST_DIR@@|$(HOST_DIR)|" -e "s|@@STAGING_DIR@@|$(STAGING_DIR)|" \ $(QT5BASE_PKGDIR)/qt.conf.in > $(HOST_DIR)/bin/qt.conf endef
Signature · 6 lines
2.31.1 _______________________________________________ buildroot mailing list [email protected] http://lists.busybox.net/mailman/listinfo/buildroot
Re: [Buildroot] [PATCH v3 05/16] qt5: Fix sporadic build failure during top-level parallel build
Hervé, All,
On 2021-08-17 10:39 +0200, Herve Codina spake thusly:
Quoted 8 lines
From: Andreas Naumann <[email protected]>
When using top level parallel build, independent qt5 packages may be built in parallel. Because of their staging dirs being hardlinked, they all use the same qt.conf file to manipulate during configure, while another qt5 package might already use it. This leads to weird build failures because the folders qmake is using are diverted in erratic ways. Fix this by actually recreating a non-shared qt.conf file for every package.
But with patch 10 (Makefile: breaks hardlinks in global {TARGET, HOST}_DIR on per-package build), is this patch 5 still needed?
IIUC, patch 10 makes the per-package directories actual copies rather than hardlinks, so the per-package host directory is a complete copy, thus this qt.conf is no longer a hardlink...
Did I miss something?
Regards,
Yann E. MORIN.
Quoted 32 lines
Signed-off-by Andreas Naumann <[email protected]>Reviewed-by Herve Codina <[email protected]>---
New patch in this v3 seriesThis patch was previously send by Andreas Naumann.
https://lore.kernel.org/buildroot/[email protected]/
https://lore.kernel.org/buildroot/[email protected]/
As this patch makes sense in this series, it was integrated in this v3 version.package/qt5/qt5.mk | 1 + 1 file changed, 1 insertion(+)package/qt5/qt5.mk +1
-0@@ -15,6 +15,7 @@ include $(sort $(wildcard package/qt5/*/*.mk)) # compiled into the Qt library. We need it to make "qmake" relocatable and # tweak the per-package install pathes define QT5_INSTALL_QT_CONF + rm -f $(HOST_DIR)/bin/qt.conf sed -e "s|@@HOST_DIR@@|$(HOST_DIR)|" -e "s|@@STAGING_DIR@@|$(STAGING_DIR)|" \ $(QT5BASE_PKGDIR)/qt.conf.in > $(HOST_DIR)/bin/qt.conf endefSignature · 6 lines
2.31.1 _______________________________________________ buildroot mailing list [email protected] http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 10 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' _______________________________________________ buildroot mailing list [email protected] http://lists.busybox.net/mailman/listinfo/buildroot
Re: [Buildroot] [PATCH v3 05/16] qt5: Fix sporadic build failure during top-level parallel build
Hervé, All,
On 2021-08-28 18:38 +0200, Yann E. MORIN spake thusly:
Quoted 10 lines
On 2021-08-17 10:39 +0200, Herve Codina spake thusly:
Quoted 8 lines
From: Andreas Naumann <[email protected]>
When using top level parallel build, independent qt5 packages may be built in parallel. Because of their staging dirs being hardlinked, they all use the same qt.conf file to manipulate during configure, while another qt5 package might already use it. This leads to weird build failures because the folders qmake is using are diverted in erratic ways. Fix this by actually recreating a non-shared qt.conf file for every package.
Quoted 8 lines
But with patch 10 (
Makefile: breaks hardlinks in global {TARGET, HOST}_DIR on per-package build), is this patch 5 still needed?IIUC, patch 10 makes the per-package directories actual copies rather than hardlinks, so the per-package host directory is a complete copy, thus this qt.conf is no longer a hardlink...
Did I miss something?
Yes, I missed that patch 10 is not about assembling the per-package directories from their dependencies, but to gather those per-package directories to assemble the final target/ and host/.
So, yes, this qt5.conf patch is still needed.
Regards,
Yann E. MORIN.
Quoted 47 lines
Regards,
Yann E. MORIN.Quoted 32 lines
Signed-off-by Andreas Naumann <[email protected]>Reviewed-by Herve Codina <[email protected]>---
New patch in this v3 seriesThis patch was previously send by Andreas Naumann.
https://lore.kernel.org/buildroot/[email protected]/
https://lore.kernel.org/buildroot/[email protected]/
As this patch makes sense in this series, it was integrated in this v3 version.package/qt5/qt5.mk | 1 + 1 file changed, 1 insertion(+)package/qt5/qt5.mk +1
-0@@ -15,6 +15,7 @@ include $(sort $(wildcard package/qt5/*/*.mk)) # compiled into the Qt library. We need it to make "qmake" relocatable and # tweak the per-package install pathes define QT5_INSTALL_QT_CONF + rm -f $(HOST_DIR)/bin/qt.conf sed -e "s|@@HOST_DIR@@|$(HOST_DIR)|" -e "s|@@STAGING_DIR@@|$(STAGING_DIR)|" \ $(QT5BASE_PKGDIR)/qt.conf.in > $(HOST_DIR)/bin/qt.conf endefSignature · 6 lines
2.31.1 _______________________________________________ buildroot mailing list [email protected] http://lists.busybox.net/mailman/listinfo/buildrootSignature · 10 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' _______________________________________________ buildroot mailing list [email protected] http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 10 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' _______________________________________________ buildroot mailing list [email protected] http://lists.busybox.net/mailman/listinfo/buildroot
Yann E. MORIN
AppliedRe: [Buildroot] [PATCH v3 05/16] qt5: Fix sporadic build failure during top-level parallel build
In reply to Herve CodinaHervé, Andreas, All,
On 2021-08-17 10:39 +0200, Herve Codina spake thusly:
Quoted 11 lines
From: Andreas Naumann <[email protected]>
When using top level parallel build, independent qt5 packages may be built in parallel. Because of their staging dirs being hardlinked, they all use the same qt.conf file to manipulate during configure, while another qt5 package might already use it. This leads to weird build failures because the folders qmake is using are diverted in erratic ways. Fix this by actually recreating a non-shared qt.conf file for every package.
Signed-off-by Andreas Naumann <[email protected]>Reviewed-by Herve Codina <[email protected]>
Applied to next, thanks! :-)
Regards,
Yann E. MORIN.
Quoted 30 lines
---
New patch in this v3 seriesThis patch was previously send by Andreas Naumann.
https://lore.kernel.org/buildroot/[email protected]/
https://lore.kernel.org/buildroot/[email protected]/
As this patch makes sense in this series, it was integrated in this v3 version.package/qt5/qt5.mk | 1 + 1 file changed, 1 insertion(+)package/qt5/qt5.mk +1
-0@@ -15,6 +15,7 @@ include $(sort $(wildcard package/qt5/*/*.mk)) # compiled into the Qt library. We need it to make "qmake" relocatable and # tweak the per-package install pathes define QT5_INSTALL_QT_CONF + rm -f $(HOST_DIR)/bin/qt.conf sed -e "s|@@HOST_DIR@@|$(HOST_DIR)|" -e "s|@@STAGING_DIR@@|$(STAGING_DIR)|" \ $(QT5BASE_PKGDIR)/qt.conf.in > $(HOST_DIR)/bin/qt.conf endefSignature · 6 lines
2.31.1 _______________________________________________ buildroot mailing list [email protected] http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 10 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' _______________________________________________ buildroot mailing list [email protected] http://lists.busybox.net/mailman/listinfo/buildroot
PATCH v3 6/16 package/pkg-qmake.mk: Move QT5_QT_CONF_FIXUP to post-prepare hook
In reply to Herve CodinaQT5_QT_CONF_FIXUP tweaks files for per-package directory build. This is typically the kind of operation expected to be in post-prepare hook.
---
New patch in this v3 series.
As said previously in
https://lore.kernel.org/buildroot/[email protected]/
This patch moves Qt5 fixup to <PKG>_POST_PREPARE_HOOKS (old <PKG>_PER_PACKAGE_TWEAK_HOOKS mentioned in discussion).
package/pkg-qmake.mk | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
package/pkg-qmake.mk +2 -1
@@ -56,13 +56,14 @@ $(2)_DEPENDENCIES += host-perl $(2)_PRE_CONFIGURE_HOOKS += QT_HEADERS_SYNC_HOOK endif +$(2)_POST_PREPARE_HOOKS += QT5_QT_CONF_FIXUP + # # Configure step. Only define it if not already defined by the package # .mk file. # ifndef $(2)_CONFIGURE_CMDS define $(2)_CONFIGURE_CMDS - $$(QT5_QT_CONF_FIXUP) cd $$($(2)_BUILDDIR) && \ $$(TARGET_MAKE_ENV) $$($(2)_CONF_ENV) $$(QT5_QMAKE) $$($(2)_CONF_OPTS) endef
Signature · 6 lines
2.31.1 _______________________________________________ buildroot mailing list [email protected] http://lists.busybox.net/mailman/listinfo/buildroot
Yann E. MORIN
AppliedRe: [Buildroot] [PATCH v3 06/16] package/pkg-qmake.mk: Move QT5_QT_CONF_FIXUP to post-prepare hook
Hervé, All,
On 2021-08-17 10:39 +0200, Herve Codina spake thusly:
Quoted 5 lines
QT5_QT_CONF_FIXUPtweaks files for per-package directory build. This is typically the kind of operation expected to be in post-prepare hook.Signed-off-by Herve Codina <[email protected]>
Applied to next, thanks.
Regards,
Yann E. MORIN.
Quoted 37 lines
---
New patch in this v3 series.
As said previously in
https://lore.kernel.org/buildroot/[email protected]/
This patch moves Qt5 fixup to <PKG>_POST_PREPARE_HOOKS (old <PKG>_PER_PACKAGE_TWEAK_HOOKS mentioned in discussion).package/pkg-qmake.mk | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)package/pkg-qmake.mk +2
-1@@ -56,13 +56,14 @@ $(2)_DEPENDENCIES += host-perl $(2)_PRE_CONFIGURE_HOOKS += QT_HEADERS_SYNC_HOOK endif +$(2)_POST_PREPARE_HOOKS += QT5_QT_CONF_FIXUP + # # Configure step. Only define it if not already defined by the package # .mk file. # ifndef $(2)_CONFIGURE_CMDS define $(2)_CONFIGURE_CMDS - $$(QT5_QT_CONF_FIXUP) cd $$($(2)_BUILDDIR) && \ $$(TARGET_MAKE_ENV) $$($(2)_CONF_ENV) $$(QT5_QMAKE) $$($(2)_CONF_OPTS) endefSignature · 6 lines
2.31.1 _______________________________________________ buildroot mailing list [email protected] http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 10 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' _______________________________________________ buildroot mailing list [email protected] http://lists.busybox.net/mailman/listinfo/buildroot
PATCH v3 7/16 package/pkg-generic.mk: detect files overwritten in TARGET_DIR and HOST_DIR
In reply to Herve CodinaFrom: Thomas Petazzoni <[email protected]>
Without per-package directory support, a package can happily overwrite files installed by other packages. Indeed, because the build order between packages is always guaranteed, Buildroot will always produce the same output.
However, with per-package directory support, it is absolutely critical that a given package does not overwrite files already installed by another package, due to how the final aggregation is done to create the complete target/, staging/ and host/ folders. Unfortunately, we currently don't have anything in Buildroot that detects this situation.
We used to have check-uniq-files, but it was dropped in commit 2496189a4207173e4cd5bbab90256f911175ee57.
This commit is a new implementation of such a detection, which is based on calculating and verifying MD5 hashes of installed files: the calculation is done at the beginning of the configure step, the verification during the newly introduced "install" step that takes place after all installation steps.
Since preventing file overwrites is really only needed when per-package directory support is used, and due to this verification having some overhead, it is only enabled when BR2_PER_PACKAGE_DIRECTORIES=y. This additional verification cost is however not too bad as on average, with per-package directory support, the per-package target/ and host/ directories will contain less files than with a build that doesn't use per-package directory support. This helps a bit in mitigating the additional cost of this verification.
Note that we are not handling separately HOST_DIR and STAGING_DIR, like we're doing with the pkg_size_{before,after} functions. Instead, the verification on HOST_DIR walks down into the STAGING_DIR.
During per-package build, original files are modified by fixup-libtool-files and fixup-python-files calls.
But since these fixups modify files using sed --in-place, these modifications are done using a temporary file and a call to rename. Rename breaks the hardlink to the original file and leave the temporary file in per-package TARGET dir.
As the original file is not modified, this is no longer considered as an overwrite. This patch simply considers that what is done by fixup-libtool-files and fixup-python-files is part of the original snapshot used to detect overwrites. And so, the original snapshot is taken after fixup-libtool-files and fixup-python-files calls.
---
This commit is retreived from Thomas's work.
The first version was discussed
https://patchwork.ozlabs.org/project/buildroot/patch/[email protected]/
This new version was not already submitted by Thomas or I missed it. Compared to the first version, this patch has an improved commit message and generates the md5sum snapshot using
'LC_ALL=C find $(1) -type f -print0 | xargs -0 -r md5sum > $($(PKG)_DIR)/.files$(2).md5;'
instead of
'cd $(1); LC_ALL=C find . -type f -exec md5sum {} \; > $($(PKG)_DIR)/.files$(2).md5'
Changes v1 to v2:
- Added note about why fixup-{libtool,python}-files are not considered overwrites and so take the overwrite snapshot after fixup-{libtool,python}-files call.
- Removed 'LC_ALL=C'
Changes v2 to v3:
None
package/pkg-generic.mk | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+)
package/pkg-generic.mk +23 -0
@@ -124,6 +124,25 @@ define FIXUP_PYTHON_SYSCONFIGDATA endef endif +# Functions to detect overwritten files + +ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y) +# $(1): base directory to search in +# $(2): suffix of file (optional) +define pkg_detect_overwrite_before + find $(1) -type f -print0 | xargs -0 -r md5sum > $($(PKG)_DIR)/.files$(2).md5; +endef + +# $(1): base directory to search in +# $(2): suffix of file (optional) +define pkg_detect_overwrite_after + if test -s $($(PKG)_DIR)/.files$(2).md5 ; then \ + md5sum --quiet -c $($(PKG)_DIR)/.files$(2).md5 || \ + { echo "ERROR: package $($(PKG)_NAME) has overwritten files installed by a previous package, aborting."; exit 1; } ; \ + fi +endef +endif + # Functions to collect statistics about installed files # $(1): base directory to search in
@@ -277,6 +296,8 @@ $(BUILD_DIR)/%/.stamp_configured: $(call fixup-libtool-files,$(NAME),$(HOST_DIR)) $(call fixup-libtool-files,$(NAME),$(STAGING_DIR)) $(foreach hook,$($(PKG)_POST_PREPARE_HOOKS),$(call $(hook))$(sep)) + @$(call pkg_detect_overwrite_before,$(TARGET_DIR)) + @$(call pkg_detect_overwrite_before,$(HOST_DIR),-host) $(foreach hook,$($(PKG)_PRE_CONFIGURE_HOOKS),$(call $(hook))$(sep)) $($(PKG)_CONFIGURE_CMDS) $(foreach hook,$($(PKG)_POST_CONFIGURE_HOOKS),$(call $(hook))$(sep))
@@ -401,6 +422,8 @@ $(BUILD_DIR)/%/.stamp_installed: @$(call pkg_size_after,$(STAGING_DIR),-staging) @$(call pkg_size_after,$(HOST_DIR),-host) @$(call check_bin_arch) + @$(call pkg_detect_overwrite_after,$(TARGET_DIR)) + @$(call pkg_detect_overwrite_after,$(HOST_DIR),-host) $(Q)touch $@ # Remove package sources
Signature · 6 lines
2.31.1 _______________________________________________ buildroot mailing list [email protected] http://lists.busybox.net/mailman/listinfo/buildroot
Re: [Buildroot] [PATCH v3 07/16] package/pkg-generic.mk: detect files overwritten in TARGET_DIR and HOST_DIR
Hervé, All,
On 2021-08-17 10:39 +0200, Herve Codina spake thusly:
Quoted 36 lines
From: Thomas Petazzoni <[email protected]>
Without per-package directory support, a package can happily overwrite files installed by other packages. Indeed, because the build order between packages is always guaranteed, Buildroot will always produce the same output.
However, with per-package directory support, it is absolutely critical that a given package does not overwrite files already installed by another package, due to how the final aggregation is done to create the complete target/, staging/ and host/ folders. Unfortunately, we currently don't have anything in Buildroot that detects this situation.
We used to have check-uniq-files, but it was dropped in commit
2496189a4207173e4cd5bbab90256f911175ee57.This commit is a new implementation of such a detection, which is based on calculating and verifying MD5 hashes of installed files: the calculation is done at the beginning of the configure step, the verification during the newly introduced "install" step that takes place after all installation steps.
Since preventing file overwrites is really only needed when per-package directory support is used, and due to this verification having some overhead, it is only enabled when BR2_PER_PACKAGE_DIRECTORIES=y. This additional verification cost is however not too bad as on average, with per-package directory support, the per-package target/ and host/ directories will contain less files than with a build that doesn't use per-package directory support. This helps a bit in mitigating the additional cost of this verification.
Note that we are not handling separately
HOST_DIRandSTAGING_DIR, like we're doing with the pkg_size_{before,after} functions. Instead, the verification onHOST_DIRwalks down into theSTAGING_DIR.
Quoted 11 lines
During per-package build, original files are modified by fixup-libtool-files and fixup-python-files calls.
But since these fixups modify files using sed --in-place, these modifications are done using a temporary file and a call to rename. Rename breaks the hardlink to the original file and leave the temporary file in per-package TARGET dir.
As the original file is not modified, this is no longer considered as an overwrite. This patch simply considers that what is done by fixup-libtool-files and fixup-python-files is part of the original snapshot used to detect overwrites. And so, the original snapshot is taken after fixup-libtool-files and fixup-python-files calls.
This paragraph should take into consideration the fact that we also now have those _POST_PREPARE_HOOKS (and first moving the libtool fixups to be hooks would have simplified this paragraph).
However, what prompted me from applying for now, is that this new detection is a hard error.
Previously, check-uniq-files was just emitting warnings, but would not prevent the build from failing. Now, with this patch, even an innocuous overwrite (e.g. because a post-build script deletes the file, or the content of the file really does not matter at runtime), the build will fail.
I.e. configurations that are currently working with PPD, despite the overwrite, will suddenly no longer build.
OTOH, if we do not make that a hard-error, we will never detect most issues, because users will never spot those warnings and wil enver report issues, and the autobuilders will not fail and we will not notice either...
One solution is to add a configuration knob to make that a hard-error, like we have the paranoid libs/headers check:
config BR2_PPD_OVERWRITE_STRICT
bool "Strict file overwrite detection" depends on BR2_PER_PACKAGE_DIRECTORIES help
Say 'y' here to turn the file overwrite detection to a hard error. By default, only warnings will be printed.
And use that to exit in error only if set:
[...]
{ \
echo "ERROR: [...]"; \
$(if $(BR2_PPD_OVERWRITE_STRICT),exit 1;) \
} ; \
[...]And then, ensure it is always enabled in our autobuilders, so in utils/genrandconfig:
diff --git a/utils/genrandconfig b/utils/genrandconfig index 622cfd4891..a891f93b6d 100755
--- a/utils/genrandconfig
+++ b/utils/genrandconfig
@@ -353,6 +353,7 @@ def gen_config(args):
# Per-package folder
if randint(0, 15) == 0:
configlines.append("BR2_PER_PACKAGE_DIRECTORIES=y\n")
+ configlines.append("BR2_PPD_OVERWRITE_STRICT=y\n")
# Amend the configuration with a few things.
if randint(0, 20) == 0:I am not totally sold to the new config knob either, but I'd still rather have it than unconditionally break the build...
After all, we know that there are some people using PPD (and TLPB) and are happy with how it works for them, so breaking the build on purpose would probably not be seen as an improvement...
Regards,
Yann E. MORIN.
Quoted 79 lines
Signed-off-by Thomas Petazzoni <[email protected]>Signed-off-by Herve Codina <[email protected]>---
This commit is retreived from Thomas's work.
The first version was discussed
https://patchwork.ozlabs.org/project/buildroot/patch/[email protected]/
This new version was not already submitted by Thomas or I missed it. Compared to the first version, this patch has an improved commit message and generates the md5sum snapshot using
'LC_ALL=C find $(1) -type f -print0 | xargs -0 -r md5sum > $($(PKG)_DIR)/.files$(2).md5;'
instead of
'cd $(1); LC_ALL=C find . -type f -exec md5sum {} \; > $($(PKG)_DIR)/.files$(2).md5'Changes v1 to v2:
- Added note about why fixup-{libtool,python}-files are not considered overwrites and so take the overwrite snapshot after fixup-{libtool,python}-files call.
- Removed 'LC_ALL=C'Changes v2 to v3:
Nonepackage/pkg-generic.mk | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+)package/pkg-generic.mk +23
-0@@ -124,6 +124,25 @@ define FIXUP_PYTHON_SYSCONFIGDATA endef endif +# Functions to detect overwritten files + +ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y) +# $(1): base directory to search in +# $(2): suffix of file (optional) +define pkg_detect_overwrite_before + find $(1) -type f -print0 | xargs -0 -r md5sum > $($(PKG)_DIR)/.files$(2).md5; +endef + +# $(1): base directory to search in +# $(2): suffix of file (optional) +define pkg_detect_overwrite_after + if test -s $($(PKG)_DIR)/.files$(2).md5 ; then \ + md5sum --quiet -c $($(PKG)_DIR)/.files$(2).md5 || \ + { echo "ERROR: package $($(PKG)_NAME) has overwritten files installed by a previous package, aborting."; exit 1; } ; \ + fi +endef +endif + # Functions to collect statistics about installed files # $(1): base directory to search in@@ -277,6 +296,8 @@ $(BUILD_DIR)/%/.stamp_configured: $(call fixup-libtool-files,$(NAME),$(HOST_DIR)) $(call fixup-libtool-files,$(NAME),$(STAGING_DIR)) $(foreach hook,$($(PKG)_POST_PREPARE_HOOKS),$(call $(hook))$(sep)) + @$(call pkg_detect_overwrite_before,$(TARGET_DIR)) + @$(call pkg_detect_overwrite_before,$(HOST_DIR),-host) $(foreach hook,$($(PKG)_PRE_CONFIGURE_HOOKS),$(call $(hook))$(sep)) $($(PKG)_CONFIGURE_CMDS) $(foreach hook,$($(PKG)_POST_CONFIGURE_HOOKS),$(call $(hook))$(sep))@@ -401,6 +422,8 @@ $(BUILD_DIR)/%/.stamp_installed: @$(call pkg_size_after,$(STAGING_DIR),-staging) @$(call pkg_size_after,$(HOST_DIR),-host) @$(call check_bin_arch) + @$(call pkg_detect_overwrite_after,$(TARGET_DIR)) + @$(call pkg_detect_overwrite_after,$(HOST_DIR),-host) $(Q)touch $@ # Remove package sourcesSignature · 6 lines
2.31.1 _______________________________________________ buildroot mailing list [email protected] http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 10 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' _______________________________________________ buildroot mailing list [email protected] http://lists.busybox.net/mailman/listinfo/buildroot
Re: [Buildroot] [PATCH v3 07/16] package/pkg-generic.mk: detect files overwritten in TARGET_DIR and HOST_DIR
Hello Yann,
First of all, thanks a lot for reviewing and merging bits of this patch series, I'm glad to see we're making progress with the TLP stuff.
Quoted 20 lines · On Sun, 29 Aug 2021 00:47:40 +0200 "Yann E. MORIN" <[email protected]>…
However, what prompted me from applying for now, is that this new detection is a hard error.
Previously, check-uniq-files was just emitting warnings, but would not prevent the build from failing. Now, with this patch, even an innocuous overwrite (e.g. because a post-build script deletes the file, or the content of the file really does not matter at runtime), the build will fail.
I.e. configurations that are currently working with PPD, despite the overwrite, will suddenly no longer build.
OTOH, if we do not make that a hard-error, we will never detect most issues, because users will never spot those warnings and wil enver report issues, and the autobuilders will not fail and we will not notice either...
One solution is to add a configuration knob to make that a hard-error, like we have the paranoid libs/headers check:
Quoted 7 lines
config
BR2_PPD_OVERWRITE_STRICT
bool "Strict file overwrite detection" depends onBR2_PER_PACKAGE_DIRECTORIEShelp
Say 'y' here to turn the file overwrite detection to a hard error. By default, only warnings will be printed.
We had some discussion with Hervé back when he worked on this, and I disagreed with adding an option. When BR2_PER_PACKAGE_DIRECTORIES=y, a file overwrite must be a hard error, as the result of the build is incorrect if there is an overwrite. It's not the "latest" package that wins in an overwrite situation, like it does in a non-PPD case.
So I really think this must be a hard error for PPD builds, and just a warning for non-PPD builds.
Yes, for PPD builds, it means users will get failures, but those failures are pointing to real problems.
So, my preference would be to merge as an unconditional check, and see how it goes. Perhaps the situation will be so bad that we will have to make it conditional, but I would prefer to have it unconditional first and see the impact.
Best regards,
Thomas
Signature · 7 lines
Thomas Petazzoni, co-owner and CEO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com _______________________________________________ buildroot mailing list [email protected] http://lists.busybox.net/mailman/listinfo/buildroot
Re: [Buildroot] [PATCH v3 07/16] package/pkg-generic.mk: detect files overwritten in TARGET_DIR and HOST_DIR
Thomas, Hervé, All,
On 2021-08-29 13:39 +0200, Thomas Petazzoni spake thusly:
Quoted 4 lines
Quoted 2 lines · On Sun, 29 Aug 2021 00:47:40 +0200 "Yann E. MORIN" <[email protected]>…
However, what prompted me from applying for now, is that this new detection is a hard error.
[--SNIP--]
Quoted 5 lines
We had some discussion with Hervé back when he worked on this, and I disagreed with adding an option. When BR2_PER_PACKAGE_DIRECTORIES=y, a file overwrite must be a hard error, as the result of the build is incorrect if there is an overwrite. It's not the "latest" package that wins in an overwrite situation, like it does in a non-PPD case.
So, existing builds that succeed with PPD, and for which an overwrite is not an issue even at runtime, will now be broken at build time.
After all, we _know_ there are users that already use PPD (for TLPB), and those users have working systems, but I highly doubt these builds have no file overwrite whatsoever...
Quoted 2 lines
So I really think this must be a hard error for PPD builds, and just a warning for non-PPD builds.
For non-PPD, that code is not even used, because the macros are only defined if BR2_PER_PACKAGE_DIRECTORIES=y; so for non-PPD, we will not even get a warning.
Quoted 2 lines
Yes, for PPD builds, it means users will get failures, but those failures are pointing to real problems.
I fully agree, but in practice, some of those overwrites may be totally innocusous when running the system. Think the gnu info db: we 100% don't care about it at runtime, but this is an overwrite (yes, we already fixed that one, but it was in this cycle, so users of previous releases who use PPD do have overwrites that they don't care about).
Quoted 2 lines
So, my preference would be to merge as an unconditional check, and see how it goes.
OK, I am aligning with that position. Unless someone else beats me to it, I'll resume work on this series later in the WE, or early next week...
Quoted 3 lines
Perhaps the situation will be so bad that we will have to make it conditional, but I would prefer to have it unconditional first and see the impact.
Note that my proposal would have unconditionally enabled it in the autobuilders, so we would have noticed as well, so we can still do that if we later decide to make the hard error conditional.
Finally, as a side note: there is still a case of file overwrite that we do not detect, even with this series: if two packages that are not part of the same dependency chain (i.e. they do not depend one on the other), and they both install the same file, then the file-overwrite will only happen when we eventually assemble the global target/ and host/ from the individual PPD target/ and host/ and we still have to add a detection for that case (which is not a pre-requisite before we apply the current series, of course).
Regards,
Yann E. MORIN.
Quoted 11 lines
Best regards,
Thomas
Signature · 7 lines
Thomas Petazzoni, co-owner and CEO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com _______________________________________________ buildroot mailing list [email protected] http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 10 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' _______________________________________________ buildroot mailing list [email protected] http://lists.busybox.net/mailman/listinfo/buildroot
Re: [Buildroot] [PATCH v3 07/16] package/pkg-generic.mk: detect files overwritten in TARGET_DIR and HOST_DIR
Thomas, All,
On 2021-08-29 14:51 +0200, Yann E. MORIN spake thusly:
[--SNIP--]
Quoted 8 lines
Finally, as a side note: there is still a case of file overwrite that we do not detect, even with this series: if two packages that are not part of the same dependency chain (i.e. they do not depend one on the other), and they both install the same file, then the file-overwrite will only happen when we eventually assemble the global target/ and host/ from the individual PPD target/ and host/ and we still have to add a detection for that case (which is not a pre-requisite before we apply the current series, of course).
And of course, this will also cause an overwrite when two such packages are also dependencies of a third patckage, and so when the PPD of that third package is prepared, we would have to detect file overwrite.
Note: let's assume the following hypotetical dependency graph (where the notation A --<-- B means that B depends on A):
.-<-- A --<-. S: skeleton
/ \ F: target-finalize
S --<-- B --<-- C
\ \
`-<-- D --<-----<-- FIf A and B both install the same file, then we do not have a sane PPD to build C.
If D and any package in the dependency chain of C (i.e. S, A, B, or C itself) install the same file, then we can only detect it in target-finalize.
So, let's take a step back, and have a wider look. I wonder if the end of a package installation step is the best location to detected file overwrite...
Indeed, let's assume that we do file overwrite detection:
1. in PPD preparation (when we rsync the dependencies, in the macro per-package-rsync): we can also detect that a package overwrites one or more files of another package, report those files, *and* prevent building a package with unsound dependencies;
2. in {target,host}-finalize: we can detect that a package overwrites one or more files provided by another package, and report those files.
So, the above (still theoretical implementation) is all we would need, in the end, and we would not need to do the detection at the end of a package installation step of each package, because it would not provide any information that we couldn't have with the above.
However, a benefit of also doing it at the end of the installation step, of a package, is that we can cut short on the build time. However, I am not sure that benefit is that important or major...
So, I'd like we think a bit harder before we apply this patch...
Regards,
Yann E. MORIN.
Signature · 10 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' _______________________________________________ buildroot mailing list [email protected] http://lists.busybox.net/mailman/listinfo/buildroot
Re: [Buildroot] [PATCH v3 07/16] package/pkg-generic.mk: detect files overwritten in TARGET_DIR and HOST_DIR
Quoted 44 lines · Yann E. MORIN
Thomas, All,
On 2021-08-29 14:51 +0200, Yann E. MORIN spake thusly:
[--SNIP--]Quoted 8 lines
Finally, as a side note: there is still a case of file overwrite that we do not detect, even with this series: if two packages that are not part of the same dependency chain (i.e. they do not depend one on the other), and they both install the same file, then the file-overwrite will only happen when we eventually assemble the global target/ and host/ from the individual PPD target/ and host/ and we still have to add a detection for that case (which is not a pre-requisite before we apply the current series, of course).
And of course, this will also cause an overwrite when two such packages are also dependencies of a third patckage, and so when the PPD of that third package is prepared, we would have to detect file overwrite.
Note: let's assume the following hypotetical dependency graph (where the notation A --<-- B means that B depends on A):
.-<-- A --<-. S: skeleton / \ F: target-finalize S --<-- B --<-- C \ \ `-<-- D --<-----<-- FIf A and B both install the same file, then we do not have a sane PPD to build C.
If D and any package in the dependency chain of C (i.e. S, A, B, or C itself) install the same file, then we can only detect it in target-finalize.
So, let's take a step back, and have a wider look. I wonder if the end of a package installation step is the best location to detected file overwrite...
Indeed, let's assume that we do file overwrite detection:
1. in PPD preparation (when we rsync the dependencies, in the macro per-package-rsync): we can also detect that a package overwrites one or more files of another package, report those files, *and* prevent building a package with unsound dependencies;
Quoted 8 lines
2. in {target,host}-finalize: we can detect that a package overwrites one or more files provided by another package, and report those files.
So, the above (still theoretical implementation) is all we would need, in the end, and we would not need to do the detection at the end of a package installation step of each package, because it would not provide any information that we couldn't have with the above.
Fabrice's patches have shown that the overwritten file detection is too
aggressive in the sense to that it *will* cause (many?) failures in the current situation.
So, taking a step further back, what really is the problem we're trying to
solve here? Why can't we overwrite files from other packages?
I think there are three distinct problems. Let's take Yann's example from above.
1. A file written in A and overwritten in C is only a problem because the finalize step doesn't take into account dependencies when combining the trees. We could solve that by taking into the account the dependencies.
2. A file written by both A and B is a problem for package C, because it is not predictable which version it will see. This problem can be solved by adding a dependency between A and B.
3. A file written by S and overwritten by A - this, I think, is the big problem. At first sight it's like case 1 - by taking into account the dependencies, the finalize stage will see the version from A. However, the problem is for stuff installed into staging, where B sees the version from S while C sees the version from A or from S, depending on order of A/B installation. This has two problems:
3a. C's version depends on the order of A/B installation;
3b. B sees a different version of the file than A - this could be a problem for e.g. header files that affect compilation.
The original scheme we discussed three or four years ago already took into
account situations 1 and 2, I think, but not yet situation 3. Note that back then, the overwritten file detection was assumed to happen only in finalize and accept overwrites in case there's a dependency.
The curses and crypt issues that Fabrice detected fall in category 3. And
they're extra nasty, because one of the packages involved is the toolchain that everybody depends on, so you can't just add a dependency to break it.
I haven't given sufficient thought to it yet, but one idea is to allow packages
to add a post-install hook to another package. That way, ncurses could remove curses.h from the toolchain, and libxcrypt could remove crypt.h. Still tricky though because it would mean that any package that uses crypt.h would need to optionally depend on libxcrypt - so that particular case would probably have to be solved by making libxcrypt be part of the toolchain... Or maybe making sure it doesn't overwrite crypt.h and teach its dependencies to include the libxcrypt-specific crypt.h.
One thing is certain: it's complicated :-)
Regards,
Arnout
Quoted 10 lines
However, a benefit of also doing it at the end of the installation step, of a package, is that we can cut short on the build time. However, I am not sure that benefit is that important or major...
So, I'd like we think a bit harder before we apply this patch...
Regards,
Yann E. MORIN.
_______________________________________________
buildroot mailing list
[email protected]
http://lists.busybox.net/mailman/listinfo/buildroot
Re: [Buildroot] [PATCH v3 07/16] package/pkg-generic.mk: detect files overwritten in TARGET_DIR and HOST_DIR
In reply to Thomas PetazzoniQuoted 45 lines · Thomas Petazzoni
Hello Yann,
First of all, thanks a lot for reviewing and merging bits of this patch series, I'm glad to see we're making progress with the TLP stuff.
Quoted 20 lines · On Sun, 29 Aug 2021 00:47:40 +0200 "Yann E. MORIN" <[email protected]>…
However, what prompted me from applying for now, is that this new detection is a hard error.
Previously, check-uniq-files was just emitting warnings, but would not prevent the build from failing. Now, with this patch, even an innocuous overwrite (e.g. because a post-build script deletes the file, or the content of the file really does not matter at runtime), the build will fail.
I.e. configurations that are currently working with PPD, despite the overwrite, will suddenly no longer build.
OTOH, if we do not make that a hard-error, we will never detect most issues, because users will never spot those warnings and wil enver report issues, and the autobuilders will not fail and we will not notice either...
One solution is to add a configuration knob to make that a hard-error, like we have the paranoid libs/headers check:
Quoted 7 lines
config
BR2_PPD_OVERWRITE_STRICT
bool "Strict file overwrite detection" depends onBR2_PER_PACKAGE_DIRECTORIEShelp
Say 'y' here to turn the file overwrite detection to a hard error. By default, only warnings will be printed.We had some discussion with Hervé back when he worked on this, and I disagreed with adding an option. When BR2_PER_PACKAGE_DIRECTORIES=y, a file overwrite must be a hard error, as the result of the build is incorrect if there is an overwrite. It's not the "latest" package that wins in an overwrite situation, like it does in a non-PPD case.
So I really think this must be a hard error for PPD builds, and just a warning for non-PPD builds.
Quoted 7 lines
Yes, for PPD builds, it means users will get failures, but those failures are pointing to real problems.
So, my preference would be to merge as an unconditional check, and see how it goes. Perhaps the situation will be so bad that we will have to make it conditional, but I would prefer to have it unconditional first and see the impact.
I was originally with Yann, but these arguments convinced that it is indeed
better to not have the option (for now).
Regards,
Arnout
_______________________________________________
buildroot mailing list
[email protected]
http://lists.busybox.net/mailman/listinfo/buildroot
Re: [Buildroot] [PATCH v3 07/16] package/pkg-generic.mk: detect files overwritten in TARGET_DIR and HOST_DIR
Hi all,
Quoted 49 lines · Arnout Vandecappelle
Quoted 46 lines · Thomas Petazzoni
Hello Yann,
First of all, thanks a lot for reviewing and merging bits of this patch series, I'm glad to see we're making progress with the TLP stuff.
Quoted 17 lines · On Sun, 29 Aug 2021 00:47:40 +0200 "Yann E. MORIN" <[email protected]>…
However, what prompted me from applying for now, is that this new detection is a hard error.
Previously, check-uniq-files was just emitting warnings, but would not prevent the build from failing. Now, with this patch, even an innocuous overwrite (e.g. because a post-build script deletes the file, or the content of the file really does not matter at runtime), the build will fail.
I.e. configurations that are currently working with PPD, despite the overwrite, will suddenly no longer build.
OTOH, if we do not make that a hard-error, we will never detect most issues, because users will never spot those warnings and wil enver report issues, and the autobuilders will not fail and we will not notice either...
Quoted 7 lines
One solution is to add a configuration knob to make that a hard-error, like we have the paranoid libs/headers check:
config
BR2_PPD_OVERWRITE_STRICTbool "Strict file overwrite detection" depends onBR2_PER_PACKAGE_DIRECTORIEShelp Say 'y' here to turn the file overwrite detection to a hard error. By default, only warnings will be printed.We had some discussion with Hervé back when he worked on this, and I disagreed with adding an option. When BR2_PER_PACKAGE_DIRECTORIES=y, a file overwrite must be a hard error, as the result of the build is incorrect if there is an overwrite. It's not the "latest" package that wins in an overwrite situation, like it does in a non-PPD case.
So I really think this must be a hard error for PPD builds, and just a warning for non-PPD builds.
Yes, for PPD builds, it means users will get failures, but those failures are pointing to real problems.
Quoted 7 lines
Quoted 4 lines
So, my preference would be to merge as an unconditional check, and see how it goes. Perhaps the situation will be so bad that we will have to make it conditional, but I would prefer to have it unconditional first and see the impact.
I was originally with Yann, but these arguments convinced that it is indeed better to not have the option (for now).
So I'm one of those actively using TLPD and I'm happy to have this producing hard errors to find possible issues in my configs as well as helping to fix them.
So I gave this a spin and immediately found util-linux recompiling some libs which util-linux-libs already installed. Looking for the reason I came upon Carlos Santos Commit 8bafc6dc "package/util-linux: build programs and libraries in separate packages" where the commit msg says ".... Installing util-linux overrides files installed by util-linux-libs but this is not a problem: it's allowed for a package to overwrite files from another package, as long as there is a dependency between the two."
So if this is still true, the logic should be changed to take dependencies into account like suggested in the other discussion.
However, the check also found a conflict where GL/glext.h of mesa3d is overwritten by gst1-plugins-base. It is a dependency, thus it's deterministic, but still we probably end up with an unexpected version glext.h in the SDK, so it's nice that things like this are brought to attention.
Now the build of our standard qt-demo config is almost done and another
conflict pops up when some kernel modules of Wifi-Sticks are installed.
Of course modules.dep, modules.alias and so on are modified. That will have to be taken care of.
However, only 3 problems up to just before the target-finalize step. So far that's not too aggressive for me.
regards,
Andreas
Quoted 6 lines
Regards, Arnout
_______________________________________________ buildroot mailing list [email protected] http://lists.busybox.net/mailman/listinfo/buildroot
_______________________________________________
buildroot mailing list
[email protected]
http://lists.busybox.net/mailman/listinfo/buildroot
Re: [Buildroot] [PATCH v3 07/16] package/pkg-generic.mk: detect files overwritten in TARGET_DIR and HOST_DIR
In reply to Herve CodinaHervé, Thomas, All,
On 2021-08-17 10:39 +0200, Herve Codina spake thusly:
Quoted 5 lines
From: Thomas Petazzoni <[email protected]> Without per-package directory support, a package can happily overwrite files installed by other packages. Indeed, because the build order between packages is always guaranteed, Buildroot will always produce the same output.
[--SNIP--]
Quoted 5 lines
This commit is a new implementation of such a detection, which is based on calculating and verifying MD5 hashes of installed files: the calculation is done at the beginning of the configure step, the verification during the newly introduced "install" step that takes place after all installation steps.
Following the various mails in the thread, and the discussion we had on Tuesday, I'm markign this patch as Changes Requested.
I'll try to scavenge all I can from the remaining patches, and mark the rest as Changes Requested as well.
So, to summarise the discussion, taking my previous example as a basis, and with Arnout's comment in mind (Aha, pun!):
.-<-- A --<-. S: skeleton
/ \ F: target-finalize
S --<-- B --<-- C
\ \
`-<-- D --<-----<-- F1. If C overwrites a file from (e.g.) A: we don't know the order in which the target-finalize will copy files.
- we can guarantee that ordering by taking the dependency chain into account when doing target-finalize.
2. If A and B install the same package: we don't know what to provide to package C:
- adding an artificial dependency between A and B is not acceptable, or we could end up with a lot of such unmanageable dependencies, not even accounting for packages in a br32-external tree;
- if the file is a DB (like the GNU info DB), stuff is "added" to that DB; only providing an arbitrary version of the file (from A or from B) is incorrect.
3. If a package in the C dependency chain (C included), and package D install the same file, then we're in the same boat as the above: we do not know what version to use in tergaet-finalize.
If I missed something, then feel free to complete. ;-)
Regards,
Yann E. MORIN.
Signature · 10 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
PATCH v3 8/16 package/pkg-generic.mk: generate final rsync exclude file list
In reply to Herve CodinaThe final rsync performed at {host,target}-finalize steps need to be done by rsyncing files generated by each packages without looking at files generated by other packages the current package depends on. This is needed to avoid overwrites in final {HOST,TARGET}_DIR.
In order to prepare the final rsync, an exclusion list is generated. This list lists files that are not generated by the current package and so files that need to be excluded from the final rsync.
Note also that the files list was not based on .files-list.{before,after}. During .file-list.{before,after} built for host directory, staging sub-directory (ie <toolchain>/sysroot) is filtered out.
The final rsync exclusion list needs to take into account the full {host,target} directory to avoid final overwrites.
Using an empty directory for per-package installation directory would be the simplest way to find what a package installs.
However, as it has been discussed in the past, this is fraught with unworkable issues. For example, some paths may be hard-coded at configure time and/or build time, and thus the package would still install in the original stagin we presented it with (for target/, this is not an isue, because target/ is never looked at during configure or build, only at install time). Or a package installation process just assumes that the directory structure exists (bad, but eh...).
---
Changes v1 to v2:
- Added a note in the commit message related to why an empty directory is not used to find what a package installs.
Changes v2 to v3:
None
package/pkg-generic.mk | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+)
package/pkg-generic.mk +28 -0
@@ -168,6 +168,30 @@ define pkg_size_after rm -f $($(PKG)_DIR)/.files-list$(2).after endef +# Functions to collect final rsync exclusion files + +# $(1): base directory to search in +# $(2): suffix of file (optional) +define pkg_final_rsync_before + cd $(1); \ + LC_ALL=C find . \( -type f -o -type l \) -printf '%T@:%i:%#m:%y:%s,%p\n' \ + | LC_ALL=C sort > $($(PKG)_DIR)/.files-final-rsync$(2).before +endef + +# $(1): base directory to search in +# $(2): suffix of file (optional) +define pkg_final_rsync_after + cd $(1); \ + LC_ALL=C find . \( -type f -o -type l \) -printf '%T@:%i:%#m:%y:%s,%p\n' \ + | LC_ALL=C sort > $($(PKG)_DIR)/.files-final-rsync$(2).after + LC_ALL=C comm -2 \ + $($(PKG)_DIR)/.files-final-rsync$(2).before \ + $($(PKG)_DIR)/.files-final-rsync$(2).after \ + | sed -r -e 's/^[^,]+,./- /' \ + > $($(PKG)_DIR)/.files-final-rsync$(2).exclude_rsync + rm -f $($(PKG)_DIR)/.files-final-rsync$(2).after +endef + define check_bin_arch support/scripts/check-bin-arch -p $($(PKG)_NAME) \ -l $($(PKG)_DIR)/.files-list.txt \
@@ -290,6 +314,8 @@ $(BUILD_DIR)/%/.stamp_configured: @$(call MESSAGE,"Configuring") $(Q)mkdir -p $(HOST_DIR) $(TARGET_DIR) $(STAGING_DIR) $(BINARIES_DIR) $(call prepare-per-package-directory,$($(PKG)_FINAL_DEPENDENCIES)) + @$(call pkg_final_rsync_before,$(TARGET_DIR)) + @$(call pkg_final_rsync_before,$(HOST_DIR),-host) @$(call pkg_size_before,$(TARGET_DIR)) @$(call pkg_size_before,$(STAGING_DIR),-staging) @$(call pkg_size_before,$(HOST_DIR),-host)
@@ -422,6 +448,8 @@ $(BUILD_DIR)/%/.stamp_installed: @$(call pkg_size_after,$(STAGING_DIR),-staging) @$(call pkg_size_after,$(HOST_DIR),-host) @$(call check_bin_arch) + @$(call pkg_final_rsync_after,$(TARGET_DIR)) + @$(call pkg_final_rsync_after,$(HOST_DIR),-host) @$(call pkg_detect_overwrite_after,$(TARGET_DIR)) @$(call pkg_detect_overwrite_after,$(HOST_DIR),-host) $(Q)touch $@
Signature · 6 lines
2.31.1 _______________________________________________ buildroot mailing list [email protected] http://lists.busybox.net/mailman/listinfo/buildroot
PATCH v3 9/16 Makefile: rsync global {TARGET, HOST}_DIR using exclusion file list
In reply to Herve CodinaOn a per-package build, rsync final {TARGET,HOST}_DIR using exclusion file list computed for each packages.
This exclusion list allows to rsync only files generated by a given package. This is needed to avoid any overwrites in final {TARGET,HOST}_DIR.
As we rsync only files generated by each packages, we need to to do this rsync recursively on each dependencies to collect all needed files. This is done based on existing <PKG>_FINAL_RECURSIVE_DEPENDENCIES
---
Changes v1 to v2:
- Added a note in the commit message related to why this exclusion list is needed.
- Added $(Q) to mkdir and rsync
- Changed indentations to make calls more readable
Changes v2 to v3:
None
Makefile | 34 ++++++++++++++++++++++++++++++++-- 1 file changed, 32 insertions(+), 2 deletions(-)
Makefile +32 -2
@@ -725,10 +725,33 @@ TARGET_DIR_FILES_LISTS = $(sort $(wildcard $(BUILD_DIR)/*/.files-list.txt)) HOST_DIR_FILES_LISTS = $(sort $(wildcard $(BUILD_DIR)/*/.files-list-host.txt)) STAGING_DIR_FILES_LISTS = $(sort $(wildcard $(BUILD_DIR)/*/.files-list-staging.txt)) +ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y) +# rsync the contents of per-package directories +# $1: space-separated list of packages to rsync from +# $2: 'host' or 'target' +# $3: destination directory +# $4: exclude file list to use +define per-package-rsync-delta + $(Q)mkdir -p $(3) + $(foreach pkg,$(1),\ + $(Q)rsync -a --link-dest=$(PER_PACKAGE_DIR)/$(pkg)/$(strip $(2))/ \ + --filter='merge $($(call UPPERCASE,$(pkg))_DIR)/$(strip $(4))' \ + $(PER_PACKAGE_DIR)/$(pkg)/$(strip $(2))/ \ + $(3)$(sep)) +endef +endif + .PHONY: host-finalize host-finalize: $(PACKAGES) $(HOST_DIR) $(HOST_DIR_SYMLINK) @$(call MESSAGE,"Finalizing host directory") - $(call per-package-rsync,$(sort $(PACKAGES)),host,$(HOST_DIR)) + $(call per-package-rsync-delta, \ + $(sort $(foreach pkg, $(PACKAGES), \ + $(pkg) $($(call UPPERCASE,$(pkg))_FINAL_RECURSIVE_DEPENDENCIES)) \ + ), \ + host, \ + $(HOST_DIR), \ + .files-final-rsync-host.exclude_rsync \ + ) .PHONY: staging-finalize staging-finalize: $(STAGING_DIR_SYMLINK)
@@ -736,7 +759,14 @@ staging-finalize: $(STAGING_DIR_SYMLINK) .PHONY: target-finalize target-finalize: $(PACKAGES) $(TARGET_DIR) host-finalize @$(call MESSAGE,"Finalizing target directory") - $(call per-package-rsync,$(sort $(PACKAGES)),target,$(TARGET_DIR)) + $(call per-package-rsync-delta, \ + $(sort $(foreach pkg, $(PACKAGES), \ + $(pkg) $($(call UPPERCASE,$(pkg))_FINAL_RECURSIVE_DEPENDENCIES)) \ + ), \ + target, \ + $(TARGET_DIR), \ + .files-final-rsync.exclude_rsync \ + ) $(foreach hook,$(TARGET_FINALIZE_HOOKS),$($(hook))$(sep)) rm -rf $(TARGET_DIR)/usr/include $(TARGET_DIR)/usr/share/aclocal \ $(TARGET_DIR)/usr/lib/pkgconfig $(TARGET_DIR)/usr/share/pkgconfig \
Signature · 6 lines
2.31.1 _______________________________________________ buildroot mailing list [email protected] http://lists.busybox.net/mailman/listinfo/buildroot
PATCH v3 10/16 Makefile: breaks hardlinks in global {TARGET, HOST}_DIR on per-package build
In reply to Herve CodinaWithout this patch, a make <pkg>_rebuild detects overwrites. Indeed, in target_finalize steps some modifications are done on installed files (ie strip or TARGET_FINALIZE_HOOKS for instance).
In order to avoid these modifications seen from per-package {TARGET,HOST}_DIR and so been analyzed as some overwrites, global {TARGET,HOST}_DIR is built using a full copy of the involved per-package files instead of hardlinks.
---
Changes v1 to v2:
- Added 'Reviewed-by: Yann E. MORIN'
Changes v2 to v2:
None
Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Makefile +1 -1
@@ -734,7 +734,7 @@ ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y) define per-package-rsync-delta $(Q)mkdir -p $(3) $(foreach pkg,$(1),\ - $(Q)rsync -a --link-dest=$(PER_PACKAGE_DIR)/$(pkg)/$(strip $(2))/ \ + $(Q)rsync -a \ --filter='merge $($(call UPPERCASE,$(pkg))_DIR)/$(strip $(4))' \ $(PER_PACKAGE_DIR)/$(pkg)/$(strip $(2))/ \ $(3)$(sep))
Signature · 6 lines
2.31.1 _______________________________________________ buildroot mailing list [email protected] http://lists.busybox.net/mailman/listinfo/buildroot
Re: [Buildroot] [PATCH v3 10/16] Makefile: breaks hardlinks in global {TARGET, HOST}_DIR on per-package build
Hervé, All,
On 2021-08-17 10:39 +0200, Herve Codina spake thusly:
Quoted 25 lines
Without this patch, a make <pkg>_rebuild detects overwrites. Indeed, in
target_finalizesteps some modifications are done on installed files (ie strip orTARGET_FINALIZE_HOOKSfor instance).In order to avoid these modifications seen from per-package {TARGET,HOST}_DIR and so been analyzed as some overwrites, global {TARGET,HOST}_DIR is built using a full copy of the involved per-package files instead of hardlinks.
Signed-off-by Herve Codina <[email protected]>Reviewed-by Yann E. MORIN <[email protected]>---
Changes v1 to v2:
- Added 'Reviewed-by: Yann E. MORIN'Changes v2 to v2:
NoneMakefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)Makefile +0
-0
Makefile ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y) · line 737
Quoted 5 lines
Makefile +1
-1@@ -734,7 +734,7 @@ ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y) define per-package-rsync-delta $(Q)mkdir -p $(3) $(foreach pkg,$(1),\ - $(Q)rsync -a --link-dest=$(PER_PACKAGE_DIR)/$(pkg)/$(strip $(2))/ \ + $(Q)rsync -a \
I was about to apply that one, because it does not look like it (semantically) depends on the previous patches, but of course it does not apply cleanly... I found it too dangerous to apply and tweak locally without further consideration. So it follows the fate of the rest of the
series: marked Changes Requested.
Regards,
Yann E. MORIN.
Quoted 10 lines
Makefile +0
-0--filter='merge $($(call UPPERCASE,$(pkg))_DIR)/$(strip $(4))' \ $(PER_PACKAGE_DIR)/$(pkg)/$(strip $(2))/ \ $(3)$(sep))Signature · 6 lines
2.31.1 _______________________________________________ buildroot mailing list [email protected] http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 10 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Re: [Buildroot] [PATCH v3 10/16] Makefile: breaks hardlinks in global {TARGET, HOST}_DIR on per-package build
In reply to Herve CodinaOn 17/08/2021 10.39, Herve Codina wrote:
Hi,
Sorry, very old patch.
Quoted 8 lines
Without this patch, a make <pkg>_rebuild detects overwrites. Indeed, in
target_finalizesteps some modifications are done on installed files (ie strip orTARGET_FINALIZE_HOOKSfor instance).In order to avoid these modifications seen from per-package {TARGET,HOST}_DIR and so been analyzed as some overwrites, global {TARGET,HOST}_DIR is built using a full copy of the involved per-package files instead of hardlinks.
Quoted 5 lines
Signed-off-by Herve Codina <[email protected]>Reviewed-by Yann E. MORIN <[email protected]>---
Changes v1 to v2:
- Added 'Reviewed-by: Yann E. MORIN'
We have similar issues if you have a post-build script making changes with sed or similar, so committed to master after fixing it up (the logic was in the mean time moved to package/pkg-utils.mk), thanks.
Quoted 20 lines
Changes v2 to v2:
NoneMakefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)Makefile +1
-1@@ -734,7 +734,7 @@ ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y) define per-package-rsync-delta $(Q)mkdir -p $(3) $(foreach pkg,$(1),\ - $(Q)rsync -a --link-dest=$(PER_PACKAGE_DIR)/$(pkg)/$(strip $(2))/ \ + $(Q)rsync -a \ --filter='merge $($(call UPPERCASE,$(pkg))_DIR)/$(strip $(4))' \ $(PER_PACKAGE_DIR)/$(pkg)/$(strip $(2))/ \ $(3)$(sep))
Signature · 6 lines
Bye, Peter Korsgaard _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Re: [Buildroot] [PATCH v3 10/16] Makefile: breaks hardlinks in global {TARGET, HOST}_DIR on per-package build
Quoted 21 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Peter" == Peter Korsgaard <[email protected]> writes:
On 17/08/2021 10.39, Herve Codina wrote:
Hi,Sorry, very old patch.
Quoted 7 lines
Without this patch, a make <pkg>_rebuild detects overwrites. Indeed, in
target_finalizesteps some modifications are done on installed files (ie strip orTARGET_FINALIZE_HOOKSfor instance).
In order to avoid these modifications seen from per-package {TARGET,HOST}_DIR
and so been analyzed as some overwrites, global {TARGET,HOST}_DIR is built using a full copy of the involved per-package files instead of hardlinks.Quoted 5 lines
Signed-off-by Herve Codina <[email protected]>Reviewed-by Yann E. MORIN <[email protected]>---
Changes v1 to v2:
- Added 'Reviewed-by: Yann E. MORIN'
Quoted 3 lines
We have similar issues if you have a post-build script making changes with sed or similar, so committed to master after fixing it up (the logic was in the mean time moved to
package/pkg-utils.mk), thanks.
Committed to 2023.02.x and 2023.08.x, thanks.
Signature · 5 lines
Bye, Peter Korsgaard _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
PATCH v3 11/16 package/pkg-generic.mk: fix per-package <pkg>-{reconfigure, rebuild, reinstall}
In reply to Herve CodinaMany overwrites are detected on <pkg>-{reconfigure,rebuild,reinstall}. Indeed, files previously installed by a package were detected as overwritten on next reconfigure, rebuild or reinstall.
To avoid this, we recreate per-package host and target dir from scratch as it was done during the first configure step.
In order not to duplicate the code, this commit shares common code in prepare-pre-configure macro.
---
Changes v1 to v2:
- Avoid code duplication using prepare-pre-configure macro
Changes v2 to v3:
None
package/pkg-generic.mk | 40 ++++++++++++++++++++++++++++++---------- 1 file changed, 30 insertions(+), 10 deletions(-)
package/pkg-generic.mk +30 -10
@@ -217,6 +217,21 @@ define REMOVE_CONFLICTING_USELESS_FILES_IN_TARGET $(call remove-conflicting-useless-files,$(TARGET_DIR)) endef +# Function to prepare package + +define prepare-pre-configure + @$(call pkg_final_rsync_before,$(TARGET_DIR)) + @$(call pkg_final_rsync_before,$(HOST_DIR),-host) + @$(call pkg_size_before,$(TARGET_DIR)) + @$(call pkg_size_before,$(STAGING_DIR),-staging) + @$(call pkg_size_before,$(HOST_DIR),-host) + $(call fixup-libtool-files,$(NAME),$(HOST_DIR)) + $(call fixup-libtool-files,$(NAME),$(STAGING_DIR)) + $(foreach hook,$($(PKG)_POST_PREPARE_HOOKS),$(call $(hook))$(sep)) + @$(call pkg_detect_overwrite_before,$(TARGET_DIR)) + @$(call pkg_detect_overwrite_before,$(HOST_DIR),-host) +endef + ################################################################################ # Implicit targets -- produce a stamp file for each step of a package build ################################################################################
@@ -314,16 +329,7 @@ $(BUILD_DIR)/%/.stamp_configured: @$(call MESSAGE,"Configuring") $(Q)mkdir -p $(HOST_DIR) $(TARGET_DIR) $(STAGING_DIR) $(BINARIES_DIR) $(call prepare-per-package-directory,$($(PKG)_FINAL_DEPENDENCIES)) - @$(call pkg_final_rsync_before,$(TARGET_DIR)) - @$(call pkg_final_rsync_before,$(HOST_DIR),-host) - @$(call pkg_size_before,$(TARGET_DIR)) - @$(call pkg_size_before,$(STAGING_DIR),-staging) - @$(call pkg_size_before,$(HOST_DIR),-host) - $(call fixup-libtool-files,$(NAME),$(HOST_DIR)) - $(call fixup-libtool-files,$(NAME),$(STAGING_DIR)) - $(foreach hook,$($(PKG)_POST_PREPARE_HOOKS),$(call $(hook))$(sep)) - @$(call pkg_detect_overwrite_before,$(TARGET_DIR)) - @$(call pkg_detect_overwrite_before,$(HOST_DIR),-host) + $(call prepare-pre-configure) $(foreach hook,$($(PKG)_PRE_CONFIGURE_HOOKS),$(call $(hook))$(sep)) $($(PKG)_CONFIGURE_CMDS) $(foreach hook,$($(PKG)_POST_CONFIGURE_HOOKS),$(call $(hook))$(sep))
@@ -488,6 +494,13 @@ define pkg-graph-depends $$(GRAPHS_DIR)/$$(@).dot endef +ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y) +define empty-per-package-directory + rm -rf $(HOST_DIR) $(TARGET_DIR) + mkdir -p $(HOST_DIR) $(TARGET_DIR) +endef +endif + ################################################################################ # inner-generic-package -- generates the make targets needed to build a # generic package
@@ -1092,6 +1105,8 @@ $(1)-all-legal-info: $$(foreach p,$$($(2)_FINAL_ALL_DEPENDENCIES),$$(p)-all-lega $(1)-dirclean: $$($(2)_TARGET_DIRCLEAN) +$(1)-clean-for-reinstall: PKG=$(2) +$(1)-clean-for-reinstall: NAME=$(1) $(1)-clean-for-reinstall: ifneq ($$($(2)_OVERRIDE_SRCDIR),) rm -f $$($(2)_TARGET_RSYNC)
@@ -1101,6 +1116,11 @@ endif rm -f $$($(2)_TARGET_INSTALL_TARGET) rm -f $$($(2)_TARGET_INSTALL_IMAGES) rm -f $$($(2)_TARGET_INSTALL_HOST) + $$(call empty-per-package-directory) + $$(call prepare-per-package-directory,$$($(2)_FINAL_DOWNLOAD_DEPENDENCIES)) + $$(call prepare-per-package-directory,$$($(2)_FINAL_EXTRACT_DEPENDENCIES)) + $$(call prepare-per-package-directory,$$($(2)_FINAL_DEPENDENCIES)) + $$(call prepare-pre-configure) $(1)-reinstall: $(1)-clean-for-reinstall $(1)
Signature · 6 lines
2.31.1 _______________________________________________ buildroot mailing list [email protected] http://lists.busybox.net/mailman/listinfo/buildroot
PATCH v3 12/16 package/pkg-generic.mk: remove .files-final-rsync.before temporary file
In reply to Herve CodinaKeeping the .files-final-rsync.before file between make invocations was needed to ensure that this file will be present in case of 'make <pkg>_{reconfigure,rebuild,reinstall}' invocation.
The .files-final-rsync.before is now recreated on <pkg>_{reconfigure,rebuild,reinstall} and we don't need to keep it between make invocation.
This patch simply removes this file as soon as it is no longer needed.
---
New patch in this v2 series
Changes v2 to v3:
None
package/pkg-generic.mk | 1 + 1 file changed, 1 insertion(+)
package/pkg-generic.mk +1 -0
@@ -189,6 +189,7 @@ define pkg_final_rsync_after $($(PKG)_DIR)/.files-final-rsync$(2).after \ | sed -r -e 's/^[^,]+,./- /' \ > $($(PKG)_DIR)/.files-final-rsync$(2).exclude_rsync + rm -f $($(PKG)_DIR)/.files-final-rsync$(2).before rm -f $($(PKG)_DIR)/.files-final-rsync$(2).after endef
Signature · 6 lines
2.31.1 _______________________________________________ buildroot mailing list [email protected] http://lists.busybox.net/mailman/listinfo/buildroot
PATCH v3 13/16 support/testing/infra: add log_file_path() function
In reply to Herve CodinaFrom: Thomas Petazzoni <[email protected]>
Some tests will need to grep through the build log to verify that some features are working are expected. In order to allow them to open the build log, we provide a new function called log_file_path(), which returns the path to the log file if available.
We also use this function in open_log_file().
Note that open_log_file() cannot be used directly to grep through the log file at the end of a build: because it opens in "a+" mode, it greps starting from the end of the file.
---
New patch in this v2 series
This patch is retrieved from Thomas's work.
The first version was discussed
https://patchwork.ozlabs.org/project/buildroot/patch/[email protected]/
This new version was not already submitted by Thomas or I missed it. Compared to the first version, this patch use a more python-like syntax as proposed by Yann in the previous review.
Changes v2 to v3:
None
support/testing/infra/__init__.py | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-)
support/testing/infra/__init__.py +6 -5
@@ -10,17 +10,18 @@ ARTIFACTS_URL = "http://autobuild.buildroot.net/artefacts/" BASE_DIR = os.path.realpath(os.path.join(os.path.dirname(__file__), "../../..")) +def log_file_path(builddir, stage, logtofile=True): + """Return path to log file""" + return "{}-{}.log".format(builddir, stage) if logtofile else None + + def open_log_file(builddir, stage, logtofile=True): """ Open a file for logging and return its handler. If logtofile is True, returns sys.stdout. Otherwise opens a file with a suitable name in the build directory. """ - if logtofile: - fhandle = open("{}-{}.log".format(builddir, stage), 'a+') - else: - fhandle = sys.stdout - return fhandle + return open(log_file_path(builddir, stage, logtofile), 'a+') if logtofile else sys.stdout def basepath(relpath=""):
Signature · 6 lines
2.31.1 _______________________________________________ buildroot mailing list [email protected] http://lists.busybox.net/mailman/listinfo/buildroot
Re: [Buildroot] [PATCH v3 13/16] support/testing/infra: add log_file_path() function
Hervé, All,
On 2021-08-17 10:39 +0200, Herve Codina spake thusly:
Quoted 7 lines
From: Thomas Petazzoni <[email protected]>
Some tests will need to grep through the build log to verify that some features are working are expected. In order to allow them to open the build log, we provide a new function called
log_file_path(), which returns the path to the log file if available.
Quoted 8 lines
We also use this function in
open_log_file().Note that
open_log_file()cannot be used directly to grep through the log file at the end of a build: because it opens in "a+" mode, it greps starting from the end of the file.Signed-off-by Thomas Petazzoni <[email protected]>Signed-off-by Herve Codina <[email protected]>
This was not really part of the PPD/TLPB stuff, and did not seem to depend on it in any way, so: applied to next, thanks.
Regards,
Yann E. MORIN.
Quoted 52 lines
---
New patch in this v2 seriesThis patch is retrieved from Thomas's work.
The first version was discussed
https://patchwork.ozlabs.org/project/buildroot/patch/[email protected]/This new version was not already submitted by Thomas or I missed it. Compared to the first version, this patch use a more python-like syntax as proposed by Yann in the previous review.
Changes v2 to v3:
Nonesupport/testing/infra/__init__.py | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-)support/testing/infra/__init__.py +6
-5@@ -10,17 +10,18 @@ ARTIFACTS_URL = "http://autobuild.buildroot.net/artefacts/" BASE_DIR = os.path.realpath(os.path.join(os.path.dirname(__file__), "../../..")) +def log_file_path(builddir, stage, logtofile=True): + """Return path to log file""" + return "{}-{}.log".format(builddir, stage) if logtofile else None + + def open_log_file(builddir, stage, logtofile=True): """ Open a file for logging and return its handler. If logtofile is True, returns sys.stdout. Otherwise opens a file with a suitable name in the build directory. """ - if logtofile: - fhandle = open("{}-{}.log".format(builddir, stage), 'a+') - else: - fhandle = sys.stdout - return fhandle + return open(log_file_path(builddir, stage, logtofile), 'a+') if logtofile else sys.stdout def basepath(relpath=""):Signature · 6 lines
2.31.1 _______________________________________________ buildroot mailing list [email protected] http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 10 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' _______________________________________________ buildroot mailing list [email protected] http://lists.busybox.net/mailman/listinfo/buildroot
PATCH v3 14/16 support/testing/tests: add test for check_bin_arch
In reply to Herve CodinaFrom: Thomas Petazzoni <[email protected]>
This tests build a bogus package that installs a binary built for the host architecture into $(TARGET_DIR), which should cause a build failure, at least as long as the host architecture isn't ARM.
---
New patch in this v2 series
This patch is retrieved from Thomas's work.
The first version was discussed
https://patchwork.ozlabs.org/project/buildroot/patch/[email protected]/
Compared to the first version, as proposed by Yann in the previous review, this patch does not launch a subprocess (grep) to search the string.
Note also that the .gitlab-ci.yml modification present in the previous version is not present in this patch.
Changes v2 to v3:
- Fixed the test (https://lore.kernel.org/buildroot/[email protected]/)
.../br2-external/detect-bad-arch/Config.in | 1 + .../detect-bad-arch/external.desc | 1 + .../br2-external/detect-bad-arch/external.mk | 1 + .../package/detect-bad-arch/Config.in | 4 ++++ .../detect-bad-arch/detect-bad-arch.mk | 15 +++++++++++++++ support/testing/tests/core/test_bad_arch.py | 19 +++++++++++++++++++ 6 files changed, 41 insertions(+) create mode 100644 support/testing/tests/core/br2-external/detect-bad-arch/Config.in create mode 100644 support/testing/tests/core/br2-external/detect-bad-arch/external.desc create mode 100644 support/testing/tests/core/br2-external/detect-bad-arch/external.mk create mode 100644 support/testing/tests/core/br2-external/detect-bad-arch/package/detect-bad-arch/Config.in create mode 100644 support/testing/tests/core/br2-external/detect-bad-arch/package/detect-bad-arch/detect-bad-arch.mk create mode 100644 support/testing/tests/core/test_bad_arch.py
support/testing/tests/core/br2-external/detect-bad-arch/Config.in new file +1 -0
@@ -0,0 +1 @@ +source "$BR2_EXTERNAL_DETECT_BAD_ARCH_PATH/package/detect-bad-arch/Config.in"
support/testing/tests/core/br2-external/detect-bad-arch/external.desc new file +1 -0
@@ -0,0 +1 @@ +name: DETECT_BAD_ARCH
support/testing/tests/core/br2-external/detect-bad-arch/external.mk new file +1 -0
@@ -0,0 +1 @@ +include $(sort $(wildcard $(BR2_EXTERNAL_DETECT_BAD_ARCH_PATH)/package/*/*.mk))
support/testing/tests/core/br2-external/detect-bad-arch/package/detect-bad-arch/Config.in new file +4 -0
@@ -0,0 +1,4 @@ +config BR2_PACKAGE_DETECT_BAD_ARCH + bool + default y +
support/testing/tests/core/br2-external/detect-bad-arch/package/detect-bad-arch/detect-bad-arch.mk new file +15 -0
@@ -0,0 +1,15 @@ +################################################################################ +# +# detect-bad-arch +# +################################################################################ + +define DETECT_BAD_ARCH_BUILD_CMDS + echo "int main(void) { return 0; }" | $(HOSTCC) -x c -o $(@D)/foo - +endef + +define DETECT_BAD_ARCH_INSTALL_TARGET_CMDS + $(INSTALL) -D -m 0755 $(@D)/foo $(TARGET_DIR)/usr/bin/foo +endef + +$(eval $(generic-package))
support/testing/tests/core/test_bad_arch.py new file +19 -0
@@ -0,0 +1,19 @@ +import infra +import infra.basetest +import subprocess + + +class DetectBadArchTest(infra.basetest.BRConfigTest): + config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + infra.basetest.MINIMAL_CONFIG + br2_external = [infra.filepath("tests/core/br2-external/detect-bad-arch")] + + def test_run(self): + with self.assertRaises(SystemError): + self.b.build() + logf_path = infra.log_file_path(self.b.builddir, "build", + infra.basetest.BRConfigTest.logtofile) + if logf_path: + s = 'ERROR: architecture for "/usr/bin/foo" is' + with open(logf_path, "r") as f: + lines = [l for l in f.readlines() if l.startswith(s)] + self.assertEqual(len(lines), 1)
Signature · 6 lines
2.31.1 _______________________________________________ buildroot mailing list [email protected] http://lists.busybox.net/mailman/listinfo/buildroot
Re: [Buildroot] [PATCH v3 14/16] support/testing/tests: add test for check_bin_arch
On 2021-08-17 10:39 +0200, Herve Codina spake thusly:
Quoted 8 lines
From: Thomas Petazzoni <[email protected]>
This tests build a bogus package that installs a binary built for the host architecture into $(TARGET_DIR), which should cause a build failure, at least as long as the host architecture isn't ARM.
Signed-off-by Thomas Petazzoni <[email protected]>Signed-off-by Herve Codina <[email protected]>
This was not really part of the PPD/TLPB stuff, and did not seem to depend on it in any way, so: applied to next, thanks.
Seem a small nit below, though...
Quoted 1 line
---
[--SNIP--]
support/testing/tests/core/test_bad_arch.py line 3
Quoted 9 lines
support/testing/tests/core/test_bad_arch.py new file +3
-0@@ -0,0 +1,19 @@ +import infra +import infra.basetest +import subprocess
$ make check-flake8
support/testing/tests/core/test_bad_arch.py:3:1: F401 'subprocess' imported but unused
1 F401 'subprocess' imported but unusedI fixed that when applying.
Regards,
Yann E. MORIN.
Signature · 10 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' _______________________________________________ buildroot mailing list [email protected] http://lists.busybox.net/mailman/listinfo/buildroot
PATCH v3 15/16 support/testing/tests: add test for file overwrite detection
In reply to Herve CodinaFrom: Thomas Petazzoni <[email protected]>
---
New patch in this v2 series
This patch is retrieved from Thomas's work.
The first version was discussed
https://patchwork.ozlabs.org/project/buildroot/patch/[email protected]/
Compared to the first version, as proposed by Yann in the previous review, this patch does not launch a subprocess (grep) to find the string.
Additionally, the strings searched were changed from a specific file ('path/file: FAILED') to the "ERROR: package ..." overwrite detection message.
Note also that the .gitlab-ci.yml modification present in the previous version is not present in this patch.
Changes v2 to v3:
None
.../br2-external/detect-overwrite/Config.in | 1 + .../detect-overwrite/external.desc | 1 + .../br2-external/detect-overwrite/external.mk | 1 + .../package/detect-overwrite/Config.in | 5 ++ .../detect-overwrite/detect-overwrite.mk | 19 ++++++++ .../testing/tests/core/test_file_overwrite.py | 48 +++++++++++++++++++ 6 files changed, 75 insertions(+) create mode 100644 support/testing/tests/core/br2-external/detect-overwrite/Config.in create mode 100644 support/testing/tests/core/br2-external/detect-overwrite/external.desc create mode 100644 support/testing/tests/core/br2-external/detect-overwrite/external.mk create mode 100644 support/testing/tests/core/br2-external/detect-overwrite/package/detect-overwrite/Config.in create mode 100644 support/testing/tests/core/br2-external/detect-overwrite/package/detect-overwrite/detect-overwrite.mk create mode 100644 support/testing/tests/core/test_file_overwrite.py
support/testing/tests/core/br2-external/detect-overwrite/Config.in new file +1 -0
@@ -0,0 +1 @@ +source "$BR2_EXTERNAL_DETECT_OVERWRITE_PATH/package/detect-overwrite/Config.in"
support/testing/tests/core/br2-external/detect-overwrite/external.desc new file +1 -0
@@ -0,0 +1 @@ +name: DETECT_OVERWRITE
support/testing/tests/core/br2-external/detect-overwrite/external.mk new file +1 -0
@@ -0,0 +1 @@ +include $(sort $(wildcard $(BR2_EXTERNAL_DETECT_OVERWRITE_PATH)/package/*/*.mk))
support/testing/tests/core/br2-external/detect-overwrite/package/detect-overwrite/Config.in new file +5 -0
@@ -0,0 +1,5 @@ +config BR2_PACKAGE_DETECT_OVERWRITE + bool "detect-overwrite" + +config BR2_PACKAGE_HOST_DETECT_OVERWRITE + bool "host-detect-overwrite"
support/testing/tests/core/br2-external/detect-overwrite/package/detect-overwrite/detect-overwrite.mk new file +19 -0
@@ -0,0 +1,19 @@ +################################################################################ +# +# detect-overwrite +# +################################################################################ + +define DETECT_OVERWRITE_INSTALL_TARGET_CMDS + grep -q "^foo" $(TARGET_DIR)/etc/passwd || \ + echo "foo" >> $(TARGET_DIR)/etc/passwd +endef + +HOST_DETECT_OVERWRITE_DEPENDENCIES = host-pkgconf + +define HOST_DETECT_OVERWRITE_INSTALL_CMDS + $(SED) 's/manipulating/tweaking/' $(HOST_DIR)/lib/pkgconfig/libpkgconf.pc +endef + +$(eval $(generic-package)) +$(eval $(host-generic-package))
support/testing/tests/core/test_file_overwrite.py new file +48 -0
48 lines not on this page · raw message
Signature · 6 lines
2.31.1 _______________________________________________ buildroot mailing list [email protected] http://lists.busybox.net/mailman/listinfo/buildroot
PATCH v3 16/16 package/pkg-generic.mk: move fixup-libtool-files to post-prepare hook
In reply to Herve CodinaThis commit moves fixup-libtool-files to the post-prepare hook infrastructure.
---
New patch in this v3 series.
This modification was mentionned in the previous review
https://lore.kernel.org/buildroot/20210625092704.GJ2852@scaer/
package/pkg-generic.mk | 20 ++++++++++++-------- 1 file changed, 12 insertions(+), 8 deletions(-)
package/pkg-generic.mk +12 -8
@@ -92,14 +92,19 @@ endif # Make sure .la files only reference the current per-package # directory. - -# $1: package name (lower case) -# $2: staging directory of the package +# +# Can't use $(foreach d, $(HOST_DIR)/lib* $(STAGING_DIR)/usr/lib*, ...) +# because those directories may be created in the same recipe this macro will +# be expanded in. +# Additionally, either or both may be missing, which would make find whine and +# fail. +# So we just use HOST_DIR as a starting point, and filter on the two directories +# of interest. ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y) -define fixup-libtool-files - $(Q)find $(2) \( -path '$(2)/lib*' -o -path '$(2)/usr/lib*' \) \ +define FIXUP_LIBTOOL_FILES + $(Q)find $(HOST_DIR) \( -path '$(HOST_DIR)/lib*' -o -path '$(STAGING_DIR)/usr/lib*' \) \ -name "*.la" -print0 | xargs -0 --no-run-if-empty \ - $(SED) "s:$(PER_PACKAGE_DIR)/[^/]\+/:$(PER_PACKAGE_DIR)/$(1)/:g" + $(SED) "s:$(PER_PACKAGE_DIR)/[^/]\+/:$(PER_PACKAGE_DIR)/$($(PKG)_NAME)/:g" endef endif
@@ -226,8 +231,6 @@ define prepare-pre-configure @$(call pkg_size_before,$(TARGET_DIR)) @$(call pkg_size_before,$(STAGING_DIR),-staging) @$(call pkg_size_before,$(HOST_DIR),-host) - $(call fixup-libtool-files,$(NAME),$(HOST_DIR)) - $(call fixup-libtool-files,$(NAME),$(STAGING_DIR)) $(foreach hook,$($(PKG)_POST_PREPARE_HOOKS),$(call $(hook))$(sep)) @$(call pkg_detect_overwrite_before,$(TARGET_DIR)) @$(call pkg_detect_overwrite_before,$(HOST_DIR),-host)
@@ -923,6 +926,7 @@ $(2)_POST_LEGAL_INFO_HOOKS ?= $(2)_TARGET_FINALIZE_HOOKS ?= $(2)_ROOTFS_PRE_CMD_HOOKS ?= +$(2)_POST_PREPARE_HOOKS += FIXUP_LIBTOOL_FILES $(2)_POST_PREPARE_HOOKS += FIXUP_PYTHON_SYSCONFIGDATA ifeq ($$($(2)_TYPE),target)
Signature · 6 lines
2.31.1 _______________________________________________ buildroot mailing list [email protected] http://lists.busybox.net/mailman/listinfo/buildroot
Yann E. MORIN
AppliedHervé, All,
On 2021-08-17 10:39 +0200, Herve Codina spake thusly:
Quoted 24 lines
This v3 series adds a new patch (patch 1) to fix a find failure already present in
PKG_PYTHON_FIXUP_SYSCONFIGthat was highlighted in the previous version of this series.
It also reworks the python fixup according to previous review (patches 2 and 3) and fixes the test given in patch 12.Two patches are added related to Qt5. The first one (patch 5) was previously sent by Andreas Naumann and fixes a Qt5 overwrite.
https://lore.kernel.org/buildroot/[email protected]/
https://lore.kernel.org/buildroot/[email protected]/
The second one (patch 6) moves the fixup function to the post-prepare hooks.One more patch was added in this serie (patch 14) to move fixup-libtool-file to the post-prepare hooks. This was mentionned during the review of the first version of this series (https://lore.kernel.org/buildroot/20210625092704.GJ2852@scaer/).
More changes details are given in each individual patches.
Best regards,
Hervé CodinaAndreas Naumann (1):
qt5: Fix sporadic build failure during top-level parallel build
Quoted 6 lines
Herve Codina (11):
package/pkg-python: fixPKG_PYTHON_FIXUP_SYSCONFIGDATApackage/pkg-python: invalidate precompiled _sysconfigdata*.pycpackage/pkg-generic.mk: move python fixup to generic package
infrastructure
package/owfs: remove Python sysconfigdata fixup
I've applied those four python-related fixes to next, thanks!
I'll resume the rest of the series later in the WE; no need to respin!
Regards,
Yann E. MORIN.
Quoted 54 lines
package/pkg-qmake.mk: MoveQT5_QT_CONF_FIXUPto post-prepare hookpackage/pkg-generic.mk: generate final rsync exclude file list
Makefile: rsync global {TARGET,HOST}_DIR using exclusion file list
Makefile: breaks hardlinks in global {TARGET,HOST}_DIR on per-package
build
package/pkg-generic.mk: fix per-package<pkg>-{reconfigure,rebuild,reinstall}
package/pkg-generic.mk: remove .files-final-rsync.before temporary
file
package/pkg-generic.mk: move fixup-libtool-files to post-prepare hookThomas Petazzoni (4):
package/pkg-generic.mk: detect files overwritten inTARGET_DIRand
HOST_DIR
support/testing/infra: addlog_file_path()function support/testing/tests: add test forcheck_bin_archsupport/testing/tests: add test for file overwrite detectionMakefile | 34 ++++- package/owfs/owfs.mk | 9 -- package/pkg-generic.mk | 125 ++++++++++++++++-- package/pkg-python.mk | 10 -- package/pkg-qmake.mk | 3 +- package/qt5/qt5.mk | 1 + support/testing/infra/__init__.py | 11 +- .../br2-external/detect-bad-arch/Config.in | 1 + .../detect-bad-arch/external.desc | 1 + .../br2-external/detect-bad-arch/external.mk | 1 + .../package/detect-bad-arch/Config.in | 4 + .../detect-bad-arch/detect-bad-arch.mk | 15 +++ .../br2-external/detect-overwrite/Config.in | 1 + .../detect-overwrite/external.desc | 1 + .../br2-external/detect-overwrite/external.mk | 1 + .../package/detect-overwrite/Config.in | 5 + .../detect-overwrite/detect-overwrite.mk | 19 +++ support/testing/tests/core/test_bad_arch.py | 19 +++ .../testing/tests/core/test_file_overwrite.py | 48 +++++++ 19 files changed, 269 insertions(+), 40 deletions(-) create mode 100644 support/testing/tests/core/br2-external/detect-bad-arch/Config.in create mode 100644 support/testing/tests/core/br2-external/detect-bad-arch/external.desc create mode 100644 support/testing/tests/core/br2-external/detect-bad-arch/external.mk create mode 100644 support/testing/tests/core/br2-external/detect-bad-arch/package/detect-bad-arch/Config.in create mode 100644 support/testing/tests/core/br2-external/detect-bad-arch/package/detect-bad-arch/detect-bad-arch.mk create mode 100644 support/testing/tests/core/br2-external/detect-overwrite/Config.in create mode 100644 support/testing/tests/core/br2-external/detect-overwrite/external.desc create mode 100644 support/testing/tests/core/br2-external/detect-overwrite/external.mk create mode 100644 support/testing/tests/core/br2-external/detect-overwrite/package/detect-overwrite/Config.in create mode 100644 support/testing/tests/core/br2-external/detect-overwrite/package/detect-overwrite/detect-overwrite.mk create mode 100644 support/testing/tests/core/test_bad_arch.py create mode 100644 support/testing/tests/core/test_file_overwrite.pySignature · 1 line
2.31.1
Signature · 10 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' _______________________________________________ buildroot mailing list [email protected] http://lists.busybox.net/mailman/listinfo/buildroot