Overwritten file detection and fixes, one more step to TLP build
cover Overwritten file detection and fixes, one more step to TLP build
Hi,
This series fixes some issues in the TLP build feature.
The starting point was the work done by Thomas Petazzoni on overwrites detection (PATCH 1).
Based on this work, this series fixes some overwrites, introduces and uses <PKG>_PER_PACKAGE_TWEAK_HOOKS. This variable is used to collect specific per-package path tweaks needed by per-package build.
With this tweaks collected in this variable, they can be re-applied when they are needed (<pkg>_{reconfigure,rebuild,reinstall}).
It also reworks the final {HOST,TARGET}_DIR rsync.
For each packages only files generated by the package are rsynced (PATCH 13). This is done using an exclusion list computed in (PATCH 12).
In order to avoid modifications performed in target-finalize step to be seen as overwrites on next build, the final {HOST,TARGET}_DIR do not contain any hardlink anymore and are a full copy of expected files and directories (PATCH 14).
The last patch (PATCH 15) fixes <pkg>_{reconfigure,rebuild,reinstall} recreating per-package {HOST,TARGET}_DIR from scratch.
Indeed, for a given package, re-generating the same file is detected as an overwrite. The simplest way to avoid this is to fully discard previous per-package {HOST,TARGET}_DIR directory and recreate them.
To have a TLP build fully fonctionnal, several things are still missing:
- Be sure that all buildroot packages do not perform any overwrites.
- Undo tweak stuff that have be done for per-package build (fixup-libtool-files, fixup-python-files, <PKG>_PER_PACKAGE_TWEAK_HOOKS) after collecting files in final {HOST,STAGING}_DIR.
This mainly means reverting .../per-package/xxxxx/... path to point to the correct ones in final {HOST,STAGING}_DIR.
I hope this series will help in TLP build feature.
Best regards,
Herv? Codina
Herve Codina (14):
package/e2fsprogs: fix fsck overwrite in HOST_DIR package/pkg-generic.mk: Remove Info documents dir entry package/pkg-generic.mk: Fix .la files overwrite detection package/pkg-generic.mk: Perform .la files fixup in per-package
HOST_DIR
package/pkg-generic.mk: Introduce <PKG>_PER_PACKAGE_TWEAK_HOOKS package/apr-util: Use <PKG>_PER_PACKAGE_TWEAK_HOOKS package/apache: Move APACHE_FIXUP_APR_LIBTOOL to
<PKG>_PER_PACKAGE_TWEAK_HOOKS
package/pkg-python: Remove _sysconfigdata*.pyc files when
_sysconfigdata*.py are changed
package/pkg-generic.mk: Move python fixup to generic package
infrastructure
package/owfs: Remove Python sysconfigdata fixup package/pkg-generic.mk: Generate final rsync exclude file list
Makefile: Rsync global {TARGET,HOST}_DIR using exclusion file list
Makefile: Breaks hardlinks in global {TARGET,HOST}_DIR on per-package
build
package/pkg-generic.mk: Fix per-package
<pkg>-{reconfigure,rebuild,reinstall}Thomas Petazzoni (1):
package/pkg-generic.mk: detect files overwritten in TARGET_DIR and
HOST_DIR
Makefile | 24 +++++++- package/apache/apache.mk | 2 +- package/apr-util/apr-util.mk | 12 ++-- package/e2fsprogs/e2fsprogs.mk | 1 + package/owfs/owfs.mk | 9 --- package/pkg-generic.mk | 103 ++++++++++++++++++++++++++++++++- package/pkg-python.mk | 10 ---- 7 files changed, 134 insertions(+), 27 deletions(-)
Signature · 1 line
2.31.1
PATCH 1/15 package/pkg-generic.mk: detect files overwritten in TARGET_DIR and HOST_DIR
From: Thomas Petazzoni <[email protected]>
Without per-package directory support, a package can happily overwrite files installed by other packages. Indeed, because the build order between packages is always guaranteed, Buildroot will always produce the same output.
However, with per-package directory support, it is absolutely critical that a given package does not overwrite files already installed by another package, due to how the final aggregation is done to create the complete target/, staging/ and host/ folders. Unfortunately, we currently don't have anything in Buildroot that detects this situation.
We used to have check-uniq-files, but it was dropped in commit 2496189a4207173e4cd5bbab90256f911175ee57.
This commit is a new implementation of such a detection, which is based on calculating and verifying MD5 hashes of installed files: the calculation is done at the beginning of the configure step, the verification during the newly introduced "install" step that takes place after all installation steps.
Since preventing file overwrites is really only needed when per-package directory support is used, and due to this verification having some overhead, it is only enabled when BR2_PER_PACKAGE_DIRECTORIES=y. This additional verification cost is however not too bad as on average, with per-package directory support, the per-package target/ and host/ directories will contain less files than with a build that doesn't use per-package directory support. This helps a bit in mitigating the additional cost of this verification.
Note that we are not handling separately HOST_DIR and STAGING_DIR, like we're doing with the pkg_size_{before,after} functions. Instead, the verification on HOST_DIR walks down into the STAGING_DIR.
This commit is retreived from Thomas's work.
The first version was discussed
https://patchwork.ozlabs.org/project/buildroot/patch/20200430095249.782597-9-thomas.petazzoni at bootlin.com/ This new version was not already submitted by Thomas or I missed it.
--- package/pkg-generic.mk | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+)
package/pkg-generic.mk +23 -0
@@ -102,6 +102,25 @@ define fixup-libtool-files endef endif +# Functions to detect overwritten files + +ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y) +# $(1): base directory to search in +# $(2): suffix of file (optional) +define pkg_detect_overwrite_before + LC_ALL=C find $(1) -type f -print0 | xargs -0 -r md5sum > $($(PKG)_DIR)/.files$(2).md5; +endef + +# $(1): base directory to search in +# $(2): suffix of file (optional) +define pkg_detect_overwrite_after + if test -s $($(PKG)_DIR)/.files$(2).md5 ; then \ + LC_ALL=C md5sum --quiet -c $($(PKG)_DIR)/.files$(2).md5 || \ + { echo "ERROR: package $($(PKG)_NAME) has overwritten files installed by a previous package, aborting."; exit 1; } ; \ + fi +endef +endif + # Functions to collect statistics about installed files # $(1): base directory to search in
@@ -235,6 +254,8 @@ $(BUILD_DIR)/%/.stamp_configured: @$(call pkg_size_before,$(TARGET_DIR)) @$(call pkg_size_before,$(STAGING_DIR),-staging) @$(call pkg_size_before,$(HOST_DIR),-host) + @$(call pkg_detect_overwrite_before,$(TARGET_DIR)) + @$(call pkg_detect_overwrite_before,$(HOST_DIR),-host) $(call fixup-libtool-files,$(NAME),$(STAGING_DIR)) $(foreach hook,$($(PKG)_PRE_CONFIGURE_HOOKS),$(call $(hook))$(sep)) $($(PKG)_CONFIGURE_CMDS)
@@ -360,6 +381,8 @@ $(BUILD_DIR)/%/.stamp_installed: @$(call pkg_size_after,$(STAGING_DIR),-staging) @$(call pkg_size_after,$(HOST_DIR),-host) @$(call check_bin_arch) + @$(call pkg_detect_overwrite_after,$(TARGET_DIR)) + @$(call pkg_detect_overwrite_after,$(HOST_DIR),-host) $(Q)touch $@ # Remove package sources
Signature · 1 line
2.31.1
PATCH 1/15 package/pkg-generic.mk: detect files overwritten in TARGET_DIR and HOST_DIR
Herv?, All,
On 2021-06-21 16:11 +0200, Herve Codina spake thusly:
Quoted 32 lines
From: Thomas Petazzoni <[email protected]>
Without per-package directory support, a package can happily overwrite files installed by other packages. Indeed, because the build order between packages is always guaranteed, Buildroot will always produce the same output.
However, with per-package directory support, it is absolutely critical that a given package does not overwrite files already installed by another package, due to how the final aggregation is done to create the complete target/, staging/ and host/ folders. Unfortunately, we currently don't have anything in Buildroot that detects this situation.
We used to have check-uniq-files, but it was dropped in commit
2496189a4207173e4cd5bbab90256f911175ee57.This commit is a new implementation of such a detection, which is based on calculating and verifying MD5 hashes of installed files: the calculation is done at the beginning of the configure step, the verification during the newly introduced "install" step that takes place after all installation steps.
Since preventing file overwrites is really only needed when per-package directory support is used, and due to this verification having some overhead, it is only enabled when BR2_PER_PACKAGE_DIRECTORIES=y. This additional verification cost is however not too bad as on average, with per-package directory support, the per-package target/ and host/ directories will contain less files than with a build that doesn't use per-package directory support. This helps a bit in mitigating the additional cost of this verification.
Quoted 5 lines
Note that we are not handling separately
HOST_DIRandSTAGING_DIR, like we're doing with the pkg_size_{before,after} functions. Instead, the verification onHOST_DIRwalks down into theSTAGING_DIR.Signed-off-by Thomas Petazzoni <[email protected]>
From here...
Quoted 4 lines
This commit is retreived from Thomas's work.
The first version was discussed
https://patchwork.ozlabs.org/project/buildroot/patch/20200430095249.782597-9-thomas.petazzoni at bootlin.com/ This new version was not already submitted by Thomas or I missed it.
... to here: this should have been a post-commit note, after the --- line...
Quoted 2 lines
Signed-off-by Herve Codina <[email protected]>---
... here.
Additionally, it would have been nice to summarise the changes made between the original submission and htis new one, and how the previous review was handled.
Quoted 19 lines
package/pkg-generic.mk | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+)package/pkg-generic.mk +8
-0@@ -102,6 +102,25 @@ define fixup-libtool-files endef endif +# Functions to detect overwritten files + +ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y) +# $(1): base directory to search in +# $(2): suffix of file (optional) +define pkg_detect_overwrite_before + LC_ALL=C find $(1) -type f -print0 | xargs -0 -r md5sum > $($(PKG)_DIR)/.files$(2).md5; +endef
package/pkg-generic.mk define fixup-libtool-files · line 122
Quoted 10 lines
package/pkg-generic.mk +10
-0+ +# $(1): base directory to search in +# $(2): suffix of file (optional) +define pkg_detect_overwrite_after + if test -s $($(PKG)_DIR)/.files$(2).md5 ; then \ + LC_ALL=C md5sum --quiet -c $($(PKG)_DIR)/.files$(2).md5 || \ + { echo "ERROR: package $($(PKG)_NAME) has overwritten files installed by a previous package, aborting."; exit 1; } ; \ + fi +endef +endif
And now I see that only part of the problem is handled; it only tries and detect files which content changes; it does not account for files which type did change. I.e. a symlink that was onverted over to a file, or the other way around.
So I think we could change the pkg_detect_overwrite_before macro thusly;
LC_ALL=C find -L $(1) -type f -print0 |...
That should cover both cases, with just the esception that a symlink is replaced with a file of the same content (or the other way around). There's just a little quirk, though:
find: File system loop detected; ?host/usr? is part of the same file system loop as ?host?.
Meh, that's starting to be a bit hairy... We can just ignore it explicitly, maybe? For merged-usr in target, we should not have that problem, but there is still the option for a custom skeleton, where people could provide it as well...
So, to summarise: this patch does not cover all cases, but it is still acceptable, and brings in the necessary infra that we can later extend should the need arises.
Regards,
Yann E. MORIN.
Quoted 28 lines
package/pkg-generic.mk +4
-0# Functions to collect statistics about installed files # $(1): base directory to search in@@ -235,6 +254,8 @@ $(BUILD_DIR)/%/.stamp_configured: @$(call pkg_size_before,$(TARGET_DIR)) @$(call pkg_size_before,$(STAGING_DIR),-staging) @$(call pkg_size_before,$(HOST_DIR),-host) + @$(call pkg_detect_overwrite_before,$(TARGET_DIR)) + @$(call pkg_detect_overwrite_before,$(HOST_DIR),-host) $(call fixup-libtool-files,$(NAME),$(STAGING_DIR)) $(foreach hook,$($(PKG)_PRE_CONFIGURE_HOOKS),$(call $(hook))$(sep)) $($(PKG)_CONFIGURE_CMDS)@@ -360,6 +381,8 @@ $(BUILD_DIR)/%/.stamp_installed: @$(call pkg_size_after,$(STAGING_DIR),-staging) @$(call pkg_size_after,$(HOST_DIR),-host) @$(call check_bin_arch) + @$(call pkg_detect_overwrite_after,$(TARGET_DIR)) + @$(call pkg_detect_overwrite_after,$(HOST_DIR),-host) $(Q)touch $@ # Remove package sourcesSignature · 6 lines
2.31.1 _______________________________________________ buildroot mailing list buildroot at busybox.net http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 1/15 package/pkg-generic.mk: detect files overwritten in TARGET_DIR and HOST_DIR
Hi,
On Mon, 21 Jun 2021 23:31:40 +0200
"Yann E. MORIN" <[email protected]> wrote:
[...]
Quoted 12 lines
Quoted 1 line
Signed-off-by Thomas Petazzoni <[email protected]>From here...
Quoted 4 lines
This commit is retreived from Thomas's work.
The first version was discussed
https://patchwork.ozlabs.org/project/buildroot/patch/20200430095249.782597-9-thomas.petazzoni at bootlin.com/ This new version was not already submitted by Thomas or I missed it.... to here: this should have been a post-commit note, after the --- line...
Quoted 8 lines
Quoted 2 lines
Signed-off-by Herve Codina <[email protected]>---
... here.
Additionally, it would have been nice to summarise the changes made between the original submission and htis new one, and how the previous review was handled.
I moved the note after a '---' line and added:
Compared to the first version, this patch has an improved commit message and
generates the md5sum snapshot using
'find -L $(1) -type f -print0 | xargs -0 -r md5sum > $($(PKG)_DIR)/.files$(2).md5;'
instead of
'cd $(1); LC_ALL=C find . -type f -exec md5sum {} \; > $($(PKG)_DIR)/.files$(2).md5'Quoted 36 lines
Quoted 20 lines
package/pkg-generic.mk | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+)package/pkg-generic.mk +8
-0@@ -102,6 +102,25 @@ define fixup-libtool-files endef endif +# Functions to detect overwritten files + +ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y) +# $(1): base directory to search in +# $(2): suffix of file (optional) +define pkg_detect_overwrite_before + LC_ALL=C find $(1) -type f -print0 | xargs -0 -r md5sum > $($(PKG)_DIR)/.files$(2).md5; +endefpackage/pkg-generic.mk define fixup-libtool-files · line 122
Quoted 10 lines
package/pkg-generic.mk +10
-0+ +# $(1): base directory to search in +# $(2): suffix of file (optional) +define pkg_detect_overwrite_after + if test -s $($(PKG)_DIR)/.files$(2).md5 ; then \ + LC_ALL=C md5sum --quiet -c $($(PKG)_DIR)/.files$(2).md5 || \ + { echo "ERROR: package $($(PKG)_NAME) has overwritten files installed by a previous package, aborting."; exit 1; } ; \ + fi +endef +endifAnd now I see that only part of the problem is handled; it only tries and detect files which content changes; it does not account for files which type did change. I.e. a symlink that was onverted over to a file, or the other way around.
Quoted 3 lines
So I think we could change the
pkg_detect_overwrite_beforemacro thusly;LC_ALL=C find -L $(1) -type f -print0 |...
I added '-L' option.
Quoted 8 lines
That should cover both cases, with just the esception that a symlink is replaced with a file of the same content (or the other way around). There's just a little quirk, though:
find: File system loop detected; ?host/usr? is part of the same file system loop as ?host?.
Quoted 9 lines
Meh, that's starting to be a bit hairy... We can just ignore it explicitly, maybe? For merged-usr in target, we should not have that problem, but there is still the option for a custom skeleton, where people could provide it as well...
So, to summarise: this patch does not cover all cases, but it is still acceptable, and brings in the necessary infra that we can later extend should the need arises.
Ok, v2 with the changes I mentioned will be sent.
Thanks for the review,
Herv? Codina
Signature · 3 lines
Herv? Codina, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 1/15 package/pkg-generic.mk: detect files overwritten in TARGET_DIR and HOST_DIR
Quoted 3 lines · On Tue, 22 Jun 2021 09:40:52 +0200 Herve Codina <[email protected]>…
Compared to the first version, this patch has an improved commit message and generates the md5sum snapshot using
'find -L $(1) -type f -print0 | xargs -0 -r md5sum > $($(PKG)_DIR)/.files$(2).md5;'
But is this better ? Due to not cd-ing into the directory, you will have full absolute paths in the .md5 file, and adding LC_ALL=C is quite customary too. Ditto for using | xargs instead of -exec.
Quoted 2 lines
instead of
'cd $(1); LC_ALL=C find . -type f -exec md5sum {} \; > $($(PKG)_DIR)/.files$(2).md5'All in all this version, looked better. Any reason for the change ?
Thomas
Signature · 3 lines
Thomas Petazzoni, co-owner and CEO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 1/15 package/pkg-generic.mk: detect files overwritten in TARGET_DIR and HOST_DIR
Quoted 10 lines · Thomas Petazzoni
Quoted 3 lines · On Tue, 22 Jun 2021 09:40:52 +0200 Herve Codina <[email protected]>…
Compared to the first version, this patch has an improved commit message and generates the md5sum snapshot using
'find -L $(1) -type f -print0 | xargs -0 -r md5sum > $($(PKG)_DIR)/.files$(2).md5;'But is this better ? Due to not cd-ing into the directory, you will have full absolute paths in the .md5 file, and adding LC_ALL=C is quite customary too. Ditto for using | xargs instead of -exec.
xargs will run only start one md5sum process if the command line size permits it, whereas "-exec md5sum {} ;" will start one md5sum per file.
An alternative is "-exec md5sum {} +", which is essentially xargs-like.
PATCH 1/15 package/pkg-generic.mk: detect files overwritten in TARGET_DIR and HOST_DIR
Nicolas, All,
On 2021-06-22 11:57 +0200, Nicolas Cavallari spake thusly:
Quoted 10 lines · Thomas Petazzoni
Quoted 9 lines
Quoted 3 lines · On Tue, 22 Jun 2021 09:40:52 +0200 Herve Codina <[email protected]>…
Compared to the first version, this patch has an improved commit message and generates the md5sum snapshot using
'find -L $(1) -type f -print0 | xargs -0 -r md5sum > $($(PKG)_DIR)/.files$(2).md5;'But is this better ? Due to not cd-ing into the directory, you will have full absolute paths in the .md5 file,
But is that even an issue? Those paths are only used for one per-package directory; they do not follow files around into another package's ppd.
So, relative or absolute, we don't much care.
Quoted 2 lines
Quoted 2 lines
and adding LC_ALL=C is quite
customary too.
In this case, do we also even care or need it? I pretty sure the output of find is not sorted at all; it most probably depend on the order of dentries in the directory, and those have absolutely no ordering guarantee...
Note that if you really want to use LC_ALL, passing it in front of 'find' will not make LC_ALL available to anythin after the pipeline, so you'd still have xargs (and md5sum) run without it...
Quoted 3 lines
Quoted 1 line
Ditto for using | xargs instead of -exec.
xargs will run only start one md5sum process if the command line size permits it, whereas "-exec md5sum {} ;" will start one md5sum per file.
Agreed. But I think this is exactly what Thomas suggested to use, right?
Quoted 1 line
An alternative is "-exec md5sum {} +", which is essentially xargs-like.
I once used that in the past, and it breaks on non-GNU find, because '+' is a GNUism. So I think using xargs is still better.
Regards,
Yann E. MORIN.
Quoted 4 lines
_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 1/15 package/pkg-generic.mk: detect files overwritten in TARGET_DIR and HOST_DIR
Hi,
Quoted 35 lines · On Tue, 22 Jun 2021 12:24:33 +0200 "Yann E. MORIN" <[email protected]>…
Nicolas, All,
On 2021-06-22 11:57 +0200, Nicolas Cavallari spake thusly:
Quoted 10 lines · Thomas Petazzoni
Quoted 9 lines
Quoted 3 lines · On Tue, 22 Jun 2021 09:40:52 +0200 Herve Codina <[email protected]>…
Compared to the first version, this patch has an improved commit message and generates the md5sum snapshot using
'find -L $(1) -type f -print0 | xargs -0 -r md5sum > $($(PKG)_DIR)/.files$(2).md5;'But is this better ? Due to not cd-ing into the directory, you will have full absolute paths in the .md5 file,
But is that even an issue? Those paths are only used for one per-package directory; they do not follow files around into another package's ppd.
So, relative or absolute, we don't much care.
Quoted 2 lines
Quoted 2 lines
and adding LC_ALL=C is quite
customary too.In this case, do we also even care or need it? I pretty sure the output of find is not sorted at all; it most probably depend on the order of dentries in the directory, and those have absolutely no ordering guarantee...
Note that if you really want to use
LC_ALL, passing it in front of 'find' will not makeLC_ALLavailable to anythin after the pipeline, so you'd still have xargs (and md5sum) run without it...Quoted 3 lines
Quoted 1 line
Ditto for using | xargs instead of -exec.
xargs will run only start one md5sum process if the command line size permits it, whereas "-exec md5sum {} ;" will start one md5sum per file.
Quoted 7 lines
Agreed. But I think this is exactly what Thomas suggested to use, right?
Quoted 1 line
An alternative is "-exec md5sum {} +", which is essentially xargs-like.
I once used that in the past, and it breaks on non-GNU find, because '+' is a GNUism. So I think using xargs is still better.
Well, according to the different exchanges,
I think I can remove LC_ALL (we don't care about any order), and I keep '... | xargs -0 -r md5sum'.
'cd' is not needed (we do not care about absolute or relative path).
So, the snapshot is taken using:
find -L $(1) -type f -print0 | xargs -0 -r md5sum > $($(PKG)_DIR)/.files$(2).md5;
and the detection is done using:
md5sum --quiet -c $($(PKG)_DIR)/.files$(2).md5
Is that ok for everyone ?
Regards,
Herv?
Signature · 3 lines
Herv? Codina, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 1/15 package/pkg-generic.mk: detect files overwritten in TARGET_DIR and HOST_DIR
Herv?, All,
On 2021-06-24 16:09 +0200, Herve Codina spake thusly:
Quoted 2 lines
On Tue, 22 Jun 2021 12:24:33 +0200
"Yann E. MORIN" <[email protected]> wrote:
[--SNIP--]
Quoted 8 lines
Well, according to the different exchanges,
I think I can remove
LC_ALL(we don't care about any order), and I keep '... | xargs -0 -r md5sum'.
'cd' is not needed (we do not care about absolute or relative path).So, the snapshot is taken using:
find -L $(1) -type f -print0 | xargs -0 -r md5sum > $($(PKG)_DIR)/.files$(2).md5;
I spoke about -L as a suggestion to follow and monitor symlinks. However, as I said, that does not work, at least for host/, because it will emit a warning about a directory loop. Also, there is no telling what users will install i their target/ and we may have a similar situation for target/.
This warning is not nice as it will happen for each and every packages.
So, drop -L. At the very least, we have the infrastructure in place, and we can extend it later should we need additional details.
But now that I think about it, the entry-changed-to-omething-else case is already covered by the existing instrumenttion, as part of the pkg_size_before/after hooks (find's %y printf directive)
So, yes, just drop the -L.
Quoted 4 lines
and the detection is done using:
md5sum --quiet -c $($(PKG)_DIR)/.files$(2).md5Is that ok for everyone ?
Except for -L, yes, this is a proper summary from my point if view.
Thanks! :-)
Regards,
Yann E. MORIN.
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 2/15 package/e2fsprogs: fix fsck overwrite in HOST_DIR
In reply to Herve Codinahost-e2fsprog overwrites fsck program and some man related files previously installed by host-util-linux.
This patch simply disable fsck in host-e2fsprog.
host-e2fsprog is used to build final ext{2,3,4} images. The missing host-e2fsprog fsck tool (filesystem integrity check tool) in HOST_DIR should not lead to issues.
--- package/e2fsprogs/e2fsprogs.mk | 1 + 1 file changed, 1 insertion(+)
package/e2fsprogs/e2fsprogs.mk +1 -0
@@ -27,6 +27,7 @@ HOST_E2FSPROGS_CONF_OPTS = \ --disable-defrag \ --disable-e2initrd-helper \ --disable-fuse2fs \ + --disable-fsck \ --disable-libblkid \ --disable-libuuid \ --disable-testio-debug \
Signature · 1 line
2.31.1
PATCH 2/15 package/e2fsprogs: fix fsck overwrite in HOST_DIR
Quoted 2 lines · On Mon, 21 Jun 2021 16:11:17 +0200 Herve Codina <[email protected]>…
host-e2fsprog overwrites fsck program and some man related files previously installed by host-util-linux.
"The host-e2fsprogs package overwrites the fsck program and some manpages previously installed by the host-util-linux package".
Quoted 2 lines
This patch simply disable fsck in host-e2fsprog.
disableS
host-e2fsprogs
Quoted 1 line
host-e2fsprog is used to build final ext{2,3,4} images.
host-e2fsprogs
Quoted 1 line
The missing host-e2fsprog fsck tool (filesystem integrity check
host-e2fsprogs
Quoted 3 lines
tool) in
HOST_DIRshould not lead to issues.Signed-off-by Herve Codina <[email protected]>
With that:
Quoted 16 lines
--- package/e2fsprogs/e2fsprogs.mk | 1 + 1 file changed, 1 insertion(+)package/e2fsprogs/e2fsprogs.mk +1
-0@@ -27,6 +27,7 @@ HOST_E2FSPROGS_CONF_OPTS = \ --disable-defrag \ --disable-e2initrd-helper \ --disable-fuse2fs \ + --disable-fsck \ --disable-libblkid \ --disable-libuuid \ --disable-testio-debug \
Signature · 3 lines
Thomas Petazzoni, co-owner and CEO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 2/15 package/e2fsprogs: fix fsck overwrite in HOST_DIR
Hi,
Quoted 9 lines · On Mon, 21 Jun 2021 22:52:54 +0200 Thomas Petazzoni…
Quoted 2 lines · On Mon, 21 Jun 2021 16:11:17 +0200 Herve Codina <[email protected]>…
host-e2fsprog overwrites fsck program and some man related files previously installed by host-util-linux.
"The host-e2fsprogs package overwrites the fsck program and some manpages previously installed by the host-util-linux package".
Ok, changed
Quoted 6 lines
Quoted 2 lines
This patch simply disable fsck in host-e2fsprog.
disableS
host-e2fsprogs
Changed and s/host-e2fsprog/host-e2fsprogs/g
Quoted 9 lines
Quoted 2 lines
host-e2fsprog is used to build final ext{2,3,4} images.
host-e2fsprogs
Quoted 1 line
The missing host-e2fsprog fsck tool (filesystem integrity check
host-e2fsprogs
Quoted 7 lines
Quoted 3 lines
tool) in
HOST_DIRshould not lead to issues.Signed-off-by Herve Codina <[email protected]>With that:
Reviewed-by Thomas Petazzoni <[email protected]>
Added.
I will remove this Reviewed-by in my v2 patch if more modifications are needed for this patch.
Thanks for the review,
Herv? Codina
Signature · 3 lines
Herv? Codina, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 2/15 package/e2fsprogs: fix fsck overwrite in HOST_DIR
In reply to Herve CodinaHerv?, All,
On 2021-06-21 16:11 +0200, Herve Codina spake thusly:
Quoted 8 lines
host-e2fsprog overwrites fsck program and some man related files previously installed by host-util-linux.
This patch simply disable fsck in host-e2fsprog.
host-e2fsprog is used to build final ext{2,3,4} images. The missing host-e2fsprog fsck tool (filesystem integrity check tool) in
HOST_DIRshould not lead to issues.
And the target variant is not susceptible to that issue, because e2fsprogs' fsck depends on !BR2_PACKAGE_UTIL_LINUX_FSCK. So we're clean there.
Except for the comments by Thomas, you can carry my:
when you respin this patch.
Regards,
Yann E. MORIN.
Quoted 24 lines
Signed-off-by Herve Codina <[email protected]>--- package/e2fsprogs/e2fsprogs.mk | 1 + 1 file changed, 1 insertion(+)package/e2fsprogs/e2fsprogs.mk +1
-0@@ -27,6 +27,7 @@ HOST_E2FSPROGS_CONF_OPTS = \ --disable-defrag \ --disable-e2initrd-helper \ --disable-fuse2fs \ + --disable-fsck \ --disable-libblkid \ --disable-libuuid \ --disable-testio-debug \Signature · 6 lines
2.31.1 _______________________________________________ buildroot mailing list buildroot at busybox.net http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 2/15 package/e2fsprogs: fix fsck overwrite in HOST_DIR
Quoted 16 lines · On Tue, 22 Jun 2021 21:40:24 +0200 "Yann E. MORIN" <[email protected]>…
Herv?, All,
On 2021-06-21 16:11 +0200, Herve Codina spake thusly:
Quoted 8 lines
host-e2fsprog overwrites fsck program and some man related files previously installed by host-util-linux.
This patch simply disable fsck in host-e2fsprog.
host-e2fsprog is used to build final ext{2,3,4} images. The missing host-e2fsprog fsck tool (filesystem integrity check tool) in
HOST_DIRshould not lead to issues.And the target variant is not susceptible to that issue, because e2fsprogs' fsck depends on !BR2_PACKAGE_UTIL_LINUX_FSCK. So we're clean there.
Quoted 5 lines
Except for the comments by Thomas, you can carry my:
Reviewed-by Yann E. MORIN <[email protected]>when you respin this patch.
Ok,
Reviewed-by added.
Thanks for the review,
Herv?
Signature · 3 lines
Herv? Codina, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 3/15 package/pkg-generic.mk: Remove Info documents dir entry
In reply to Herve CodinaSome packages (autotools for instance) install documentation files using install-info. This program adds an entry in the Info directory file (share/info/dir) and this causes TARGET_DIR and/or HOST_DIR overwrite.
In order to avoid this overwrite this patch removes the Info directory file right after any installation.
--- package/pkg-generic.mk | 3 +++ 1 file changed, 3 insertions(+)
package/pkg-generic.mk +3 -0
@@ -280,6 +280,7 @@ $(BUILD_DIR)/%/.stamp_host_installed: $(foreach hook,$($(PKG)_PRE_INSTALL_HOOKS),$(call $(hook))$(sep)) +$($(PKG)_INSTALL_CMDS) $(foreach hook,$($(PKG)_POST_INSTALL_HOOKS),$(call $(hook))$(sep)) + $(Q) rm -f $(HOST_DIR)/share/info/dir @$(call step_end,install-host) $(Q)touch $@
@@ -309,6 +310,7 @@ $(BUILD_DIR)/%/.stamp_staging_installed: $(foreach hook,$($(PKG)_PRE_INSTALL_STAGING_HOOKS),$(call $(hook))$(sep)) +$($(PKG)_INSTALL_STAGING_CMDS) $(foreach hook,$($(PKG)_POST_INSTALL_STAGING_HOOKS),$(call $(hook))$(sep)) + $(Q) rm -f $(STAGING_DIR)/share/info/dir $(Q)if test -n "$($(PKG)_CONFIG_SCRIPTS)" ; then \ $(call MESSAGE,"Fixing package configuration files") ;\ $(SED) "s,$(HOST_DIR), at HOST_DIR@,g" \
@@ -368,6 +370,7 @@ $(BUILD_DIR)/%/.stamp_target_installed: $(or $($(PKG)_INSTALL_INIT_OPENRC), \ $($(PKG)_INSTALL_INIT_SYSV))) $(foreach hook,$($(PKG)_POST_INSTALL_TARGET_HOOKS),$(call $(hook))$(sep)) + $(Q) rm -f $(TARGET_DIR)/share/info/dir $(Q)if test -n "$($(PKG)_CONFIG_SCRIPTS)" ; then \ $(RM) -f $(addprefix $(TARGET_DIR)/usr/bin/,$($(PKG)_CONFIG_SCRIPTS)) ; \ fi
Signature · 1 line
2.31.1
PATCH 3/15 package/pkg-generic.mk: Remove Info documents dir entry
Quoted 18 lines · On Mon, 21 Jun 2021 16:11:18 +0200 Herve Codina <[email protected]>…
Some packages (autotools for instance) install documentation files using install-info. This program adds an entry in the Info directory file (share/info/dir) and this causes
TARGET_DIRand/orHOST_DIRoverwrite.In order to avoid this overwrite this patch removes the Info directory file right after any installation.
Signed-off-by Herve Codina <[email protected]>--- package/pkg-generic.mk | 3 +++ 1 file changed, 3 insertions(+)package/pkg-generic.mk +0
-0
package/pkg-generic.mk $(BUILD_DIR)/%/.stamp_host_installed: · line 283
Quoted 4 lines
package/pkg-generic.mk +1
-0@@ -280,6 +280,7 @@ $(BUILD_DIR)/%/.stamp_host_installed: $(foreach hook,$($(PKG)_PRE_INSTALL_HOOKS),$(call $(hook))$(sep)) +$($(PKG)_INSTALL_CMDS) $(foreach hook,$($(PKG)_POST_INSTALL_HOOKS),$(call $(hook))$(sep)) + $(Q) rm -f $(HOST_DIR)/share/info/dir
No space between $(Q) and rm. This should perhaps use $(RM) in fact.
However, I'm not a huge fan of having this right in the middle of the infrastructure. It feels like a small detail that gets handled in the middle of super generic infrastructure code.
The issue is that I don't really have a good alternative proposal :-/
Instead of removing that file, ignore it in the overwrite detection, perhaps?
Thomas
Signature · 3 lines
Thomas Petazzoni, co-owner and CEO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 3/15 package/pkg-generic.mk: Remove Info documents dir entry
Hi,
Quoted 16 lines · On Mon, 21 Jun 2021 22:51:20 +0200 Thomas Petazzoni…
Quoted 13 lines · On Mon, 21 Jun 2021 16:11:18 +0200 Herve Codina <[email protected]>…
Some packages (autotools for instance) install documentation files using install-info. This program adds an entry in the Info directory file (share/info/dir) and this causes
TARGET_DIRand/orHOST_DIRoverwrite.In order to avoid this overwrite this patch removes the Info directory file right after any installation.
Signed-off-by Herve Codina <[email protected]>--- package/pkg-generic.mk | 3 +++ 1 file changed, 3 insertions(+)
Quoted 11 lines
package/pkg-generic.mk $(BUILD_DIR)/%/.stamp_host_installed: · line 283
Quoted 9 lines
package/pkg-generic.mk +1
-0@@ -280,6 +280,7 @@ $(BUILD_DIR)/%/.stamp_host_installed: $(foreach hook,$($(PKG)_PRE_INSTALL_HOOKS),$(call $(hook))$(sep)) +$($(PKG)_INSTALL_CMDS) $(foreach hook,$($(PKG)_POST_INSTALL_HOOKS),$(call $(hook))$(sep)) + $(Q) rm -f $(HOST_DIR)/share/info/dirNo space between $(Q) and rm. This should perhaps use $(RM) in fact.
Space removed
Quoted 6 lines
However, I'm not a huge fan of having this right in the middle of the infrastructure. It feels like a small detail that gets handled in the middle of super generic infrastructure code.
The issue is that I don't really have a good alternative proposal :-/
Maybe using a macro defined closed to fixup-libtool-files and calling this macro here instead of '$(Q)rm ...' will help.
Do you think it will
be better ?
Quoted 3 lines
Instead of removing that file, ignore it in the overwrite detection, perhaps?
This add a little complexity in overwrite detection (filter out) and I prefer having overwrite detection quite stupid. It checks for overwrites without any exception.
Adding exception now in the detection mechanism is opening the door to more and more exceptions.
Herv?
Signature · 3 lines
Herv? Codina, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 3/15 package/pkg-generic.mk: Remove Info documents dir entry
Quoted 9 lines · On Tue, 22 Jun 2021 10:43:43 +0200 Herve Codina <[email protected]>…
Quoted 5 lines
However, I'm not a huge fan of having this right in the middle of the infrastructure. It feels like a small detail that gets handled in the middle of super generic infrastructure code.
The issue is that I don't really have a good alternative proposal :-/
Maybe using a macro defined closed to fixup-libtool-files and calling this macro here instead of '$(Q)rm ...' will help.
Do you think it will be better ?
Either a macro, or a list of files that are removed, perhaps?
Quoted 5 lines
This add a little complexity in overwrite detection (filter out) and I prefer having overwrite detection quite stupid. It checks for overwrites without any exception.
Adding exception now in the detection mechanism is opening the door to more and more exceptions.
Yes, I understand your argument. As I stated: I'm also not sure which solution to propose here. I was just not a big fan of this removal of one specific file, there, in the middle of a highly generic piece of infrastructure.
Thomas
Signature · 3 lines
Thomas Petazzoni, co-owner and CEO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 3/15 package/pkg-generic.mk: Remove Info documents dir entry
Thomas, Herv?, All,
On 2021-06-22 11:34 +0200, Thomas Petazzoni spake thusly:
Quoted 7 lines
Quoted 5 lines · On Tue, 22 Jun 2021 10:43:43 +0200 Herve Codina <[email protected]>…
Quoted 5 lines
However, I'm not a huge fan of having this right in the middle of the infrastructure. It feels like a small detail that gets handled in the middle of super generic infrastructure code.
The issue is that I don't really have a good alternative proposal :-/
I too am not too fond of this, but I don't have an obvious better solution in mind...
Quoted 5 lines
Quoted 3 lines
Maybe using a macro defined closed to fixup-libtool-files and calling this macro here instead of '$(Q)rm ...' will help.
Do you think it will be better ?Either a macro, or a list of files that are removed, perhaps?
I think a list+macro would be a better solution.
Alternatively, we could append to post-install hooks, like we do for pre-configure hooks in the autotools infra for autoreconf et al., for example...
# Outside generic-package-inner:
# $1: base directory (target, staging, host)
define remove-conflicting-useless-files
$(Q)$(RM) -rf $(patsubst %, $(1)/%, $($(PKG)_DROP_FILES_OR_DIRS))
endef
define REMOVE_CONFLICTING_USELESS_FILES_IN_HOST
$(call remove-conflicting-useless-files,$(HOST_DIR))
endef
define REMOVE_CONFLICTING_USELESS_FILES_IN_STAGING
$(call remove-conflicting-useless-files,$(STAGING_DIR))
endef
define REMOVE_CONFLICTING_USELESS_FILES_IN_TARGET
$(call remove-conflicting-useless-files,$(TARGET_DIR))
endef
# In generic-package-inner:
$(2)_DROP_FILES_OR_DIRS += /share/info/dir
# For host packages:
$(2)_POST_INSTALL_HOOKS += REMOVE_CONFLICTING_USELESS_FILES_IN_HOST
# For target packages:
$(2)_POST_INSTALL_STAGING_HOOKS += REMOVE_CONFLICTING_USELESS_FILES_IN_STAGING
$(2)_POST_INSTALL_TARGET_HOOKS += REMOVE_CONFLICTING_USELESS_FILES_IN_TARGETThis way, it also paves the way to add other hooks to actually fix some files instead of removing them...
For example, assume that packages (e.g. 'wonders') can register themselves against another package (e.g. 'manager') by appending a line to a text file, like so (over-simplified, of course):
echo "name=wonders path=/usr/lib/wonders/v42" >>/usr/share/manager/registry
Then we could have a hook that detects that, extracts the delta installed by the package, stash that somewhere in staging, and have a target-finalise hook that aggregates all the packages in the end.
Quoted 10 lines
Quoted 5 lines
This add a little complexity in overwrite detection (filter out) and I prefer having overwrite detection quite stupid. It checks for overwrites without any exception.
Adding exception now in the detection mechanism is opening the door to more and more exceptions.Yes, I understand your argument. As I stated: I'm also not sure which solution to propose here. I was just not a big fan of this removal of one specific file, there, in the middle of a highly generic piece of infrastructure.
Yes, this bothered me too...
Regards,
Yann E. MORIN.
Quoted 9 lines
Thomas
Signature · 7 lines
Thomas Petazzoni, co-owner and CEO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com _______________________________________________ buildroot mailing list buildroot at busybox.net http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 3/15 package/pkg-generic.mk: Remove Info documents dir entry
Hi,
Quoted 41 lines · On Tue, 22 Jun 2021 22:18:18 +0200 "Yann E. MORIN" <[email protected]>…
I think a list+macro would be a better solution.
Alternatively, we could append to post-install hooks, like we do for pre-configure hooks in the autotools infra for autoreconf et al., for example...
# Outside generic-package-inner: # $1: base directory (target, staging, host) define remove-conflicting-useless-files $(Q)$(RM) -rf $(patsubst %, $(1)/%, $($(PKG)_DROP_FILES_OR_DIRS)) endef define REMOVE_CONFLICTING_USELESS_FILES_IN_HOST $(call remove-conflicting-useless-files,$(HOST_DIR)) endef define REMOVE_CONFLICTING_USELESS_FILES_IN_STAGING $(call remove-conflicting-useless-files,$(STAGING_DIR)) endef define REMOVE_CONFLICTING_USELESS_FILES_IN_TARGET $(call remove-conflicting-useless-files,$(TARGET_DIR)) endef # In generic-package-inner: $(2)_DROP_FILES_OR_DIRS += /share/info/dir # For host packages: $(2)_POST_INSTALL_HOOKS += REMOVE_CONFLICTING_USELESS_FILES_IN_HOST # For target packages: $(2)_POST_INSTALL_STAGING_HOOKS += REMOVE_CONFLICTING_USELESS_FILES_IN_STAGING $(2)_POST_INSTALL_TARGET_HOOKS += REMOVE_CONFLICTING_USELESS_FILES_IN_TARGETThis way, it also paves the way to add other hooks to actually fix some files instead of removing them...
For example, assume that packages (e.g. 'wonders') can register themselves against another package (e.g. 'manager') by appending a line to a text file, like so (over-simplified, of course):
Quoted 6 lines
echo "name=wonders path=/usr/lib/wonders/v42" >>/usr/share/manager/registry
Then we could have a hook that detects that, extracts the delta installed by the package, stash that somewhere in staging, and have a target-finalise hook that aggregates all the packages in the end.
I like this solution.
If it is okay for everyone, I will rework my patch accordingly.
Thomas, your opinion ?
Regards,
Herv?
Signature · 3 lines
Herv? Codina, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 4/15 package/pkg-generic.mk: Fix .la files overwrite detection
In reply to Herve CodinaDuring per-package build, original .la files are modified by fixup-libtool-files calls.
But since fixup-libtool-files modifies files using sed --in-place, these modification are done using a temporary file and a call to rename. Rename breaks the hardlink to the original file and leave the temporary file in per-package TARGET dir.
As the original file is not modified, this is no longer considered as an overwrite.
To fix this detection, this patch simply considers the what is done by fixup-libtool-files is part of the original snapshot used to detect overwrites. And so, the original snapshot is taken after fixup-libtool-files call.
--- package/pkg-generic.mk | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
package/pkg-generic.mk +1 -1
@@ -254,9 +254,9 @@ $(BUILD_DIR)/%/.stamp_configured: @$(call pkg_size_before,$(TARGET_DIR)) @$(call pkg_size_before,$(STAGING_DIR),-staging) @$(call pkg_size_before,$(HOST_DIR),-host) + $(call fixup-libtool-files,$(NAME),$(STAGING_DIR)) @$(call pkg_detect_overwrite_before,$(TARGET_DIR)) @$(call pkg_detect_overwrite_before,$(HOST_DIR),-host) - $(call fixup-libtool-files,$(NAME),$(STAGING_DIR)) $(foreach hook,$($(PKG)_PRE_CONFIGURE_HOOKS),$(call $(hook))$(sep)) $($(PKG)_CONFIGURE_CMDS) $(foreach hook,$($(PKG)_POST_CONFIGURE_HOOKS),$(call $(hook))$(sep))
Signature · 1 line
2.31.1
PATCH 4/15 package/pkg-generic.mk: Fix .la files overwrite detection
Quoted 10 lines · On Mon, 21 Jun 2021 16:11:19 +0200 Herve Codina <[email protected]>…
During per-package build, original .la files are modified by fixup-libtool-files calls.
But since fixup-libtool-files modifies files using sed --in-place, these modification are done using a temporary file and a call to rename. Rename breaks the hardlink to the original file and leave the temporary file in per-package TARGET dir.
As the original file is not modified, this is no longer considered as an overwrite.To fix this detection, this patch simply considers the what is done
"the what" -> "that what"
Quoted 5 lines
by fixup-libtool-files is part of the original snapshot used to detect overwrites. And so, the original snapshot is taken after fixup-libtool-files call.
Signed-off-by Herve Codina <[email protected]>
With that:
Thomas
Signature · 3 lines
Thomas Petazzoni, co-owner and CEO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 4/15 package/pkg-generic.mk: Fix .la files overwrite detection
Quoted 12 lines · On Mon, 21 Jun 2021 22:54:11 +0200 Thomas Petazzoni…
Quoted 9 lines · On Mon, 21 Jun 2021 16:11:19 +0200 Herve Codina <[email protected]>…
During per-package build, original .la files are modified by fixup-libtool-files calls.
But since fixup-libtool-files modifies files using sed --in-place, these modification are done using a temporary file and a call to rename. Rename breaks the hardlink to the original file and leave the temporary file in per-package TARGET dir.
As the original file is not modified, this is no longer considered as an overwrite.
Quoted 3 lines
Quoted 1 line
To fix this detection, this patch simply considers the what is done
"the what" -> "that what"
Fixed.
Quoted 10 lines
Quoted 6 lines
by fixup-libtool-files is part of the original snapshot used to detect overwrites. And so, the original snapshot is taken after fixup-libtool-files call.
Signed-off-by Herve Codina <[email protected]>With that:
Reviewed-by Thomas Petazzoni <[email protected]>
Thanks for this first review.
Herv?
Signature · 3 lines
Herv? Codina, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 4/15 package/pkg-generic.mk: Fix .la files overwrite detection
In reply to Herve CodinaOn 2021-06-21 16:11 +0200, Herve Codina spake thusly:
Quoted 9 lines
During per-package build, original .la files are modified by fixup-libtool-files calls.
But since fixup-libtool-files modifies files using sed --in-place, these modification are done using a temporary file and a call to rename. Rename breaks the hardlink to the original file and leave the temporary file in per-package TARGET dir.
As the original file is not modified, this is no longer considered as an overwrite.
Quoted 4 lines
To fix this detection, this patch simply considers the what is done by fixup-libtool-files is part of the original snapshot used to detect overwrites. And so, the original snapshot is taken after fixup-libtool-files call.
Then this should be squashed together with the first patch, to avoid introducing the issue just to fix it a few patches down the series.
You should however add a note about that in the commit log of the first patch, of course, to explain why the overwrite ifnra is inserted after the .la tweaks.
So, I agree with the explanations, which make sense, but I disagree that it should be a separate patch...
Regards,
Yann E. MORIN.
Quoted 27 lines
Signed-off-by Herve Codina <[email protected]>--- package/pkg-generic.mk | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)package/pkg-generic.mk +1
-1@@ -254,9 +254,9 @@ $(BUILD_DIR)/%/.stamp_configured: @$(call pkg_size_before,$(TARGET_DIR)) @$(call pkg_size_before,$(STAGING_DIR),-staging) @$(call pkg_size_before,$(HOST_DIR),-host) + $(call fixup-libtool-files,$(NAME),$(STAGING_DIR)) @$(call pkg_detect_overwrite_before,$(TARGET_DIR)) @$(call pkg_detect_overwrite_before,$(HOST_DIR),-host) - $(call fixup-libtool-files,$(NAME),$(STAGING_DIR)) $(foreach hook,$($(PKG)_PRE_CONFIGURE_HOOKS),$(call $(hook))$(sep)) $($(PKG)_CONFIGURE_CMDS) $(foreach hook,$($(PKG)_POST_CONFIGURE_HOOKS),$(call $(hook))$(sep))Signature · 6 lines
2.31.1 _______________________________________________ buildroot mailing list buildroot at busybox.net http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 4/15 package/pkg-generic.mk: Fix .la files overwrite detection
Hi,
Quoted 18 lines · On Mon, 21 Jun 2021 23:42:23 +0200 "Yann E. MORIN" <[email protected]>…
On 2021-06-21 16:11 +0200, Herve Codina spake thusly:
Quoted 9 lines
During per-package build, original .la files are modified by fixup-libtool-files calls.
But since fixup-libtool-files modifies files using sed --in-place, these modification are done using a temporary file and a call to rename. Rename breaks the hardlink to the original file and leave the temporary file in per-package TARGET dir.
As the original file is not modified, this is no longer considered as an overwrite.Quoted 4 lines
To fix this detection, this patch simply considers the what is done by fixup-libtool-files is part of the original snapshot used to detect overwrites. And so, the original snapshot is taken after fixup-libtool-files call.
Then this should be squashed together with the first patch, to avoid introducing the issue just to fix it a few patches down the series.
Quoted 7 lines
You should however add a note about that in the commit log of the first patch, of course, to explain why the overwrite ifnra is inserted after the .la tweaks.
So, I agree with the explanations, which make sense, but I disagree that it should be a separate patch...
Well, I have seen this when I created the patches.
I kept them separate because on the first patch, I introduced the tool to check the overwrites and i would like it to take its snapshot as soon as possible in the build sequence (ie right after collecting dependencies files and taking snapshots for current package statistics). Then I fixed the issue seen by the overwrites detection and I put at the same level fixing host-e2fsprogs, fixing .la files or a bit later fixing python with one patch per fix to detail (or try to detail) the issue and the way I fixed it.
Squashing the 2 patches leads to one patch that introduces the tool and fixes one of the issues detected by the tool. What about the others issues detected ? Squash also together with the first patch ? I think it will produce a huge patch quite complicate to understand even with all individual commit message squashed.
However, that being said, I can squash this patch (Fix .la files overwrite detection) with the 1st one (detect files overwritten in TARGET_DIR and HOST_DIR) if you still think it will be better.
Regards,
Herv? Codina
Signature · 3 lines
Herv? Codina, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 4/15 package/pkg-generic.mk: Fix .la files overwrite detection
Herv?, All,
On 2021-06-22 11:31 +0200, Herve Codina spake thusly:
Quoted 26 lines
Quoted 17 lines · On Mon, 21 Jun 2021 23:42:23 +0200 "Yann E. MORIN" <[email protected]>…
On 2021-06-21 16:11 +0200, Herve Codina spake thusly:
Quoted 9 lines
During per-package build, original .la files are modified by fixup-libtool-files calls.
But since fixup-libtool-files modifies files using sed --in-place, these modification are done using a temporary file and a call to rename. Rename breaks the hardlink to the original file and leave the temporary file in per-package TARGET dir.
As the original file is not modified, this is no longer considered as an overwrite.Quoted 4 lines
To fix this detection, this patch simply considers the what is done by fixup-libtool-files is part of the original snapshot used to detect overwrites. And so, the original snapshot is taken after fixup-libtool-files call.
Then this should be squashed together with the first patch, to avoid introducing the issue just to fix it a few patches down the series.
Quoted 7 lines
You should however add a note about that in the commit log of the first patch, of course, to explain why the overwrite ifnra is inserted after the .la tweaks.
So, I agree with the explanations, which make sense, but I disagree that it should be a separate patch...
Quoted 9 lines
Well, I have seen this when I created the patches.
I kept them separate because on the first patch, I introduced the tool to check the overwrites and i would like it to take its snapshot as soon as possible in the build sequence (ie right after collecting dependencies files and taking snapshots for current package statistics). Then I fixed the issue seen by the overwrites detection and I put at the same level fixing host-e2fsprogs, fixing .la files or a bit later fixing python with one patch per fix to detail (or try to detail) the issue and the way I fixed it.
But really, it *is* the first patch that introduces the issue, so it should be fixed from the onset, rather than after-the-fact.
Quoted 2 lines
Squashing the 2 patches leads to one patch that introduces the tool and fixes one of the issues detected by the tool.
Sorry, but this patch (4/15) is fixing an issue introduced by the first patch, with:
@$(call pkg_size_before,$(TARGET_DIR))
@$(call pkg_size_before,$(STAGING_DIR),-staging)
@$(call pkg_size_before,$(HOST_DIR),-host)
+ @$(call pkg_detect_overwrite_before,$(TARGET_DIR))
+ @$(call pkg_detect_overwrite_before,$(HOST_DIR),-host)
$(call fixup-libtool-files,$(NAME),$(STAGING_DIR))
$(foreach hook,$($(PKG)_PRE_CONFIGURE_HOOKS),$(call $(hook))$(sep))
$($(PKG)_CONFIGURE_CMDS)... when it should directly be:
@$(call pkg_size_before,$(STAGING_DIR),-staging)
@$(call pkg_size_before,$(HOST_DIR),-host)
$(call fixup-libtool-files,$(NAME),$(STAGING_DIR))
+ @$(call pkg_detect_overwrite_before,$(TARGET_DIR))
+ @$(call pkg_detect_overwrite_before,$(HOST_DIR),-host)
$(foreach hook,$($(PKG)_PRE_CONFIGURE_HOOKS),$(call $(hook))$(sep))
$($(PKG)_CONFIGURE_CMDS)
$(foreach hook,$($(PKG)_POST_CONFIGURE_HOOKS),$(call $(hook))$(sep))And as I said, the commit log should explain why the 'overwrite' calls are inserted after the .la fixup.
Quoted 4 lines
What about the others issues
detected ? Squash also together with the first patch ? I think it will produce a huge patch quite complicate to understand even with all individual commit message squashed.
Ideally, I would say a series should first fix the issues, then introduce the tooling.
Otherwise, if only the first patch(es) are applied, the tree is broken:
indeed the tooling has been applied, and thus is used, but the issues are still there.
Also, it is usually easier and less controversial to get fixes applied, than new tooling.
Quoted 3 lines
However, that being said, I can squash this patch (Fix .la files overwrite detection) with the 1st one (detect files overwritten in
TARGET_DIRandHOST_DIR) if you still think it will be better.
Yes, I still think that it is better.
Regards,
Yann E. MORIN.
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 4/15 package/pkg-generic.mk: Fix .la files overwrite detection
Quoted 2 lines · On Tue, 22 Jun 2021 11:56:09 +0200 "Yann E. MORIN" <[email protected]>…
Sorry, but this patch (4/15) is fixing an issue introduced by the first patch, with:
I agree in principle. But here, it's really a new requirement that we are adding on packages, and we *know* that even with Herv? patches there will be some other packages that will need fixing.
So to me it's not really like introducing a regression in a patch, and fixing it later.
Quoted 7 lines
Quoted 4 lines
What about the others issues
detected ? Squash also together with the first patch ? I think it will produce a huge patch quite complicate to understand even with all individual commit message squashed.Ideally, I would say a series should first fix the issues, then introduce the tooling.
No, please keep one series, but having the fixes *before* we introduce the check. And as stated above, the fixes will not fix all problems, they will only fix the problems we know about. More problems will be detected by the autobuilders thanks to the overwrite check.
Quoted 5 lines
Quoted 3 lines
However, that being said, I can squash this patch (Fix .la files overwrite detection) with the 1st one (detect files overwritten in
TARGET_DIRandHOST_DIR) if you still think it will be better.Yes, I still think that it is better.
No, please don't squash, have fixes added before the overwrite check instead.
Thomas
Signature · 3 lines
Thomas Petazzoni, co-owner and CEO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 4/15 package/pkg-generic.mk: Fix .la files overwrite detection
Thomas, All,
On 2021-06-22 12:12 +0200, Thomas Petazzoni spake thusly:
Quoted 7 lines
Quoted 2 lines · On Tue, 22 Jun 2021 11:56:09 +0200 "Yann E. MORIN" <[email protected]>…
Sorry, but this patch (4/15) is fixing an issue introduced by the first patch, with:
I agree in principle. But here, it's really a new requirement that we are adding on packages, and we *know* that even with Herv? patches there will be some other packages that will need fixing.
Sorry, but this patch is fixing the infra, not the packages.
Quoted 2 lines
So to me it's not really like introducing a regression in a patch, and fixing it later.
I never said it was a regression. ;-)
Quoted 10 lines
Quoted 7 lines
Quoted 4 lines
What about the others issues
detected ? Squash also together with the first patch ? I think it will produce a huge patch quite complicate to understand even with all individual commit message squashed.Ideally, I would say a series should first fix the issues, then introduce the tooling.
No, please keep one series, but having the fixes *before* we introduce the check.
Exactly; I never said to split the series into two series. It should still be one.
Quoted 3 lines
And as stated above, the fixes will not fix all problems, they will only fix the problems we know about. More problems will be detected by the autobuilders thanks to the overwrite check.
And this is totally OK. [0]
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
However, that being said, I can squash this patch (Fix .la files overwrite detection) with the 1st one (detect files overwritten in
TARGET_DIRandHOST_DIR) if you still think it will be better.Yes, I still think that it is better.
No, please don't squash, have fixes added before the overwrite check instead.
Well, I still disagree, because this patch really fixes an issue introduced *in the infra* by the first patch.
Regards,
Yann E. MORIN.
[0] but brace for impact! ;-]
Quoted 5 lines
Thomas
Signature · 3 lines
Thomas Petazzoni, co-owner and CEO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 4/15 package/pkg-generic.mk: Fix .la files overwrite detection
Hi,
Quoted 40 lines · On Tue, 22 Jun 2021 12:30:39 +0200 "Yann E. MORIN" <[email protected]>…
On 2021-06-22 12:12 +0200, Thomas Petazzoni spake thusly:
Quoted 7 lines
Quoted 2 lines · On Tue, 22 Jun 2021 11:56:09 +0200 "Yann E. MORIN" <[email protected]>…
Sorry, but this patch (4/15) is fixing an issue introduced by the first patch, with:
I agree in principle. But here, it's really a new requirement that we are adding on packages, and we *know* that even with Herv? patches there will be some other packages that will need fixing.
Sorry, but this patch is fixing the infra, not the packages.
Quoted 2 lines
So to me it's not really like introducing a regression in a patch, and fixing it later.
I never said it was a regression. ;-)
Quoted 10 lines
Quoted 7 lines
Quoted 4 lines
What about the others issues
detected ? Squash also together with the first patch ? I think it will produce a huge patch quite complicate to understand even with all individual commit message squashed.Ideally, I would say a series should first fix the issues, then introduce the tooling.
No, please keep one series, but having the fixes *before* we introduce the check.
Exactly; I never said to split the series into two series. It should still be one.
Quoted 3 lines
And as stated above, the fixes will not fix all problems, they will only fix the problems we know about. More problems will be detected by the autobuilders thanks to the overwrite check.
And this is totally OK. [0]
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
However, that being said, I can squash this patch (Fix .la files overwrite detection) with the 1st one (detect files overwritten in
TARGET_DIRandHOST_DIR) if you still think it will be better.
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
Yes, I still think that it is better.
No, please don't squash, have fixes added before the overwrite check instead.
Well, I still disagree, because this patch really fixes an issue introduced *in the infra* by the first patch.
Well, my bad.
I think that the better thing to do is to fully rework the history. First fixing overwrites and then introducing the overwrite detection tooling (ie the actual PATCH 1 will be moved just before actual PATCH 12 and so actual PATCH 4 simply disappears).
The patch introducing the tooling will explain in its commit message why the calls to fixup-libtool-files and fixup-python-files are performed before taking the overwrite snapshot and why it is safe (sed --in-place).
The variable <PKG>_PER_PACKAGE_TWEAK_HOOKS (or other name) will be introduced before the tooling. The patches changing some packages to use this variable (move tweaks from <PKG>_POST_CONFIGURE_HOOKS to <PKG>_PER_PACKAGE_TWEAK_HOOKS) will also be introduced before the tooling even if these changes really make sense after the tooling introduction.
Indeed, the notion of action done before taking the overwrite detection snapshot (<PKG>_PER_PACKAGE_TWEAK_HOOKS) and action done after (<PKG>_PRE_CONFIGURE_HOOKS, configure and <PKG>_POST_CONFIGURE_HOOKS) makes sense only with the overwrite detection tool.
Is that seem better for everyone ?
Regards,
Herv?
Signature · 3 lines
Herv? Codina, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 4/15 package/pkg-generic.mk: Fix .la files overwrite detection
Herv?, All,
On 2021-06-24 17:44 +0200, Herve Codina spake thusly:
Quoted 2 lines
On Tue, 22 Jun 2021 12:30:39 +0200
"Yann E. MORIN" <[email protected]> wrote:
[--SNIP--]
Quoted 3 lines
Quoted 2 lines
Well, I still disagree, because this patch really fixes an issue introduced *in the infra* by the first patch.
Well, my bad.
No worries, really! This series is quite the complex one. ;-)
Quoted 8 lines
I think that the better thing to do is to fully rework the history. First fixing overwrites and then introducing the overwrite detection tooling (ie the actual PATCH 1 will be moved just before actual PATCH 12 and so actual PATCH 4 simply disappears).
The patch introducing the tooling will explain in its commit message why the calls to fixup-libtool-files and fixup-python-files are performed before taking the overwrite snapshot and why it is safe (sed --in-place).
Exactly! :-)
Quoted 11 lines
The variable <PKG>_PER_PACKAGE_TWEAK_HOOKS (or other name) will be introduced before the tooling. The patches changing some packages to use this variable (move tweaks from <PKG>_POST_CONFIGURE_HOOKS to <PKG>_PER_PACKAGE_TWEAK_HOOKS) will also be introduced before the tooling even if these changes really make sense after the tooling introduction.
Indeed, the notion of action done before taking the overwrite detection snapshot (<PKG>_PER_PACKAGE_TWEAK_HOOKS) and action done after (<PKG>_PRE_CONFIGURE_HOOKS, configure and <PKG>_POST_CONFIGURE_HOOKS) makes sense only with the overwrite detection tool.Is that seem better for everyone ?
It looks a sane ordering to me, yes. THanks for working on this!
Regards,
Yann E. MORIN.
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 5/15 package/pkg-generic.mk: Perform .la files fixup in per-package HOST_DIR
In reply to Herve Codinafixup-libtool-files was called on per-package STAGING_DIR. Some host-xxxx packages have their .la files with directories pointing outside their own per-package directory.
This commit, calling fixup-libtool-files on HOST_DIR, fixes this issue.
--- package/pkg-generic.mk | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
package/pkg-generic.mk +3 -1
@@ -97,7 +97,8 @@ endif # $2: staging directory of the package ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y) define fixup-libtool-files - $(Q)find $(2)/usr/lib* -name "*.la" | xargs --no-run-if-empty \ + $(Q)find $(2) \( -path '$(2)/lib*' -o -path '$(2)/usr/lib*' \) \ + -name "*.la" | xargs --no-run-if-empty \ $(SED) "s:$(PER_PACKAGE_DIR)/[^/]\+/:$(PER_PACKAGE_DIR)/$(1)/:g" endef endif
@@ -254,6 +255,7 @@ $(BUILD_DIR)/%/.stamp_configured: @$(call pkg_size_before,$(TARGET_DIR)) @$(call pkg_size_before,$(STAGING_DIR),-staging) @$(call pkg_size_before,$(HOST_DIR),-host) + $(call fixup-libtool-files,$(NAME),$(HOST_DIR)) $(call fixup-libtool-files,$(NAME),$(STAGING_DIR)) @$(call pkg_detect_overwrite_before,$(TARGET_DIR)) @$(call pkg_detect_overwrite_before,$(HOST_DIR),-host)
Signature · 1 line
2.31.1
PATCH 5/15 package/pkg-generic.mk: Perform .la files fixup in per-package HOST_DIR
Quoted 3 lines · On Mon, 21 Jun 2021 16:11:20 +0200 Herve Codina <[email protected]>…
fixup-libtool-files was called on per-package
STAGING_DIR. Some host-xxxx packages have their .la files with directories pointing outside their own per-package directory.
Was this causing a practical issue, or is being done just to be cleaner?
Thomas
Signature · 3 lines
Thomas Petazzoni, co-owner and CEO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 5/15 package/pkg-generic.mk: Perform .la files fixup in per-package HOST_DIR
Hi,
Quoted 9 lines · On Mon, 21 Jun 2021 22:48:12 +0200 Thomas Petazzoni…
Quoted 3 lines · On Mon, 21 Jun 2021 16:11:20 +0200 Herve Codina <[email protected]>…
fixup-libtool-files was called on per-package
STAGING_DIR. Some host-xxxx packages have their .la files with directories pointing outside their own per-package directory.Was this causing a practical issue, or is being done just to be cleaner?
I didn't see any practical issue on the builds I did and I didn't try to find any use-case that leads to an issue.
So this patch is to be cleaner.
Herv? Codina
Signature · 3 lines
Herv? Codina, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 5/15 package/pkg-generic.mk: Perform .la files fixup in per-package HOST_DIR
In reply to Herve CodinaHerv?, All,
On 2021-06-21 16:11 +0200, Herve Codina spake thusly:
Quoted 6 lines
fixup-libtool-files was called on per-package
STAGING_DIR. Some host-xxxx packages have their .la files with directories pointing outside their own per-package directory.This commit, calling fixup-libtool-files on
HOST_DIR, fixes this issue.
Usually, I don't much like that we have a fix for some theoretical bug, but in this case I think it just does make sense.
And because a .la file is only used to find the libraries at link time, not runtime, this is has most probably gone unnoticed because, at build time, the libraries are present in the out-of-package tree, and are present at runtime in the per-package tree...
Quoted 10 lines
Signed-off-by Herve Codina <[email protected]>--- package/pkg-generic.mk | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)package/pkg-generic.mk +0
-0
package/pkg-generic.mk endif · line 102
Quoted 7 lines
package/pkg-generic.mk +2
-1@@ -97,7 +97,8 @@ endif # $2: staging directory of the package ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y) define fixup-libtool-files - $(Q)find $(2)/usr/lib* -name "*.la" | xargs --no-run-if-empty \ + $(Q)find $(2) \( -path '$(2)/lib*' -o -path '$(2)/usr/lib*' \) \ + -name "*.la" | xargs --no-run-if-empty \ $(SED) "s:$(PER_PACKAGE_DIR)/[^/]\+/:$(PER_PACKAGE_DIR)/$(1)/:g"
While at it, can you switch over to using find's -print0 and xarg's -0 ?
package/pkg-generic.mk $(BUILD_DIR)/%/.stamp_configured: · line 259
Quoted 8 lines
package/pkg-generic.mk +1
-0endef endif@@ -254,6 +255,7 @@ $(BUILD_DIR)/%/.stamp_configured: @$(call pkg_size_before,$(TARGET_DIR)) @$(call pkg_size_before,$(STAGING_DIR),-staging) @$(call pkg_size_before,$(HOST_DIR),-host) + $(call fixup-libtool-files,$(NAME),$(HOST_DIR)) $(call fixup-libtool-files,$(NAME),$(STAGING_DIR))
Regards,
Yann E. MORIN.
Quoted 9 lines
package/pkg-generic.mk +0
-0@$(call pkg_detect_overwrite_before,$(TARGET_DIR)) @$(call pkg_detect_overwrite_before,$(HOST_DIR),-host)Signature · 6 lines
2.31.1 _______________________________________________ buildroot mailing list buildroot at busybox.net http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 5/15 package/pkg-generic.mk: Perform .la files fixup in per-package HOST_DIR
Hi,
Quoted 9 lines · On Tue, 22 Jun 2021 12:12:02 +0200 "Yann E. MORIN" <[email protected]>…
package/pkg-generic.mk endif · line 102
Quoted 7 lines
# $2: staging directory of the package
ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y)
define fixup-libtool-filespackage/pkg-generic.mk +2
-1- $(Q)find $(2)/usr/lib* -name "*.la" | xargs --no-run-if-empty \ + $(Q)find $(2) \( -path '$(2)/lib*' -o -path '$(2)/usr/lib*' \) \ + -name "*.la" | xargs --no-run-if-empty \ $(SED) "s:$(PER_PACKAGE_DIR)/[^/]\+/:$(PER_PACKAGE_DIR)/$(1)/:g"While at it, can you switch over to using find's -print0 and xarg's -0 ?
Yes, changed.
I will also change PATCH 10 of this series. Same construction in fixup-python-files same sanction.
Regards,
Herv?
Signature · 3 lines
Herv? Codina, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 6/15 package/pkg-generic.mk: Introduce <PKG>_PER_PACKAGE_TWEAK_HOOKS
In reply to Herve CodinaThis hook will be used by package infrastucture or some packages to tweak external files (ie files not provides by the package itself) when we build with BR2_PER_PACKAGE_DIRECTORIES set.
This allows to have these tweaks in a specific hook instead of <PKG>_POST_CONFIGURE_HOOK. And so we can call these specific tweaks whenever we need them.
--- package/pkg-generic.mk | 2 ++ 1 file changed, 2 insertions(+)
package/pkg-generic.mk +2 -0
@@ -257,6 +257,7 @@ $(BUILD_DIR)/%/.stamp_configured: @$(call pkg_size_before,$(HOST_DIR),-host) $(call fixup-libtool-files,$(NAME),$(HOST_DIR)) $(call fixup-libtool-files,$(NAME),$(STAGING_DIR)) + $(foreach hook,$($(PKG)_PER_PACKAGE_TWEAK_HOOKS),$(call $(hook))$(sep)) @$(call pkg_detect_overwrite_before,$(TARGET_DIR)) @$(call pkg_detect_overwrite_before,$(HOST_DIR),-host) $(foreach hook,$($(PKG)_PRE_CONFIGURE_HOOKS),$(call $(hook))$(sep))
@@ -844,6 +845,7 @@ $(2)_PRE_LEGAL_INFO_HOOKS ?= $(2)_POST_LEGAL_INFO_HOOKS ?= $(2)_TARGET_FINALIZE_HOOKS ?= $(2)_ROOTFS_PRE_CMD_HOOKS ?= +$(2)_PER_PACKAGE_TWEAK_HOOKS ?= ifeq ($$($(2)_TYPE),target) ifneq ($$(HOST_$(2)_KCONFIG_VAR),)
Signature · 1 line
2.31.1
PATCH 6/15 package/pkg-generic.mk: Introduce <PKG>_PER_PACKAGE_TWEAK_HOOKS
Quoted 9 lines · On Mon, 21 Jun 2021 16:11:21 +0200 Herve Codina <[email protected]>…
This hook will be used by package infrastucture or some packages to tweak external files (ie files not provides by the package itself) when we build with
BR2_PER_PACKAGE_DIRECTORIESset.This allows to have these tweaks in a specific hook instead of <PKG>_POST_CONFIGURE_HOOK. And so we can call these specific tweaks whenever we need them.
Signed-off-by Herve Codina <[email protected]>
I think this commit log needs an explanation as to why this is necessary, i.e why isn't the current solution of POST_CONFIGURE_HOOKS working.
Thanks!
Thomas
Signature · 3 lines
Thomas Petazzoni, co-owner and CEO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 6/15 package/pkg-generic.mk: Introduce <PKG>_PER_PACKAGE_TWEAK_HOOKS
In reply to Herve CodinaHerv?, All,
On 2021-06-21 16:11 +0200, Herve Codina spake thusly:
Quoted 7 lines
This hook will be used by package infrastucture or some packages to tweak external files (ie files not provides by the package itself) when we build with
BR2_PER_PACKAGE_DIRECTORIESset.This allows to have these tweaks in a specific hook instead of <PKG>_POST_CONFIGURE_HOOK. And so we can call these specific tweaks whenever we need them.
I'm a bit confused, because those new hooks are even before _PRE_CONFIGURE_HOOKS, so smeantically, they are different from _POST_CONFIGURE_HOOKS...
So I fail to see how this articulates with POST_CONFIGURE_HOOKS. Looking at the next patch, about apr-utils, does not explain why the existing apr-utils fixups need to change, with this new hooks...
Ah, yes, yes, now I see. OK, so here's a suggestion for a commit log:
package/pkg-generic: add early pre-configure hooks
Currently, when a package needs to modify files it inherits from its dependencies, because they contain paths, we can only do that in a pre- or post-configure hook.
However, whatever is done as part of those hooks, will be accounted to the package itself, and thus will trigger file-overwrite detection.
So, we need a way to be able to actually modify files before we start monitoring for them.
We introduce a new set of hooks that an individual package can set, or that a package infra can set, and that are called right before we snapshot the state of target, and host (to which staging belongs),
Quoted 10 lines
Signed-off-by Herve Codina <[email protected]>--- package/pkg-generic.mk | 2 ++ 1 file changed, 2 insertions(+)package/pkg-generic.mk +0
-0
package/pkg-generic.mk $(BUILD_DIR)/%/.stamp_configured: · line 260
Quoted 4 lines
package/pkg-generic.mk +1
-0@@ -257,6 +257,7 @@ $(BUILD_DIR)/%/.stamp_configured: @$(call pkg_size_before,$(HOST_DIR),-host) $(call fixup-libtool-files,$(NAME),$(HOST_DIR)) $(call fixup-libtool-files,$(NAME),$(STAGING_DIR)) + $(foreach hook,$($(PKG)_PER_PACKAGE_TWEAK_HOOKS),$(call $(hook))$(sep))
I am going to bikeshed on the naming of this variable (which is a good sing that I am pretty much OK with the feature).
I think $(2)_EARLY_PRE_CONFIGURE_HOOKS is more sensible.
package/pkg-generic.mk $(2)_PRE_LEGAL_INFO_HOOKS ?= · line 848
Quoted 8 lines
package/pkg-generic.mk +1
-0@$(call pkg_detect_overwrite_before,$(TARGET_DIR)) @$(call pkg_detect_overwrite_before,$(HOST_DIR),-host) $(foreach hook,$($(PKG)_PRE_CONFIGURE_HOOKS),$(call $(hook))$(sep))@@ -844,6 +845,7 @@ $(2)_PRE_LEGAL_INFO_HOOKS ?= $(2)_POST_LEGAL_INFO_HOOKS ?= $(2)_TARGET_FINALIZE_HOOKS ?= $(2)_ROOTFS_PRE_CMD_HOOKS ?= +$(2)_PER_PACKAGE_TWEAK_HOOKS ?=
Actually, this is totally useless: if the variable is unset, this would set it to empty, which for 'make' is totally identical (an unset variable can be expanded, and the expansion is the emopty string).
I know you are just adhering to existing code, and so you should indeed keep it for the next series, but a cleanup in the infra would be welcome too! ;-)
Regards,
Yann E. MORIN.
Quoted 9 lines
package/pkg-generic.mk +0
-0ifeq ($$($(2)_TYPE),target) ifneq ($$(HOST_$(2)_KCONFIG_VAR),)Signature · 6 lines
2.31.1 _______________________________________________ buildroot mailing list buildroot at busybox.net http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 6/15 package/pkg-generic.mk: Introduce <PKG>_PER_PACKAGE_TWEAK_HOOKS
Quoted 8 lines · On Tue, 22 Jun 2021 22:39:09 +0200 "Yann E. MORIN" <[email protected]>…
Ah, yes, yes, now I see. OK, so here's a suggestion for a commit log:
package/pkg-generic: add early pre-configure hooks
Currently, when a package needs to modify files it inherits from its dependencies, because they contain paths, we can only do that in a pre- or post-configure hook.
Quoted 5 lines
However, whatever is done as part of those hooks, will be accounted to the package itself, and thus will trigger file-overwrite detection.
So, we need a way to be able to actually modify files before we start monitoring for them.
"before we start monitoring changes in those files".
Quoted 3 lines
We introduce a new set of hooks that an individual package can set, or that a package infra can set, and that are called right before we snapshot the state of target, and host (to which staging belongs),
But yes, you've perfectly understood the problem being solved here!
Thomas
Signature · 3 lines
Thomas Petazzoni, co-owner and CEO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 6/15 package/pkg-generic.mk: Introduce <PKG>_PER_PACKAGE_TWEAK_HOOKS
Hi,
Quoted 17 lines · On Wed, 23 Jun 2021 14:40:20 +0200 Thomas Petazzoni…
Quoted 8 lines · On Tue, 22 Jun 2021 22:39:09 +0200 "Yann E. MORIN" <[email protected]>…
Ah, yes, yes, now I see. OK, so here's a suggestion for a commit log:
package/pkg-generic: add early pre-configure hooks
Currently, when a package needs to modify files it inherits from its dependencies, because they contain paths, we can only do that in a pre- or post-configure hook.
Quoted 5 lines
However, whatever is done as part of those hooks, will be accounted to the package itself, and thus will trigger file-overwrite detection.
So, we need a way to be able to actually modify files before we start monitoring for them.
Quoted 8 lines
"before we start monitoring changes in those files".
Quoted 3 lines
We introduce a new set of hooks that an individual package can set, or that a package infra can set, and that are called right before we snapshot the state of target, and host (to which staging belongs),
But yes, you've perfectly understood the problem being solved here!
This will be the new commit log.
Herv?
Signature · 3 lines
Herv? Codina, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 6/15 package/pkg-generic.mk: Introduce <PKG>_PER_PACKAGE_TWEAK_HOOKS
In reply to Yann E. MORINHi,
Quoted 7 lines · On Tue, 22 Jun 2021 22:39:09 +0200 "Yann E. MORIN" <[email protected]>…
package/pkg-generic.mk $(BUILD_DIR)/%/.stamp_configured: · line 260
Quoted 2 lines
package/pkg-generic.mk +1
-0$(call fixup-libtool-files,$(NAME),$(STAGING_DIR)) + $(foreach hook,$($(PKG)_PER_PACKAGE_TWEAK_HOOKS),$(call $(hook))$(sep))I am going to bikeshed on the naming of this variable (which is a good sing that I am pretty much OK with the feature).
I think $(2)_EARLY_PRE_CONFIGURE_HOOKS is more sensible.
That's good for me.
Okay for everyone ?
Regards,
Herve
Signature · 3 lines
Herv? Codina, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 6/15 package/pkg-generic.mk: Introduce <PKG>_PER_PACKAGE_TWEAK_HOOKS
Hi Yann, All
Quoted 11 lines · On Fri, 25 Jun 2021 09:21:20 +0200 Herve Codina <[email protected]>…
Hi,
Quoted 6 lines · On Tue, 22 Jun 2021 22:39:09 +0200 "Yann E. MORIN" <[email protected]>…
package/pkg-generic.mk $(BUILD_DIR)/%/.stamp_configured: · line 260
Quoted 2 lines
package/pkg-generic.mk +1
-0$(call fixup-libtool-files,$(NAME),$(STAGING_DIR)) + $(foreach hook,$($(PKG)_PER_PACKAGE_TWEAK_HOOKS),$(call $(hook))$(sep))I am going to bikeshed on the naming of this variable (which is a good sing that I am pretty much OK with the feature).
Quoted 6 lines
Quoted 1 line
I think $(2)_EARLY_PRE_CONFIGURE_HOOKS is more sensible.
That's good for me.
Okay for everyone ?
The more I read the code, the more I think $(2)_EARLY_PRE_CONFIGURE_HOOKS is not a good name.
Actions done in this hook are for package preparation.
Yann talks about $(2)-prepare step and even if this step is not implemented as a buildroot step (.stamp_xxxxxx) this is really a package preparation.
Indeed, we prepare the package for the next step (retrieving dependencies files for per-package host and target dir) and finally we do the fixups. Once done, the package is ready and so we take the snapshot for overwrite detection.
What do you think about $(2)_POST_PREPARE_HOOKS ?
Regards,
Herv?
Signature · 3 lines
Herv? Codina, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 6/15 package/pkg-generic.mk: Introduce <PKG>_PER_PACKAGE_TWEAK_HOOKS
Herv?, All,
On 2021-07-02 09:18 +0200, Herve Codina spake thusly:
Quoted 15 lines
Quoted 8 lines · On Fri, 25 Jun 2021 09:21:20 +0200 Herve Codina <[email protected]>…
Quoted 5 lines · On Tue, 22 Jun 2021 22:39:09 +0200 "Yann E. MORIN" <[email protected]>…
package/pkg-generic.mk $(BUILD_DIR)/%/.stamp_configured: · line 260
Quoted 2 lines
package/pkg-generic.mk +1
-0$(call fixup-libtool-files,$(NAME),$(STAGING_DIR)) + $(foreach hook,$($(PKG)_PER_PACKAGE_TWEAK_HOOKS),$(call $(hook))$(sep))I am going to bikeshed on the naming of this variable (which is a good sing that I am pretty much OK with the feature).
I think $(2)_EARLY_PRE_CONFIGURE_HOOKS is more sensible.That's good for me.
Actions done in this hook are for package preparation.
Yann talks about $(2)-prepare step and even if this step is not implemented as a buildroot step (.stamp_xxxxxx) this is really a package preparation.
Quoted 6 lines
Indeed, we prepare the package for the next step (retrieving dependencies files for per-package host and target dir) and finally we do the fixups. Once done, the package is ready and so we take the snapshot for overwrite detection.
What do you think about $(2)_POST_PREPARE_HOOKS ?
Works for me.
Regards,
Yann E. MORIN.
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 7/15 package/apr-util: Use <PKG>_PER_PACKAGE_TWEAK_HOOKS
In reply to Herve CodinaOriginal APR_UTIL_FIX_RULES_MK_LIBTOOL tweaked libtool and rules.mk.
libtool is provided by a dependency (apr). It needs to be tweaked and, as an apr-util external file, this tweak is relevant in <PKG>_PER_PACKAGE_TWEAK_HOOKS.
rules.mk is generated by apr-util configure step and it is private to apr-util. The modification performed needs to be kept in <PKG>_POST_CONFIGURE_HOOKS.
This commit splits original APR_UTIL_FIX_RULES_MK_LIBTOOL and attaches each part to the correct hook.
--- package/apr-util/apr-util.mk | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-)
package/apr-util/apr-util.mk +8 -4
@@ -18,13 +18,17 @@ APR_UTIL_CONF_OPTS = \ APR_UTIL_CONFIG_SCRIPTS = apu-1-config ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y) -define APR_UTIL_FIX_RULES_MK_LIBTOOL - $(SED) 's,$(PER_PACKAGE_DIR)/apr/,$(PER_PACKAGE_DIR)/apr-util/,g' \ - $(@D)/build/rules.mk +define APR_UTIL_FIX_LIBTOOL $(SED) 's,$(PER_PACKAGE_DIR)/apr/,$(PER_PACKAGE_DIR)/apr-util/,g' \ $(STAGING_DIR)/usr/build-1/libtool endef -APR_UTIL_POST_CONFIGURE_HOOKS += APR_UTIL_FIX_RULES_MK_LIBTOOL +APR_UTIL_PER_PACKAGE_TWEAK_HOOKS += APR_UTIL_FIX_LIBTOOL + +define APR_UTIL_FIX_RULES_MK + $(SED) 's,$(PER_PACKAGE_DIR)/apr/,$(PER_PACKAGE_DIR)/apr-util/,g' \ + $(@D)/build/rules.mk +endef +APR_UTIL_POST_CONFIGURE_HOOKS += APR_UTIL_FIX_RULES_MK endif # When iconv is available, then use it to provide charset conversion
Signature · 1 line
2.31.1
PATCH 7/15 package/apr-util: Use <PKG>_PER_PACKAGE_TWEAK_HOOKS
Quoted 3 lines · On Mon, 21 Jun 2021 16:11:22 +0200 Herve Codina <[email protected]>…
rules.mk is generated by apr-util configure step and it is private to apr-util. The modification performed needs to be kept in <PKG>_POST_CONFIGURE_HOOKS.
Are you sure ? Both the tweak to build-1/libtool and rules.mk were introduced in commit 84b4c19e551288911a230c2b73e96bc6e2ed12f9 to solve per-package issues. So I would strongly suspect that both of them need to be moved to APR_UTIL_PER_PACKAGE_TWEAK_HOOKS.
Thomas
Signature · 3 lines
Thomas Petazzoni, co-owner and CEO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 7/15 package/apr-util: Use <PKG>_PER_PACKAGE_TWEAK_HOOKS
Hi,
Quoted 12 lines · On Mon, 21 Jun 2021 22:56:32 +0200 Thomas Petazzoni…
Quoted 3 lines · On Mon, 21 Jun 2021 16:11:22 +0200 Herve Codina <[email protected]>…
rules.mk is generated by apr-util configure step and it is private to apr-util. The modification performed needs to be kept in <PKG>_POST_CONFIGURE_HOOKS.
Are you sure ? Both the tweak to build-1/libtool and rules.mk were introduced in commit
84b4c19e551288911a230c2b73e96bc6e2ed12f9to solve per-package issues. So I would strongly suspect that both of them need to be moved toAPR_UTIL_PER_PACKAGE_TWEAK_HOOKS.
Yes, both of them need to be done but
rules.mk is generated by apr-util itself at configure step.
This file is not present (not yet generated) when APR_UTIL_PER_PACKAGE_TWEAK_HOOKS are called.
XXX_PER_PACKAGE_TWEAK_HOOKS are for tweaking files external to current package that leads to overwrite -> Ok for build-1/libtool
rules.mk is not correct for per-package build but it is generated by apr-util package for apr-util package without any overwrites -> Keep tweak in <PKG>_POST_CONFIGURE_HOOKS.
Herv?
Signature · 3 lines
Herv? Codina, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 7/15 package/apr-util: Use <PKG>_PER_PACKAGE_TWEAK_HOOKS
Herv?, All,
On 2021-06-22 11:47 +0200, Herve Codina spake thusly:
Quoted 20 lines
Quoted 12 lines · On Mon, 21 Jun 2021 22:56:32 +0200 Thomas Petazzoni…
Quoted 3 lines · On Mon, 21 Jun 2021 16:11:22 +0200 Herve Codina <[email protected]>…
rules.mk is generated by apr-util configure step and it is private to apr-util. The modification performed needs to be kept in <PKG>_POST_CONFIGURE_HOOKS.
Are you sure ? Both the tweak to build-1/libtool and rules.mk were introduced in commit
84b4c19e551288911a230c2b73e96bc6e2ed12f9to solve per-package issues. So I would strongly suspect that both of them need to be moved toAPR_UTIL_PER_PACKAGE_TWEAK_HOOKS.Yes, both of them need to be done but
rules.mk is generated by apr-util itself at configure step.
This file is not present (not yet generated) whenAPR_UTIL_PER_PACKAGE_TWEAK_HOOKSare called.
Quoted 6 lines
XXX_PER_PACKAGE_TWEAK_HOOKSare for tweaking files external to current package that leads to overwrite -> Ok for build-1/libtoolrules.mk is not correct for per-package build but it is generated by apr-util package for apr-util package without any overwrites -> Keep tweak in <PKG>_POST_CONFIGURE_HOOKS.
Yes, this all makes sense now.
Also, for apr and apr-utils, we are fortunate enough that the tweak is not required at configure time, but at build time, which so far allowed us to use POST_CONFIGURE hooks.
However, for some other packages, we might not be so lucky, and we would have needed pre- and -postconfigure hooks. The pre-hooks can't be used anymore for such tweaks, because of file-overwrite detection, so we need those earlier hook.
Makes sense, makes sense.
So, except for the naming of the variable, when you respin, you can carry my:
Regards,
Yann E. MORIN.
Quoted 10 lines
Herv?
Signature · 7 lines
Herv? Codina, Bootlin Embedded Linux and Kernel engineering https://bootlin.com _______________________________________________ buildroot mailing list buildroot at busybox.net http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 7/15 package/apr-util: Use <PKG>_PER_PACKAGE_TWEAK_HOOKS
Hi,
Quoted 5 lines · On Tue, 22 Jun 2021 22:42:18 +0200 "Yann E. MORIN" <[email protected]>…
So, except for the naming of the variable, when you respin, you can carry my:
Reviewed-by Yann E. MORIN <[email protected]>
This will be done.
Herv?
Signature · 3 lines
Herv? Codina, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 8/15 package/apache: Move APACHE_FIXUP_APR_LIBTOOL to <PKG>_PER_PACKAGE_TWEAK_HOOKS
In reply to Herve CodinaAPACHE_FIXUP_APR_LIBTOOL tweaks files for per package directory build. This is typically the kind of operation expected to be in <PKG>_PER_PACKAGE_TWEAK_HOOKS
--- package/apache/apache.mk | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
package/apache/apache.mk +1 -1
@@ -23,7 +23,7 @@ define APACHE_FIXUP_APR_LIBTOOL $(SED) "s@$(PER_PACKAGE_DIR)/[^/]\+/@$(PER_PACKAGE_DIR)/apache/@g" \ $(STAGING_DIR)/usr/build-1/libtool endef -APACHE_POST_CONFIGURE_HOOKS += APACHE_FIXUP_APR_LIBTOOL +APACHE_PER_PACKAGE_TWEAK_HOOKS += APACHE_FIXUP_APR_LIBTOOL endif APACHE_CONF_ENV= \
Signature · 1 line
2.31.1
PATCH 8/15 package/apache: Move APACHE_FIXUP_APR_LIBTOOL to <PKG>_PER_PACKAGE_TWEAK_HOOKS
Herv?, All,
On 2021-06-21 16:11 +0200, Herve Codina spake thusly:
Quoted 14 lines
APACHE_FIXUP_APR_LIBTOOLtweaks files for per package directory build. This is typically the kind of operation expected to be in <PKG>_PER_PACKAGE_TWEAK_HOOKSSigned-off-by Herve Codina <[email protected]>--- package/apache/apache.mk | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)package/apache/apache.mk +0
-0
package/apache/apache.mk define APACHE_FIXUP_APR_LIBTOOL · line 26
Quoted 5 lines
package/apache/apache.mk +1
-1@@ -23,7 +23,7 @@ define APACHE_FIXUP_APR_LIBTOOL $(SED) "s@$(PER_PACKAGE_DIR)/[^/]\+/@$(PER_PACKAGE_DIR)/apache/@g" \ $(STAGING_DIR)/usr/build-1/libtool endef -APACHE_POST_CONFIGURE_HOOKS += APACHE_FIXUP_APR_LIBTOOL +APACHE_PER_PACKAGE_TWEAK_HOOKS += APACHE_FIXUP_APR_LIBTOOL
Same, with the variable name:
Regards,
Yann E. MORIN.
Quoted 10 lines
package/apache/apache.mk +0
-0endif APACHE_CONF_ENV= \Signature · 6 lines
2.31.1 _______________________________________________ buildroot mailing list buildroot at busybox.net http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 9/15 package/pkg-python: Remove _sysconfigdata*.pyc files when _sysconfigdata*.py are changed
In reply to Herve CodinaIn order to avoid _sysconfigdata*.pyc overwrites when they are generated based on _sysconfigdata*.py changes, this commit simply removes _sysconfigdata*.pyc whenever _sysconfigdata*.py are changed.
As they are removed, overwrite detection will no longer trig and coherency between the two files (.py and .pyc) is ensured.
--- package/pkg-python.mk | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
package/pkg-python.mk +3 -1
@@ -96,7 +96,9 @@ ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y) define PKG_PYTHON_FIXUP_SYSCONFIGDATA find $(HOST_DIR)/lib/python* $(STAGING_DIR)/usr/lib/python* \ -name "_sysconfigdata*.py" | xargs --no-run-if-empty \ - $(SED) "s:$(PER_PACKAGE_DIR)/[^/]\+/:$(PER_PACKAGE_DIR)/$($(PKG)_NAME)/:g" + $(SED) "s:$(PER_PACKAGE_DIR)/[^/]\+/:$(PER_PACKAGE_DIR)/$($(PKG)_NAME)/:g" ;\ + find $(HOST_DIR)/lib/python* $(STAGING_DIR)/usr/lib/python* \ + -name "_sysconfigdata*.pyc" -print0 | xargs -0 -r rm -f endef endif
Signature · 1 line
2.31.1
PATCH 9/15 package/pkg-python: Remove _sysconfigdata*.pyc files when _sysconfigdata*.py are changed
Quoted 17 lines · On Mon, 21 Jun 2021 16:11:24 +0200 Herve Codina <[email protected]>…
In order to avoid _sysconfigdata*.pyc overwrites when they are generated based on _sysconfigdata*.py changes, this commit simply removes _sysconfigdata*.pyc whenever _sysconfigdata*.py are changed.
As they are removed, overwrite detection will no longer trig and coherency between the two files (.py and .pyc) is ensured.
Signed-off-by Herve Codina <[email protected]>--- package/pkg-python.mk | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)package/pkg-python.mk +0
-0
package/pkg-python.mk ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y) · line 99
Quoted 5 lines
package/pkg-python.mk +1
-1@@ -96,7 +96,9 @@ ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y) define PKG_PYTHON_FIXUP_SYSCONFIGDATA find $(HOST_DIR)/lib/python* $(STAGING_DIR)/usr/lib/python* \ -name "_sysconfigdata*.py" | xargs --no-run-if-empty \ - $(SED) "s:$(PER_PACKAGE_DIR)/[^/]\+/:$(PER_PACKAGE_DIR)/$($(PKG)_NAME)/:g" + $(SED) "s:$(PER_PACKAGE_DIR)/[^/]\+/:$(PER_PACKAGE_DIR)/$($(PKG)_NAME)/:g" ;\
The semicolon + backslash is not needed, these two commands can (I believe) be executed as separate shell commands.
package/pkg-python.mk ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y) · line 101
Quoted 2 lines
package/pkg-python.mk +2
-0+ find $(HOST_DIR)/lib/python* $(STAGING_DIR)/usr/lib/python* \ + -name "_sysconfigdata*.pyc" -print0 | xargs -0 -r rm -f
Do we need a --no-run-if-empty on the xargs ? Or perhaps it should be using find directly, i.e:
find $(HOST_DIR)/lib/python* $(STAGING_DIR)/usr/lib/python* \
-name "_sysconfigdata*.pyc" -exec rm -f {} \;(or something like that)
Thanks!
Thomas
Signature · 3 lines
Thomas Petazzoni, co-owner and CEO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 9/15 package/pkg-python: Remove _sysconfigdata*.pyc files when _sysconfigdata*.py are changed
Quoted 15 lines · On Mon, 21 Jun 2021 17:12:02 +0200 Thomas Petazzoni…
Quoted 12 lines · On Mon, 21 Jun 2021 16:11:24 +0200 Herve Codina <[email protected]>…
In order to avoid _sysconfigdata*.pyc overwrites when they are generated based on _sysconfigdata*.py changes, this commit simply removes _sysconfigdata*.pyc whenever _sysconfigdata*.py are changed.
As they are removed, overwrite detection will no longer trig and coherency between the two files (.py and .pyc) is ensured.
Signed-off-by Herve Codina <[email protected]>--- package/pkg-python.mk | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
Quoted 13 lines
package/pkg-python.mk ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y) · line 99
Quoted 10 lines
package/pkg-python.mk +1
-1@@ -96,7 +96,9 @@ ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y) define PKG_PYTHON_FIXUP_SYSCONFIGDATA find $(HOST_DIR)/lib/python* $(STAGING_DIR)/usr/lib/python* \ -name "_sysconfigdata*.py" | xargs --no-run-if-empty \ - $(SED) "s:$(PER_PACKAGE_DIR)/[^/]\+/:$(PER_PACKAGE_DIR)/$($(PKG)_NAME)/:g" + $(SED) "s:$(PER_PACKAGE_DIR)/[^/]\+/:$(PER_PACKAGE_DIR)/$($(PKG)_NAME)/:g" ;\The semicolon + backslash is not needed, these two commands can (I believe) be executed as separate shell commands.
Yes, they can be executed in separate shell commands -> I removed the semicolon and backslash.
Quoted 5 lines
package/pkg-python.mk ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y) · line 101
Quoted 3 lines
package/pkg-python.mk +2
-0+ find $(HOST_DIR)/lib/python* $(STAGING_DIR)/usr/lib/python* \ + -name "_sysconfigdata*.pyc" -print0 | xargs -0 -r rm -fDo we need a --no-run-if-empty on the xargs ? Or perhaps it should be
It is cleaner to avoid running rm is there is nothing to remove. So, I prefer keeping the xargs -r option.
Quoted 6 lines
using find directly, i.e:
find $(HOST_DIR)/lib/python* $(STAGING_DIR)/usr/lib/python* \ -name "_sysconfigdata*.pyc" -exec rm -f {} \;(or something like that)
Using "-exec rm -f {} \;" will lead to start as many rm as there are files to remove whereas xargs will run only one rm.
Herv?
Signature · 3 lines
Herv? Codina, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 9/15 package/pkg-python: Remove _sysconfigdata*.pyc files when _sysconfigdata*.py are changed
Herv?, All,
On 2021-06-22 19:52 +0200, Herve Codina spake thusly:
Quoted 16 lines
Quoted 14 lines · On Mon, 21 Jun 2021 17:12:02 +0200 Thomas Petazzoni…
Quoted 12 lines · On Mon, 21 Jun 2021 16:11:24 +0200 Herve Codina <[email protected]>…
In order to avoid _sysconfigdata*.pyc overwrites when they are generated based on _sysconfigdata*.py changes, this commit simply removes _sysconfigdata*.pyc whenever _sysconfigdata*.py are changed.
As they are removed, overwrite detection will no longer trig and coherency between the two files (.py and .pyc) is ensured.
Signed-off-by Herve Codina <[email protected]>--- package/pkg-python.mk | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
Quoted 10 lines
Quoted 10 lines
package/pkg-python.mk ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y) · line 99
Quoted 10 lines
package/pkg-python.mk +1
-1@@ -96,7 +96,9 @@ ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y) define PKG_PYTHON_FIXUP_SYSCONFIGDATA find $(HOST_DIR)/lib/python* $(STAGING_DIR)/usr/lib/python* \ -name "_sysconfigdata*.py" | xargs --no-run-if-empty \ - $(SED) "s:$(PER_PACKAGE_DIR)/[^/]\+/:$(PER_PACKAGE_DIR)/$($(PKG)_NAME)/:g" + $(SED) "s:$(PER_PACKAGE_DIR)/[^/]\+/:$(PER_PACKAGE_DIR)/$($(PKG)_NAME)/:g" ;\
[--SNIP--]
Quoted 8 lines
Quoted 3 lines
package/pkg-python.mk ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y) · line 101
Quoted 2 lines
package/pkg-python.mk +2
-0+ find $(HOST_DIR)/lib/python* $(STAGING_DIR)/usr/lib/python* \ + -name "_sysconfigdata*.pyc" -print0 | xargs -0 -r rm -fDo we need a --no-run-if-empty on the xargs ? Or perhaps it should be
It is cleaner to avoid running rm is there is nothing to remove. So, I prefer keeping the xargs -r option.
Quoted 2 lines
using find directly, i.e:
Quoted 7 lines
Quoted 4 lines
find $(HOST_DIR)/lib/python* $(STAGING_DIR)/usr/lib/python* \ -name "_sysconfigdata*.pyc" -exec rm -f {} \;(or something like that)
Using "-exec rm -f {} \;" will lead to start as many rm as there are files to remove whereas xargs will run only one rm.
So, 1: wrong. xargs may have to exec more than one rm if there are too many arguments, or the command line is too long. But of course, this is inconsequential, because even on a huge build, this would be just a few exec instead of tens of thousands.
And 2: why not using find's -delete option to begin with:
find $(HOST_DIR)/lib/python* $(STAGING_DIR)/usr/lib/python* \
-name "_sysconfigdata*.pyc" -delete
There, not even a pipe, or an exec of anything else...
Regards,
Yann E. MORIN.
Quoted 11 lines
Herv?
Signature · 7 lines
Herv? Codina, Bootlin Embedded Linux and Kernel engineering https://bootlin.com _______________________________________________ buildroot mailing list buildroot at busybox.net http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 9/15 package/pkg-python: Remove _sysconfigdata*.pyc files when _sysconfigdata*.py are changed
Hi,
Quoted 7 lines · On Tue, 22 Jun 2021 22:50:40 +0200 "Yann E. MORIN" <[email protected]>…
And 2: why not using find's -delete option to begin with:
find $(HOST_DIR)/lib/python* $(STAGING_DIR)/usr/lib/python* \
-name "_sysconfigdata*.pyc" -deleteThere, not even a pipe, or an exec of anything else...
Yes, indeed.
I just didn't think about '-delete' when I wrote the "find ...".
Changed to '-delete' option.
Herv?
Signature · 3 lines
Herv? Codina, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 10/15 package/pkg-generic.mk: Move python fixup to generic package infrastructure
In reply to Herve CodinaFixing _sysconfigdata*.{py,pyc} was previously done by python package infrastructure. Some packages use python stuff without using python package infrastructure.
These packages perform overwrites and need the specific python fixup to fix them.
In order to be sure to fix all of these packages, the python fixup is moved to the generic package infrastructure and applied to all packages.
This follows the same principle as for the .la libtool files fixup.
--- package/pkg-generic.mk | 17 +++++++++++++++++ package/pkg-python.mk | 12 ------------ 2 files changed, 17 insertions(+), 12 deletions(-)
package/pkg-generic.mk +17 -0
@@ -103,6 +103,21 @@ define fixup-libtool-files endef endif +# Make sure python _sysconfigdata*.py files only reference the current +# per-package directory. + +# $1: package name (lower case) +# $2: staging or host directory of the package +ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y) +define fixup-python-files + $(Q)find $(2) \( -path '$(2)/lib/python*' -o -path '$(2)/usr/lib/python*' \) \ + -name "_sysconfigdata*.py" | xargs --no-run-if-empty \ + $(SED) "s:$(PER_PACKAGE_DIR)/[^/]\+/:$(PER_PACKAGE_DIR)/$(1)/:g" ;\ + find $(2) \( -path '$(2)/lib/python*' -o -path '$(2)/usr/lib/python*' \) \ + -name "_sysconfigdata*.pyc" -print0 | xargs -0 -r rm -f +endef +endif + # Functions to detect overwritten files ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y)
@@ -257,6 +272,8 @@ $(BUILD_DIR)/%/.stamp_configured: @$(call pkg_size_before,$(HOST_DIR),-host) $(call fixup-libtool-files,$(NAME),$(HOST_DIR)) $(call fixup-libtool-files,$(NAME),$(STAGING_DIR)) + $(call fixup-python-files,$(NAME),$(HOST_DIR)) + $(call fixup-python-files,$(NAME),$(STAGING_DIR)) $(foreach hook,$($(PKG)_PER_PACKAGE_TWEAK_HOOKS),$(call $(hook))$(sep)) @$(call pkg_detect_overwrite_before,$(TARGET_DIR)) @$(call pkg_detect_overwrite_before,$(HOST_DIR),-host)
package/pkg-python.mk +0 -12
@@ -92,16 +92,6 @@ HOST_PKG_PYTHON_SETUPTOOLS_INSTALL_OPTS = \ --root=/ \ --single-version-externally-managed -ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y) -define PKG_PYTHON_FIXUP_SYSCONFIGDATA - find $(HOST_DIR)/lib/python* $(STAGING_DIR)/usr/lib/python* \ - -name "_sysconfigdata*.py" | xargs --no-run-if-empty \ - $(SED) "s:$(PER_PACKAGE_DIR)/[^/]\+/:$(PER_PACKAGE_DIR)/$($(PKG)_NAME)/:g" ;\ - find $(HOST_DIR)/lib/python* $(STAGING_DIR)/usr/lib/python* \ - -name "_sysconfigdata*.pyc" -print0 | xargs -0 -r rm -f -endef -endif - ################################################################################ # inner-python-package -- defines how the configuration, compilation # and installation of a Python package should be done, implements a
@@ -246,8 +236,6 @@ $(2)_PYTHON_INTERPRETER = $$(HOST_DIR)/bin/$$($(2)_NEEDS_HOST_PYTHON) endif endif -$(2)_PRE_CONFIGURE_HOOKS += PKG_PYTHON_FIXUP_SYSCONFIGDATA - # # Build step. Only define it if not already defined by the package .mk # file.
Signature · 1 line
2.31.1
PATCH 10/15 package/pkg-generic.mk: Move python fixup to generic package infrastructure
Herv?, All,
On 2021-06-21 16:11 +0200, Herve Codina spake thusly:
Quoted 44 lines
Fixing _sysconfigdata*.{py,pyc} was previously done by python package infrastructure. Some packages use python stuff without using python package infrastructure.
These packages perform overwrites and need the specific python fixup to fix them.In order to be sure to fix all of these packages, the python fixup is moved to the generic package infrastructure and applied to all packages.
This follows the same principle as for the .la libtool files fixup.Signed-off-by Herve Codina <[email protected]>--- package/pkg-generic.mk | 17 +++++++++++++++++ package/pkg-python.mk | 12 ------------ 2 files changed, 17 insertions(+), 12 deletions(-)package/pkg-generic.mk +15
-0@@ -103,6 +103,21 @@ define fixup-libtool-files endef endif +# Make sure python _sysconfigdata*.py files only reference the current +# per-package directory. + +# $1: package name (lower case) +# $2: staging or host directory of the package +ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y) +define fixup-python-files + $(Q)find $(2) \( -path '$(2)/lib/python*' -o -path '$(2)/usr/lib/python*' \) \ + -name "_sysconfigdata*.py" | xargs --no-run-if-empty \ + $(SED) "s:$(PER_PACKAGE_DIR)/[^/]\+/:$(PER_PACKAGE_DIR)/$(1)/:g" ;\ + find $(2) \( -path '$(2)/lib/python*' -o -path '$(2)/usr/lib/python*' \) \ + -name "_sysconfigdata*.pyc" -print0 | xargs -0 -r rm -f +endef +endif + # Functions to detect overwritten files ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y)
package/pkg-generic.mk $(BUILD_DIR)/%/.stamp_configured: · line 277
Quoted 6 lines
package/pkg-generic.mk +2
-0@@ -257,6 +272,8 @@ $(BUILD_DIR)/%/.stamp_configured: @$(call pkg_size_before,$(HOST_DIR),-host) $(call fixup-libtool-files,$(NAME),$(HOST_DIR)) $(call fixup-libtool-files,$(NAME),$(STAGING_DIR)) + $(call fixup-python-files,$(NAME),$(HOST_DIR)) + $(call fixup-python-files,$(NAME),$(STAGING_DIR)) $(foreach hook,$($(PKG)_PER_PACKAGE_TWEAK_HOOKS),$(call $(hook))$(sep))
So, earlier in the series, you introduced _PER_PACKAGE_TWEAK_HOOKS (or whatever the name it will eventually bear), stating that infras could set it, but here you are not taking the chance to demonstrate this, and instead you are inserting the python fixups in an ad-hoc way.
Which also illustrates that the libtool fixups should be converted to the _PER_PACKAGE_TWEAK_HOOKS, probably...
Otherwise, this is a good change.
Regards,
Yann E. MORIN.
Quoted 39 lines
package/pkg-generic.mk +0
-0@$(call pkg_detect_overwrite_before,$(TARGET_DIR)) @$(call pkg_detect_overwrite_before,$(HOST_DIR),-host)package/pkg-python.mk +0
-12@@ -92,16 +92,6 @@ HOST_PKG_PYTHON_SETUPTOOLS_INSTALL_OPTS = \ --root=/ \ --single-version-externally-managed -ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y) -define PKG_PYTHON_FIXUP_SYSCONFIGDATA - find $(HOST_DIR)/lib/python* $(STAGING_DIR)/usr/lib/python* \ - -name "_sysconfigdata*.py" | xargs --no-run-if-empty \ - $(SED) "s:$(PER_PACKAGE_DIR)/[^/]\+/:$(PER_PACKAGE_DIR)/$($(PKG)_NAME)/:g" ;\ - find $(HOST_DIR)/lib/python* $(STAGING_DIR)/usr/lib/python* \ - -name "_sysconfigdata*.pyc" -print0 | xargs -0 -r rm -f -endef -endif - ################################################################################ # inner-python-package -- defines how the configuration, compilation # and installation of a Python package should be done, implements a@@ -246,8 +236,6 @@ $(2)_PYTHON_INTERPRETER = $$(HOST_DIR)/bin/$$($(2)_NEEDS_HOST_PYTHON) endif endif -$(2)_PRE_CONFIGURE_HOOKS += PKG_PYTHON_FIXUP_SYSCONFIGDATA - # # Build step. Only define it if not already defined by the package .mk # file.Signature · 6 lines
2.31.1 _______________________________________________ buildroot mailing list buildroot at busybox.net http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 10/15 package/pkg-generic.mk: Move python fixup to generic package infrastructure
Hi,
Quoted 8 lines · On Tue, 22 Jun 2021 23:01:03 +0200 "Yann E. MORIN" <[email protected]>…
So, earlier in the series, you introduced
_PER_PACKAGE_TWEAK_HOOKS(or whatever the name it will eventually bear), stating that infras could set it, but here you are not taking the chance to demonstrate this, and instead you are inserting the python fixups in an ad-hoc way.Which also illustrates that the libtool fixups should be converted to the
_PER_PACKAGE_TWEAK_HOOKS, probably...
Yes that's perfectly true.
But do you think we need it right now in this patch series. I really would prefer to keep it as it is for this patch series.
The series is already quite complex and I think that converting libtool and python fixups to PER_PACKAGE_TWEAK_HOOKS (or other name) can be done afterward.
Herve
Signature · 3 lines
Herv? Codina, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 10/15 package/pkg-generic.mk: Move python fixup to generic package infrastructure
Herv?, All,
On 2021-06-25 10:22 +0200, Herve Codina spake thusly:
Quoted 9 lines
Quoted 7 lines · On Tue, 22 Jun 2021 23:01:03 +0200 "Yann E. MORIN" <[email protected]>…
So, earlier in the series, you introduced
_PER_PACKAGE_TWEAK_HOOKS(or whatever the name it will eventually bear), stating that infras could set it, but here you are not taking the chance to demonstrate this, and instead you are inserting the python fixups in an ad-hoc way.Which also illustrates that the libtool fixups should be converted to the
_PER_PACKAGE_TWEAK_HOOKS, probably...
Quoted 8 lines
Yes that's perfectly true.
But do you think we need it right now in this patch series. I really would prefer to keep it as it is for this patch series.
The series is already quite complex and I think that converting libtool and python fixups to
PER_PACKAGE_TWEAK_HOOKS(or other name) can be done afterward.
I am always a bit wary about the "cleanup can be done in a later pass" excuse, because that later pass almost never manifests.
But still, the issue is real, and we better have a solution rather than nothing, and we better have it earlier rather than later. So yes, keep things as they are.
Regards,
Yann E. MORIN, first-shoted in the morning!
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 11/15 package/owfs: Remove Python sysconfigdata fixup
In reply to Herve CodinaThis fixup is done at pkg-generic level for all packages. So, it is no more needed in owfs package.
--- package/owfs/owfs.mk | 9 --------- 1 file changed, 9 deletions(-)
package/owfs/owfs.mk +0 -9
@@ -88,15 +88,6 @@ OWFS_DEPENDENCIES += python host-swig # override the PYSITEDIR variable in make. OWFS_EXTRA_MAKE_OPTS += PYSITEDIR=/usr/lib/python$(PYTHON_VERSION_MAJOR)/site-packages -ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y) -define OWFS_FIXUP_PYTHON_SYSCONFIGDATA - find $(HOST_DIR)/lib/python* $(STAGING_DIR)/usr/lib/python* \ - -name "_sysconfigdata*.py" | xargs --no-run-if-empty \ - $(SED) "s:$(PER_PACKAGE_DIR)/[^/]\+/:$(PER_PACKAGE_DIR)/owfs/:g" -endef -OWFS_PRE_CONFIGURE_HOOKS += OWFS_FIXUP_PYTHON_SYSCONFIGDATA -endif - else OWFS_CONF_OPTS += --disable-owpython --without-python endif
Signature · 1 line
2.31.1
PATCH 11/15 package/owfs: Remove Python sysconfigdata fixup
Herv?, All,
On 2021-06-21 16:11 +0200, Herve Codina spake thusly:
Quoted 4 lines
This fixup is done at pkg-generic level for all packages. So, it is no more needed in owfs package.
Signed-off-by Herve Codina <[email protected]>
When you respin the series, you can carry my:
Regards,
Yann E. MORIN.
Quoted 31 lines
--- package/owfs/owfs.mk | 9 --------- 1 file changed, 9 deletions(-)package/owfs/owfs.mk +0
-9@@ -88,15 +88,6 @@ OWFS_DEPENDENCIES += python host-swig # override the PYSITEDIR variable in make. OWFS_EXTRA_MAKE_OPTS += PYSITEDIR=/usr/lib/python$(PYTHON_VERSION_MAJOR)/site-packages -ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y) -define OWFS_FIXUP_PYTHON_SYSCONFIGDATA - find $(HOST_DIR)/lib/python* $(STAGING_DIR)/usr/lib/python* \ - -name "_sysconfigdata*.py" | xargs --no-run-if-empty \ - $(SED) "s:$(PER_PACKAGE_DIR)/[^/]\+/:$(PER_PACKAGE_DIR)/owfs/:g" -endef -OWFS_PRE_CONFIGURE_HOOKS += OWFS_FIXUP_PYTHON_SYSCONFIGDATA -endif - else OWFS_CONF_OPTS += --disable-owpython --without-python endifSignature · 6 lines
2.31.1 _______________________________________________ buildroot mailing list buildroot at busybox.net http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 11/15 package/owfs: Remove Python sysconfigdata fixup
Quoted 12 lines · On Tue, 22 Jun 2021 23:02:26 +0200 "Yann E. MORIN" <[email protected]>…
Herv?, All,
On 2021-06-21 16:11 +0200, Herve Codina spake thusly:
Quoted 4 lines
This fixup is done at pkg-generic level for all packages. So, it is no more needed in owfs package.
Signed-off-by Herve Codina <[email protected]>When you respin the series, you can carry my:
Reviewed-by Yann E. MORIN <[email protected]>
This will be done.
Herv?
Signature · 3 lines
Herv? Codina, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 12/15 package/pkg-generic.mk: Generate final rsync exclude file list
In reply to Herve CodinaThe final rsync performed at {host,target}-finalize steps need to be done by rsyncing files generated by each packages without looking at files generated by other packages the current package depends on. This is needed to avoid overwrites in final {HOST,TARGET}_DIR.
In order to prepare the final rsync, an exclusion list is generated. This list lists files that are not generated by the current package and so files that need to be excluded from the final rsync.
Note also that the files list was not based on .files-list.{before,after}. During .file-list.{before,after} built for host directory, staging sub-directory (ie <toolchain>/sysroot) is filtered out.
The final rsync exclusion list needs to take into account the full {host,target} directory to avoid final overwrites.
--- package/pkg-generic.mk | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+)
package/pkg-generic.mk +28 -0
@@ -162,6 +162,30 @@ define pkg_size_after rm -f $($(PKG)_DIR)/.files-list$(2).after endef +# Functions to collect final rsync exclusion files + +# $(1): base directory to search in +# $(2): suffix of file (optional) +define pkg_final_rsync_before + cd $(1); \ + LC_ALL=C find . \( -type f -o -type l \) -printf '%T@:%i:%#m:%y:%s,%p\n' \ + | LC_ALL=C sort > $($(PKG)_DIR)/.files-final-rsync$(2).before +endef + +# $(1): base directory to search in +# $(2): suffix of file (optional) +define pkg_final_rsync_after + cd $(1); \ + LC_ALL=C find . \( -type f -o -type l \) -printf '%T@:%i:%#m:%y:%s,%p\n' \ + | LC_ALL=C sort > $($(PKG)_DIR)/.files-final-rsync$(2).after + LC_ALL=C comm -2 \ + $($(PKG)_DIR)/.files-final-rsync$(2).before \ + $($(PKG)_DIR)/.files-final-rsync$(2).after \ + | sed -r -e 's/^[^,]+,./- /' \ + > $($(PKG)_DIR)/.files-final-rsync$(2).exclude_rsync + rm -f $($(PKG)_DIR)/.files-final-rsync$(2).after +endef + define check_bin_arch support/scripts/check-bin-arch -p $($(PKG)_NAME) \ -l $($(PKG)_DIR)/.files-list.txt \
@@ -267,6 +291,8 @@ $(BUILD_DIR)/%/.stamp_configured: @$(call MESSAGE,"Configuring") $(Q)mkdir -p $(HOST_DIR) $(TARGET_DIR) $(STAGING_DIR) $(BINARIES_DIR) $(call prepare-per-package-directory,$($(PKG)_FINAL_DEPENDENCIES)) + @$(call pkg_final_rsync_before,$(TARGET_DIR)) + @$(call pkg_final_rsync_before,$(HOST_DIR),-host) @$(call pkg_size_before,$(TARGET_DIR)) @$(call pkg_size_before,$(STAGING_DIR),-staging) @$(call pkg_size_before,$(HOST_DIR),-host)
@@ -404,6 +430,8 @@ $(BUILD_DIR)/%/.stamp_installed: @$(call pkg_size_after,$(STAGING_DIR),-staging) @$(call pkg_size_after,$(HOST_DIR),-host) @$(call check_bin_arch) + @$(call pkg_final_rsync_after,$(TARGET_DIR)) + @$(call pkg_final_rsync_after,$(HOST_DIR),-host) @$(call pkg_detect_overwrite_after,$(TARGET_DIR)) @$(call pkg_detect_overwrite_after,$(HOST_DIR),-host) $(Q)touch $@
Signature · 1 line
2.31.1
PATCH 12/15 package/pkg-generic.mk: Generate final rsync exclude file list
Herv?, All,
On 2021-06-21 16:11 +0200, Herve Codina spake thusly:
Quoted 10 lines
The final rsync performed at {host,target}-finalize steps need to be done by rsyncing files generated by each packages without looking at files generated by other packages the current package depends on. This is needed to avoid overwrites in final {HOST,TARGET}_DIR.
In order to prepare the final rsync, an exclusion list is generated. This list lists files that are not generated by the current package and so files that need to be excluded from the final rsync.
Quoted 5 lines
Note also that the files list was not based on .files-list.{before,after}. During .file-list.{before,after} built for host directory, staging sub-directory (ie <toolchain>/sysroot) is filtered out.
The final rsync exclusion list needs to take into account the full {host,target} directory to avoid final overwrites.
Ideally, we would in fact redirect the package installation to an emoty directory, so that we can actually find what it installs.
However, as has been discussed in the past, this is fraught with unworkable issues. For example, some paths may be hard-coded at configure time and/or build time, and thus the package would still install in the original stagin we presented it with (for target/, this is not an isue, because target/ is never looked at during configure or build, only at install time). Or a package may try to modify an existing file (bad, but still). Or a file just assumes that tdirectory structure exists (bad, but eh...)
So, yes, the rsync exclusion list is a good workaround.
I see you have provided detailed comit logs, that is great. Still, for such core stuff, I think they still miss the bigger picture, like I explained above for example, and which should have been part of the commit log to explain why we resort to an exclusion list rather than the more obvious and simple empty-DESTDIR.
Quoted 25 lines
Signed-off-by Herve Codina <[email protected]>--- package/pkg-generic.mk | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+)package/pkg-generic.mk +12
-0@@ -162,6 +162,30 @@ define pkg_size_after rm -f $($(PKG)_DIR)/.files-list$(2).after endef +# Functions to collect final rsync exclusion files + +# $(1): base directory to search in +# $(2): suffix of file (optional) +define pkg_final_rsync_before + cd $(1); \ + LC_ALL=C find . \( -type f -o -type l \) -printf '%T@:%i:%#m:%y:%s,%p\n' \ + | LC_ALL=C sort > $($(PKG)_DIR)/.files-final-rsync$(2).before +endef + +# $(1): base directory to search in +# $(2): suffix of file (optional)
package/pkg-generic.mk define pkg_size_after · line 186
Quoted 10 lines
package/pkg-generic.mk +10
-0+define pkg_final_rsync_after + cd $(1); \ + LC_ALL=C find . \( -type f -o -type l \) -printf '%T@:%i:%#m:%y:%s,%p\n' \ + | LC_ALL=C sort > $($(PKG)_DIR)/.files-final-rsync$(2).after + LC_ALL=C comm -2 \ + $($(PKG)_DIR)/.files-final-rsync$(2).before \ + $($(PKG)_DIR)/.files-final-rsync$(2).after \ + | sed -r -e 's/^[^,]+,./- /' \ + > $($(PKG)_DIR)/.files-final-rsync$(2).exclude_rsync + rm -f $($(PKG)_DIR)/.files-final-rsync$(2).after
You forgot to remove .files-final-rsync$(2).before
package/pkg-generic.mk $(BUILD_DIR)/%/.stamp_configured: · line 295
Quoted 11 lines
package/pkg-generic.mk +4
-0+endef + define check_bin_arch support/scripts/check-bin-arch -p $($(PKG)_NAME) \ -l $($(PKG)_DIR)/.files-list.txt \@@ -267,6 +291,8 @@ $(BUILD_DIR)/%/.stamp_configured: @$(call MESSAGE,"Configuring") $(Q)mkdir -p $(HOST_DIR) $(TARGET_DIR) $(STAGING_DIR) $(BINARIES_DIR) $(call prepare-per-package-directory,$($(PKG)_FINAL_DEPENDENCIES)) + @$(call pkg_final_rsync_before,$(TARGET_DIR)) + @$(call pkg_final_rsync_before,$(HOST_DIR),-host)
The more I look at this, the more I believe we should resurrect the idea of a 'prepare' step that goes in-between 'patch' and 'configure'...
Regards,
Yann E. MORIN.
Quoted 19 lines
package/pkg-generic.mk +2
-0@$(call pkg_size_before,$(TARGET_DIR)) @$(call pkg_size_before,$(STAGING_DIR),-staging) @$(call pkg_size_before,$(HOST_DIR),-host)@@ -404,6 +430,8 @@ $(BUILD_DIR)/%/.stamp_installed: @$(call pkg_size_after,$(STAGING_DIR),-staging) @$(call pkg_size_after,$(HOST_DIR),-host) @$(call check_bin_arch) + @$(call pkg_final_rsync_after,$(TARGET_DIR)) + @$(call pkg_final_rsync_after,$(HOST_DIR),-host) @$(call pkg_detect_overwrite_after,$(TARGET_DIR)) @$(call pkg_detect_overwrite_after,$(HOST_DIR),-host) $(Q)touch $@Signature · 6 lines
2.31.1 _______________________________________________ buildroot mailing list buildroot at busybox.net http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 12/15 package/pkg-generic.mk: Generate final rsync exclude file list
Hi,
Quoted 11 lines · On Tue, 22 Jun 2021 23:15:10 +0200 "Yann E. MORIN" <[email protected]>…
Ideally, we would in fact redirect the package installation to an emoty directory, so that we can actually find what it installs.
However, as has been discussed in the past, this is fraught with unworkable issues. For example, some paths may be hard-coded at configure time and/or build time, and thus the package would still install in the original stagin we presented it with (for target/, this is not an isue, because target/ is never looked at during configure or build, only at install time). Or a package may try to modify an existing file (bad, but still). Or a file just assumes that tdirectory structure exists (bad, but eh...)
Modifying existing files is no more allowed -> Overwrites detection.
Quoted 9 lines
So, yes, the rsync exclusion list is a good workaround.
I see you have provided detailed comit logs, that is great. Still, for such core stuff, I think they still miss the bigger picture, like I explained above for example, and which should have been part of the commit log to explain why we resort to an exclusion list rather than the more obvious and simple empty-DESTDIR.
I can add at the end of commit log :
--- 8< ---
Using an empty directory for per-package installation directory would be the simplest way to find what a package installs.
However, as it has been discussed in the past, this is fraught with unworkable issues. For example, some paths may be hard-coded at configure time and/or build time, and thus the package would still install in the original stagin we presented it with (for target/, this is not an isue, because target/ is never looked at during configure or build, only@install time). Or a package installation process just assumes that the directory structure exists (bad, but eh...). --- 8< ---
Is that ok for you ?
Quoted 4 lines
Quoted 4 lines
package/pkg-generic.mk +4
-0+define pkg_final_rsync_after + cd $(1); \ + LC_ALL=C find . \( -type f -o -type l \) -printf '%T@:%i:%#m:%y:%s,%p\n' \ + | LC_ALL=C sort > $($(PKG)_DIR)/.files-final-rsync$(2).after
Quoted 8 lines
package/pkg-generic.mk define pkg_size_after · line 186
Quoted 6 lines
package/pkg-generic.mk +6
-0+ LC_ALL=C comm -2 \ + $($(PKG)_DIR)/.files-final-rsync$(2).before \ + $($(PKG)_DIR)/.files-final-rsync$(2).after \ + | sed -r -e 's/^[^,]+,./- /' \ + > $($(PKG)_DIR)/.files-final-rsync$(2).exclude_rsync + rm -f $($(PKG)_DIR)/.files-final-rsync$(2).afterYou forgot to remove .files-final-rsync$(2).before
I cannot remove it in this patch.
.files-final-rsync$(2).before is generated by .stamp_configured rule calling pkg_final_rsync_before call and if we run 'make && make foo-rebuild' the file generated by first make invocation will be no more present for second make invocation.
We can remove .files-final-rsync$(2).before only after patch "package/pkg-generic.mk: Fix per-package <pkg>-{reconfigure,rebuild,reinstall}" Indeed, in this patch, pkg_final_rsync_after is called after the per-package host and target dir are recreated from scratch.
I will remove .files-final-rsync$(2).before in a dedicated patch after "package/pkg-generic.mk: Fix per-package <pkg>-{reconfigure,rebuild,reinstall}"
Is that ok for you ?
Herv?
Signature · 3 lines
Herv? Codina, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 12/15 package/pkg-generic.mk: Generate final rsync exclude file list
Herv?, All,
On 2021-06-25 11:05 +0200, Herve Codina spake thusly:
Quoted 13 lines
Quoted 11 lines · On Tue, 22 Jun 2021 23:15:10 +0200 "Yann E. MORIN" <[email protected]>…
Ideally, we would in fact redirect the package installation to an emoty directory, so that we can actually find what it installs.
However, as has been discussed in the past, this is fraught with unworkable issues. For example, some paths may be hard-coded at configure time and/or build time, and thus the package would still install in the original stagin we presented it with (for target/, this is not an isue, because target/ is never looked at during configure or build, only at install time). Or a package may try to modify an existing file (bad, but still). Or a file just assumes that tdirectory structure exists (bad, but eh...)
Quoted 1 line
Modifying existing files is no more allowed -> Overwrites detection.
Exactly. But what I mean is that, for packages that really need to modify an existing file (e.g. to register themselves against another package's registry), providing an empty target/ is not viable. Yes, that will trigger the overwrite-detection, and we will have to find a proper solution for those. Just, installing in an empty target would not work.
So yes, the list of installed files is still the best solution.
Quoted 12 lines
I can add at the end of commit log :
--- 8< ---
Using an empty directory for per-package installation directory would be the simplest way to find what a package installs.
However, as it has been discussed in the past, this is fraught with unworkable issues. For example, some paths may be hard-coded at configure time and/or build time, and thus the package would still install in the original stagin we presented it with (for target/, this is not an isue, because target/ is never looked at during configure or build, only at install time). Or a package installation process just assumes that the directory structure exists (bad, but eh...). --- 8< ---
Quoted 1 line
Is that ok for you ?
Yes. The registration thing is only just a corner case, so needs not be in the commit log.
Quoted 13 lines
Quoted 4 lines
Quoted 4 lines
package/pkg-generic.mk +4
-0+define pkg_final_rsync_after + cd $(1); \ + LC_ALL=C find . \( -type f -o -type l \) -printf '%T@:%i:%#m:%y:%s,%p\n' \ + | LC_ALL=C sort > $($(PKG)_DIR)/.files-final-rsync$(2).afterQuoted 8 lines
package/pkg-generic.mk define pkg_size_after · line 186
Quoted 6 lines
package/pkg-generic.mk +6
-0+ LC_ALL=C comm -2 \ + $($(PKG)_DIR)/.files-final-rsync$(2).before \ + $($(PKG)_DIR)/.files-final-rsync$(2).after \ + | sed -r -e 's/^[^,]+,./- /' \ + > $($(PKG)_DIR)/.files-final-rsync$(2).exclude_rsync + rm -f $($(PKG)_DIR)/.files-final-rsync$(2).afterYou forgot to remove .files-final-rsync$(2).before
Quoted 6 lines
I cannot remove it in this patch.
.files-final-rsync$(2).before is generated by .stamp_configured rule calling
pkg_final_rsync_beforecall and if we run 'make && make foo-rebuild' the file generated by first make invocation will be no more present for second make invocation.
Ah, yes, I see now. Thanks.
Quoted 9 lines
We can remove .files-final-rsync$(2).before only after patch "
package/pkg-generic.mk: Fix per-package <pkg>-{reconfigure,rebuild,reinstall}" Indeed, in this patch,pkg_final_rsync_afteris called after the per-package host and target dir are recreated from scratch.I will remove .files-final-rsync$(2).before in a dedicated patch after "
package/pkg-generic.mk: Fix per-package <pkg>-{reconfigure,rebuild,reinstall}"Is that ok for you ?
Yes, it is. ?
Regards,
Yann E. MORIN.
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 13/15 Makefile: Rsync global {TARGET, HOST}_DIR using exclusion file list
In reply to Herve CodinaOn a per-package build, rsync final {TARGET,HOST}_DIR using exclusion file list computed for each packages.
As we rsync only files generated by each packages, we need to to do this rsync recursively on each dependencies to collect all needed files. This is done based on existing <PKG>_FINAL_RECURSIVE_DEPENDENCIES
--- Makefile | 24 ++++++++++++++++++++++-- 1 file changed, 22 insertions(+), 2 deletions(-)
Makefile +22 -2
@@ -740,10 +740,28 @@ TARGET_DIR_FILES_LISTS = $(sort $(wildcard $(BUILD_DIR)/*/.files-list.txt)) HOST_DIR_FILES_LISTS = $(sort $(wildcard $(BUILD_DIR)/*/.files-list-host.txt)) STAGING_DIR_FILES_LISTS = $(sort $(wildcard $(BUILD_DIR)/*/.files-list-staging.txt)) +ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y) +# rsync the contents of per-package directories +# $1: space-separated list of packages to rsync from +# $2: 'host' or 'target' +# $3: destination directory +# $4: exclude file list to use +define per-package-rsync-delta + mkdir -p $(3) + $(foreach pkg,$(1),\ + rsync -a --link-dest=$(PER_PACKAGE_DIR)/$(pkg)/$(strip $(2))/ \ + --filter='merge $($(call UPPERCASE,$(pkg))_DIR)/$(strip $(4))' \ + $(PER_PACKAGE_DIR)/$(pkg)/$(strip $(2))/ \ + $(3)$(sep)) +endef +endif + .PHONY: host-finalize host-finalize: $(PACKAGES) $(HOST_DIR) $(HOST_DIR_SYMLINK) @$(call MESSAGE,"Finalizing host directory") - $(call per-package-rsync,$(sort $(PACKAGES)),host,$(HOST_DIR)) + $(call per-package-rsync-delta, $(sort $(foreach pkg, $(PACKAGES), \ + $(pkg) $($(call UPPERCASE,$(pkg))_FINAL_RECURSIVE_DEPENDENCIES))), \ + host,$(HOST_DIR),.files-final-rsync-host.exclude_rsync) .PHONY: staging-finalize staging-finalize: $(STAGING_DIR_SYMLINK)
@@ -751,7 +769,9 @@ staging-finalize: $(STAGING_DIR_SYMLINK) .PHONY: target-finalize target-finalize: $(PACKAGES) $(TARGET_DIR) host-finalize @$(call MESSAGE,"Finalizing target directory") - $(call per-package-rsync,$(sort $(PACKAGES)),target,$(TARGET_DIR)) + $(call per-package-rsync-delta, $(sort $(foreach pkg, $(PACKAGES), \ + $(pkg) $($(call UPPERCASE,$(pkg))_FINAL_RECURSIVE_DEPENDENCIES))), \ + target,$(TARGET_DIR),.files-final-rsync.exclude_rsync) $(foreach hook,$(TARGET_FINALIZE_HOOKS),$($(hook))$(sep)) rm -rf $(TARGET_DIR)/usr/include $(TARGET_DIR)/usr/share/aclocal \ $(TARGET_DIR)/usr/lib/pkgconfig $(TARGET_DIR)/usr/share/pkgconfig \
Signature · 1 line
2.31.1
PATCH 13/15 Makefile: Rsync global {TARGET, HOST}_DIR using exclusion file list
Herv?, All,
Nitpick, applicable to most of your patches in this series: there should not be an uppercase in the title, after the colon sign, and the tense should be imperative, e.g:
Makefile: rsync global {TARGET,HOST}_DIR using exclusion file list
Makefile: break hardlinks in global {TARGET,HOST}_DIR on per-package build package/pkg-generic.mk: fix per-package <pkg>-{reconfigure, rebuild, reinstall}
On 2021-06-21 16:11 +0200, Herve Codina spake thusly:
Quoted 6 lines
On a per-package build, rsync final {TARGET,HOST}_DIR using exclusion file list computed for each packages.
As we rsync only files generated by each packages, we need to to do this rsync recursively on each dependencies to collect all needed files. This is done based on existing <PKG>_FINAL_RECURSIVE_DEPENDENCIES
I am not sure I understand why this is needed...
One thing that comes to mind is speedup. Indeed, now that we can ensure that packages only install new files and don;t change existing files, we can speedup the final aggregation by just handling the new files.
But since we are using hardlinks, the aggregation is rather fast...
Can you expand on the rationale in the commit log when you respin, please?
Quoted 11 lines
Signed-off-by Herve Codina <[email protected]>--- Makefile | 24 ++++++++++++++++++++++-- 1 file changed, 22 insertions(+), 2 deletions(-)Makefile +0
-0@@ -740,10 +740,28 @@ TARGET_DIR_FILES_LISTS = $(sort $(wildcard $(BUILD_DIR)/*/.files-list.txt)) HOST_DIR_FILES_LISTS = $(sort $(wildcard $(BUILD_DIR)/*/.files-list-host.txt))
Makefile TARGET_DIR_FILES_LISTS = $(sort $(wildcard $(BUILD_DIR)/*/.files-list.txt)) · line 750
Quoted 10 lines
Makefile +8
-0STAGING_DIR_FILES_LISTS = $(sort $(wildcard $(BUILD_DIR)/*/.files-list-staging.txt)) +ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y) +# rsync the contents of per-package directories +# $1: space-separated list of packages to rsync from +# $2: 'host' or 'target' +# $3: destination directory +# $4: exclude file list to use +define per-package-rsync-delta + mkdir -p $(3)
$(Q)mkdir -p $(3)
Makefile TARGET_DIR_FILES_LISTS = $(sort $(wildcard $(BUILD_DIR)/*/.files-list.txt)) · line 755
Quoted 5 lines
Makefile +5
-0+ $(foreach pkg,$(1),\ + rsync -a --link-dest=$(PER_PACKAGE_DIR)/$(pkg)/$(strip $(2))/ \ + --filter='merge $($(call UPPERCASE,$(pkg))_DIR)/$(strip $(4))' \ + $(PER_PACKAGE_DIR)/$(pkg)/$(strip $(2))/ \ + $(3)$(sep))
I would read it more easily if the lines after rsync are indented furhter, to represent the fact that they are stilll rsync arguments. Also, proably it would be nice to have this quiet by default. Indeed, with rather-deep dependency chains, there can be quite a lot of rsync lines...
$(foreach pkg,$(1),\
$(Q)rsync -a --link-dest=$(PER_PACKAGE_DIR)/$(pkg)/$(strip $(2))/ \
--filter='merge $($(call UPPERCASE,$(pkg))_DIR)/$(strip $(4))' \
$(PER_PACKAGE_DIR)/$(pkg)/$(strip $(2))/ \
$(3)$(sep))Furthermore, this rsync filter is non-obvious. It would be nice to explain the commit log that the list is already a proper filer, not just a list.
Additionally, in the commit that generates that list, the commit log should also mention that a list of filters is created, not a list of filenames.
Makefile TARGET_DIR_FILES_LISTS = $(sort $(wildcard $(BUILD_DIR)/*/.files-list.txt)) · line 764
Quoted 10 lines
Makefile +6
-1+endef +endif + .PHONY: host-finalize host-finalize: $(PACKAGES) $(HOST_DIR) $(HOST_DIR_SYMLINK) @$(call MESSAGE,"Finalizing host directory") - $(call per-package-rsync,$(sort $(PACKAGES)),host,$(HOST_DIR)) + $(call per-package-rsync-delta, $(sort $(foreach pkg, $(PACKAGES), \ + $(pkg) $($(call UPPERCASE,$(pkg))_FINAL_RECURSIVE_DEPENDENCIES))), \ + host,$(HOST_DIR),.files-final-rsync-host.exclude_rsync)
This is unparseable by a human... :-(
$(call per-package-rsync-delta, \
$(sort $(foreach pkg, $(PACKAGES), \
$(pkg) $($(call UPPERCASE,$(pkg))_FINAL_RECURSIVE_DEPENDENCIES)) \
), \
host, \
$(HOST_DIR), \
.files-final-rsync-host.exclude_rsync \
)Makefile staging-finalize: $(STAGING_DIR_SYMLINK) · line 774
Quoted 10 lines
Makefile +3
-1.PHONY: staging-finalize staging-finalize: $(STAGING_DIR_SYMLINK)@@ -751,7 +769,9 @@ staging-finalize: $(STAGING_DIR_SYMLINK) .PHONY: target-finalize target-finalize: $(PACKAGES) $(TARGET_DIR) host-finalize @$(call MESSAGE,"Finalizing target directory") - $(call per-package-rsync,$(sort $(PACKAGES)),target,$(TARGET_DIR)) + $(call per-package-rsync-delta, $(sort $(foreach pkg, $(PACKAGES), \ + $(pkg) $($(call UPPERCASE,$(pkg))_FINAL_RECURSIVE_DEPENDENCIES))), \ + target,$(TARGET_DIR),.files-final-rsync.exclude_rsync)
Ditto.
But now that I review this patch, I notice that there is still a hole in our file-overwrite detection. It only handles files modified by a package, that comes from it dependencies.
However, if two packages in two different depednency chains, install the same file, then we're still toast. For example:
A --> B --> C
`-> DIf C and D install the same file, then we don't detect the conflict, and we can't easil predict which the final file will come from...
Of course, such a detection will have to be *in addition* to what this series already does, so this new issue does not invalidate this series.
Regards,
Yann E. MORIN.
Quoted 10 lines
Makefile +0
-0$(foreach hook,$(TARGET_FINALIZE_HOOKS),$($(hook))$(sep)) rm -rf $(TARGET_DIR)/usr/include $(TARGET_DIR)/usr/share/aclocal \ $(TARGET_DIR)/usr/lib/pkgconfig $(TARGET_DIR)/usr/share/pkgconfig \Signature · 6 lines
2.31.1 _______________________________________________ buildroot mailing list buildroot at busybox.net http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 13/15 Makefile: Rsync global {TARGET, HOST}_DIR using exclusion file list
Herv?, All,
Additional revview I forgot previously...
On 2021-06-24 22:20 +0200, Yann E. MORIN spake thusly:
Quoted 9 lines
On 2021-06-21 16:11 +0200, Herve Codina spake thusly:
Quoted 6 lines
On a per-package build, rsync final {TARGET,HOST}_DIR using exclusion file list computed for each packages.
As we rsync only files generated by each packages, we need to to do this rsync recursively on each dependencies to collect all needed files. This is done based on existing <PKG>_FINAL_RECURSIVE_DEPENDENCIESI am not sure I understand why this is needed...
Quoted 8 lines
One thing that comes to mind is speedup. Indeed, now that we can ensure that packages only install new files and don;t change existing files, we can speedup the final aggregation by just handling the new files.
But since we are using hardlinks, the aggregation is rather fast...
Can you expand on the rationale in the commit log when you respin, please?
Now I have seen the next patch, it makes sense. Yet, it should be explained here nonetheless. ;-)
[--SNIP--]
Quoted 5 lines
Makefile TARGET_DIR_FILES_LISTS = $(sort $(wildcard $(BUILD_DIR)/*/.files-list.txt)) · line 755
Quoted 5 lines
Makefile +5
-0+ $(foreach pkg,$(1),\ + rsync -a --link-dest=$(PER_PACKAGE_DIR)/$(pkg)/$(strip $(2))/ \ + --filter='merge $($(call UPPERCASE,$(pkg))_DIR)/$(strip $(4))' \ + $(PER_PACKAGE_DIR)/$(pkg)/$(strip $(2))/ \ + $(3)$(sep))
[--SNIP--]
Quoted 6 lines
Furthermore, this rsync filter is non-obvious. It would be nice to explain the commit log that the list is already a proper filer, not just a list.
Additionally, in the commit that generates that list, the commit log should also mention that a list of filters is created, not a list of filenames.
And to begin with, why are we generating an _exclude_ list, rather than an _include_ list?
I.e. rather than rsync everything except what we don't want, why not just rsync just what we want, and generate a list of filters about exactly just the files installed by a package?
I.e. in previous patch, crea create '+' filters rather than '-' filters. Also, I think using the full-length filter 'include' is nicer than the shorter '+'.
Probably something like:
LC_ALL=C comm -1 \
$($(PKG)_DIR)/.files-final-rsync$(2).before \
$($(PKG)_DIR)/.files-final-rsync$(2).after \
| sed -r -e 's/^[^,]+,./include /' \
> $($(PKG)_DIR)/.files-final-rsync$(2).exclude_rsync(Totally untested, slippery code, excersise caution.)
Regards,
Yann E. MORIN.
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 13/15 Makefile: Rsync global {TARGET, HOST}_DIR using exclusion file list
Hi,
Quoted 18 lines · On Thu, 24 Jun 2021 22:34:10 +0200 "Yann E. MORIN" <[email protected]>…
Herv?, All,
Additional revview I forgot previously...
On 2021-06-24 22:20 +0200, Yann E. MORIN spake thusly:
Quoted 13 lines
On 2021-06-21 16:11 +0200, Herve Codina spake thusly:
Quoted 6 lines
On a per-package build, rsync final {TARGET,HOST}_DIR using exclusion file list computed for each packages.
As we rsync only files generated by each packages, we need to to do this rsync recursively on each dependencies to collect all needed files. This is done based on existing <PKG>_FINAL_RECURSIVE_DEPENDENCIESI am not sure I understand why this is needed...
One thing that comes to mind is speedup. Indeed, now that we can ensure that packages only install new files and don;t change existing files, we can speedup the final aggregation by just handling the new files.
Quoted 7 lines
Quoted 4 lines
But since we are using hardlinks, the aggregation is rather fast...
Can you expand on the rationale in the commit log when you respin, please?
Now I have seen the next patch, it makes sense. Yet, it should be explained here nonetheless. ;-)
The next patch just breaks the hardlinks had nothing to do with the use of <PKG>_FINAL_RECURSIVE_DEPENDENCIES here.
Previously files were collected using a simple rsync and iterating on $(PACKAGES). The rsync collected all files from current package host or target so it collected files generated by the current package and the packages it depends on.
Now, we collect only files generated by the current package and we discard files generated by the packages it depends on.
Iterating on $(PACKAGES) is no longer enough.
Indeed $(PACKAGES) is the list of packages that are enabled in .config file (ie selected during 'make menuconfig' or defconfig file)
But some packages (sure for host packages) can have some "hidden" dependencies by hidden I means not seen or selected by config files.
This "hidden" dependencies are created at Makefile level.
HOST_FOO_DEPENDENCIES = host-bar
We need to collect files from this kind on dependency to generate the final HOST_DIR.
<PKG>_FINAL_RECURSIVE_DEPENDENCIES contains all makefile level dependencies a given package depends on.
Finally, we do:
for pkg selected in .config ($(PACKAGES))
# Add current pkg and packages it depends on at makefile level FULL_PACKAGES_AND_DEPENDS += pkg + $(<PKG>_FINAL_RECURSIVE_DEPENDENCIES)
And then we call sort $(FULL_PACKAGES_AND_DEPENDS) to remove duplicates
I can add in the commit log message:
---- 8< ----
Using only packages present in $(PACKAGES) is no longer enough. Indeed, we collect only files generated for a given package
and $(PACKAGES) contains the list of packages selected in .config file Some dependencies exists at Makefile level that are no [email protected] file level (at least for host packages).HOST_FOO_DEPENDENCIES += host-bar
To take into account these dependencies, <PKG>_FINAL_RECURSIVE_DEPENDENCIES is used in addition to $(PACKAGES).
---- 8< ----
Is that ok for you ?
Quoted 28 lines
[--SNIP--]
Quoted 5 lines
Makefile TARGET_DIR_FILES_LISTS = $(sort $(wildcard $(BUILD_DIR)/*/.files-list.txt)) · line 755
Quoted 5 lines
Makefile +5
-0+ $(foreach pkg,$(1),\ + rsync -a --link-dest=$(PER_PACKAGE_DIR)/$(pkg)/$(strip $(2))/ \ + --filter='merge $($(call UPPERCASE,$(pkg))_DIR)/$(strip $(4))' \ + $(PER_PACKAGE_DIR)/$(pkg)/$(strip $(2))/ \ + $(3)$(sep))[--SNIP--]
Quoted 6 lines
Furthermore, this rsync filter is non-obvious. It would be nice to explain the commit log that the list is already a proper filer, not just a list.
Additionally, in the commit that generates that list, the commit log should also mention that a list of filters is created, not a list of filenames.And to begin with, why are we generating an _exclude_ list, rather than an _include_ list?
I.e. rather than rsync everything except what we don't want, why not just rsync just what we want, and generate a list of filters about exactly just the files installed by a package?
I.e. in previous patch, crea create '+' filters rather than '-' filters. Also, I think using the full-length filter 'include' is nicer than the shorter '+'.
Probably something like:
Quoted 8 lines
LC_ALL=C comm -1 \ $($(PKG)_DIR)/.files-final-rsync$(2).before \ $($(PKG)_DIR)/.files-final-rsync$(2).after \ | sed -r -e 's/^[^,]+,./include /' \ > $($(PKG)_DIR)/.files-final-rsync$(2).exclude_rsync(Totally untested, slippery code, excersise caution.)
Inclusion filter is not so simple ...
In rsync man, we can read:
---- 8< ----
Note that, when using the --recursive (-r) option (which is implied by -a), every subdir component of every path is visited left to right, with each directory having a chance for exclusion before its content. In this way include/exclude patterns are applied recursively to the pathname of each node in the filesystem's tree (those inside the transfer).
The exclude patterns short-circuit the directory traversal
stage as rsync finds the files to send.
For instance, to include "/foo/bar/baz", the directories "/foo" and "/foo/bar" must not be excluded. Excluding one of those parent directories prevents the examination of its content, cutting off rsync's recursion into those paths and rendering the include for "/foo/bar/baz" ineffectual (since rsync can't match something it never sees in the cut-off section of the directory hierarchy).
The concept path exclusion is particularly important when using a trailing '*' rule. For instance, this won't work:
+ /some/path/this-file-will-not-be-found
+ /file-is-included
- *
This fails because the parent directory "some" is excluded by the '*' rule, so rsync never visits any of the files in the "some" or "some/path" directories. One solution is to ask for all directories in the hierarchy to be included by using a single rule: "+ */" (put it somewhere before the "- *" rule), and perhaps use the --prune-empty-dirs option.
Another solution is to add specific include rules for all the parent dirs that need to be visited. For instance, this set of rules works fine:
+ /some/
+ /some/path/
+ /some/path/this-file-is-found
+ /file-also-included
- *
---- 8< ----
I feel tricky and error prone to use include filter and really simpler to take the problem on the other side.
Instead of including what we want, exclude what we don't want.
The filter file passed to rsync is as simple as:
- /foo/bar/not_thisone
- /foo/not_thisone_too
- /not_thisone
With this filter file, rsync will include all files that are not mentioned without any more needs.
Regards,
Herv?
Signature · 3 lines
Herv? Codina, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 13/15 Makefile: Rsync global {TARGET, HOST}_DIR using exclusion file list
Herv?, All,
On 2021-06-25 13:59 +0200, Herve Codina spake thusly:
Quoted 18 lines
Quoted 16 lines · On Thu, 24 Jun 2021 22:34:10 +0200 "Yann E. MORIN" <[email protected]>…
On 2021-06-24 22:20 +0200, Yann E. MORIN spake thusly:
Quoted 15 lines
On 2021-06-21 16:11 +0200, Herve Codina spake thusly:
Quoted 6 lines
On a per-package build, rsync final {TARGET,HOST}_DIR using exclusion file list computed for each packages.
As we rsync only files generated by each packages, we need to to do this rsync recursively on each dependencies to collect all needed files. This is done based on existing <PKG>_FINAL_RECURSIVE_DEPENDENCIESI am not sure I understand why this is needed...
One thing that comes to mind is speedup. Indeed, now that we can ensure that packages only install new files and don;t change existing files, we can speedup the final aggregation by just handling the new files.
But since we are using hardlinks, the aggregation is rather fast...
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
Can you expand on the rationale in the commit log when you respin, please?
Now I have seen the next patch, it makes sense. Yet, it should be explained here nonetheless. ;-)
The next patch just breaks the hardlinks had nothing to do with the use of <PKG>_FINAL_RECURSIVE_DEPENDENCIES here.
Oh, I was not commenting about the _FINAL_RECURSIVE_DEPENDENCIES, but about the need for an exclusion list to begin with. I totally agree witht the rationale for using _FINAL_RECURSIVE_DEPENDENCIES.
[--SNIP--]
Quoted 2 lines
Quoted 2 lines
And to begin with, why are we generating an _exclude_ list, rather than an _include_ list?
[--SNIP--]
Quoted 4 lines
Quoted 1 line
(Totally untested, slippery code, excersise caution.)
Inclusion filter is not so simple ...
In rsync man, we can read:
[--SNIP--]
Indeed, I see now. The rsync filter is not about listing files; it is deciding what should be or should not be transferred. Anything that is not filtered-out is transferred.
So OK, using an exclude list toally makes sense.
Regards,
Yann E. MORIN.
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 13/15 Makefile: Rsync global {TARGET, HOST}_DIR using exclusion file list
In reply to Yann E. MORINQuoted 6 lines · On Thu, 24 Jun 2021 22:20:10 +0200 "Yann E. MORIN" <[email protected]>…
Herv?, All,
Nitpick, applicable to most of your patches in this series: there should not be an uppercase in the title, after the colon sign, and the tense should be imperative, e.g:
Ok, thanks.
Quoted 33 lines
On 2021-06-21 16:11 +0200, Herve Codina spake thusly:
Quoted 6 lines
On a per-package build, rsync final {TARGET,HOST}_DIR using exclusion file list computed for each packages.
As we rsync only files generated by each packages, we need to to do this rsync recursively on each dependencies to collect all needed files. This is done based on existing <PKG>_FINAL_RECURSIVE_DEPENDENCIESI am not sure I understand why this is needed...
One thing that comes to mind is speedup. Indeed, now that we can ensure that packages only install new files and don;t change existing files, we can speedup the final aggregation by just handling the new files.
But since we are using hardlinks, the aggregation is rather fast...
Can you expand on the rationale in the commit log when you respin, please?
Quoted 13 lines
Signed-off-by Herve Codina <[email protected]>--- Makefile | 24 ++++++++++++++++++++++-- 1 file changed, 22 insertions(+), 2 deletions(-)Makefile +0
-0@@ -740,10 +740,28 @@ TARGET_DIR_FILES_LISTS = $(sort $(wildcard $(BUILD_DIR)/*/.files-list.txt)) HOST_DIR_FILES_LISTS = $(sort $(wildcard $(BUILD_DIR)/*/.files-list-host.txt)) STAGING_DIR_FILES_LISTS = $(sort $(wildcard $(BUILD_DIR)/*/.files-list-staging.txt))
Quoted 10 lines
Makefile TARGET_DIR_FILES_LISTS = $(sort $(wildcard $(BUILD_DIR)/*/.files-list.txt)) · line 750
Quoted 8 lines
Makefile +8
-0+ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y) +# rsync the contents of per-package directories +# $1: space-separated list of packages to rsync from +# $2: 'host' or 'target' +# $3: destination directory +# $4: exclude file list to use +define per-package-rsync-delta + mkdir -p $(3)$(Q)mkdir -p $(3)
Changed.
Quoted 13 lines
Makefile TARGET_DIR_FILES_LISTS = $(sort $(wildcard $(BUILD_DIR)/*/.files-list.txt)) · line 755
Quoted 6 lines
Makefile +5
-0+ $(foreach pkg,$(1),\ + rsync -a --link-dest=$(PER_PACKAGE_DIR)/$(pkg)/$(strip $(2))/ \ + --filter='merge $($(call UPPERCASE,$(pkg))_DIR)/$(strip $(4))' \ + $(PER_PACKAGE_DIR)/$(pkg)/$(strip $(2))/ \ + $(3)$(sep))I would read it more easily if the lines after rsync are indented furhter, to represent the fact that they are stilll rsync arguments. Also, proably it would be nice to have this quiet by default. Indeed, with rather-deep dependency chains, there can be quite a lot of rsync lines...
Quoted 5 lines
$(foreach pkg,$(1),\
$(Q)rsync -a --link-dest=$(PER_PACKAGE_DIR)/$(pkg)/$(strip $(2))/ \
--filter='merge $($(call UPPERCASE,$(pkg))_DIR)/$(strip $(4))' \
$(PER_PACKAGE_DIR)/$(pkg)/$(strip $(2))/ \
$(3)$(sep))Changed
Quoted 22 lines
Furthermore, this rsync filter is non-obvious. It would be nice to explain the commit log that the list is already a proper filer, not just a list.
Additionally, in the commit that generates that list, the commit log should also mention that a list of filters is created, not a list of filenames.
Makefile TARGET_DIR_FILES_LISTS = $(sort $(wildcard $(BUILD_DIR)/*/.files-list.txt)) · line 764
Quoted 10 lines
Makefile +6
-1+endef +endif + .PHONY: host-finalize host-finalize: $(PACKAGES) $(HOST_DIR) $(HOST_DIR_SYMLINK) @$(call MESSAGE,"Finalizing host directory") - $(call per-package-rsync,$(sort $(PACKAGES)),host,$(HOST_DIR)) + $(call per-package-rsync-delta, $(sort $(foreach pkg, $(PACKAGES), \ + $(pkg) $($(call UPPERCASE,$(pkg))_FINAL_RECURSIVE_DEPENDENCIES))), \ + host,$(HOST_DIR),.files-final-rsync-host.exclude_rsync)This is unparseable by a human... :-(
Quoted 8 lines
$(call per-package-rsync-delta, \
$(sort $(foreach pkg, $(PACKAGES), \
$(pkg) $($(call UPPERCASE,$(pkg))_FINAL_RECURSIVE_DEPENDENCIES)) \
), \
host, \
$(HOST_DIR), \
.files-final-rsync-host.exclude_rsync \
)Changed.
Quoted 13 lines
Makefile staging-finalize: $(STAGING_DIR_SYMLINK) · line 774
Quoted 11 lines
Makefile +3
-1.PHONY: staging-finalize staging-finalize: $(STAGING_DIR_SYMLINK)@@ -751,7 +769,9 @@ staging-finalize: $(STAGING_DIR_SYMLINK) .PHONY: target-finalize target-finalize: $(PACKAGES) $(TARGET_DIR) host-finalize @$(call MESSAGE,"Finalizing target directory") - $(call per-package-rsync,$(sort $(PACKAGES)),target,$(TARGET_DIR)) + $(call per-package-rsync-delta, $(sort $(foreach pkg, $(PACKAGES), \ + $(pkg) $($(call UPPERCASE,$(pkg))_FINAL_RECURSIVE_DEPENDENCIES))), \ + target,$(TARGET_DIR),.files-final-rsync.exclude_rsync)Ditto.
Changed
Quoted 8 lines
But now that I review this patch, I notice that there is still a hole in our file-overwrite detection. It only handles files modified by a package, that comes from it dependencies.
However, if two packages in two different depednency chains, install the same file, then we're still toast. For example:
Quoted 9 lines
A --> B --> C `-> DIf C and D install the same file, then we don't detect the conflict, and we can't easil predict which the final file will come from...
Of course, such a detection will have to be *in addition* to what this series already does, so this new issue does not invalidate this series.
I missed this use case.
You you think it can help to keep in mind if
(1) I add it in my v2 my cover letter
or
(2) I add it in this patch commit log
Herv?
Signature · 3 lines
Herv? Codina, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 14/15 Makefile: Breaks hardlinks in global {TARGET, HOST}_DIR on per-package build
In reply to Herve CodinaWithout this patch, a make <pkg>_rebuild detects overwrites. Indeed, in target_finalize steps some modifications are done on installed files (ie strip or TARGET_FINALIZE_HOOKS for instance).
In order to avoid these modifications seen from per-package {TARGET,HOST}_DIR and so been analyzed as some overwrites, global {TARGET,HOST}_DIR is built using a full copy of the involved per-package files instead of hardlinks.
--- Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Makefile +1 -1
@@ -749,7 +749,7 @@ ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y) define per-package-rsync-delta mkdir -p $(3) $(foreach pkg,$(1),\ - rsync -a --link-dest=$(PER_PACKAGE_DIR)/$(pkg)/$(strip $(2))/ \ + rsync -a \ --filter='merge $($(call UPPERCASE,$(pkg))_DIR)/$(strip $(4))' \ $(PER_PACKAGE_DIR)/$(pkg)/$(strip $(2))/ \ $(3)$(sep))
Signature · 1 line
2.31.1
PATCH 14/15 Makefile: Breaks hardlinks in global {TARGET, HOST}_DIR on per-package build
Herv?, All,
On 2021-06-21 16:11 +0200, Herve Codina spake thusly:
Quoted 7 lines
Without this patch, a make <pkg>_rebuild detects overwrites. Indeed, in
target_finalizesteps some modifications are done on installed files (ie strip orTARGET_FINALIZE_HOOKSfor instance).In order to avoid these modifications seen from per-package {TARGET,HOST}_DIR and so been analyzed as some overwrites, global {TARGET,HOST}_DIR is built using a full copy of the involved per-package files instead of hardlinks.
OK, now it makes sense to do a partial rsync with just the needed files.
When you respin:
Regards,
Yann E. MORIN.
Quoted 25 lines
Signed-off-by Herve Codina <[email protected]>--- Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)Makefile +1
-1@@ -749,7 +749,7 @@ ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y) define per-package-rsync-delta mkdir -p $(3) $(foreach pkg,$(1),\ - rsync -a --link-dest=$(PER_PACKAGE_DIR)/$(pkg)/$(strip $(2))/ \ + rsync -a \ --filter='merge $($(call UPPERCASE,$(pkg))_DIR)/$(strip $(4))' \ $(PER_PACKAGE_DIR)/$(pkg)/$(strip $(2))/ \ $(3)$(sep))Signature · 6 lines
2.31.1 _______________________________________________ buildroot mailing list buildroot at busybox.net http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 15/15 package/pkg-generic.mk: Fix per-package <pkg>-{reconfigure, rebuild, reinstall}
In reply to Herve CodinaMany overwrites are detected on <pkg>-{reconfigure,rebuild,reinstall}. Indeed, files previously installed by a package were detected as overwritten on next reconfigure, rebuild or reinstall.
To avoid this, we recreate per-package host and target dir from scratch as it was done during the first configure step.
--- package/pkg-generic.mk | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+)
package/pkg-generic.mk +26 -0
@@ -470,6 +470,13 @@ define pkg-graph-depends $$(GRAPHS_DIR)/$$(@).dot endef +ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y) +define empty-per-package-directory + rm -rf $(HOST_DIR) $(TARGET_DIR) + mkdir -p $(HOST_DIR) $(TARGET_DIR) +endef +endif + ################################################################################ # inner-generic-package -- generates the make targets needed to build a # generic package
@@ -1072,6 +1079,25 @@ endif rm -f $$($(2)_TARGET_INSTALL_TARGET) rm -f $$($(2)_TARGET_INSTALL_IMAGES) rm -f $$($(2)_TARGET_INSTALL_HOST) + $$(call empty-per-package-directory) + $$(call prepare-per-package-directory,$$($(2)_FINAL_DOWNLOAD_DEPENDENCIES)) + $$(call prepare-per-package-directory,$$($(2)_FINAL_EXTRACT_DEPENDENCIES)) + $$(call prepare-per-package-directory,$$($(2)_FINAL_DEPENDENCIES)) + @$$(call pkg_final_rsync_before,$$(TARGET_DIR)) + @$$(call pkg_final_rsync_before,$$(HOST_DIR),-host) + @$$(call pkg_size_before,$$(TARGET_DIR)) + @$$(call pkg_size_before,$$(STAGING_DIR),-staging) + @$$(call pkg_size_before,$$(HOST_DIR),-host) + $$(call fixup-libtool-files,$$(NAME),$$(HOST_DIR)) + $$(call fixup-libtool-files,$$(NAME),$$(STAGING_DIR)) + $$(call fixup-python-files,$$(NAME),$$(HOST_DIR)) + $$(call fixup-python-files,$$(NAME),$$(STAGING_DIR)) + $$(foreach hook,$$($(2)_PER_PACKAGE_TWEAK_HOOKS),$$(call $$(hook))$$(sep)) + @$$(call pkg_detect_overwrite_before,$$(TARGET_DIR)) + @$$(call pkg_detect_overwrite_before,$$(HOST_DIR),-host) + +$(1)-clean-for-reinstall: PKG=$(2) +$(1)-clean-for-reinstall: NAME=$(1) $(1)-reinstall: $(1)-clean-for-reinstall $(1)
Signature · 1 line
2.31.1
PATCH 15/15 package/pkg-generic.mk: Fix per-package <pkg>-{reconfigure, rebuild, reinstall}
Herv?, All,
On 2021-06-21 16:11 +0200, Herve Codina spake thusly:
Quoted 40 lines
Many overwrites are detected on <pkg>-{reconfigure,rebuild,reinstall}. Indeed, files previously installed by a package were detected as overwritten on next reconfigure, rebuild or reinstall.
To avoid this, we recreate per-package host and target dir from scratch as it was done during the first configure step.
Signed-off-by Herve Codina <[email protected]>--- package/pkg-generic.mk | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+)package/pkg-generic.mk +13
-0@@ -470,6 +470,13 @@ define pkg-graph-depends $$(GRAPHS_DIR)/$$(@).dot endef +ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y) +define empty-per-package-directory + rm -rf $(HOST_DIR) $(TARGET_DIR) + mkdir -p $(HOST_DIR) $(TARGET_DIR) +endef +endif + ################################################################################ # inner-generic-package -- generates the make targets needed to build a # generic package@@ -1072,6 +1079,25 @@ endif rm -f $$($(2)_TARGET_INSTALL_TARGET) rm -f $$($(2)_TARGET_INSTALL_IMAGES) rm -f $$($(2)_TARGET_INSTALL_HOST) + $$(call empty-per-package-directory) + $$(call prepare-per-package-directory,$$($(2)_FINAL_DOWNLOAD_DEPENDENCIES)) + $$(call prepare-per-package-directory,$$($(2)_FINAL_EXTRACT_DEPENDENCIES)) + $$(call prepare-per-package-directory,$$($(2)_FINAL_DEPENDENCIES)) + @$$(call pkg_final_rsync_before,$$(TARGET_DIR)) + @$$(call pkg_final_rsync_before,$$(HOST_DIR),-host)
package/pkg-generic.mk endif · line 1097
Quoted 10 lines
package/pkg-generic.mk +10
-0+ @$$(call pkg_size_before,$$(TARGET_DIR)) + @$$(call pkg_size_before,$$(STAGING_DIR),-staging) + @$$(call pkg_size_before,$$(HOST_DIR),-host) + $$(call fixup-libtool-files,$$(NAME),$$(HOST_DIR)) + $$(call fixup-libtool-files,$$(NAME),$$(STAGING_DIR)) + $$(call fixup-python-files,$$(NAME),$$(HOST_DIR)) + $$(call fixup-python-files,$$(NAME),$$(STAGING_DIR)) + $$(foreach hook,$$($(2)_PER_PACKAGE_TWEAK_HOOKS),$$(call $$(hook))$$(sep)) + @$$(call pkg_detect_overwrite_before,$$(TARGET_DIR)) + @$$(call pkg_detect_overwrite_before,$$(HOST_DIR),-host)
I don;t like this duplication, because it will inevitably diverge. And already, it is different: rather than rely on $(PKG)_FINAL_DEPENDENCIES) and a single call to prepare-per-package-directory, there are now three calls to prepare-per-package-directory with three types of depenencies.
Why can't we comonalise this into a big macro that gets expanded twise, once in the original $(BUILD_DIR)/%/.stamp_configured rule, and once here?
This would also termendously help into factorising this away and into its own $(1)-prepare step, eventually.
package/pkg-generic.mk endif · line 1100
Quoted 2 lines
package/pkg-generic.mk +2
-0+$(1)-clean-for-reinstall: PKG=$(2) +$(1)-clean-for-reinstall: NAME=$(1)
Please move those two right before the actual rule (i.e. just before line 991 in mastr as of today.
Regards,
Yann E. MORIN.
Quoted 9 lines
package/pkg-generic.mk +0
-0$(1)-reinstall: $(1)-clean-for-reinstall $(1)Signature · 6 lines
2.31.1 _______________________________________________ buildroot mailing list buildroot at busybox.net http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 15/15 package/pkg-generic.mk: Fix per-package <pkg>-{reconfigure, rebuild, reinstall}
Quoted 50 lines · On Thu, 24 Jun 2021 22:44:42 +0200 "Yann E. MORIN" <[email protected]>…
Herv?, All,
On 2021-06-21 16:11 +0200, Herve Codina spake thusly:
Quoted 47 lines
Many overwrites are detected on <pkg>-{reconfigure,rebuild,reinstall}. Indeed, files previously installed by a package were detected as overwritten on next reconfigure, rebuild or reinstall.
To avoid this, we recreate per-package host and target dir from scratch as it was done during the first configure step.
Signed-off-by Herve Codina <[email protected]>--- package/pkg-generic.mk | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+)package/pkg-generic.mk +20
-0@@ -470,6 +470,13 @@ define pkg-graph-depends $$(GRAPHS_DIR)/$$(@).dot endef +ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y) +define empty-per-package-directory + rm -rf $(HOST_DIR) $(TARGET_DIR) + mkdir -p $(HOST_DIR) $(TARGET_DIR) +endef +endif + ################################################################################ # inner-generic-package -- generates the make targets needed to build a # generic package@@ -1072,6 +1079,25 @@ endif rm -f $$($(2)_TARGET_INSTALL_TARGET) rm -f $$($(2)_TARGET_INSTALL_IMAGES) rm -f $$($(2)_TARGET_INSTALL_HOST) + $$(call empty-per-package-directory) + $$(call prepare-per-package-directory,$$($(2)_FINAL_DOWNLOAD_DEPENDENCIES)) + $$(call prepare-per-package-directory,$$($(2)_FINAL_EXTRACT_DEPENDENCIES)) + $$(call prepare-per-package-directory,$$($(2)_FINAL_DEPENDENCIES)) + @$$(call pkg_final_rsync_before,$$(TARGET_DIR)) + @$$(call pkg_final_rsync_before,$$(HOST_DIR),-host) + @$$(call pkg_size_before,$$(TARGET_DIR)) + @$$(call pkg_size_before,$$(STAGING_DIR),-staging) + @$$(call pkg_size_before,$$(HOST_DIR),-host) + $$(call fixup-libtool-files,$$(NAME),$$(HOST_DIR)) + $$(call fixup-libtool-files,$$(NAME),$$(STAGING_DIR)) + $$(call fixup-python-files,$$(NAME),$$(HOST_DIR)) + $$(call fixup-python-files,$$(NAME),$$(STAGING_DIR))
Quoted 8 lines
package/pkg-generic.mk endif · line 1097
Quoted 3 lines
package/pkg-generic.mk +3
-0+ $$(foreach hook,$$($(2)_PER_PACKAGE_TWEAK_HOOKS),$$(call $$(hook))$$(sep)) + @$$(call pkg_detect_overwrite_before,$$(TARGET_DIR)) + @$$(call pkg_detect_overwrite_before,$$(HOST_DIR),-host)I don;t like this duplication, because it will inevitably diverge. And already, it is different: rather than rely on $(PKG)_FINAL_DEPENDENCIES) and a single call to prepare-per-package-directory, there are now three calls to prepare-per-package-directory with three types of depenencies.
The basic sequence (out of <pkg>_{reconfigure, rebuild, reinstall} is:
.stamp_downloaded
-> call prepare-per-package-directory with <PKG>_FINAL_DOWNLOAD_DEPENDENCIES
.stamp_extracted
-> call prepare-per-package-directory with <PKG>_FINAL_EXTRACT_DEPENDENCIES
.stamp_configured
-> call prepare-per-package-directory with <PKG>_FINAL_FINAL_DEPENDENCIES
Here, I wanted to recreate host and target dir with all dependencies as they were at previous .stamp_configure and so did the 3 calls.
Quoted 4 lines
Why can't we comonalise this into a big macro that gets expanded twise, once in the original $(BUILD_DIR)/%/.stamp_configured rule, and once here?
Excluding call to prepare-per-package-directory as they are not the same.
What name for this macro ? PREPARE_PRE_CONFIGURE ?
Quoted 9 lines
This would also termendously help into factorising this away and into its own $(1)-prepare step, eventually.
package/pkg-generic.mk endif · line 1100
Quoted 2 lines
package/pkg-generic.mk +2
-0+$(1)-clean-for-reinstall: PKG=$(2) +$(1)-clean-for-reinstall: NAME=$(1)Please move those two right before the actual rule (i.e. just before line 991 in mastr as of today.
Done
Quoted 14 lines
Regards,
Yann E. MORIN.Quoted 9 lines
package/pkg-generic.mk +0
-0$(1)-reinstall: $(1)-clean-for-reinstall $(1)Signature · 6 lines
2.31.1 _______________________________________________ buildroot mailing list buildroot at busybox.net http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 3 lines
Herv? Codina, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
cover Overwritten file detection and fixes, one more step to TLP build
In reply to Herve CodinaHi Herve?,
Great that someone is picking thuis up!
Quoted 5 lines · Herve Codina
Hi,
This series fixes some issues in the TLP build feature.
The starting point was the work done by Thomas Petazzoni on overwrites detection (PATCH 1).
I've marked [1] as Changes Requested.
However, it seems the tests [2][3][4] are missing from this series. Could you
include them next time you post it?
Quoted 5 lines
Based on this work, this series fixes some overwrites, introduces and uses <PKG>_PER_PACKAGE_TWEAK_HOOKS. This variable is used to collect specific per-package path tweaks needed by per-package build.
With this tweaks collected in this variable, they can be re-applied when they are needed (<pkg>_{reconfigure,rebuild,reinstall}).
That's excellent!
Quoted 6 lines
It also reworks the final {HOST,TARGET}_DIR rsync.
For each packages only files generated by the package are rsynced (PATCH 13). This is done using an exclusion list computed in (PATCH 12).
In order to avoid modifications performed in target-finalize step to be seen as overwrites on next build, the final {HOST,TARGET}_DIR do not contain any hardlink anymore and are a full copy of expected files and directories (PATCH 14).
Hm, that will have quite the impact on disk size, no? In non-PPS builds,HOST_DIR is typically about 1/3 of the total size (the other 2/3 is BUILD_DIR, the rest is negligible). By doing a non-hardlinked copy, the size of HOST_DIR is basically going to double, right?
I still think it's acceptable, but it's something to take into account...
Quoted 5 lines
The last patch (PATCH 15) fixes <pkg>_{reconfigure,rebuild,reinstall} recreating per-package {HOST,TARGET}_DIR from scratch.
Indeed, for a given package, re-generating the same file is detected as an overwrite. The simplest way to avoid this is to fully discard previous per-package {HOST,TARGET}_DIR directory and recreate them.
+1 to that!
Quoted 2 lines
To have a TLP build fully fonctionnal, several things are still missing:
- Be sure that all buildroot packages do not perform any overwrites.
I guess that's easy: increase the chance of PER_PACKAGE enabled in
genrandconfig, and see what happens to the autobuilders. Or is there anything else missing?
Quoted 7 lines
- Undo tweak stuff that have be done for per-package build (fixup-libtool-files, fixup-python-files, <PKG>_PER_PACKAGE_TWEAK_HOOKS) after collecting files in final {HOST,STAGING}_DIR.
This mainly means reverting .../per-package/xxxxx/... path to point to the correct ones in final {HOST,STAGING}_DIR.I hope this series will help in TLP build feature.
Regards,
Arnout
[1]
https://patchwork.ozlabs.org/project/buildroot/patch/20200430095249.782597-9-thomas.petazzoni at bootlin.com/
[2]
https://patchwork.ozlabs.org/project/buildroot/patch/20200430095249.782597-10-thomas.petazzoni at bootlin.com/
[3]
https://patchwork.ozlabs.org/project/buildroot/patch/20200430095249.782597-11-thomas.petazzoni at bootlin.com/
[4]
https://patchwork.ozlabs.org/project/buildroot/patch/20200430095249.782597-12-thomas.petazzoni at bootlin.com/
Quoted 39 lines
Best regards,
Herv? CodinaHerve Codina (14):
package/e2fsprogs: fix fsck overwrite inHOST_DIRpackage/pkg-generic.mk: Remove Info documents dir entrypackage/pkg-generic.mk: Fix .la files overwrite detectionpackage/pkg-generic.mk: Perform .la files fixup in per-package
HOST_DIR
package/pkg-generic.mk: Introduce <PKG>_PER_PACKAGE_TWEAK_HOOKS package/apr-util: Use <PKG>_PER_PACKAGE_TWEAK_HOOKS package/apache: MoveAPACHE_FIXUP_APR_LIBTOOLto
<PKG>_PER_PACKAGE_TWEAK_HOOKS
package/pkg-python: Remove _sysconfigdata*.pyc files when
_sysconfigdata*.py are changed
package/pkg-generic.mk: Move python fixup to generic package
infrastructure
package/owfs: Remove Python sysconfigdata fixuppackage/pkg-generic.mk: Generate final rsync exclude file list
Makefile: Rsync global {TARGET,HOST}_DIR using exclusion file list
Makefile: Breaks hardlinks in global {TARGET,HOST}_DIR on per-package
build
package/pkg-generic.mk: Fix per-package<pkg>-{reconfigure,rebuild,reinstall}Thomas Petazzoni (1):
package/pkg-generic.mk: detect files overwritten inTARGET_DIRand
HOST_DIRMakefile | 24 +++++++- package/apache/apache.mk | 2 +- package/apr-util/apr-util.mk | 12 ++-- package/e2fsprogs/e2fsprogs.mk | 1 + package/owfs/owfs.mk | 9 --- package/pkg-generic.mk | 103 ++++++++++++++++++++++++++++++++- package/pkg-python.mk | 10 ---- 7 files changed, 134 insertions(+), 27 deletions(-)
cover Overwritten file detection and fixes, one more step to TLP build
Hi,
Quoted 2 lines · On Mon, 21 Jun 2021 22:42:18 +0200 Arnout Vandecappelle <[email protected]> wrote:
However, it seems the tests [2][3][4] are missing from this series. Could you
include them next time you post it?
Added in v2 series.
Signature · 3 lines
Herv? Codina, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
cover Overwritten file detection and fixes, one more step to TLP build
In reply to Herve CodinaHerv?, All,
On 2021-06-21 16:11 +0200, Herve Codina spake thusly:
Quoted 3 lines
This series fixes some issues in the TLP build feature.
The starting point was the work done by Thomas Petazzoni on overwrites detection (PATCH 1).
[--SNIP--]
Thanks for working on this complex topic. :-)
I'm now done with my review; I may have missed details of lesser importance, so we'll deal with those later; Thomas also provided some feedback. I see that you have already taken initial reviews into account, and that you seem nicely on track for a v2.
As such, I've marked this entire series as "changes requested" in patchwork.
Can you add me in Cc of the full series when you later respin, please?
Regards,
Yann E. MORIN.
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
cover Overwritten file detection and fixes, one more step to TLP build
In reply to Herve CodinaHi Herve, all,
Quoted 5 lines · Herve Codina
Hi,
This series fixes some issues in the TLP build feature.
The starting point was the work done by Thomas Petazzoni on overwrites detection (PATCH 1).
after a long time I have recently picked up playing with TLP build again. As part of that I found a problem in qt5 where I sent a patch two days ago "qt5: Fix sporadic build failure during top-level parallel build" which is caused by manipulation of the hard-linked qt.conf in HOST_DIR from different qt5 packages.
Now I see your promising set, but the matter is quite complex and so I wonder if the "overwritten file detection" would a) uncover the qt.conf problem and b)if your patch series somehow fixes it in a generic way?
regards,
Andreas
Quoted 68 lines
Based on this work, this series fixes some overwrites, introduces and uses <PKG>_PER_PACKAGE_TWEAK_HOOKS. This variable is used to collect specific per-package path tweaks needed by per-package build.
With this tweaks collected in this variable, they can be re-applied when they are needed (<pkg>_{reconfigure,rebuild,reinstall}).It also reworks the final {HOST,TARGET}_DIR rsync.
For each packages only files generated by the package are rsynced (PATCH 13). This is done using an exclusion list computed in (PATCH 12).
In order to avoid modifications performed in target-finalize step to be seen as overwrites on next build, the final {HOST,TARGET}_DIR do not contain any hardlink anymore and are a full copy of expected files and directories (PATCH 14).The last patch (PATCH 15) fixes <pkg>_{reconfigure,rebuild,reinstall} recreating per-package {HOST,TARGET}_DIR from scratch.
Indeed, for a given package, re-generating the same file is detected as an overwrite. The simplest way to avoid this is to fully discard previous per-package {HOST,TARGET}_DIR directory and recreate them.To have a TLP build fully fonctionnal, several things are still missing:
- Be sure that all buildroot packages do not perform any overwrites.
- Undo tweak stuff that have be done for per-package build (fixup-libtool-files, fixup-python-files, <PKG>_PER_PACKAGE_TWEAK_HOOKS) after collecting files in final {HOST,STAGING}_DIR.
This mainly means reverting .../per-package/xxxxx/... path to point to the correct ones in final {HOST,STAGING}_DIR.I hope this series will help in TLP build feature.
Best regards,
Herv? CodinaHerve Codina (14):
package/e2fsprogs: fix fsck overwrite inHOST_DIRpackage/pkg-generic.mk: Remove Info documents dir entrypackage/pkg-generic.mk: Fix .la files overwrite detectionpackage/pkg-generic.mk: Perform .la files fixup in per-package
HOST_DIR
package/pkg-generic.mk: Introduce <PKG>_PER_PACKAGE_TWEAK_HOOKS package/apr-util: Use <PKG>_PER_PACKAGE_TWEAK_HOOKS package/apache: MoveAPACHE_FIXUP_APR_LIBTOOLto
<PKG>_PER_PACKAGE_TWEAK_HOOKS
package/pkg-python: Remove _sysconfigdata*.pyc files when
_sysconfigdata*.py are changed
package/pkg-generic.mk: Move python fixup to generic package
infrastructure
package/owfs: Remove Python sysconfigdata fixuppackage/pkg-generic.mk: Generate final rsync exclude file list
Makefile: Rsync global {TARGET,HOST}_DIR using exclusion file list
Makefile: Breaks hardlinks in global {TARGET,HOST}_DIR on per-package
build
package/pkg-generic.mk: Fix per-package<pkg>-{reconfigure,rebuild,reinstall}Thomas Petazzoni (1):
package/pkg-generic.mk: detect files overwritten inTARGET_DIRand
HOST_DIRMakefile | 24 +++++++- package/apache/apache.mk | 2 +- package/apr-util/apr-util.mk | 12 ++-- package/e2fsprogs/e2fsprogs.mk | 1 + package/owfs/owfs.mk | 9 --- package/pkg-generic.mk | 103 ++++++++++++++++++++++++++++++++- package/pkg-python.mk | 10 ---- 7 files changed, 134 insertions(+), 27 deletions(-)
cover Overwritten file detection and fixes, one more step to TLP build
Hi,
Quoted 11 lines · On Fri, 25 Jun 2021 11:08:43 +0200 Andreas Naumann <[email protected]> wrote:
after a long time I have recently picked up playing with TLP build again. As part of that I found a problem in qt5 where I sent a patch two days ago "qt5: Fix sporadic build failure during top-level parallel build" which is caused by manipulation of the hard-linked qt.conf in
HOST_DIRfrom different qt5 packages.Now I see your promising set, but the matter is quite complex and so I wonder if the "overwritten file detection" would a) uncover the qt.conf problem and b)if your patch series somehow fixes it in a generic way?
I quickly looked at your patch and it looks like an overwrite issue.
Your patch:
---- 8< ----
# compiled into the Qt library. We need it to make "qmake" relocatable and # tweak the per-package install pathes define QT5_INSTALL_QT_CONF + rm -f $(HOST_DIR)/bin/qt.conf sed -e "s|@@HOST_DIR@@|$(HOST_DIR)|" -e "s|@@STAGING_DIR@@|$(STAGING_DIR)|" \ $(QT5BASE_PKGDIR)/qt.conf.in > $(HOST_DIR)/bin/qt.conf endef ---- 8< ----
This overwrite is done by sed whose output is redirected $(HOST_DIR)/bin/qt.conf
With TLP this overwite leads to package A seeing modification done by package B
Without your patch and with overwrite detection present in my series the overwrite should be detected and the build stopped.
My series do not contains any fixes for this specific Qt5 overwrite. This is a concrete example of what I said still missing ...
Your 'rm' breaks the hardlink and so the following sed does not perform overwrite anymore.
That's the correct fix.
To go further with interactions with my current series still ongoing and not ready to be merged, the entire macro (QT5_INSTALL_QT_CONF) should be moved to <PKG>_PER_PACKAGE_TWEAK_HOOKS and QT5_QT_CONF_FIXUP is no more needed.
Herv?
Signature · 3 lines
Herv? Codina, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
cover Overwritten file detection and fixes, one more step to TLP build
Hi Herve,
Quoted 32 lines · Herve Codina
Hi,
Quoted 11 lines · On Fri, 25 Jun 2021 11:08:43 +0200 Andreas Naumann <[email protected]> wrote:
after a long time I have recently picked up playing with TLP build again. As part of that I found a problem in qt5 where I sent a patch two days ago "qt5: Fix sporadic build failure during top-level parallel build" which is caused by manipulation of the hard-linked qt.conf in
HOST_DIRfrom different qt5 packages.Now I see your promising set, but the matter is quite complex and so I wonder if the "overwritten file detection" would a) uncover the qt.conf problem and b)if your patch series somehow fixes it in a generic way?
I quickly looked at your patch and it looks like an overwrite issue.
Your patch:
---- 8< ----# compiled into the Qt library. We need it to make "qmake" relocatable and # tweak the per-package install pathes define QT5_INSTALL_QT_CONF(quoted diff) +1
-1+ rm -f $(HOST_DIR)/bin/qt.conf sed -e "s|@@HOST_DIR@@|$(HOST_DIR)|" -e "s|@@STAGING_DIR@@|$(STAGING_DIR)|" \ $(QT5BASE_PKGDIR)/qt.conf.in > $(HOST_DIR)/bin/qt.conf endef ---- 8< ----This overwrite is done by sed whose output is redirected $(HOST_DIR)/bin/qt.conf
Quoted 7 lines
With TLP this overwite leads to package A seeing modification done by package B
Without your patch and with overwrite detection present in my series the overwrite should be detected and the build stopped.
My series do not contains any fixes for this specific Qt5 overwrite. This is a concrete example of what I said still missing ...
Thank you for the confirmation.
Quoted 8 lines
Your 'rm' breaks the hardlink and so the following sed does not perform overwrite anymore.
That's the correct fix.To go further with interactions with my current series still ongoing and not ready to be merged, the entire macro (
QT5_INSTALL_QT_CONF) should be moved to <PKG>_PER_PACKAGE_TWEAK_HOOKS andQT5_QT_CONF_FIXUPis no more needed.
Ok, I'll try to catch the point when your set is accepted and the repost an updated version.
have a nice weekend,
Andreas
Quoted 3 lines
Herv?