package/pkg-utils: fix regression in size and build time with PPD (branch yem/rsync-copy)

Commit 21d52e52d8de (package/pkg-utils.mk: break hardlinks in global {TARGET, HOST}_DIR on per-package build) stopped hardlink the source and destination when rsyncing per-package directory, on the rationale that modifying files in-place after the rsync would also modify the original file, and that break foo-rebuild and can cause issues with post-build scripts.

However, what 21d52e52d8de did not envision, is that copying instead of hard-linking has two nasty side effects:

- the size increase for the build directory increase with the number of packages and with the depth of th dependency chains for those
packages: a (relatively small) build that was previously totalling ~13GiB in output/, now totals north of 122GiB, an almost 10-time increase;

- the build time increases, as it takes mopre time to read+write files

than it takes to create a new link to a file; the same build saw a

increase of build time from 5min 10s to 7min 30s, a 45% increase.

These regressions are both serious, so revert the change; we'll come up with a stopgap measure in follow-up commits.

This reverts commit 21d52e52d8dee0940d28b3a38551eb183be37813.

Signed-off-by Yann E. MORIN <[email protected]>

Cc Herve Codina <[email protected]>

Cc Peter Korsgaard <[email protected]>

Cc Thomas Petazzoni <[email protected]>

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

"Yann" == Yann E MORIN <[email protected]> writes:

Commit 21d52e52d8de (package/pkg-utils.mk: break hardlinks in global {TARGET, HOST}_DIR on per-package build) stopped hardlink the source and destination when rsyncing per-package directory, on the rationale that modifying files in-place after the rsync would also modify the original file, and that break foo-rebuild and can cause issues with post-build scripts.

However, what 21d52e52d8de did not envision, is that copying instead of hard-linking has two nasty side effects:

- the size increase for the build directory increase with the number of packages and with the depth of th dependency chains for those
packages: a (relatively small) build that was previously totalling ~13GiB in output/, now totals north of 122GiB, an almost 10-time increase;

- the build time increases, as it takes mopre time to read+write files than it takes to create a new link to a file; the same build saw a increase of build time from 5min 10s to 7min 30s, a 45% increase.

These regressions are both serious, so revert the change; we'll come up with a stopgap measure in follow-up commits.

This reverts commit 21d52e52d8dee0940d28b3a38551eb183be37813.

Signed-off-by Yann E. MORIN <[email protected]>

Cc Herve Codina <[email protected]>

Cc Peter Korsgaard <[email protected]>

Cc Thomas Petazzoni <[email protected]>

commit 21d52e52d8de (package/pkg-utils.mk: break hardlinks in global {TARGET, HOST}_DIR on per-package build) was recently reverted, so we are back to a situation where it is possible for packages and post-build scripts to modify files in-place, and thus impact files in any arbitrary per-package directory, which may break things on rebuild for example.

21d52e52d8de wsa too big a hammer, but we can still apply the reasoning

from it, to the aggregation of the final target and host directories.

This solves the case for post-build scripts at least. We leave the case of inter-package modification aside, as it is a bigger issue that will need more than jsut copying files around.

Signed-off-by Yann E. MORIN <[email protected]>

Cc Herve Codina <[email protected]>

Cc Peter Korsgaard <[email protected]>

Cc Thomas Petazzoni <[email protected]>

---
 Makefile             |  4 ++--
 package/pkg-utils.mk | 15 ++++++++++++---
 2 files changed, 14 insertions(+), 5 deletions(-)

On Tue, 17 Oct 2023 23:01:20 +0200
"Yann E. MORIN" <[email protected]> wrote:

Quoted 7 lines

package/pkg-utils.mk +1 -0

@@ -214,10 +214,19 @@ ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y)
 # $1: space-separated list of packages to rsync from
 # $2: 'host' or 'target'
 # $3: destination directory
+# $4: literal "copy" or "hardlink" to copy or hardlink files from src to dest

package/pkg-utils.mk ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y) · line 222

Quoted 6 lines

package/pkg-utils.mk +2 -1

 define per-package-rsync
 	mkdir -p $(3)
 	$(foreach pkg,$(1),\
-		rsync -a --link-dest=$(PER_PACKAGE_DIR)/$(pkg)/$(2)/ \
+		rsync -a \
+			--hard-links \

You preserve hard links (--hard-links) in all cases (copy and hardlink). In case of copy, is it correct to preserve hard links ?

package/pkg-utils.mk ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y) · line 232

Quoted 10 lines

package/pkg-utils.mk +7 -0

+			$(if $(filter hardlink,$(4)), \
+				--link-dest=$(PER_PACKAGE_DIR)/$(pkg)/$(2)/, \
+				$(if $(filter copy,$(4)), \
+					$(empty), \
+					$(error per-package-rsync can only "copy" or "hardlink", not "$(4)") \
+				) \
+			) \
 			$(PER_PACKAGE_DIR)/$(pkg)/$(2)/ \
 			$(3)$(sep))
 endef

[...]

Best regards,
Hervé
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

Hervé, All

Quoted 16 lines

Quoted 4 lines

package/pkg-utils.mk ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y) · line 222

Quoted 10 lines

package/pkg-utils.mk +3 -1

@@ -214,10 +214,19 @@ ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y)
 # $1: space-separated list of packages to rsync from
 # $2: 'host' or 'target'
 # $3: destination directory
+# $4: literal "copy" or "hardlink" to copy or hardlink files from src to dest
 define per-package-rsync
 	mkdir -p $(3)
 	$(foreach pkg,$(1),\
-		rsync -a --link-dest=$(PER_PACKAGE_DIR)/$(pkg)/$(2)/ \
+		rsync -a \
+			--hard-links \  

You preserve hard links (--hard-links) in all cases (copy and hardlink). In case of copy, is it correct to preserve hard links ?

Yes, --hard-links is:

This tells rsync to look for hard-linked files in the source and link together the corresponding files on the destination.

Regards,
Yann E. MORIN.

Quoted 13 lines

Quoted 10 lines

package/pkg-utils.mk +7 -0

+			$(if $(filter hardlink,$(4)), \
+				--link-dest=$(PER_PACKAGE_DIR)/$(pkg)/$(2)/, \
+				$(if $(filter copy,$(4)), \
+					$(empty), \
+					$(error per-package-rsync can only "copy" or "hardlink", not "$(4)") \
+				) \
+			) \
 			$(PER_PACKAGE_DIR)/$(pkg)/$(2)/ \
 			$(3)$(sep))
 endef  

[...]

Quoted 16 lines · On Wed, 18 Oct 2023 17:20:47 +0200 "Yann E. MORIN" <[email protected]>…

Hervé, All

Quoted 15 lines

package/pkg-utils.mk ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y) · line 222

Quoted 13 lines

package/pkg-utils.mk +3 -1

@@ -214,10 +214,19 @@ ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y)
 # $1: space-separated list of packages to rsync from
 # $2: 'host' or 'target'
 # $3: destination directory
+# $4: literal "copy" or "hardlink" to copy or hardlink files from src to dest
 define per-package-rsync
 	mkdir -p $(3)
 	$(foreach pkg,$(1),\
-		rsync -a --link-dest=$(PER_PACKAGE_DIR)/$(pkg)/$(2)/ \
+		rsync -a \
+			--hard-links \  

You preserve hard links (--hard-links) in all cases (copy and hardlink). In case of copy, is it correct to preserve hard links ?

Quoted 4 lines

Yes, --hard-links is:

This tells rsync to look for hard-linked files in the source and link together the corresponding files on the destination.

In case of 'copy', you keep hard-links already present in source and so the destination (final host/target dirs) is not a full copy.

Some hard-links can be present and post-build scripts can still corrupt some per-package sources.

I don't understand why keeping hard-links on this last copy is needed and I probably miss something...

Regards,
Hervé

Quoted 17 lines

Regards,
Yann E. MORIN.

Quoted 13 lines

Quoted 10 lines

package/pkg-utils.mk +7 -0

+			$(if $(filter hardlink,$(4)), \
+				--link-dest=$(PER_PACKAGE_DIR)/$(pkg)/$(2)/, \
+				$(if $(filter copy,$(4)), \
+					$(empty), \
+					$(error per-package-rsync can only "copy" or "hardlink", not "$(4)") \
+				) \
+			) \
 			$(PER_PACKAGE_DIR)/$(pkg)/$(2)/ \
 			$(3)$(sep))
 endef  

[...]

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

Hervé, All,

On 2023-10-18 17:38 +0200, Herve Codina via buildroot spake thusly:

Quoted 18 lines

Quoted 16 lines · On Wed, 18 Oct 2023 17:20:47 +0200 "Yann E. MORIN" <[email protected]>…

Hervé, All

Quoted 15 lines

package/pkg-utils.mk ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y) · line 222

Quoted 13 lines

package/pkg-utils.mk +3 -1

@@ -214,10 +214,19 @@ ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y)
 # $1: space-separated list of packages to rsync from
 # $2: 'host' or 'target'
 # $3: destination directory
+# $4: literal "copy" or "hardlink" to copy or hardlink files from src to dest
 define per-package-rsync
 	mkdir -p $(3)
 	$(foreach pkg,$(1),\
-		rsync -a --link-dest=$(PER_PACKAGE_DIR)/$(pkg)/$(2)/ \
+		rsync -a \
+			--hard-links \    

You preserve hard links (--hard-links) in all cases (copy and hardlink). In case of copy, is it correct to preserve hard links ?

Quoted 6 lines

Quoted 4 lines

Yes, --hard-links is:

This tells rsync to look for hard-linked files in the source and link together the corresponding files on the destination.

In case of 'copy', you keep hard-links already present in source and so the destination (final host/target dirs) is not a full copy.

Ah, it does not hard-link something in dest into src. The hardlinks are only in the destination. See below...

Quoted 5 lines

Some hard-links can be present and post-build scripts can still corrupt some per-package sources.

I don't understand why keeping hard-links on this last copy is needed and I probably miss something...

That is because we can have hard-links in host or target. For example, git will install a lot of hard-links in /usr/libexec/git-core/ (excerpt):

    $ ls -li per-package/git/target/usr/libexec/git-core/
    9588770 -rwxr-xr-x 141 ymorin ymorin 3318828 Oct 16 17:45 git-var
    9588770 -rwxr-xr-x 141 ymorin ymorin 3318828 Oct 16 17:45 git-verify-commit
    9588770 -rwxr-xr-x 141 ymorin ymorin 3318828 Oct 16 17:45 git-verify-pack
    9588770 -rwxr-xr-x 141 ymorin ymorin 3318828 Oct 16 17:45 git-verify-tag
    9588770 -rwxr-xr-x 141 ymorin ymorin 3318828 Oct 16 17:45 git-version

    $ ls -li target/usr/libexec/git-core/
    9600190 -rwxr-xr-x 141 ymorin ymorin 2832860 Oct 16 17:45 git-var
    9600190 -rwxr-xr-x 141 ymorin ymorin 2832860 Oct 16 17:45 git-verify-commit
    9600190 -rwxr-xr-x 141 ymorin ymorin 2832860 Oct 16 17:45 git-verify-pack
    9600190 -rwxr-xr-x 141 ymorin ymorin 2832860 Oct 16 17:45 git-verify-tag
    9600190 -rwxr-xr-x 141 ymorin ymorin 2832860 Oct 16 17:45 git-version

As you can see, in git's PPD, git tools are hard-links, and in target/ they also are hard-links. But the inode is different, so the hard-links in target/ are different from the hard-links in the PPD.

However, in all the PPDs of packages that have git as a dependency, the hard-links would all be to the same inode, 9588770.

(The size delta is because the files in target/ are stripped.)

So, in case of "copy", this is actually a copy, that keeps existing hard-links in the source, as new hard-links in the destination. Without --hard-links, the files in target/usr/libexec/git-core/ would all have a different inode, as they would be different files, and that would tremendously increase the filesystem size (squashfs would notice, and would store the content only once, but would still create as many inodes as needed, though).

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

"Yann" == Yann E MORIN <[email protected]> writes:

commit 21d52e52d8de (package/pkg-utils.mk: break hardlinks in global {TARGET, HOST}_DIR on per-package build) was recently reverted, so we are back to a situation where it is possible for packages and post-build scripts to modify files in-place, and thus impact files in any arbitrary per-package directory, which may break things on rebuild for example.

21d52e52d8de wsa too big a hammer, but we can still apply the reasoning from it, to the aggregation of the final target and host directories.

This solves the case for post-build scripts at least. We leave the case of inter-package modification aside, as it is a bigger issue that will need more than jsut copying files around.

Signed-off-by Yann E. MORIN <[email protected]>

Cc Herve Codina <[email protected]>

Cc Peter Korsgaard <[email protected]>

Cc Thomas Petazzoni <[email protected]>

This small series is a proposal to revert the huge regression in size and (to a lesser extend) build dtime due to not hard-linking source files whn preparing per-package directories, or when assembling the final target+host directories.

Regards,
Yann E. MORIN.

----------------------------------------------------------------
Yann E. MORIN (2):
Revert "package/pkg-utils.mk: break hardlinks in global {TARGET, HOST}_DIR on per-package build" package/pkg-utils: teach per-package-rsync to copy or hardlink dest

 Makefile             |  4 ++--
 package/pkg-utils.mk | 13 +++++++++++--
 2 files changed, 13 insertions(+), 4 deletions(-)

Signature · 10 lines

.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot