core/pkg-generic: only save latest package list

When rebuilding a package, simply appending the package's file list to the global list means that the package list grows for every rebuild, as does the time taken to check for files installed by multiple packages. Furthermore, we get false positives where a file is reported as being installed by multiple copies of the same package.

With this approach we may end up with orphaned files in the target filesystem if a package that has been updated and rebuilt no longer installs the same set of files, but we know that only a clean build will produce reliable results. In fact it may be helpful to identify these orphaned files as evidence that the build is not clean.

Signed-off-by John Keeping <[email protected]>

---
 package/pkg-generic.mk | 1 +
 1 file changed, 1 insertion(+)

package/pkg-generic.mk +0 -0

index 1c9dd1d734..edc2c9349c 100644

package/pkg-generic.mk +0 -0

 	find . \( -type f -o -type l \) \
 		-newer $($(PKG)_DIR)/.stamp_built \
 		-exec printf '$(1),%s\n' {} + \

Signature · 1 line

2.17.0

John, All,

On 2018-04-29 14:07 +0100, John Keeping spake thusly:

Quoted 24 lines

When rebuilding a package, simply appending the package's file list to the global list means that the package list grows for every rebuild, as does the time taken to check for files installed by multiple packages. Furthermore, we get false positives where a file is reported as being installed by multiple copies of the same package.

With this approach we may end up with orphaned files in the target filesystem if a package that has been updated and rebuilt no longer installs the same set of files, but we know that only a clean build will produce reliable results. In fact it may be helpful to identify these orphaned files as evidence that the build is not clean.

Signed-off-by John Keeping <[email protected]>

---
 package/pkg-generic.mk | 1 +
 1 file changed, 1 insertion(+)

package/pkg-generic.mk +0 -0

index 1c9dd1d734..edc2c9349c 100644

package/pkg-generic.mk GLOBAL_INSTRUMENTATION_HOOKS += step_time · line 68

Quoted 5 lines

package/pkg-generic.mk +1 -0

@@ -64,6 +64,7 @@ GLOBAL_INSTRUMENTATION_HOOKS += step_time
 # $(3): suffix of file  (optional)
 define step_pkg_size_inner
 	cd $(2); \
+	$(SED) '/^$(1),/d'
 $(BUILD_DIR)/packages-file-list$(3).txt; \  

Since BUILD_DIR is a fully-qualified path, I would have put it as the first line of the macros, and thus it would not have required the trailing '\':

    $(SED) '/^$(1),/d' $(BUILD_DIR)/packages-file-list$(3).txt
    cd $(2); \
    find [...]

Quoted 35 lines · On Mon, 30 Apr 2018 18:47:28 +0200 "Yann E. MORIN" <[email protected]>…

On 2018-04-29 14:07 +0100, John Keeping spake thusly:

Quoted 24 lines

When rebuilding a package, simply appending the package's file list to the global list means that the package list grows for every rebuild, as does the time taken to check for files installed by multiple packages. Furthermore, we get false positives where a file is reported as being installed by multiple copies of the same package.

With this approach we may end up with orphaned files in the target filesystem if a package that has been updated and rebuilt no longer installs the same set of files, but we know that only a clean build will produce reliable results. In fact it may be helpful to identify these orphaned files as evidence that the build is not clean.

Signed-off-by John Keeping <[email protected]>

---
 package/pkg-generic.mk | 1 +
 1 file changed, 1 insertion(+)

package/pkg-generic.mk +0 -0

index 1c9dd1d734..edc2c9349c 100644

package/pkg-generic.mk GLOBAL_INSTRUMENTATION_HOOKS += step_time · line 68

Quoted 5 lines

package/pkg-generic.mk +1 -0

@@ -64,6 +64,7 @@ GLOBAL_INSTRUMENTATION_HOOKS += step_time
 # $(3): suffix of file  (optional)
 define step_pkg_size_inner
 	cd $(2); \
+	$(SED) '/^$(1),/d'
 $(BUILD_DIR)/packages-file-list$(3).txt; \  

Since BUILD_DIR is a fully-qualified path, I would have put it as the first line of the macros, and thus it would not have required the trailing '\':

Quoted 3 lines

    $(SED) '/^$(1),/d' $(BUILD_DIR)/packages-file-list$(3).txt
    cd $(2); \
    find [...]

This will cause a build error if packages-file-list$(3).txt doesn't exist, which will be the case for the first package built.

We could use:

      -$(SED) '/^$(1),/d' $(BUILD_DIR)/packages-file-list$(3).txt

but then make will log that it is ignoring an error.

When rebuilding a package, simply appending the package's file list to the global list means that the package list grows for every rebuild, as does the time taken to check for files installed by multiple packages. Furthermore, we get false positives where a file is reported as being installed by multiple copies of the same package.

With this approach we may end up with orphaned files in the target filesystem if a package that has been updated and rebuilt no longer installs the same set of files, but we know that only a clean build will produce reliable results. In fact it may be helpful to identify these orphaned files as evidence that the build is not clean.

Signed-off-by John Keeping <[email protected]>

---
Changes in v2:
- Do the sed as a separate step rather than in the same shell invocation as the find (we know that $(BUILD_DIR) is an absolute path); since we no longer ignore the exit status of sed, we also have to ensure that the file exists before executing sed

 package/pkg-generic.mk | 2 ++
 1 file changed, 2 insertions(+)

package/pkg-generic.mk +0 -0

index bd47ca1964..8ebbacc180 100644

package/pkg-generic.mk +1 -0

+	$(SED) '/^$(1),/d' $(BUILD_DIR)/packages-file-list$(3).txt
 	cd $(2); \
 	find . \( -type f -o -type l \) \
 		-newer $($(PKG)_DIR)/.stamp_built \

Signature · 1 line

2.17.0

John, All,

On 2018-05-01 12:13 +0100, John Keeping spake thusly:

Quoted 30 lines

When rebuilding a package, simply appending the package's file list to the global list means that the package list grows for every rebuild, as does the time taken to check for files installed by multiple packages. Furthermore, we get false positives where a file is reported as being installed by multiple copies of the same package.

With this approach we may end up with orphaned files in the target filesystem if a package that has been updated and rebuilt no longer installs the same set of files, but we know that only a clean build will produce reliable results. In fact it may be helpful to identify these orphaned files as evidence that the build is not clean.

Signed-off-by John Keeping <[email protected]>

---
Changes in v2:
- Do the sed as a separate step rather than in the same shell invocation as the find (we know that $(BUILD_DIR) is an absolute path); since we no longer ignore the exit status of sed, we also have to ensure that the file exists before executing sed

 package/pkg-generic.mk | 2 ++
 1 file changed, 2 insertions(+)

package/pkg-generic.mk +0 -0

index bd47ca1964..8ebbacc180 100644

package/pkg-generic.mk GLOBAL_INSTRUMENTATION_HOOKS += step_time · line 66

Quoted 4 lines

package/pkg-generic.mk +1 -0

@@ -63,6 +63,8 @@ GLOBAL_INSTRUMENTATION_HOOKS += step_time
 # $(2): base directory to search in
 # $(3): suffix of file  (optional)
 define step_pkg_size_inner
+	@>$(BUILD_DIR)/packages-file-list$(3).txt  

Have you tested that? In my little experience with dealing with the shell, this actually truncates the file to a zero-size, and thus we loose the existing package->file list, even for other packages.

So, just do as I suggested: use touch to ensure the file exists.

(Yeah, one could also use the append-redirection >> instead of touch, but is that trick really an improvement over touch? touch is well known, and virtually everyone will recognise a construct that uses touch before sed-ing a file.)

Quoted 7 lines

package/pkg-generic.mk +1 -0

+	$(SED) '/^$(1),/d' $(BUILD_DIR)/packages-file-list$(3).txt
 	cd $(2); \
 	find . \( -type f -o -type l \) \
 		-newer $($(PKG)_DIR)/.stamp_built \

Signature · 1 line

2.17.0

When rebuilding a package, simply appending the package's file list to the global list means that the package list grows for every rebuild, as does the time taken to check for files installed by multiple packages. Furthermore, we get false positives where a file is reported as being installed by multiple copies of the same package.

With this approach we may end up with orphaned files in the target filesystem if a package that has been updated and rebuilt no longer installs the same set of files, but we know that only a clean build will produce reliable results. In fact it may be helpful to identify these orphaned files as evidence that the build is not clean.

Signed-off-by John Keeping <[email protected]>

---
Changes in v3:
- Use touch instead of redirect to avoid truncating the file

 package/pkg-generic.mk | 2 ++
 1 file changed, 2 insertions(+)

package/pkg-generic.mk +2 -0

index bd47ca1964..87c856407d 100644
@@ -63,6 +63,8 @@ GLOBAL_INSTRUMENTATION_HOOKS += step_time
 # $(2): base directory to search in
 # $(3): suffix of file  (optional)
 define step_pkg_size_inner
+	@touch $(BUILD_DIR)/packages-file-list$(3).txt
+	$(SED) '/^$(1),/d' $(BUILD_DIR)/packages-file-list$(3).txt
 	cd $(2); \
 	find . \( -type f -o -type l \) \
 		-newer $($(PKG)_DIR)/.stamp_built \

Signature · 1 line

2.17.0

When rebuilding a package, simply appending the package's file list to the global list means that the package list grows for every rebuild, as does the time taken to check for files installed by multiple packages. Furthermore, we get false positives where a file is reported as being installed by multiple copies of the same package.

With this approach we may end up with orphaned files in the target filesystem if a package that has been updated and rebuilt no longer installs the same set of files, but we know that only a clean build will produce reliable results. In fact it may be helpful to identify these orphaned files as evidence that the build is not clean.

Signed-off-by John Keeping <[email protected]>

---
Changes in v3:
- Use touch instead of redirect to avoid truncating the file

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

"John" == John Keeping <[email protected]> writes:

When rebuilding a package, simply appending the package's file list to the global list means that the package list grows for every rebuild, as does the time taken to check for files installed by multiple packages. Furthermore, we get false positives where a file is reported as being installed by multiple copies of the same package.

With this approach we may end up with orphaned files in the target filesystem if a package that has been updated and rebuilt no longer installs the same set of files, but we know that only a clean build will produce reliable results. In fact it may be helpful to identify these orphaned files as evidence that the build is not clean.

Signed-off-by John Keeping <[email protected]>

---
Changes in v3:
- Use touch instead of redirect to avoid truncating the file

When rebuilding a package, simply appending the package's file list to the global list means that the package list grows for every rebuild, as

does the time taken to check for files installed by multiple packages. Furthermore, we get false positives where a file is reported as being installed by multiple copies of the same package.

With this approach we may end up with orphaned files in the target filesystem if a package that has been updated and rebuilt no longer installs the same set of files, but we know that only a clean build will produce reliable results. In fact it may be helpful to identify these orphaned files as evidence that the build is not clean.

Signed-off-by John Keeping <[email protected]>

---
Changes in v3:
- Use touch instead of redirect to avoid truncating the file

 package/pkg-generic.mk | 2 ++
 1 file changed, 2 insertions(+)

package/pkg-generic.mk +2 -0

index bd47ca1964..87c856407d 100644
@@ -63,6 +63,8 @@ GLOBAL_INSTRUMENTATION_HOOKS += step_time
 # $(2): base directory to search in
 # $(3): suffix of file  (optional)
 define step_pkg_size_inner
+	@touch $(BUILD_DIR)/packages-file-list$(3).txt
+	$(SED) '/^$(1),/d' $(BUILD_DIR)/packages-file-list$(3).txt
 	cd $(2); \
 	find . \( -type f -o -type l \) \
 		-newer $($(PKG)_DIR)/.stamp_built \

Signature · 1 line

2.17.0

John. All,

On 2018-05-01 13:28 +0100, John Keeping spake thusly:

Quoted 2 lines

When rebuilding a package, simply appending the package's file list to the global list means that the package list grows for every rebuild, as

Is there a particular issue with those three files growing on rebuild?

Compared to the rest of the build tree, those three files have very small sizes in fact, and if they eventually end up big enough to be a substantial part of the build tree, it's most probably time for a clean build from scratch anyway, no?

Quoted 4 lines

On 2018-05-01 13:28 +0100, John Keeping spake thusly:

Quoted 2 lines

When rebuilding a package, simply appending the package's file list to the global list means that the package list grows for every rebuild, as

Quoted 1 line

Is there a particular issue with those three files growing on rebuild?

The problem is that not only does this makes the file list grow, it also adds loads of incorrect data: when rebuilding a package that was compiled early, all files modified since the package was built will be included. So the list will include files that have been installed by this package and all packages compiled after it. For some core packages, this usually means "almost everything".

Quoted 4 lines

Compared to the rest of the build tree, those three files have very small sizes in fact, and if they eventually end up big enough to be a substantial part of the build tree, it's most probably time for a clean build from scratch anyway, no?

Having a big list would not a problem if nobody used the content of this file, but some tools scans it and use it to check files. After many recompiling, I remember having problems with one tool taking a considerable amount of time for each build because of all the duplicates in the list. But i don't remember which tool it was.

Nicolas, All,

On 2019-01-04 16:30 +0100, Nicolas Cavallari spake thusly:

Quoted 5 lines · Yann E. MORIN

Quoted 4 lines

On 2018-05-01 13:28 +0100, John Keeping spake thusly:

Quoted 2 lines

When rebuilding a package, simply appending the package's file list to the global list means that the package list grows for every rebuild, as

Quoted 8 lines

Quoted 1 line

Is there a particular issue with those three files growing on rebuild?

The problem is that not only does this makes the file list grow, it also adds loads of incorrect data: when rebuilding a package that was compiled early, all files modified since the package was built will be included. So the list will include files that have been installed by this package and all packages compiled after it. For some core packages, this usually means "almost everything".

[...]>
Still, I am not usre I understood your problem anyway, so can you explain in more details, and provide an easy reproducing sequence (based on the git master)?

Quoted 10 lines

Quoted 4 lines

Compared to the rest of the build tree, those three files have very small sizes in fact, and if they eventually end up big enough to be a substantial part of the build tree, it's most probably time for a clean build from scratch anyway, no?

Having a big list would not a problem if nobody used the content of this file, but some tools scans it and use it to check files. After many recompiling, I remember having problems with one tool taking a considerable amount of time for each build because of all the duplicates in the list. But i don't remember which tool it was.

Are you talking about a tool in Buildroot? Can you try to provide more details, please? Are you still able to reproduce the slowness on the current git master?

The only two tools that make use of the packages-file-list.txt are check-uniq-files (which actually checks that files are not installed by two or more packages), and support/scripts/size-stats, which graphs a pie-chart representing the size contribution of each package to the content of target/. The former is always run at the end of the build, in the target-finalize step, while the latter is only run on-demand.

And in any case, if the list becomes *so big* that reading it takes an insane amount of time, then probably your host/ staging/ and target/ directories are in a state that is so far from being clean that you probably should condider a rebuild from scratch at that point.

make qemu_x86_defconfig
make
make ifupdown-scripts-reinstall
grep "ifupdown-scripts" output/build/packages-file-list.txt

=>
ifupdown-scripts,./bin/busybox

ifupdown-scripts,./lib/libuClibc-1.0.31.so ifupdown-scripts,./lib/modules/4.16.7/modules.dep.bin

ifupdown-scripts,./lib32

Which is almost the whole system.

Now why is that ? The answer is here:

find . \( -type f -o -type l \) \
          -newer $($(PKG)_DIR)/.stamp_built \
          -exec printf '$(1),%s\n' {} + \
          >> $(BUILD_DIR)/packages-file-list$(3).txt

Since .stamp_built has not been modified since the first time the package was built, this will list all files that have been installed after ifupdown-scripts was built.

With a bigger system, the list of files can be much larger. And without this patch, the list would grow by maybe the size of the whole system each time an early package is reinstalled.

Quoted 15 lines

Quoted 10 lines

Quoted 4 lines

Compared to the rest of the build tree, those three files have very small sizes in fact, and if they eventually end up big enough to be a substantial part of the build tree, it's most probably time for a clean build from scratch anyway, no?

Having a big list would not a problem if nobody used the content of this file, but some tools scans it and use it to check files. After many recompiling, I remember having problems with one tool taking a considerable amount of time for each build because of all the duplicates in the list. But i don't remember which tool it was.

Are you talking about a tool in Buildroot? Can you try to provide more details, please? Are you still able to reproduce the slowness on the current git master?

Quoted 6 lines

The only two tools that make use of the packages-file-list.txt are check-uniq-files (which actually checks that files are not installed by two or more packages), and support/scripts/size-stats, which graphs a pie-chart representing the size contribution of each package to the content of target/. The former is always run at the end of the build, in the target-finalize step, while the latter is only run on-demand.

I don't remember which tool it was, but it was probably check-bin-arch.

It will scan the package file list and call readelf on all files. If the package file list has duplicates, then readelf will be called multiple times on the same files. And it is also run at make $pkg-reinstall times, which fits my memory of make $pkg-reinstall being slower over time.