core/pkg-generic: store package<->files for staging and host

To compute the list of files added by a package, we first store the list of files before the install, do the install, list the files after the install, and finally compare the two lists. The two lists are stored in dot-files, hidden in the package's build dir.

To compute the list of files added by a package, we first store the list of files before the install, do the install, list the files after the install, and finally compare the two lists. The two lists are stored in dot-files, hidden in the package's build dir.

We currently keep those two files, and only list the files installed in target/

In followup patches, we'll also list files installed in staging/ as well as files installed in host/.

Rather than add even more internal, hidden files in the package build dir, we'll just re-use the same two temporary files to store the bfore

and after lists.

So, remove them after the comparison is done.

Signed-off-by "Yann E. MORIN" <[email protected]>

Cc Thomas Petazzoni <[email protected]>

---
 package/pkg-generic.mk | 1 +
 1 file changed, 1 insertion(+)

package/pkg-generic.mk +1 -0

index 31dbc54..6b3c69c 100644
@@ -78,6 +78,7 @@ define step_pkg_size_end
 		while read hash file ; do \
 			echo "$(1),$${file}" >> $(BUILD_DIR)/packages-file-list.txt ; \
 		done
+	rm -f $($(PKG)_DIR)/.br_filelist_before $($(PKG)_DIR)/.br_filelist_after
 endef
 
 define step_pkg_size

s/rmove/remove/

Quoted 5 lines · Yann E. MORIN

To compute the list of files added by a package, we first store the list of files before the install, do the install, list the files after the install, and finally compare the two lists. The two lists are stored in dot-files, hidden in the package's build dir.

Quoted 8 lines

We currently keep those two files, and only list the files installed in target/

In followup patches, we'll also list files installed in staging/ as well as files installed in host/.

Rather than add even more internal, hidden files in the package build dir, we'll just re-use the same two temporary files to store the bfore

                                                                    ^e

Quoted 1 line

and after lists.

Actually, I like to have as much as possible build info lingering around, so
I'd tend not to do this removal and instead rename to something with 'target' in it. For example, I think we have no other way at the moment to detect when two packages write the same file.

IMO having internal hidden files in the package
build dir doesn't hurt anyone. I certainly do like the applied patches list!

That said, I don't see any other reason to keep those files, and we probably
want explicit support for detecting collisions, so I'm OK with removing them after all.

There is no need to redirect again and again for each new file added to the list; we can just redirect once and for all.

Quoted 2 lines

There is no need to redirect again and again for each new file added to the list; we can just redirect once and for all.

Typoes in the commit title:

pk-generic -> pkg-generic

then I'm not even sure was you meant with "redirect only wsa when"

And isntalled -> installed.

Thanks,

Thomas

Signature · 3 lines

Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

There is no need to redirect again and again for each new file added to the list; we can just redirect once and for all.

Signed-off-by "Yann E. MORIN" <[email protected]>

Cc Thomas Petazzoni <[email protected]>

---
 package/pkg-generic.mk | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

package/pkg-generic.mk +2 -2

index 6b3c69c..166efd6 100644
@@ -76,8 +76,8 @@ define step_pkg_size_end
 		$($(PKG)_DIR)/.br_filelist_after
 	comm -13 $($(PKG)_DIR)/.br_filelist_before $($(PKG)_DIR)/.br_filelist_after | \
 		while read hash file ; do \
-			echo "$(1),$${file}" >> $(BUILD_DIR)/packages-file-list.txt ; \
-		done
+			echo "$(1),$${file}" ; \
+		done >> $(BUILD_DIR)/packages-file-list.txt
 	rm -f $($(PKG)_DIR)/.br_filelist_before $($(PKG)_DIR)/.br_filelist_after
 endef
 

Currently, we store the list of files installed in target/ and associate each of them to the package that installed it.

However, we sometime may need to know what package installed which file

in staging/, for example to debug header collision, or in host/, to debug what package installed what host tool.

Enhance the step instrumentation to also generate the list for staging/ and host/.

Signed-off-by "Yann E. MORIN" <[email protected]>

Cc Thomas Petazzoni <[email protected]>

---
 package/pkg-generic.mk | 16 +++++++++++-----
 1 file changed, 11 insertions(+), 5 deletions(-)

package/pkg-generic.mk +3 -3

index 166efd6..725a543 100644
@@ -62,7 +62,7 @@ GLOBAL_INSTRUMENTATION_HOOKS += step_time
 # files currently installed in the target. Note that the MD5 is also
 # stored, in order to identify if the files are overwritten.
 define step_pkg_size_start
-	(cd $(TARGET_DIR) ; find . -type f -print0 | xargs -0 md5sum) | sort > \
+	(cd $(2) ; find . -type f -print0 | xargs -0 md5sum) | sort > \
 		$($(PKG)_DIR)/.br_filelist_before
 endef
 

@@ -72,19 +72,25 @@ endef
 # a diff with the .br_filelist_before to compute the list of files
 # installed by this package.
 define step_pkg_size_end
-	(cd $(TARGET_DIR); find . -type f -print0 | xargs -0 md5sum) | sort > \
+	(cd $(2); find . -type f -print0 | xargs -0 md5sum) | sort > \
 		$($(PKG)_DIR)/.br_filelist_after
 	comm -13 $($(PKG)_DIR)/.br_filelist_before $($(PKG)_DIR)/.br_filelist_after | \
 		while read hash file ; do \
 			echo "$(1),$${file}" ; \
-		done >> $(BUILD_DIR)/packages-file-list.txt
+		done >> $(BUILD_DIR)/packages-file-list$(3).txt
 	rm -f $($(PKG)_DIR)/.br_filelist_before $($(PKG)_DIR)/.br_filelist_after
 endef
 
 define step_pkg_size
 	$(if $(filter install-target,$(2)),\

Hello All!

This small series does for staging and host as we do for target:
store the list of installed files and relate them to the package that installed each.

---
Changes v1 -> v2:
- split in three patches
- in addition to staging, also store for host

Regards,
Yann E. MORIN.

The following changes since commit 4f50f4928ca3855d9d1b1b9699d2cb37e4ca65da

Add out-of-source U-Boot device tree support (2017-04-05 22:29:23 +0200)

are available in the git repository at:

git://git.buildroot.org/~ymorin/git/buildroot.git

for you to fetch changes up to 44910f2975b83c91148d08f3df11ea048eea0609

core/pkg-generic: store file->package list for staging and host too (2017-04-05 23:15:27 +0200)

----------------------------------------------------------------
Yann E. MORIN (3):
core/pkg-generic: rmove intermediate file-list files core/pk-generic: redirect only wsa when listing package's isntalled files core/pkg-generic: store file->package list for staging and host too

 package/pkg-generic.mk | 19 +++++++++++++------
 1 file changed, 13 insertions(+), 6 deletions(-)