Skip to contentSkip to conversation

instrumentation: extend packages-file-list.txt

PATCH New Thomas De Schampheleire 2 participants 8 messages Feb 5, 2017 – Aug 3, 2017

lore
Share & tools
lore
#0 lore raw

v2 cover instrumentation: extend packages-file-list.txt

From: Thomas De Schampheleire <[email protected]>

The main goal of this patch is to add symbolic link entries in packages-file-list.txt, in addition to regular files.

In v2, as suggested by Peter Korsgaard, directories and links to directories are added too.

Thomas De Schampheleire (2):
instrumentation: extract duplication to get list of installed files
instrumentation: extend packages-file-list.txt with symlinks and
directories

 package/pkg-generic.mk | 15 +++++++++++----
 1 file changed, 11 insertions(+), 4 deletions(-)
Signature · 1 line
2.10.2
#1 lore raw

PATCH v2 1/2 instrumentation: extract duplication to get list of installed files

From: Thomas De Schampheleire <[email protected]>

Before and after the building of each package, the instrumentation hooks are run. One of these hooks obtains the list of files installed by a package.

The code to obtain this list is currently duplicated in the start and end part of the hook. While the amount of duplication is currently small, a subsequent patch will make more changes to this code, increasing the duplication.

Therefore, split off into a helper function.

---
v2: new patch

 package/pkg-generic.mk | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)
package/pkg-generic.mk +7 -4
index 3ca71b0..077b02b 100644
@@ -57,13 +57,17 @@ GLOBAL_INSTRUMENTATION_HOOKS += step_time
 
 # Hooks to collect statistics about installed files
 
+define _step_pkg_size_get_file_list
+	(cd $(TARGET_DIR) ; find . -type f -print0 | xargs -0 md5sum) | sort > \
+		$1
+endef
+
 # This hook will be called before the target installation of a
 # package. We store in a file named .br_filelist_before the list of
 # files currently installed in the target. Note that the MD5 is also
 # stored, in order to identify if the files are overwritten.
 define step_pkg_size_start
-	(cd $(TARGET_DIR) ; find . -type f -print0 | xargs -0 md5sum) | sort > \
-		$($(PKG)_DIR)/.br_filelist_before
+	$(call _step_pkg_size_get_file_list,$($(PKG)_DIR)/.br_filelist_before)
 endef
 
 # This hook will be called after the target installation of a
@@ -72,8 +76,7 @@ endef
 # a diff with the .br_filelist_before to compute the list of files
 # installed by this package.
 define step_pkg_size_end
-	(cd $(TARGET_DIR); find . -type f -print0 | xargs -0 md5sum) | sort > \
-		$($(PKG)_DIR)/.br_filelist_after
+	$(call _step_pkg_size_get_file_list,$($(PKG)_DIR)/.br_filelist_after)
 	comm -13 $($(PKG)_DIR)/.br_filelist_before $($(PKG)_DIR)/.br_filelist_after | \
 		while read hash file ; do \
 			echo "$(1),$${file}" >> $(BUILD_DIR)/packages-file-list.txt ; \
Signature · 1 line
2.10.2
#2 lore raw

PATCH v2 1/2 instrumentation: extract duplication to get list of installed files

Quoted 5 lines · Thomas De Schampheleire

From: Thomas De Schampheleire <[email protected]>

Before and after the building of each package, the instrumentation hooks are run. One of these hooks obtains the list of files installed by a package.

Quoted 8 lines

The code to obtain this list is currently duplicated in the start and end part of the hook. While the amount of duplication is currently small, a subsequent patch will make more changes to this code, increasing the duplication.

Therefore, split off into a helper function.

Regards,
Arnout

Quoted 41 lines

---
v2: new patch

 package/pkg-generic.mk | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)
package/pkg-generic.mk +7 -4
index 3ca71b0..077b02b 100644
@@ -57,13 +57,17 @@ GLOBAL_INSTRUMENTATION_HOOKS += step_time
 
 # Hooks to collect statistics about installed files
 
+define _step_pkg_size_get_file_list
+	(cd $(TARGET_DIR) ; find . -type f -print0 | xargs -0 md5sum) | sort > \
+		$1
+endef
+
 # This hook will be called before the target installation of a
 # package. We store in a file named .br_filelist_before the list of
 # files currently installed in the target. Note that the MD5 is also
 # stored, in order to identify if the files are overwritten.
 define step_pkg_size_start
-	(cd $(TARGET_DIR) ; find . -type f -print0 | xargs -0 md5sum) | sort > \
-		$($(PKG)_DIR)/.br_filelist_before
+	$(call _step_pkg_size_get_file_list,$($(PKG)_DIR)/.br_filelist_before)
 endef
 
 # This hook will be called after the target installation of a
@@ -72,8 +76,7 @@ endef
 # a diff with the .br_filelist_before to compute the list of files
 # installed by this package.
 define step_pkg_size_end
-	(cd $(TARGET_DIR); find . -type f -print0 | xargs -0 md5sum) | sort > \
-		$($(PKG)_DIR)/.br_filelist_after
+	$(call _step_pkg_size_get_file_list,$($(PKG)_DIR)/.br_filelist_after)
 	comm -13 $($(PKG)_DIR)/.br_filelist_before $($(PKG)_DIR)/.br_filelist_after | \
 		while read hash file ; do \
 			echo "$(1),$${file}" >> $(BUILD_DIR)/packages-file-list.txt ; \
Signature · 6 lines
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
#3 lore raw

PATCH v2 2/2 instrumentation: extend packages-file-list.txt with symlinks and directories

In reply to Thomas De Schampheleire

From: Thomas De Schampheleire <[email protected]>

Using the package instrumentation hooks, a file packages-file-list.txt is created containing the list of files installed by a package with their size.

Due to the use of 'find -type f', symbolic links to files are not included in this file list. Since the original purpose of this file was to calculate the total size of a package and symbolic links have file size 0, this was not a problem.

However, if packages-file-list.txt is reused for other purposes, for example to get a complete list of files installed by a package regardless of size, symbolic links to files are important too.

Likewise, to get a complete view of what a package installs, directories should be included too.

Update the instrumentation hook accordingly.
Although for files an md5sum is taken, we cannot do this for directories. Instead, mimic the output of md5sum with a fake hash string.

Note: for directories that are used by several packages, e.g. /etc/init.d/, the package that created the directory will be treated as the 'owner' of that directory. This gives a somewhat distorted view of ownership. Similarly, the package size reported by 'make graph-size' (based on packages-file-list.txt) will include the 'size' of a directory for that owner, which is reported as e.g. 4096 bytes.

---
v2: add directory information

 package/pkg-generic.mk | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)
package/pkg-generic.mk +6 -2
index 077b02b..1beea32 100644
@@ -58,8 +58,12 @@ GLOBAL_INSTRUMENTATION_HOOKS += step_time
 # Hooks to collect statistics about installed files
 
 define _step_pkg_size_get_file_list
-	(cd $(TARGET_DIR) ; find . -type f -print0 | xargs -0 md5sum) | sort > \
-		$1
+	(cd $(TARGET_DIR) ; \
+		( \
+			find . -xtype f -print0 | xargs -0 md5sum ; \
+			find . -xtype d -print0 | xargs -0 -I{} printf 'directory  {}\n'; \
+		) \
+	) | sort > $1
 endef
 
 # This hook will be called before the target installation of a
Signature · 1 line
2.10.2
#4 lore raw

PATCH v2 2/2 instrumentation: extend packages-file-list.txt with symlinks and directories

Quoted 5 lines · Thomas De Schampheleire

From: Thomas De Schampheleire <[email protected]>

Using the package instrumentation hooks, a file packages-file-list.txt is created containing the list of files installed by a package with their size.

Quoted 8 lines

Due to the use of 'find -type f', symbolic links to files are not included in this file list. Since the original purpose of this file was to calculate the total size of a package and symbolic links have file size 0, this was not a problem.

However, if packages-file-list.txt is reused for other purposes, for example to get a complete list of files installed by a package regardless of size, symbolic links to files are important too.

I'm absolutely +1 with the purpose of this patch. In fact, we should be
collecting file lists for host and staging as well. Cfr. [1].

Quoted 6 lines

Likewise, to get a complete view of what a package installs, directories should be included too.

Update the instrumentation hook accordingly.
Although for files an md5sum is taken, we cannot do this for directories. Instead, mimic the output of md5sum with a fake hash string.

The logic would be that for symlinks, you'd include the file pointed to rather
than a hash.

Quoted 4 lines

Note: for directories that are used by several packages, e.g. /etc/init.d/, the package that created the directory will be treated as the 'owner' of that directory. This gives a somewhat distorted view of ownership.

This seems reasonable to me.

Quoted 3 lines

Similarly, the package size reported by 'make graph-size' (based on packages-file-list.txt) will include the 'size' of a directory for that owner, which is reported as e.g. 4096 bytes.

I think 'make graph-size' should filter out the directories. Currently not
possible since the package file list contains just package + filename, but with a per-package file list as introduced in [1] it becomes feasible.

Quoted 13 lines

---
v2: add directory information

 package/pkg-generic.mk | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)
package/pkg-generic.mk +0 -0
index 077b02b..1beea32 100644

package/pkg-generic.mk GLOBAL_INSTRUMENTATION_HOOKS += step_time · line 64

Quoted 9 lines
package/pkg-generic.mk +4 -2
@@ -58,8 +58,12 @@ GLOBAL_INSTRUMENTATION_HOOKS += step_time
 # Hooks to collect statistics about installed files
 
 define _step_pkg_size_get_file_list
-	(cd $(TARGET_DIR) ; find . -type f -print0 | xargs -0 md5sum) | sort > \
-		$1
+	(cd $(TARGET_DIR) ; \
+		( \
+			find . -xtype f -print0 | xargs -0 md5sum ; \
+			find . -xtype d -print0 | xargs -0 -I{} printf 'directory  {}\n'; \

I don't see symlinks being handled here, or am I missing something?

Would it be too crazy to do things in a single find command?

	find . \( -xtype f -exec md5sum '{}' \; \) -o \
		\( -xtype d -exec printf 'directory %s' '{}' \;

Regards,
Arnout

[1] http://patchwork.ozlabs.org/patch/782702/

Quoted 6 lines
package/pkg-generic.mk +2 -0
+		) \
+	) | sort > $1
 endef
 
 # This hook will be called before the target installation of a
Signature · 6 lines
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
#5 lore raw

PATCH v2 2/2 instrumentation: extend packages-file-list.txt with symlinks and directories

Hi Arnout,

Thanks for your review, please see below...

2017-07-03 16:47 GMT+02:00 Arnout Vandecappelle <[email protected]>:

Quoted 23 lines · Thomas De Schampheleire
Quoted 5 lines

From: Thomas De Schampheleire <[email protected]>

Using the package instrumentation hooks, a file packages-file-list.txt is created containing the list of files installed by a package with their size.

Quoted 8 lines

Due to the use of 'find -type f', symbolic links to files are not included in this file list. Since the original purpose of this file was to calculate the total size of a package and symbolic links have file size 0, this was not a problem.

However, if packages-file-list.txt is reused for other purposes, for example to get a complete list of files installed by a package regardless of size, symbolic links to files are important too.

I'm absolutely +1 with the purpose of this patch. In fact, we should be
collecting file lists for host and staging as well. Cfr. [1].

Quoted 3 lines

Likewise, to get a complete view of what a package installs, directories should be included too.

Quoted 6 lines
Quoted 3 lines

Update the instrumentation hook accordingly.
Although for files an md5sum is taken, we cannot do this for directories. Instead, mimic the output of md5sum with a fake hash string.

The logic would be that for symlinks, you'd include the file pointed to rather
than a hash.

Yes, seems a fair option.

Quoted 31 lines
Quoted 5 lines

Note: for directories that are used by several packages, e.g. /etc/init.d/, the package that created the directory will be treated as the 'owner' of that directory. This gives a somewhat distorted view of ownership.

This seems reasonable to me.

Quoted 3 lines

Similarly, the package size reported by 'make graph-size' (based on packages-file-list.txt) will include the 'size' of a directory for that owner, which is reported as e.g. 4096 bytes.

I think 'make graph-size' should filter out the directories. Currently not
possible since the package file list contains just package + filename, but with a per-package file list as introduced in [1] it becomes feasible.

Quoted 15 lines

---
v2: add directory information

 package/pkg-generic.mk | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)
package/pkg-generic.mk +0 -0
index 077b02b..1beea32 100644
@@ -58,8 +58,12 @@ GLOBAL_INSTRUMENTATION_HOOKS += step_time
 # Hooks to collect statistics about installed files
 
Quoted 9 lines

package/pkg-generic.mk GLOBAL_INSTRUMENTATION_HOOKS += step_time · line 64

Quoted 7 lines
package/pkg-generic.mk +4 -2
 define _step_pkg_size_get_file_list
-     (cd $(TARGET_DIR) ; find . -type f -print0 | xargs -0 md5sum) | sort > \
-             $1
+     (cd $(TARGET_DIR) ; \
+             ( \
+                     find . -xtype f -print0 | xargs -0 md5sum ; \
+                     find . -xtype d -print0 | xargs -0 -I{} printf 'directory  {}\n'; \

I don't see symlinks being handled here, or am I missing something?

the clue is in the 'xtype', 'xtype f' is true if the target is a file or a link to a file, 'xtype d' is true if target is directory or link to directory.

Quoted 5 lines

Would it be too crazy to do things in a single find command?

        find . \( -xtype f -exec md5sum '{}' \; \) -o \
                \( -xtype d -exec printf 'directory %s' '{}' \;

I'm fine with either way but xargs is more efficient as it groups arguments in a single invocation, at least for the md5sum call. To get the same behavior with find one would need to use the syntax "-exec command {} +" rather than "-exec command ;" .

Best regards,
Thomas

#6 lore raw

PATCH v2 2/2 instrumentation: extend packages-file-list.txt with symlinks and directories

Quoted 8 lines · Thomas De Schampheleire

Hi Arnout,

Thanks for your review, please see below...

2017-07-03 16:47 GMT+02:00 Arnout Vandecappelle <[email protected]>:

Quoted 3 lines · Thomas De Schampheleire

[snip]

Quoted 8 lines
Quoted 8 lines
Quoted 7 lines

define _step_pkg_size_get_file_list

package/pkg-generic.mk +4 -2
-     (cd $(TARGET_DIR) ; find . -type f -print0 | xargs -0 md5sum) | sort > \
-             $1
+     (cd $(TARGET_DIR) ; \
+             ( \
+                     find . -xtype f -print0 | xargs -0 md5sum ; \
+                     find . -xtype d -print0 | xargs -0 -I{} printf 'directory  {}\n'; \
Quoted 5 lines
Quoted 1 line

I don't see symlinks being handled here, or am I missing something?

the clue is in the 'xtype', 'xtype f' is true if the target is a file or a link to a file, 'xtype d' is true if target is directory or link to directory.

Thanks, I missed that. But if we want to store symlinks explicitly in the file
with the -> construct it'd need an explicit -type l, right?

Quoted 11 lines
Quoted 6 lines

Would it be too crazy to do things in a single find command?

        find . \( -xtype f -exec md5sum '{}' \; \) -o \
                \( -xtype d -exec printf 'directory %s' '{}' \;

I'm fine with either way but xargs is more efficient as it groups arguments in a single invocation, at least for the md5sum call. To get the same behavior with find one would need to use the syntax "-exec command {} +" rather than "-exec command ;" .

Great! I like it! For the symlink part we still may need xargs though.

Regards,
Arnout

Quoted 4 lines

Best regards,
Thomas

Signature · 6 lines
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
#7 lore raw

PATCH v2 2/2 instrumentation: extend packages-file-list.txt with symlinks and directories

In reply to Thomas De Schampheleire
Quoted 21 lines · Thomas De Schampheleire

From: Thomas De Schampheleire <[email protected]>

Using the package instrumentation hooks, a file packages-file-list.txt is created containing the list of files installed by a package with their size.

Due to the use of 'find -type f', symbolic links to files are not included in this file list. Since the original purpose of this file was to calculate the total size of a package and symbolic links have file size 0, this was not a problem.

However, if packages-file-list.txt is reused for other purposes, for example to get a complete list of files installed by a package regardless of size, symbolic links to files are important too.

Likewise, to get a complete view of what a package installs, directories should be included too.

Update the instrumentation hook accordingly.
Although for files an md5sum is taken, we cannot do this for directories. Instead, mimic the output of md5sum with a fake hash string.

Quoted 8 lines

Note: for directories that are used by several packages, e.g. /etc/init.d/, the package that created the directory will be treated as the 'owner' of that directory. This gives a somewhat distorted view of ownership. Similarly, the package size reported by 'make graph-size' (based on packages-file-list.txt) will include the 'size' of a directory for that owner, which is reported as e.g. 4096 bytes.

In the end, I've applied both to next as-is, thanks!

Regards,
Arnout

Signature · 6 lines
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF