instrumentation: extend packages-file-list.txt
v2 cover instrumentation: extend packages-file-list.txt
From: Thomas De Schampheleire <[email protected]>
The main goal of this patch is to add symbolic link entries in packages-file-list.txt, in addition to regular files.
In v2, as suggested by Peter Korsgaard, directories and links to directories are added too.
Thomas De Schampheleire (2):
instrumentation: extract duplication to get list of installed files
instrumentation: extend packages-file-list.txt with symlinks and
directories
package/pkg-generic.mk | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-)
Signature · 1 line
2.10.2
PATCH v2 1/2 instrumentation: extract duplication to get list of installed files
From: Thomas De Schampheleire <[email protected]>
Before and after the building of each package, the instrumentation hooks are run. One of these hooks obtains the list of files installed by a package.
The code to obtain this list is currently duplicated in the start and end part of the hook. While the amount of duplication is currently small, a subsequent patch will make more changes to this code, increasing the duplication.
Therefore, split off into a helper function.
---
v2: new patch
package/pkg-generic.mk | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-)
package/pkg-generic.mk +7 -4
@@ -57,13 +57,17 @@ GLOBAL_INSTRUMENTATION_HOOKS += step_time # Hooks to collect statistics about installed files +define _step_pkg_size_get_file_list + (cd $(TARGET_DIR) ; find . -type f -print0 | xargs -0 md5sum) | sort > \ + $1 +endef + # This hook will be called before the target installation of a # package. We store in a file named .br_filelist_before the list of # files currently installed in the target. Note that the MD5 is also # stored, in order to identify if the files are overwritten. define step_pkg_size_start - (cd $(TARGET_DIR) ; find . -type f -print0 | xargs -0 md5sum) | sort > \ - $($(PKG)_DIR)/.br_filelist_before + $(call _step_pkg_size_get_file_list,$($(PKG)_DIR)/.br_filelist_before) endef # This hook will be called after the target installation of a
@@ -72,8 +76,7 @@ endef # a diff with the .br_filelist_before to compute the list of files # installed by this package. define step_pkg_size_end - (cd $(TARGET_DIR); find . -type f -print0 | xargs -0 md5sum) | sort > \ - $($(PKG)_DIR)/.br_filelist_after + $(call _step_pkg_size_get_file_list,$($(PKG)_DIR)/.br_filelist_after) comm -13 $($(PKG)_DIR)/.br_filelist_before $($(PKG)_DIR)/.br_filelist_after | \ while read hash file ; do \ echo "$(1),$${file}" >> $(BUILD_DIR)/packages-file-list.txt ; \
Signature · 1 line
2.10.2
PATCH v2 1/2 instrumentation: extract duplication to get list of installed files
Quoted 5 lines · Thomas De Schampheleire
From: Thomas De Schampheleire <[email protected]>
Before and after the building of each package, the instrumentation hooks are run. One of these hooks obtains the list of files installed by a package.
Quoted 8 lines
The code to obtain this list is currently duplicated in the start and end part of the hook. While the amount of duplication is currently small, a subsequent patch will make more changes to this code, increasing the duplication.
Therefore, split off into a helper function.
Signed-off-by Thomas De Schampheleire <[email protected]>
Regards,
Arnout
Quoted 41 lines
---
v2: new patchpackage/pkg-generic.mk | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-)package/pkg-generic.mk +7
-4@@ -57,13 +57,17 @@ GLOBAL_INSTRUMENTATION_HOOKS += step_time # Hooks to collect statistics about installed files +define _step_pkg_size_get_file_list + (cd $(TARGET_DIR) ; find . -type f -print0 | xargs -0 md5sum) | sort > \ + $1 +endef + # This hook will be called before the target installation of a # package. We store in a file named .br_filelist_before the list of # files currently installed in the target. Note that the MD5 is also # stored, in order to identify if the files are overwritten. define step_pkg_size_start - (cd $(TARGET_DIR) ; find . -type f -print0 | xargs -0 md5sum) | sort > \ - $($(PKG)_DIR)/.br_filelist_before + $(call _step_pkg_size_get_file_list,$($(PKG)_DIR)/.br_filelist_before) endef # This hook will be called after the target installation of a@@ -72,8 +76,7 @@ endef # a diff with the .br_filelist_before to compute the list of files # installed by this package. define step_pkg_size_end - (cd $(TARGET_DIR); find . -type f -print0 | xargs -0 md5sum) | sort > \ - $($(PKG)_DIR)/.br_filelist_after + $(call _step_pkg_size_get_file_list,$($(PKG)_DIR)/.br_filelist_after) comm -13 $($(PKG)_DIR)/.br_filelist_before $($(PKG)_DIR)/.br_filelist_after | \ while read hash file ; do \ echo "$(1),$${file}" >> $(BUILD_DIR)/packages-file-list.txt ; \
Signature · 6 lines
Arnout Vandecappelle arnout at mind be Senior Embedded Software Architect +32-16-286500 Essensium/Mind http://www.mind.be G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle GPG fingerprint: 7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
PATCH v2 2/2 instrumentation: extend packages-file-list.txt with symlinks and directories
In reply to Thomas De SchampheleireFrom: Thomas De Schampheleire <[email protected]>
Using the package instrumentation hooks, a file packages-file-list.txt is created containing the list of files installed by a package with their size.
Due to the use of 'find -type f', symbolic links to files are not included in this file list. Since the original purpose of this file was to calculate the total size of a package and symbolic links have file size 0, this was not a problem.
However, if packages-file-list.txt is reused for other purposes, for example to get a complete list of files installed by a package regardless of size, symbolic links to files are important too.
Likewise, to get a complete view of what a package installs, directories should be included too.
Update the instrumentation hook accordingly.
Although for files an md5sum is taken, we cannot do this for directories. Instead, mimic the output of md5sum with a fake hash string.
Note: for directories that are used by several packages, e.g. /etc/init.d/, the package that created the directory will be treated as the 'owner' of that directory. This gives a somewhat distorted view of ownership. Similarly, the package size reported by 'make graph-size' (based on packages-file-list.txt) will include the 'size' of a directory for that owner, which is reported as e.g. 4096 bytes.
---
v2: add directory information
package/pkg-generic.mk | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)
package/pkg-generic.mk +6 -2
@@ -58,8 +58,12 @@ GLOBAL_INSTRUMENTATION_HOOKS += step_time # Hooks to collect statistics about installed files define _step_pkg_size_get_file_list - (cd $(TARGET_DIR) ; find . -type f -print0 | xargs -0 md5sum) | sort > \ - $1 + (cd $(TARGET_DIR) ; \ + ( \ + find . -xtype f -print0 | xargs -0 md5sum ; \ + find . -xtype d -print0 | xargs -0 -I{} printf 'directory {}\n'; \ + ) \ + ) | sort > $1 endef # This hook will be called before the target installation of a
Signature · 1 line
2.10.2
PATCH v2 2/2 instrumentation: extend packages-file-list.txt with symlinks and directories
Quoted 5 lines · Thomas De Schampheleire
From: Thomas De Schampheleire <[email protected]>
Using the package instrumentation hooks, a file packages-file-list.txt is created containing the list of files installed by a package with their size.
Quoted 8 lines
Due to the use of 'find -type f', symbolic links to files are not included in this file list. Since the original purpose of this file was to calculate the total size of a package and symbolic links have file size 0, this was not a problem.
However, if packages-file-list.txt is reused for other purposes, for example to get a complete list of files installed by a package regardless of size, symbolic links to files are important too.
I'm absolutely +1 with the purpose of this patch. In fact, we should be
collecting file lists for host and staging as well. Cfr. [1].
Quoted 6 lines
Likewise, to get a complete view of what a package installs, directories should be included too.
Update the instrumentation hook accordingly.
Although for files an md5sum is taken, we cannot do this for directories. Instead, mimic the output of md5sum with a fake hash string.
The logic would be that for symlinks, you'd include the file pointed to rather
than a hash.
Quoted 4 lines
Note: for directories that are used by several packages, e.g.
/etc/init.d/, the package that created the directory will be treated as the 'owner' of that directory. This gives a somewhat distorted view of ownership.
This seems reasonable to me.
Quoted 3 lines
Similarly, the package size reported by 'make graph-size' (based on packages-file-list.txt) will include the 'size' of a directory for that owner, which is reported as e.g. 4096 bytes.
I think 'make graph-size' should filter out the directories. Currently not
possible since the package file list contains just package + filename, but with a per-package file list as introduced in [1] it becomes feasible.
Quoted 13 lines
Signed-off-by Thomas De Schampheleire <[email protected]>---
v2: add directory informationpackage/pkg-generic.mk | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)package/pkg-generic.mk +0
-0
package/pkg-generic.mk GLOBAL_INSTRUMENTATION_HOOKS += step_time · line 64
Quoted 9 lines
package/pkg-generic.mk +4
-2@@ -58,8 +58,12 @@ GLOBAL_INSTRUMENTATION_HOOKS += step_time # Hooks to collect statistics about installed files define _step_pkg_size_get_file_list - (cd $(TARGET_DIR) ; find . -type f -print0 | xargs -0 md5sum) | sort > \ - $1 + (cd $(TARGET_DIR) ; \ + ( \ + find . -xtype f -print0 | xargs -0 md5sum ; \ + find . -xtype d -print0 | xargs -0 -I{} printf 'directory {}\n'; \
I don't see symlinks being handled here, or am I missing something?
Would it be too crazy to do things in a single find command?
find . \( -xtype f -exec md5sum '{}' \; \) -o \
\( -xtype d -exec printf 'directory %s' '{}' \; Regards,
Arnout
Quoted 6 lines
package/pkg-generic.mk +2
-0+ ) \ + ) | sort > $1 endef # This hook will be called before the target installation of a
Signature · 6 lines
Arnout Vandecappelle arnout at mind be Senior Embedded Software Architect +32-16-286500 Essensium/Mind http://www.mind.be G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle GPG fingerprint: 7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
PATCH v2 2/2 instrumentation: extend packages-file-list.txt with symlinks and directories
Hi Arnout,
Thanks for your review, please see below...
2017-07-03 16:47 GMT+02:00 Arnout Vandecappelle <[email protected]>:
Quoted 23 lines · Thomas De Schampheleire
Quoted 5 lines
From: Thomas De Schampheleire <[email protected]>
Using the package instrumentation hooks, a file packages-file-list.txt is created containing the list of files installed by a package with their size.
Quoted 8 lines
Due to the use of 'find -type f', symbolic links to files are not included in this file list. Since the original purpose of this file was to calculate the total size of a package and symbolic links have file size 0, this was not a problem.
However, if packages-file-list.txt is reused for other purposes, for example to get a complete list of files installed by a package regardless of size, symbolic links to files are important too.
I'm absolutely +1 with the purpose of this patch. In fact, we should be
collecting file lists for host and staging as well. Cfr. [1].Quoted 3 lines
Likewise, to get a complete view of what a package installs, directories should be included too.
Quoted 6 lines
Quoted 3 lines
Update the instrumentation hook accordingly.
Although for files an md5sum is taken, we cannot do this for directories. Instead, mimic the output of md5sum with a fake hash string.The logic would be that for symlinks, you'd include the file pointed to rather
than a hash.
Yes, seems a fair option.
Quoted 31 lines
Quoted 5 lines
Note: for directories that are used by several packages, e.g.
/etc/init.d/, the package that created the directory will be treated as the 'owner' of that directory. This gives a somewhat distorted view of ownership.This seems reasonable to me.
Quoted 3 lines
Similarly, the package size reported by 'make graph-size' (based on packages-file-list.txt) will include the 'size' of a directory for that owner, which is reported as e.g. 4096 bytes.
I think 'make graph-size' should filter out the directories. Currently not
possible since the package file list contains just package + filename, but with a per-package file list as introduced in [1] it becomes feasible.Quoted 15 lines
Signed-off-by Thomas De Schampheleire <[email protected]>---
v2: add directory informationpackage/pkg-generic.mk | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)package/pkg-generic.mk +0
-0@@ -58,8 +58,12 @@ GLOBAL_INSTRUMENTATION_HOOKS += step_time # Hooks to collect statistics about installed files
Quoted 9 lines
package/pkg-generic.mk GLOBAL_INSTRUMENTATION_HOOKS += step_time · line 64
Quoted 7 lines
package/pkg-generic.mk +4
-2define _step_pkg_size_get_file_list - (cd $(TARGET_DIR) ; find . -type f -print0 | xargs -0 md5sum) | sort > \ - $1 + (cd $(TARGET_DIR) ; \ + ( \ + find . -xtype f -print0 | xargs -0 md5sum ; \ + find . -xtype d -print0 | xargs -0 -I{} printf 'directory {}\n'; \I don't see symlinks being handled here, or am I missing something?
the clue is in the 'xtype', 'xtype f' is true if the target is a file or a link to a file, 'xtype d' is true if target is directory or link to directory.
Quoted 5 lines
Would it be too crazy to do things in a single find command?
find . \( -xtype f -exec md5sum '{}' \; \) -o \ \( -xtype d -exec printf 'directory %s' '{}' \;
I'm fine with either way but xargs is more efficient as it groups arguments in a single invocation, at least for the md5sum call. To get the same behavior with find one would need to use the syntax "-exec command {} +" rather than "-exec command ;" .
Best regards,
Thomas
PATCH v2 2/2 instrumentation: extend packages-file-list.txt with symlinks and directories
Quoted 8 lines · Thomas De Schampheleire
Hi Arnout,
Thanks for your review, please see below...
2017-07-03 16:47 GMT+02:00 Arnout Vandecappelle <[email protected]>:
Quoted 3 lines · Thomas De Schampheleire
[snip]
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
define
_step_pkg_size_get_file_listpackage/pkg-generic.mk +4
-2- (cd $(TARGET_DIR) ; find . -type f -print0 | xargs -0 md5sum) | sort > \ - $1 + (cd $(TARGET_DIR) ; \ + ( \ + find . -xtype f -print0 | xargs -0 md5sum ; \ + find . -xtype d -print0 | xargs -0 -I{} printf 'directory {}\n'; \
Quoted 5 lines
Quoted 1 line
I don't see symlinks being handled here, or am I missing something?
the clue is in the 'xtype', 'xtype f' is true if the target is a file or a link to a file, 'xtype d' is true if target is directory or link to directory.
Thanks, I missed that. But if we want to store symlinks explicitly in the file
with the -> construct it'd need an explicit -type l, right?
Quoted 11 lines
Quoted 6 lines
Would it be too crazy to do things in a single find command?
find . \( -xtype f -exec md5sum '{}' \; \) -o \ \( -xtype d -exec printf 'directory %s' '{}' \;I'm fine with either way but xargs is more efficient as it groups arguments in a single invocation, at least for the md5sum call. To get the same behavior with find one would need to use the syntax "-exec command {} +" rather than "-exec command ;" .
Great! I like it! For the symlink part we still may need xargs though.
Regards,
Arnout
Quoted 4 lines
Best regards,
Thomas
Signature · 6 lines
Arnout Vandecappelle arnout at mind be Senior Embedded Software Architect +32-16-286500 Essensium/Mind http://www.mind.be G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle GPG fingerprint: 7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
PATCH v2 2/2 instrumentation: extend packages-file-list.txt with symlinks and directories
In reply to Thomas De SchampheleireQuoted 21 lines · Thomas De Schampheleire
From: Thomas De Schampheleire <[email protected]>
Using the package instrumentation hooks, a file packages-file-list.txt is created containing the list of files installed by a package with their size.
Due to the use of 'find -type f', symbolic links to files are not included in this file list. Since the original purpose of this file was to calculate the total size of a package and symbolic links have file size 0, this was not a problem.
However, if packages-file-list.txt is reused for other purposes, for example to get a complete list of files installed by a package regardless of size, symbolic links to files are important too.
Likewise, to get a complete view of what a package installs, directories should be included too.
Update the instrumentation hook accordingly.
Although for files an md5sum is taken, we cannot do this for directories. Instead, mimic the output of md5sum with a fake hash string.
Quoted 8 lines
Note: for directories that are used by several packages, e.g.
/etc/init.d/, the package that created the directory will be treated as the 'owner' of that directory. This gives a somewhat distorted view of ownership. Similarly, the package size reported by 'make graph-size' (based on packages-file-list.txt) will include the 'size' of a directory for that owner, which is reported as e.g. 4096 bytes.Signed-off-by Thomas De Schampheleire <[email protected]>
In the end, I've applied both to next as-is, thanks!
Regards,
Arnout
Signature · 6 lines
Arnout Vandecappelle arnout at mind be Senior Embedded Software Architect +32-16-286500 Essensium/Mind http://www.mind.be G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle GPG fingerprint: 7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF