core/pkg-generic: store package<->files for staging and host
v3 cover core/pkg-generic: store package<->files for staging and host
Hello All!
This small series does for staging and host as we do for target:
store the list of installed files and relate them to the package that installed each.
In the end, it also reports files that are touched by more than one package, for target, staging and host files.
---
Changes v2 -> v3:
- make it python2 and python3 compatible (Baruch)
- typoes (Peter S.)
Changes v1 -> v2:
- split in three patches
- in addition to staging, also store for host
- report files touched by more than one package
Regards,
Yann E. MORIN.
The following changes since commit 5868ab5f22300e920474005e3ae4f78e0b210473
package/util-linux: drop _VERSION_MINOR variable (2017-10-28 14:24:55 +0200)
are available in the git repository at:
git://git.buildroot.org/~ymorin/git/buildroot.git
for you to fetch changes up to cd4d3c26158a9dc124b3ac584da2c61e004cbc8b
core: check files are not touched by more than one package (2017-10-28 16:02:56 +0200)
----------------------------------------------------------------
Yann E. MORIN (4):
core/pkg-generic: remove intermediate file-list files core/pkg-generic: redirect only when listing package's installed files core/pkg-generic: store file->package list for staging and host too
core: check files are not touched by more than one package
Makefile | 4 ++++ package/pkg-generic.mk | 45 ++++++++++++++++++++++++------------- support/scripts/check-uniq-files.py | 40 +++++++++++++++++++++++++++++++++ 3 files changed, 73 insertions(+), 16 deletions(-) create mode 100755 support/scripts/check-uniq-files.py
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH v3 1/4 core/pkg-generic: remove intermediate file-list files
To compute the list of files added by a package, we first store the list of files before the install, do the install, list the files after the install, and finally compare the two lists. The two lists are stored in dot-files, hidden in the package's build dir.
We currently keep those two files, and only list the files installed in target/
In followup patches, we'll also list files installed in staging/ as well as files installed in host/.
Rather than add even more internal, hidden files in the package build dir, we'll just re-use the same two temporary files to store the bfore and after lists.
So, remove them after the comparison is done.
--- package/pkg-generic.mk | 1 + 1 file changed, 1 insertion(+)
package/pkg-generic.mk +1 -0
@@ -85,6 +85,7 @@ define step_pkg_size_end while read hash file ; do \ echo "$(1),$${file}" >> $(BUILD_DIR)/packages-file-list.txt ; \ done + rm -f $($(PKG)_DIR)/.br_filelist_before $($(PKG)_DIR)/.br_filelist_after endef define step_pkg_size
Signature · 1 line
2.11.0
PATCH v3 1/4 core/pkg-generic: remove intermediate file-list files
Hello,
Quoted 5 lines · Yann E. MORIN
To compute the list of files added by a package, we first store the list of files before the install, do the install, list the files after the install, and finally compare the two lists. The two lists are stored in dot-files, hidden in the package's build dir.
Quoted 8 lines
We currently keep those two files, and only list the files installed in target/
In followup patches, we'll also list files installed in staging/ as well as files installed in host/.
Rather than add even more internal, hidden files in the package build dir, we'll just re-use the same two temporary files to store the bfore
bfore -> before
Quoted 9 lines
and after lists.
So, remove them after the comparison is done.
Signed-off-by "Yann E. MORIN" <[email protected]>Cc Thomas Petazzoni <[email protected]>--- package/pkg-generic.mk | 1 + 1 file changed, 1 insertion(+)
Applied to next, thanks. Interestingly, when I submitted the first iterations of the patches adding this file list collecting mechanism, I was also removing the files. And then between v3 and v4, I decided to keep them, with the following explanation:
They are now named .br_filelist_before and .br_filelist_after. They are also no longer removed after the step_pkg_size_end hook, as they don't consume that much space and could be useful for analysis. If anyone finds that annoying, I'll be happy to change this back to remove them.
:-)
As we discussed on IRC, the idea of re-using the
same .br_filelist_before and .br_filelist_after for both the staging and target installation is not going to work nicely with top-level parallel build, because target and staging installation for a given package can happen in parallel.
However, the whole file list collection mechanism will have to be re-thought for top-level parallel build, so I took your patch anyway.
Thanks!
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons Embedded Linux, Kernel and Android engineering http://free-electrons.com
PATCH v3 2/4 core/pkg-generic: redirect only when listing package's installed files
In reply to Yann E. MORINThere is no need to redirect again and again for each new file added to the list; we can just redirect once and for all.
--- package/pkg-generic.mk | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
package/pkg-generic.mk +2 -2
@@ -83,8 +83,8 @@ define step_pkg_size_end $(call _step_pkg_size_get_file_list,$($(PKG)_DIR)/.br_filelist_after) comm -13 $($(PKG)_DIR)/.br_filelist_before $($(PKG)_DIR)/.br_filelist_after | \ while read hash file ; do \ - echo "$(1),$${file}" >> $(BUILD_DIR)/packages-file-list.txt ; \ - done + echo "$(1),$${file}" ; \ + done >> $(BUILD_DIR)/packages-file-list.txt rm -f $($(PKG)_DIR)/.br_filelist_before $($(PKG)_DIR)/.br_filelist_after endef
Signature · 1 line
2.11.0
Thomas Petazzoni
AppliedPATCH v3 2/4 core/pkg-generic: redirect only when listing package's installed files
Hello,
Quoted 8 lines · Yann E. MORIN
There is no need to redirect again and again for each new file added to the list; we can just redirect once and for all.
Signed-off-by "Yann E. MORIN" <[email protected]>Cc Thomas Petazzoni <[email protected]>--- package/pkg-generic.mk | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
Applied to next, thanks.
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons Embedded Linux, Kernel and Android engineering http://free-electrons.com
PATCH v3 3/4 core/pkg-generic: store file->package list for staging and host too
In reply to Yann E. MORINCurrently, we store the list of files installed in target/ and associate each of them to the package that installed it.
However, we sometime may need to know what package installed which file in staging/, for example to debug header collision, or in host/, to debug what package installed what host tool.
Enhance the step instrumentation to also generate the list for staging/ and host/.
We maintain backward compatibility, for external script that wanted to parse the previously existing list, by not renaming the target-related package list. Only the staging- and host-related lists are named after staging and host.
--- package/pkg-generic.mk | 42 +++++++++++++++++++++++++++--------------- 1 file changed, 27 insertions(+), 15 deletions(-)
package/pkg-generic.mk +27 -15
@@ -58,7 +58,7 @@ GLOBAL_INSTRUMENTATION_HOOKS += step_time # Hooks to collect statistics about installed files define _step_pkg_size_get_file_list - (cd $(TARGET_DIR) ; \ + (cd $(2) ; \ ( \ find . -xtype f -print0 | xargs -0 md5sum ; \ find . -xtype d -print0 | xargs -0 -I{} printf 'directory {}\n'; \
@@ -66,32 +66,44 @@ define _step_pkg_size_get_file_list ) | sort > $1 endef -# This hook will be called before the target installation of a -# package. We store in a file named .br_filelist_before the list of -# files currently installed in the target. Note that the MD5 is also -# stored, in order to identify if the files are overwritten. +# This hook will be called before the installation of a package. We store in +# a file named .br_filelist_before the list of files currently installed. +# Note that the MD5 is also stored, in order to identify if the files are +# overwritten. +# $(1): package name (ignored) +# $(2): base directory to search in define step_pkg_size_start - $(call _step_pkg_size_get_file_list,$($(PKG)_DIR)/.br_filelist_before) + $(call _step_pkg_size_get_file_list,$($(PKG)_DIR)/.br_filelist_before,$(2)) endef -# This hook will be called after the target installation of a -# package. We store in a file named .br_filelist_after the list of -# files (and their MD5) currently installed in the target. We then do -# a diff with the .br_filelist_before to compute the list of files -# installed by this package. +# This hook will be called after the installation of a package. We store in +# a file named .br_filelist_after the list of files (and their MD5) currently +# installed. We then do a diff with the .br_filelist_before to compute the +# list of files installed by this package. +# The suffix is typically for the target variant, for legacy backward +# compatibility. +# $(1): package name (ignored) +# $(2): base directory to search in +# $(3): suffix of file (optional) define step_pkg_size_end - $(call _step_pkg_size_get_file_list,$($(PKG)_DIR)/.br_filelist_after) + $(call _step_pkg_size_get_file_list,$($(PKG)_DIR)/.br_filelist_after,$(2)) comm -13 $($(PKG)_DIR)/.br_filelist_before $($(PKG)_DIR)/.br_filelist_after | \ while read hash file ; do \ echo "$(1),$${file}" ; \ - done >> $(BUILD_DIR)/packages-file-list.txt + done >> $(BUILD_DIR)/packages-file-list$(3).txt rm -f $($(PKG)_DIR)/.br_filelist_before $($(PKG)_DIR)/.br_filelist_after endef define step_pkg_size $(if $(filter install-target,$(2)),\ - $(if $(filter start,$(1)),$(call step_pkg_size_start,$(3))) \ - $(if $(filter end,$(1)),$(call step_pkg_size_end,$(3)))) + $(if $(filter start,$(1)),$(call step_pkg_size_start,$(3),$(TARGET_DIR))) \ + $(if $(filter end,$(1)),$(call step_pkg_size_end,$(3),$(TARGET_DIR)))) + $(if $(filter install-staging,$(2)),\ + $(if $(filter start,$(1)),$(call step_pkg_size_start,$(3),$(STAGING_DIR),-staging)) \ + $(if $(filter end,$(1)),$(call step_pkg_size_end,$(3),$(STAGING_DIR),-staging))) + $(if $(filter install-host,$(2)),\ + $(if $(filter start,$(1)),$(call step_pkg_size_start,$(3),$(HOST_DIR),-host)) \ + $(if $(filter end,$(1)),$(call step_pkg_size_end,$(3),$(HOST_DIR),-host))) endef GLOBAL_INSTRUMENTATION_HOOKS += step_pkg_size
Signature · 1 line
2.11.0
Thomas Petazzoni
AppliedPATCH v3 3/4 core/pkg-generic: store file->package list for staging and host too
Hello,
package/pkg-generic.mk define _step_pkg_size_get_file_list · line 83
Quoted 10 lines · Yann E. MORIN
package/pkg-generic.mk +5
-5-# This hook will be called after the target installation of a -# package. We store in a file named .br_filelist_after the list of -# files (and their MD5) currently installed in the target. We then do -# a diff with the .br_filelist_before to compute the list of files -# installed by this package. +# This hook will be called after the installation of a package. We store in +# a file named .br_filelist_after the list of files (and their MD5) currently +# installed. We then do a diff with the .br_filelist_before to compute the +# list of files installed by this package. +# The suffix is typically for the target variant, for legacy backward
The suffix is typically *empty*
Applied to next with this fixed!
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons Embedded Linux and Kernel engineering http://free-electrons.com
PATCH v3 4/4 core: check files are not touched by more than one package
In reply to Yann E. MORINCurrently, we do nothing about packages that touch the same file: given a specific configuration, the result is reproducible (even though it might not be what the user expected) because the build order is guaranteed.
However, when we later introduce top-level parallel build, we will no longer be able to guarantee a build order, by the mere way of it being parallel. Reconcialliting all those modified files will be impossible to do automatically. The only way will be to refuse such situations.
As a preliminary step, introduce a helper script that detects files that are being moified by two or more packages, and reports them and the impacted packages, at the end of the build.
The list being reported at the end of the build will make it prominently visible in autobuilder results, so we can assess the problem, if any.
Later on, calling that helper script can be done right after the package installation step, to bail out early.
Thanks Arnout for the pythonist way to write default dictionaries! ;-)
Note: doing it in python rather than a shell script is impressively
faster: where the shell script takes ~1.2s on a minimalist build, the python script only takes ~0.015s, that is about 80 times faster.
---
Changes v2 -> v3:
- make it python2 *and* python3 compatible (Baruch)
- typoes (Peter S.)
--- Makefile | 4 ++++ support/scripts/check-uniq-files.py | 40 +++++++++++++++++++++++++++++++++++++ 2 files changed, 44 insertions(+) create mode 100755 support/scripts/check-uniq-files.py
Makefile +4 -0
@@ -679,6 +679,10 @@ $(TARGETS_ROOTFS): target-finalize .PHONY: target-finalize target-finalize: $(PACKAGES) @$(call MESSAGE,"Finalizing target directory") + # Check files that are touched by more than one package + ./support/scripts/check-uniq-files.py -t target $(BUILD_DIR)/packages-file-list.txt + ./support/scripts/check-uniq-files.py -t staging $(BUILD_DIR)/packages-file-list-staging.txt + ./support/scripts/check-uniq-files.py -t host $(BUILD_DIR)/packages-file-list-host.txt $(foreach hook,$(TARGET_FINALIZE_HOOKS),$($(hook))$(sep)) rm -rf $(TARGET_DIR)/usr/include $(TARGET_DIR)/usr/share/aclocal \ $(TARGET_DIR)/usr/lib/pkgconfig $(TARGET_DIR)/usr/share/pkgconfig \
support/scripts/check-uniq-files.py new file +40 -0
@@ -0,0 +1,40 @@ +#!/usr/bin/env python + +import sys +import csv +import argparse +from collections import defaultdict + +warn = 'Warnng: {}s file "{}" is touched by more than one package: {}\n' + +def main(): + parser = argparse.ArgumentParser() + parser.add_argument('packages_file_list', nargs='*', + help='The packages-file-list to check from') + parser.add_argument('-t', '--type', metavar="TYPE", + help='Report as a TYPE file (TYPE is either target, staging, or host)') + + args = parser.parse_args() + + if not len(args.packages_file_list) == 1: + sys.stderr.write('No packages-file-list was provided.\n') + return False + + if args.type is None: + sys.stderr.write('No type was provided\n') + return False + + file_to_pkg = defaultdict(list) + with open(args.packages_file_list[0], 'r') as pkg_file_list: + r = csv.reader(pkg_file_list, delimiter=',') + for row in r: + pkg = row[0] + file = row[1] + file_to_pkg[file].append(pkg) + + for file in file_to_pkg: + if len(file_to_pkg[file]) > 1: + sys.stderr.write(warn.format(args.type, file, file_to_pkg[file])) + +if __name__ == "__main__": + sys.exit(main())
Signature · 1 line
2.11.0
Thomas Petazzoni
AppliedPATCH v3 4/4 core: check files are not touched by more than one package
Hello,
Quoted 8 lines · Yann E. MORIN
Currently, we do nothing about packages that touch the same file: given a specific configuration, the result is reproducible (even though it might not be what the user expected) because the build order is guaranteed.
However, when we later introduce top-level parallel build, we will no longer be able to guarantee a build order, by the mere way of it being parallel. Reconcialliting all those modified files will be impossible to
Reconciliating
Quoted 2 lines
Makefile | 4 ++++ support/scripts/check-uniq-files.py | 40 +++++++++++++++++++++++++++++++++++++
I dropped the .py extension for this script. Most (but not all of course) of our scripts don't have an extension.
Quoted 6 lines
support/scripts/check-uniq-files.py new file +0
-0
support/scripts/check-uniq-files.py line 8
Quoted 8 lines
support/scripts/check-uniq-files.py new file +8
-0@@ -0,0 +1,40 @@ +#!/usr/bin/env python + +import sys +import csv +import argparse +from collections import defaultdict + +warn = 'Warnng: {}s file "{}" is touched by more than one package: {}\n'
Warnng -> Warning
also the "s" at the end of {}s didn't make much sense to me "targets file", "stagings file", "hosts file" ?
"target file", "staging file", "host file" made more sense, so I dropped the s in {}s.
Applied to next with those changes.
Thanks!
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons Embedded Linux and Kernel engineering http://free-electrons.com