support/scripts/check-uniq-files: ignore reinstalled packages
PATCH 1/1 support/scripts/check-uniq-files: ignore reinstalled packages
If a package is rebuilt, then any files it installs will be listed multiple times in the file list and check-uniq-files will report that these files are touched by more than one package even though it is the same package listed multiple times.
Switch to storing the package names in a set so that each package can only appear once.
--- support/scripts/check-uniq-files | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
support/scripts/check-uniq-files +2 -2
@@ -24,11 +24,11 @@ def main(): sys.stderr.write('No type was provided\n') return False - file_to_pkg = defaultdict(list) + file_to_pkg = defaultdict(set) with open(args.packages_file_list[0], 'rb') as pkg_file_list: for line in pkg_file_list.readlines(): pkg, _, file = line.rstrip(b'\n').partition(b',') - file_to_pkg[file].append(pkg) + file_to_pkg[file].add(pkg) for file in file_to_pkg: if len(file_to_pkg[file]) > 1:
Signature · 1 line
2.17.0
PATCH 1/1 support/scripts/check-uniq-files: ignore reinstalled packages
Hello All,
----- Mensagem original -----
Quoted 40 lines
De: "John Keeping" <[email protected]>
Para: buildroot at buildroot.org
Cc: "John Keeping" <[email protected]>
Enviadas: Quinta-feira, 26 de abril de 2018 13:27:31
Assunto: [Buildroot] [PATCH] support/scripts/check-uniq-files: ignore reinstalled packagesIf a package is rebuilt, then any files it installs will be listed multiple times in the file list and check-uniq-files will report that these files are touched by more than one package even though it is the same package listed multiple times.
Switch to storing the package names in a set so that each package can only appear once.
Signed-off-by John Keeping <[email protected]>---
support/scripts/check-uniq-files | 4 ++--1 file changed, 2 insertions(+), 2 deletions(-)support/scripts/check-uniq-files +2
-3@@ -24,11 +24,11 @@ def main(): sys.stderr.write('No type was provided\n') return False - file_to_pkg = defaultdict(list) + file_to_pkg = defaultdict(set) with open(args.packages_file_list[0], 'rb') as pkg_file_list: for line in pkg_file_list.readlines(): pkg, _, file = line.rstrip(b'\n').partition(b',') - file_to_pkg[file].append(pkg) + file_to_pkg[file].add(pkg) for file in file_to_pkg: if len(file_to_pkg[file]) > 1: --2.17.0
Quoted 4 lines
_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot
I can confirm we have problems rebuilding packages using buildroot 2018-02.1. The 3 files "package-file-list" are increasing in size when we rebuild packages (make <pkg>-rebuild or make <pkg>-reinstall), and the compilation time is increasing linearly with every rebuild, but the increase is noticeable.
One build that takes one hour is increasing 40 minutes every time we rebuild the entire platform (make <pkg>-rebuild for all packages). And we can see the problem is directly associated with the instrumentation hooks, mainly the "check-bin-arch" parsing the ever increasing "package-file-list".
Thanks
Signature · 5 lines
Dr. Henrique Marks henrique.marks at datacom.ind.br R. Am?rica, 1000 - Eldorado do Sul - RS CEP: 92990-000 - Brasil Fone: +55 51 3933 3000 - Ramal 3466
PATCH 1/1 support/scripts/check-uniq-files: ignore reinstalled packages
Quoted 13 lines · On Thu, 26 Apr 2018 15:22:35 -0300 (BRT) Henrique Marks…
----- Mensagem original -----
Quoted 12 lines
De: "John Keeping" <[email protected]>
Para: buildroot at buildroot.org
Cc: "John Keeping" <[email protected]>
Enviadas: Quinta-feira, 26 de abril de 2018 13:27:31
Assunto: [Buildroot] [PATCH] support/scripts/check-uniq-files:
ignore reinstalled packagesIf a package is rebuilt, then any files it installs will be listed multiple times in the file list and check-uniq-files will report that these files are touched by more than one package even though it is the same package listed multiple times.
Quoted 8 lines
Quoted 2 lines
Switch to storing the package names in a set so that each package can only appear once.
I can confirm we have problems rebuilding packages using buildroot 2018-02.1. The 3 files "package-file-list" are increasing in size when we rebuild packages (make <pkg>-rebuild or make <pkg>-reinstall), and the compilation time is increasing linearly with every rebuild, but the increase is noticeable.
I hadn't considered this aspect of the problem. Maybe we are better fixing the root cause of the problem with a patch like this:
-- >8 --
package/pkg-generic.mk +1 -0
@@ -64,6 +64,7 @@ GLOBAL_INSTRUMENTATION_HOOKS += step_time # $(3): suffix of file (optional) define step_pkg_size_inner cd $(2); \ + $(SED) '/^$(1),/d' $(BUILD_DIR)/packages-file-list$(3).txt; \ find . \( -type f -o -type l \) \ -newer $($(PKG)_DIR)/.stamp_built \ -exec printf '$(1),%s\n' {} + \
-- 8< --
But I don't know enough about the build process to know if this is safe in the presence of top-level parallel build.
Regards,
John
PATCH 1/1 support/scripts/check-uniq-files: ignore reinstalled packages
John, Henrique, All,
On 2018-04-27 11:31 +0100, John Keeping spake thusly:
Quoted 33 lines
Quoted 13 lines · On Thu, 26 Apr 2018 15:22:35 -0300 (BRT) Henrique Marks…
----- Mensagem original -----
Quoted 12 lines
De: "John Keeping" <[email protected]>
Para: buildroot at buildroot.org
Cc: "John Keeping" <[email protected]>
Enviadas: Quinta-feira, 26 de abril de 2018 13:27:31
Assunto: [Buildroot] [PATCH] support/scripts/check-uniq-files:
ignore reinstalled packagesIf a package is rebuilt, then any files it installs will be listed multiple times in the file list and check-uniq-files will report that these files are touched by more than one package even though it is the same package listed multiple times.
Quoted 8 lines
Quoted 2 lines
Switch to storing the package names in a set so that each package can only appear once.
I can confirm we have problems rebuilding packages using buildroot 2018-02.1. The 3 files "package-file-list" are increasing in size when we rebuild packages (make <pkg>-rebuild or make <pkg>-reinstall), and the compilation time is increasing linearly with every rebuild, but the increase is noticeable.
I hadn't considered this aspect of the problem. Maybe we are better fixing the root cause of the problem with a patch like this:
-- >8 --
package/pkg-generic.mk +0
-0
package/pkg-generic.mk GLOBAL_INSTRUMENTATION_HOOKS += step_time · line 67
Quoted 4 lines
package/pkg-generic.mk +1
-0@@ -64,6 +64,7 @@ GLOBAL_INSTRUMENTATION_HOOKS += step_time # $(3): suffix of file (optional) define step_pkg_size_inner cd $(2); \ + $(SED) '/^$(1),/d' $(BUILD_DIR)/packages-file-list$(3).txt; \
This means that a file that was previously installed by that package, but no longer is (e.g. because the install comands were "fixed" to no longer install it), will now be orphaned, and belong to no package.
But I don't care about that situation, since only a clean build from scratch is known to provide good results anyway, and that is the only thing we want to ensure.
So, please resend an updated patch to use this new solution.
Please provide an extensive commit log that explains why we do remove the existign entries, and the limitations it has (e.g. orphaned files).
Regards,
Yann E. MORIN.
Quoted 15 lines
package/pkg-generic.mk +0
-1find . \( -type f -o -type l \) \ -newer $($(PKG)_DIR)/.stamp_built \ -exec printf '$(1),%s\n' {} + \ -- 8< --But I don't know enough about the build process to know if this is safe in the presence of top-level parallel build.
Regards,
John
_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 1/1 support/scripts/check-uniq-files: ignore reinstalled packages
John,
2018-04-28 23:29 GMT+02:00 Yann E. MORIN <[email protected]>:
Quoted 45 lines
John, Henrique, All,
On 2018-04-27 11:31 +0100, John Keeping spake thusly:
Quoted 33 lines
Quoted 13 lines · On Thu, 26 Apr 2018 15:22:35 -0300 (BRT) Henrique Marks…
----- Mensagem original -----
Quoted 12 lines
De: "John Keeping" <[email protected]>
Para: buildroot at buildroot.org
Cc: "John Keeping" <[email protected]>
Enviadas: Quinta-feira, 26 de abril de 2018 13:27:31
Assunto: [Buildroot] [PATCH] support/scripts/check-uniq-files:
ignore reinstalled packagesIf a package is rebuilt, then any files it installs will be listed multiple times in the file list and check-uniq-files will report that these files are touched by more than one package even though it is the same package listed multiple times.
Quoted 8 lines
Quoted 2 lines
Switch to storing the package names in a set so that each package can only appear once.
I can confirm we have problems rebuilding packages using buildroot 2018-02.1. The 3 files "package-file-list" are increasing in size when we rebuild packages (make <pkg>-rebuild or make <pkg>-reinstall), and the compilation time is increasing linearly with every rebuild, but the increase is noticeable.
I hadn't considered this aspect of the problem. Maybe we are better fixing the root cause of the problem with a patch like this:
-- >8 --
package/pkg-generic.mk +0
-0package/pkg-generic.mk GLOBAL_INSTRUMENTATION_HOOKS += step_time · line 67
Quoted 4 lines
package/pkg-generic.mk +1
-0@@ -64,6 +64,7 @@ GLOBAL_INSTRUMENTATION_HOOKS += step_time # $(3): suffix of file (optional) define step_pkg_size_inner cd $(2); \ + $(SED) '/^$(1),/d' $(BUILD_DIR)/packages-file-list$(3).txt; \This means that a file that was previously installed by that package, but no longer is (e.g. because the install comands were "fixed" to no longer install it), will now be orphaned, and belong to no package.
Quoted 8 lines
But I don't care about that situation, since only a clean build from scratch is known to provide good results anyway, and that is the only thing we want to ensure.
So, please resend an updated patch to use this new solution.
Please provide an extensive commit log that explains why we do remove the existign entries, and the limitations it has (e.g. orphaned files).
Could you respin this patch?
Thanks!
Quoted 31 lines
Regards,
Yann E. MORIN.Quoted 9 lines
package/pkg-generic.mk +0
-1find . \( -type f -o -type l \) \ -newer $($(PKG)_DIR)/.stamp_built \ -exec printf '$(1),%s\n' {} + \ -- 8< --But I don't know enough about the build process to know if this is safe in the presence of top-level parallel build.
Quoted 6 lines
Regards,
John
_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot--
.-----------------.--------------------.------------------.--------------------.| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 1 line
Profile: http://it.linkedin.com/in/compagnucciangelo
PATCH 1/1 support/scripts/check-uniq-files: ignore reinstalled packages
Hi Angelo,
Quoted 53 lines · On Thu, 3 May 2018 18:27:53 +0200 Angelo Compagnucci…
2018-04-28 23:29 GMT+02:00 Yann E. MORIN <[email protected]>:
Quoted 52 lines
John, Henrique, All,
On 2018-04-27 11:31 +0100, John Keeping spake thusly:
Quoted 34 lines
Quoted 14 lines · On Thu, 26 Apr 2018 15:22:35 -0300 (BRT) Henrique Marks…
----- Mensagem original -----
Quoted 13 lines
De: "John Keeping" <[email protected]>
Para: buildroot at buildroot.org
Cc: "John Keeping" <[email protected]>
Enviadas: Quinta-feira, 26 de abril de 2018 13:27:31
Assunto: [Buildroot] [PATCH] support/scripts/check-uniq-files:
ignore reinstalled packagesIf a package is rebuilt, then any files it installs will be listed multiple times in the file list and check-uniq-files will report that these files are touched by more than one package even though it is the same package listed multiple times.
Quoted 8 lines
Quoted 2 lines
Switch to storing the package names in a set so that each package can only appear once.
I can confirm we have problems rebuilding packages using buildroot 2018-02.1. The 3 files "package-file-list" are increasing in size when we rebuild packages (make <pkg>-rebuild or make <pkg>-reinstall), and the compilation time is increasing linearly with every rebuild, but the increase is noticeable.
I hadn't considered this aspect of the problem. Maybe we are better fixing the root cause of the problem with a patch like this:
-- >8 --
package/pkg-generic.mk +0
-0package/pkg-generic.mk GLOBAL_INSTRUMENTATION_HOOKS += step_time · line 68
Quoted 5 lines
package/pkg-generic.mk +1
-0@@ -64,6 +64,7 @@ GLOBAL_INSTRUMENTATION_HOOKS += step_time # $(3): suffix of file (optional) define step_pkg_size_inner cd $(2); \ + $(SED) '/^$(1),/d' $(BUILD_DIR)/packages-file-list$(3).txt; \This means that a file that was previously installed by that package, but no longer is (e.g. because the install comands were "fixed" to no longer install it), will now be orphaned, and belong to no package.
But I don't care about that situation, since only a clean build from scratch is known to provide good results anyway, and that is the only thing we want to ensure.
Quoted 7 lines
Quoted 5 lines
So, please resend an updated patch to use this new solution.
Please provide an extensive commit log that explains why we do remove the existign entries, and the limitations it has (e.g. orphaned files).
Could you respin this patch?
This got respun a few times on a separate thread, resulting in:
Regards,
John