fs/common.mk: enable multithreaded xz compression
PATCH 1/1 fs/common.mk: enable multithreaded xz compression
From: James Hilliard <[email protected]>
xz help indicates only 1 thread is used unless we set threads:
-T, --threads=NUM use at most NUM threads; the default is 1; set to 0
to use as many threads as there are processor cores
--- fs/common.mk | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
fs/common.mk +1 -1
@@ -106,7 +106,7 @@ endif ifeq ($$(BR2_TARGET_ROOTFS_$(2)_XZ),y) ROOTFS_$(2)_DEPENDENCIES += host-xz ROOTFS_$(2)_COMPRESS_EXT = .xz -ROOTFS_$(2)_COMPRESS_CMD = xz -9 -C crc32 -c +ROOTFS_$(2)_COMPRESS_CMD = xz -T 0 -9 -C crc32 -c endif $$(BINARIES_DIR)/$$(ROOTFS_$(2)_FINAL_IMAGE_NAME): ROOTFS=$(2)
Signature · 1 line
2.7.4
PATCH 1/1 fs/common.mk: enable multithreaded xz compression
James,
Quoted 7 lines · On Sun, Jan 20, 2019 at 8:08 PM <[email protected]> wrote:
From: James Hilliard <[email protected]>
xz help indicates only 1 thread is used unless we set threads:
-T, --threads=NUM use at most NUM threads; the default is 1; set to 0
to use as many threads as there are processor cores
It's good to see xz now has support, but I assume there is a minimum version? We ran into this outside of Builroot on a project where we conditionally used the parallel "pxz" tool when available, as "xz" didn't have the -T support consistently in distros yet.
Quoted 19 lines
Signed-off-by James Hilliard <[email protected]>--- fs/common.mk | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)fs/common.mk +1
-2@@ -106,7 +106,7 @@ endif ifeq ($$(BR2_TARGET_ROOTFS_$(2)_XZ),y) ROOTFS_$(2)_DEPENDENCIES += host-xz ROOTFS_$(2)_COMPRESS_EXT = .xz -ROOTFS_$(2)_COMPRESS_CMD = xz -9 -C crc32 -c +ROOTFS_$(2)_COMPRESS_CMD = xz -T 0 -9 -C crc32 -c endif $$(BINARIES_DIR)/$$(ROOTFS_$(2)_FINAL_IMAGE_NAME): ROOTFS=$(2) --
Quoted 2 lines
2.7.4
Thanks for sending the patch!
Matt
PATCH 1/1 fs/common.mk: enable multithreaded xz compression
Quoted 13 lines · Matthew Weber
James,
Quoted 7 lines · On Sun, Jan 20, 2019 at 8:08 PM <[email protected]> wrote:
From: James Hilliard <[email protected]>
xz help indicates only 1 thread is used unless we set threads:
-T, --threads=NUM use at most NUM threads; the default is 1; set to 0
to use as many threads as there are processor cores
Quoted 4 lines
It's good to see xz now has support, but I assume there is a minimum version? We ran into this outside of Builroot on a project where we conditionally used the parallel "pxz" tool when available, as "xz" didn't have the -T support consistently in distros yet.
I thought it was using the version compiled by buildroot. Isn't that what host-xz is being used for?
Quoted 25 lines
Quoted 20 lines
Signed-off-by James Hilliard <[email protected]>--- fs/common.mk | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)fs/common.mk +1
-2@@ -106,7 +106,7 @@ endif ifeq ($$(BR2_TARGET_ROOTFS_$(2)_XZ),y) ROOTFS_$(2)_DEPENDENCIES += host-xz ROOTFS_$(2)_COMPRESS_EXT = .xz -ROOTFS_$(2)_COMPRESS_CMD = xz -9 -C crc32 -c +ROOTFS_$(2)_COMPRESS_CMD = xz -T 0 -9 -C crc32 -c endif $$(BINARIES_DIR)/$$(ROOTFS_$(2)_FINAL_IMAGE_NAME): ROOTFS=$(2) --Quoted 2 lines
2.7.4
Thanks for sending the patch!
Matt
Quoted 15 lines · James Hilliard
Quoted 13 lines · Matthew Weber
James,
Quoted 7 lines · On Sun, Jan 20, 2019 at 8:08 PM <[email protected]> wrote:
From: James Hilliard <[email protected]>
xz help indicates only 1 thread is used unless we set threads:
-T, --threads=NUM use at most NUM threads; the default is 1; set to 0
to use as many threads as there are processor cores
Quoted 6 lines
Quoted 4 lines
It's good to see xz now has support, but I assume there is a minimum version? We ran into this outside of Builroot on a project where we conditionally used the parallel "pxz" tool when available, as "xz" didn't have the -T support consistently in distros yet.
I thought it was using the version compiled by buildroot. Isn't that what host-xz is being used for?
Ah, true. I had glanced at the dependency checking scripts but we always built it based on the fs/common.mk dependency on host-xz so that is correct.
One other comment would be that "0" wouldn't be ideal on a shared machine. Suggest limiting to $(PARALLEL_JOBS) which is used in other filesystem creation (grep fs folder for an example in iso9660 or squashfs).
PATCH 1/1 fs/common.mk: enable multithreaded xz compression
In reply to James HilliardQuoted 16 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"James" == James Hilliard <[email protected]> writes:
Quoted 13 lines · Matthew Weber
James,
Quoted 7 lines · On Sun, Jan 20, 2019 at 8:08 PM <[email protected]> wrote:
From: James Hilliard <[email protected]>
xz help indicates only 1 thread is used unless we set threads:
-T, --threads=NUM use at most NUM threads; the default is 1; set to 0
to use as many threads as there are processor cores
Quoted 7 lines
Quoted 4 lines
It's good to see xz now has support, but I assume there is a minimum version? We ran into this outside of Builroot on a project where we conditionally used the parallel "pxz" tool when available, as "xz" didn't have the -T support consistently in distros yet.
I thought it was using the version compiled by buildroot. Isn't that what host-xz is being used for?
Yes, correct. Do you know if the -T 0 option generates identical output as -T 1? Otherwise we may want to disable it for reproducible builds.
Signature · 1 line
Bye, Peter Korsgaard
PATCH 1/1 fs/common.mk: enable multithreaded xz compression
Quoted 22 lines · Peter Korsgaard
Quoted 21 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
"James" == James Hilliard <[email protected]> writes:
Quoted 17 lines · Matthew Weber
James,
Quoted 7 lines · On Sun, Jan 20, 2019 at 8:08 PM <[email protected]> wrote:
From: James Hilliard <[email protected]>
xz help indicates only 1 thread is used unless we set threads:
-T, --threads=NUM use at most NUM threads; the default is 1; set to 0
to use as many threads as there are processor coresIt's good to see xz now has support, but I assume there is a minimum version? We ran into this outside of Builroot on a project where we conditionally used the parallel "pxz" tool when available, as "xz" didn't have the -T support consistently in distros yet.
Quoted 5 lines
Quoted 2 lines
I thought it was using the version compiled by buildroot. Isn't that what host-xz is being used for?
Yes, correct. Do you know if the -T 0 option generates identical output as -T 1? Otherwise we may want to disable it for reproducible builds.
I think it doesn't generate identical output, I wonder if it would be a good idea to make all these compression flags configurable. One issue I'm running across is that my test builds take at least 5 minutes or so to compress, having a way to set different xz flags between test and production builds would be handy.
Quoted 3 lines
--
Bye, Peter Korsgaard
PATCH 1/1 fs/common.mk: enable multithreaded xz compression
In reply to Peter KorsgaardHello,
Quoted 5 lines · Peter Korsgaard
Quoted 2 lines
I thought it was using the version compiled by buildroot. Isn't that what host-xz is being used for?
Yes, correct. Do you know if the -T 0 option generates identical output as -T 1? Otherwise we may want to disable it for reproducible builds.
Does it matter than the output of -T 0 and -T 1 are identical ?
What matters for BR2_REPRODUCIBLE=y is if the same input, with -T 0, always provide the exact identical output.
Am I missing something here ?
Best regards,
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 1/1 fs/common.mk: enable multithreaded xz compression
Quoted 8 lines · Thomas Petazzoni
Hello,
Quoted 3 lines · Peter Korsgaard
Quoted 2 lines
I thought it was using the version compiled by buildroot. Isn't that what host-xz is being used for?
Quoted 7 lines
Quoted 2 lines
Yes, correct. Do you know if the -T 0 option generates identical output as -T 1? Otherwise we may want to disable it for reproducible builds.
Does it matter than the output of -T 0 and -T 1 are identical ?
What matters for BR2_REPRODUCIBLE=y is if the same input, with -T 0, always provide the exact identical output.
My assumption is that -T 0 may be inconsistent due to the autodetection.
Quoted 10 lines
Am I missing something here ?
Best regards,
Thomas
--
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
PATCH 1/1 fs/common.mk: enable multithreaded xz compression
In reply to Thomas PetazzoniQuoted 5 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Thomas" == Thomas Petazzoni <[email protected]> writes:
Hello,
On Mon, 21 Jan 2019 18:39:24 +0100, Peter Korsgaard wrote:
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
I thought it was using the version compiled by buildroot. Isn't that what host-xz is being used for?
Yes, correct. Do you know if the -T 0 option generates identical output as -T 1? Otherwise we may want to disable it for reproducible builds.
Does it matter than the output of -T 0 and -T 1 are identical ?
It would cause an extra change in output between buildroot versions using -T 0 and ones using -T 1 - But OK, probably that isn't really important given all the other changes.
What I really meant was if the -T 1 option affected the output format, and if so, if the difference involved any tradeoff (E.G. perhaps -T 1 compresses slightly worse than -T 0) and if repeated runs with -T 1 would generate the same output (on same machine or on another machine with different number of threads).
Quoted 2 lines
What matters for BR2_REPRODUCIBLE=y is if the same input, with -T 0, always provide the exact identical output.
With the new patch that only enables -T 1 for !reproducible builds, that is correct.
Signature · 1 line
Bye, Peter Korsgaard
PATCH 1/1 fs/common.mk: enable multithreaded xz compression
Quoted 18 lines · Peter Korsgaard
Quoted 6 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
"Thomas" == Thomas Petazzoni <[email protected]> writes:
Hello,
On Mon, 21 Jan 2019 18:39:24 +0100, Peter Korsgaard wrote:Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
I thought it was using the version compiled by buildroot. Isn't that what host-xz is being used for?
Yes, correct. Do you know if the -T 0 option generates identical output as -T 1? Otherwise we may want to disable it for reproducible builds.
Does it matter than the output of -T 0 and -T 1 are identical ?
It would cause an extra change in output between buildroot versions using -T 0 and ones using -T 1 - But OK, probably that isn't really important given all the other changes.
Quoted 5 lines
What I really meant was if the -T 1 option affected the output format, and if so, if the difference involved any tradeoff (E.G. perhaps -T 1 compresses slightly worse than -T 0) and if repeated runs with -T 1 would generate the same output (on same machine or on another machine with different number of threads).
The single threaded -T 1 should produce the best compression quality, although it's a negligible difference.
From my understanding that is because single threaded compression doesn't need to split up the file being compressed into separate blocks.
Quoted 9 lines
Quoted 3 lines
What matters for BR2_REPRODUCIBLE=y is if the same input, with -T 0, always provide the exact identical output.
With the new patch that only enables -T 1 for !reproducible builds, that is correct.
--
Bye, Peter Korsgaard
PATCH 1/1 fs/common.mk: enable multithreaded xz compression
In reply to Peter KorsgaardQuoted 8 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Peter" == Peter Korsgaard <[email protected]> writes:
Does it matter than the output of -T 0 and -T 1 are identical ?
It would cause an extra change in output between buildroot versions using -T 0 and ones using -T 1 - But OK, probably that isn't really important given all the other changes.
Quoted 5 lines
What I really meant was if the -T 1 option affected the output format, and if so, if the difference involved any tradeoff (E.G. perhaps -T 1 compresses slightly worse than -T 0) and if repeated runs with -T 1 would generate the same output (on same machine or on another machine with different number of threads).
Sorry, got -T 1 and -T 0 mixed up, I meant -T 0 here.
Quoted 5 lines
Quoted 2 lines
What matters for BR2_REPRODUCIBLE=y is if the same input, with -T 0, always provide the exact identical output.
With the new patch that only enables -T 1 for !reproducible builds, that is correct.
The assumption is that this is not true, especially between systems with different number of CPUs.
Signature · 1 line
Bye, Peter Korsgaard
PATCH 1/1 fs/common.mk: enable multithreaded xz compression
In reply to Peter KorsgaardQuoted 5 lines · On Sun, 27 Jan 2019 23:06:14 +0100 Peter Korsgaard <[email protected]> wrote:
Quoted 1 line
Does it matter than the output of -T 0 and -T 1 are identical ?
It would cause an extra change in output between buildroot versions using -T 0 and ones using -T 1 - But OK, probably that isn't really important given all the other changes.
Well, BR2_REPRODUCIBLE is definitely not about guaranteeing identical output between Buildroot versions. How could this even work, since we are bumping the version of dozens/hundreds of packages ?
BR2_REPRODUCIBLE is about given a fixed Buildroot version and configuration, be able to generate a binary-identical result.
Quoted 5 lines
Quoted 2 lines
What matters for BR2_REPRODUCIBLE=y is if the same input, with -T 0, always provide the exact identical output.
With the new patch that only enables -T 1 for !reproducible builds, that is correct.
I thought -T 0 was the problematic thing as it enabled automatic parallelization, while -T 1 explicitly disabled parallelization.
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 1/1 fs/common.mk: enable multithreaded xz compression
Quoted 17 lines · Thomas Petazzoni
Quoted 5 lines · On Sun, 27 Jan 2019 23:06:14 +0100 Peter Korsgaard <[email protected]> wrote:
Quoted 1 line
Does it matter than the output of -T 0 and -T 1 are identical ?
It would cause an extra change in output between buildroot versions using -T 0 and ones using -T 1 - But OK, probably that isn't really important given all the other changes.
Well,
BR2_REPRODUCIBLEis definitely not about guaranteeing identical output between Buildroot versions. How could this even work, since we are bumping the version of dozens/hundreds of packages ?
BR2_REPRODUCIBLEis about given a fixed Buildroot version and configuration, be able to generate a binary-identical result.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
What matters for BR2_REPRODUCIBLE=y is if the same input, with -T 0, always provide the exact identical output.
With the new patch that only enables -T 1 for !reproducible builds, that is correct.
I thought -T 0 was the problematic thing as it enabled automatic parallelization, while -T 1 explicitly disabled parallelization.
Right, -T 1 is the default.
Quoted 6 lines
Thomas
--
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
PATCH 1/1 fs/common.mk: enable multithreaded xz compression
In reply to Thomas PetazzoniQuoted 11 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Thomas" == Thomas Petazzoni <[email protected]> writes:
Quoted 5 lines · On Sun, 27 Jan 2019 23:06:14 +0100 Peter Korsgaard <[email protected]> wrote:
Quoted 1 line
Does it matter than the output of -T 0 and -T 1 are identical ?
It would cause an extra change in output between buildroot versions using -T 0 and ones using -T 1 - But OK, probably that isn't really important given all the other changes.
Quoted 6 lines
Well,
BR2_REPRODUCIBLEis definitely not about guaranteeing identical output between Buildroot versions. How could this even work, since we are bumping the version of dozens/hundreds of packages ?
BR2_REPRODUCIBLEis about given a fixed Buildroot version and configuration, be able to generate a binary-identical result.
Correct, hence my note "isn't really important".
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
What matters for BR2_REPRODUCIBLE=y is if the same input, with -T 0, always provide the exact identical output.
With the new patch that only enables -T 1 for !reproducible builds, that is correct.
I thought -T 0 was the problematic thing as it enabled automatic parallelization, while -T 1 explicitly disabled parallelization.
Sorry, I got T 0/1 mixed up in my mail, like I explained in the followup.
Signature · 1 line
Bye, Peter Korsgaard