Skip to contentSkip to conversation

fs/common.mk: enable multithreaded xz compression

PATCH Superseded james.hilliard1 at gmail.com 5 participants 14 messages Jan 21, 2019 – Jan 28, 2019 VFS

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 fs/common.mk: enable multithreaded xz compression

From: James Hilliard <[email protected]>

xz help indicates only 1 thread is used unless we set threads:
-T, --threads=NUM use at most NUM threads; the default is 1; set to 0
to use as many threads as there are processor cores

---
 fs/common.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
fs/common.mk +1 -1
index a560417..6a57121 100644
@@ -106,7 +106,7 @@ endif
 ifeq ($$(BR2_TARGET_ROOTFS_$(2)_XZ),y)
 ROOTFS_$(2)_DEPENDENCIES += host-xz
 ROOTFS_$(2)_COMPRESS_EXT = .xz
-ROOTFS_$(2)_COMPRESS_CMD = xz -9 -C crc32 -c
+ROOTFS_$(2)_COMPRESS_CMD = xz -T 0 -9 -C crc32 -c
 endif
 
 $$(BINARIES_DIR)/$$(ROOTFS_$(2)_FINAL_IMAGE_NAME): ROOTFS=$(2)
Signature · 1 line
2.7.4
#1 lore raw

PATCH 1/1 fs/common.mk: enable multithreaded xz compression

James,

Quoted 7 lines · On Sun, Jan 20, 2019 at 8:08 PM <[email protected]> wrote:

From: James Hilliard <[email protected]>

xz help indicates only 1 thread is used unless we set threads:
-T, --threads=NUM use at most NUM threads; the default is 1; set to 0
to use as many threads as there are processor cores

It's good to see xz now has support, but I assume there is a minimum version? We ran into this outside of Builroot on a project where we conditionally used the parallel "pxz" tool when available, as "xz" didn't have the -T support consistently in distros yet.

Quoted 19 lines
---
 fs/common.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
fs/common.mk +1 -2
index a560417..6a57121 100644
@@ -106,7 +106,7 @@ endif
 ifeq ($$(BR2_TARGET_ROOTFS_$(2)_XZ),y)
 ROOTFS_$(2)_DEPENDENCIES += host-xz
 ROOTFS_$(2)_COMPRESS_EXT = .xz
-ROOTFS_$(2)_COMPRESS_CMD = xz -9 -C crc32 -c
+ROOTFS_$(2)_COMPRESS_CMD = xz -T 0 -9 -C crc32 -c
 endif
 
 $$(BINARIES_DIR)/$$(ROOTFS_$(2)_FINAL_IMAGE_NAME): ROOTFS=$(2)
--
Quoted 2 lines

2.7.4

Thanks for sending the patch!
Matt

#2 lore raw

PATCH 1/1 fs/common.mk: enable multithreaded xz compression

Quoted 13 lines · Matthew Weber

James,

Quoted 7 lines · On Sun, Jan 20, 2019 at 8:08 PM <[email protected]> wrote:

From: James Hilliard <[email protected]>

xz help indicates only 1 thread is used unless we set threads:
-T, --threads=NUM use at most NUM threads; the default is 1; set to 0
to use as many threads as there are processor cores

Quoted 4 lines

It's good to see xz now has support, but I assume there is a minimum version? We ran into this outside of Builroot on a project where we conditionally used the parallel "pxz" tool when available, as "xz" didn't have the -T support consistently in distros yet.

I thought it was using the version compiled by buildroot. Isn't that what host-xz is being used for?

Quoted 25 lines
Quoted 20 lines
---
 fs/common.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
fs/common.mk +1 -2
index a560417..6a57121 100644
@@ -106,7 +106,7 @@ endif
 ifeq ($$(BR2_TARGET_ROOTFS_$(2)_XZ),y)
 ROOTFS_$(2)_DEPENDENCIES += host-xz
 ROOTFS_$(2)_COMPRESS_EXT = .xz
-ROOTFS_$(2)_COMPRESS_CMD = xz -9 -C crc32 -c
+ROOTFS_$(2)_COMPRESS_CMD = xz -T 0 -9 -C crc32 -c
 endif
 
 $$(BINARIES_DIR)/$$(ROOTFS_$(2)_FINAL_IMAGE_NAME): ROOTFS=$(2)
--
Quoted 2 lines

2.7.4

Thanks for sending the patch!
Matt

Reviewed-by on v1 1/1
#3 lore raw
Quoted 15 lines · James Hilliard
Quoted 13 lines · Matthew Weber

James,

Quoted 7 lines · On Sun, Jan 20, 2019 at 8:08 PM <[email protected]> wrote:

From: James Hilliard <[email protected]>

xz help indicates only 1 thread is used unless we set threads:
-T, --threads=NUM use at most NUM threads; the default is 1; set to 0
to use as many threads as there are processor cores

Quoted 6 lines
Quoted 4 lines

It's good to see xz now has support, but I assume there is a minimum version? We ran into this outside of Builroot on a project where we conditionally used the parallel "pxz" tool when available, as "xz" didn't have the -T support consistently in distros yet.

I thought it was using the version compiled by buildroot. Isn't that what host-xz is being used for?

Ah, true. I had glanced at the dependency checking scripts but we always built it based on the fs/common.mk dependency on host-xz so that is correct.

One other comment would be that "0" wouldn't be ideal on a shared machine. Suggest limiting to $(PARALLEL_JOBS) which is used in other filesystem creation (grep fs folder for an example in iso9660 or squashfs).

#4 lore raw

PATCH 1/1 fs/common.mk: enable multithreaded xz compression

In reply to James Hilliard
Quoted 16 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"James" == James Hilliard <[email protected]> writes:

Quoted 13 lines · Matthew Weber

James,

Quoted 7 lines · On Sun, Jan 20, 2019 at 8:08 PM <[email protected]> wrote:

From: James Hilliard <[email protected]>

xz help indicates only 1 thread is used unless we set threads:
-T, --threads=NUM use at most NUM threads; the default is 1; set to 0
to use as many threads as there are processor cores

Quoted 7 lines
Quoted 4 lines

It's good to see xz now has support, but I assume there is a minimum version? We ran into this outside of Builroot on a project where we conditionally used the parallel "pxz" tool when available, as "xz" didn't have the -T support consistently in distros yet.

I thought it was using the version compiled by buildroot. Isn't that what host-xz is being used for?

Yes, correct. Do you know if the -T 0 option generates identical output as -T 1? Otherwise we may want to disable it for reproducible builds.

Signature · 1 line
Bye, Peter Korsgaard
#5 lore raw

PATCH 1/1 fs/common.mk: enable multithreaded xz compression

Quoted 22 lines · Peter Korsgaard
Quoted 21 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

"James" == James Hilliard <[email protected]> writes:

Quoted 17 lines · Matthew Weber

James,

Quoted 7 lines · On Sun, Jan 20, 2019 at 8:08 PM <[email protected]> wrote:

From: James Hilliard <[email protected]>

xz help indicates only 1 thread is used unless we set threads:
-T, --threads=NUM use at most NUM threads; the default is 1; set to 0
to use as many threads as there are processor cores

It's good to see xz now has support, but I assume there is a minimum version? We ran into this outside of Builroot on a project where we conditionally used the parallel "pxz" tool when available, as "xz" didn't have the -T support consistently in distros yet.

Quoted 5 lines
Quoted 2 lines

I thought it was using the version compiled by buildroot. Isn't that what host-xz is being used for?

Yes, correct. Do you know if the -T 0 option generates identical output as -T 1? Otherwise we may want to disable it for reproducible builds.

I think it doesn't generate identical output, I wonder if it would be a good idea to make all these compression flags configurable. One issue I'm running across is that my test builds take at least 5 minutes or so to compress, having a way to set different xz flags between test and production builds would be handy.

Quoted 3 lines

--
Bye, Peter Korsgaard

#6 lore raw

PATCH 1/1 fs/common.mk: enable multithreaded xz compression

In reply to Peter Korsgaard

Hello,

Quoted 5 lines · Peter Korsgaard
Quoted 2 lines

I thought it was using the version compiled by buildroot. Isn't that what host-xz is being used for?

Yes, correct. Do you know if the -T 0 option generates identical output as -T 1? Otherwise we may want to disable it for reproducible builds.

Does it matter than the output of -T 0 and -T 1 are identical ?

What matters for BR2_REPRODUCIBLE=y is if the same input, with -T 0, always provide the exact identical output.

Am I missing something here ?

Best regards,

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#7 lore raw

PATCH 1/1 fs/common.mk: enable multithreaded xz compression

Quoted 8 lines · Thomas Petazzoni

Hello,

Quoted 3 lines · Peter Korsgaard
Quoted 2 lines

I thought it was using the version compiled by buildroot. Isn't that what host-xz is being used for?

Quoted 7 lines
Quoted 2 lines

Yes, correct. Do you know if the -T 0 option generates identical output as -T 1? Otherwise we may want to disable it for reproducible builds.

Does it matter than the output of -T 0 and -T 1 are identical ?

What matters for BR2_REPRODUCIBLE=y is if the same input, with -T 0, always provide the exact identical output.

My assumption is that -T 0 may be inconsistent due to the autodetection.

Quoted 10 lines

Am I missing something here ?

Best regards,

Thomas
--
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

#8 lore raw

PATCH 1/1 fs/common.mk: enable multithreaded xz compression

In reply to Thomas Petazzoni
Quoted 5 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Thomas" == Thomas Petazzoni <[email protected]> writes:

Hello,
On Mon, 21 Jan 2019 18:39:24 +0100, Peter Korsgaard wrote:

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

I thought it was using the version compiled by buildroot. Isn't that what host-xz is being used for?

Yes, correct. Do you know if the -T 0 option generates identical output as -T 1? Otherwise we may want to disable it for reproducible builds.

Does it matter than the output of -T 0 and -T 1 are identical ?

It would cause an extra change in output between buildroot versions using -T 0 and ones using -T 1 - But OK, probably that isn't really important given all the other changes.

What I really meant was if the -T 1 option affected the output format, and if so, if the difference involved any tradeoff (E.G. perhaps -T 1 compresses slightly worse than -T 0) and if repeated runs with -T 1 would generate the same output (on same machine or on another machine with different number of threads).

Quoted 2 lines

What matters for BR2_REPRODUCIBLE=y is if the same input, with -T 0, always provide the exact identical output.

With the new patch that only enables -T 1 for !reproducible builds, that is correct.

Signature · 1 line
Bye, Peter Korsgaard
#9 lore raw

PATCH 1/1 fs/common.mk: enable multithreaded xz compression

Quoted 18 lines · Peter Korsgaard
Quoted 6 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

"Thomas" == Thomas Petazzoni <[email protected]> writes:

Hello,
On Mon, 21 Jan 2019 18:39:24 +0100, Peter Korsgaard wrote:

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

I thought it was using the version compiled by buildroot. Isn't that what host-xz is being used for?

Yes, correct. Do you know if the -T 0 option generates identical output as -T 1? Otherwise we may want to disable it for reproducible builds.

Does it matter than the output of -T 0 and -T 1 are identical ?

It would cause an extra change in output between buildroot versions using -T 0 and ones using -T 1 - But OK, probably that isn't really important given all the other changes.

Quoted 5 lines

What I really meant was if the -T 1 option affected the output format, and if so, if the difference involved any tradeoff (E.G. perhaps -T 1 compresses slightly worse than -T 0) and if repeated runs with -T 1 would generate the same output (on same machine or on another machine with different number of threads).

The single threaded -T 1 should produce the best compression quality, although it's a negligible difference.
From my understanding that is because single threaded compression doesn't need to split up the file being compressed into separate blocks.

Quoted 9 lines
Quoted 3 lines

What matters for BR2_REPRODUCIBLE=y is if the same input, with -T 0, always provide the exact identical output.

With the new patch that only enables -T 1 for !reproducible builds, that is correct.

--
Bye, Peter Korsgaard

#10 lore raw

PATCH 1/1 fs/common.mk: enable multithreaded xz compression

In reply to Peter Korsgaard
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Peter" == Peter Korsgaard <[email protected]> writes:

Does it matter than the output of -T 0 and -T 1 are identical ?

It would cause an extra change in output between buildroot versions using -T 0 and ones using -T 1 - But OK, probably that isn't really important given all the other changes.

Quoted 5 lines

What I really meant was if the -T 1 option affected the output format, and if so, if the difference involved any tradeoff (E.G. perhaps -T 1 compresses slightly worse than -T 0) and if repeated runs with -T 1 would generate the same output (on same machine or on another machine with different number of threads).

Sorry, got -T 1 and -T 0 mixed up, I meant -T 0 here.

Quoted 5 lines
Quoted 2 lines

What matters for BR2_REPRODUCIBLE=y is if the same input, with -T 0, always provide the exact identical output.

With the new patch that only enables -T 1 for !reproducible builds, that is correct.

The assumption is that this is not true, especially between systems with different number of CPUs.

Signature · 1 line
Bye, Peter Korsgaard
#11 lore raw

PATCH 1/1 fs/common.mk: enable multithreaded xz compression

In reply to Peter Korsgaard
Quoted 5 lines · On Sun, 27 Jan 2019 23:06:14 +0100 Peter Korsgaard <[email protected]> wrote:
Quoted 1 line

Does it matter than the output of -T 0 and -T 1 are identical ?

It would cause an extra change in output between buildroot versions using -T 0 and ones using -T 1 - But OK, probably that isn't really important given all the other changes.

Well, BR2_REPRODUCIBLE is definitely not about guaranteeing identical output between Buildroot versions. How could this even work, since we are bumping the version of dozens/hundreds of packages ?

BR2_REPRODUCIBLE is about given a fixed Buildroot version and configuration, be able to generate a binary-identical result.

Quoted 5 lines
Quoted 2 lines

What matters for BR2_REPRODUCIBLE=y is if the same input, with -T 0, always provide the exact identical output.

With the new patch that only enables -T 1 for !reproducible builds, that is correct.

I thought -T 0 was the problematic thing as it enabled automatic parallelization, while -T 1 explicitly disabled parallelization.

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#12 lore raw

PATCH 1/1 fs/common.mk: enable multithreaded xz compression

Quoted 17 lines · Thomas Petazzoni
Quoted 5 lines · On Sun, 27 Jan 2019 23:06:14 +0100 Peter Korsgaard <[email protected]> wrote:
Quoted 1 line

Does it matter than the output of -T 0 and -T 1 are identical ?

It would cause an extra change in output between buildroot versions using -T 0 and ones using -T 1 - But OK, probably that isn't really important given all the other changes.

Well, BR2_REPRODUCIBLE is definitely not about guaranteeing identical output between Buildroot versions. How could this even work, since we are bumping the version of dozens/hundreds of packages ?

BR2_REPRODUCIBLE is about given a fixed Buildroot version and configuration, be able to generate a binary-identical result.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

What matters for BR2_REPRODUCIBLE=y is if the same input, with -T 0, always provide the exact identical output.

With the new patch that only enables -T 1 for !reproducible builds, that is correct.

I thought -T 0 was the problematic thing as it enabled automatic parallelization, while -T 1 explicitly disabled parallelization.

Right, -T 1 is the default.

Quoted 6 lines

Thomas
--
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

#13 lore raw

PATCH 1/1 fs/common.mk: enable multithreaded xz compression

In reply to Thomas Petazzoni
Quoted 11 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Thomas" == Thomas Petazzoni <[email protected]> writes:

Quoted 5 lines · On Sun, 27 Jan 2019 23:06:14 +0100 Peter Korsgaard <[email protected]> wrote:
Quoted 1 line

Does it matter than the output of -T 0 and -T 1 are identical ?

It would cause an extra change in output between buildroot versions using -T 0 and ones using -T 1 - But OK, probably that isn't really important given all the other changes.

Quoted 6 lines

Well, BR2_REPRODUCIBLE is definitely not about guaranteeing identical output between Buildroot versions. How could this even work, since we are bumping the version of dozens/hundreds of packages ?

BR2_REPRODUCIBLE is about given a fixed Buildroot version and configuration, be able to generate a binary-identical result.

Correct, hence my note "isn't really important".

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

What matters for BR2_REPRODUCIBLE=y is if the same input, with -T 0, always provide the exact identical output.

With the new patch that only enables -T 1 for !reproducible builds, that is correct.

I thought -T 0 was the problematic thing as it enabled automatic parallelization, while -T 1 explicitly disabled parallelization.

Sorry, I got T 0/1 mixed up in my mail, like I explained in the followup.

Signature · 1 line
Bye, Peter Korsgaard