Skip to contentSkip to conversation

fs/common.mk: enable multithreaded xz compression

PATCH New james.hilliard1 at gmail.com 4 participants 5 messages Jan 24, 2019 – Aug 3, 2019 VFS

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 fs/common.mk: enable multithreaded xz compression

From: James Hilliard <[email protected]>

xz help indicates only 1 thread is used unless we set threads:
-T, --threads=NUM use at most NUM threads; the default is 1; set to 0
to use as many threads as there are processor cores

---
Changes v1 -> v2:
- use PARALLEL_JOBS to determine how many threads to compress with
- disable multithreaded compression for reproducible builds

---
 fs/common.mk | 4 ++++
 1 file changed, 4 insertions(+)
fs/common.mk +4 -0
index a560417..76da6d8 100644
@@ -106,7 +106,11 @@ endif
 ifeq ($$(BR2_TARGET_ROOTFS_$(2)_XZ),y)
 ROOTFS_$(2)_DEPENDENCIES += host-xz
 ROOTFS_$(2)_COMPRESS_EXT = .xz
+ifeq ($(BR2_REPRODUCIBLE),y)
 ROOTFS_$(2)_COMPRESS_CMD = xz -9 -C crc32 -c
+else
+ROOTFS_$(2)_COMPRESS_CMD = xz -T $(PARALLEL_JOBS) -9 -C crc32 -c
+endif
 endif
 
 $$(BINARIES_DIR)/$$(ROOTFS_$(2)_FINAL_IMAGE_NAME): ROOTFS=$(2)
Signature · 1 line
2.7.4
#1 lore raw

PATCH v2 1/1 fs/common.mk: enable multithreaded xz compression

James,

Quoted 9 lines · On Thu, Jan 24, 2019 at 2:28 AM <[email protected]> wrote:

From: James Hilliard <[email protected]>

xz help indicates only 1 thread is used unless we set threads:
-T, --threads=NUM use at most NUM threads; the default is 1; set to 0
to use as many threads as there are processor cores

Quoted 13 lines

---
Changes v1 -> v2:
- use PARALLEL_JOBS to determine how many threads to compress with
- disable multithreaded compression for reproducible builds

---
 fs/common.mk | 4 ++++
 1 file changed, 4 insertions(+)
fs/common.mk +0 -0
index a560417..76da6d8 100644

fs/common.mk endif · line 112

Quoted 7 lines
fs/common.mk +3 -0
@@ -106,7 +106,11 @@ endif
 ifeq ($$(BR2_TARGET_ROOTFS_$(2)_XZ),y)
 ROOTFS_$(2)_DEPENDENCIES += host-xz
 ROOTFS_$(2)_COMPRESS_EXT = .xz
+ifeq ($(BR2_REPRODUCIBLE),y)
 ROOTFS_$(2)_COMPRESS_CMD = xz -9 -C crc32 -c
+else
+ROOTFS_$(2)_COMPRESS_CMD = xz -T $(PARALLEL_JOBS) -9 -C crc32 -c

I double checked this logic below in package/Makefile.in for creating PARALLEL_JOBS still made sense for the use of the value above.

# If BR2_JLEVEL is 0, scale the maximum concurrency with the number of # CPUs. An additional job is used in order to keep processors busy

# while waiting on I/O.
# If the number of processors is not available, assume one.
ifeq ($(BR2_JLEVEL),0)
PARALLEL_JOBS := $(shell echo \
        $$((1 + `getconf _NPROCESSORS_ONLN 2>/dev/null || echo 1`)))
else
PARALLEL_JOBS := $(BR2_JLEVEL)
endif

Matt

#2 lore raw

PATCH v2 1/1 fs/common.mk: enable multithreaded xz compression

Yann, Thomas,

Quoted 43 lines · Matthew Weber

James,

Quoted 9 lines · On Thu, Jan 24, 2019 at 2:28 AM <[email protected]> wrote:

From: James Hilliard <[email protected]>

xz help indicates only 1 thread is used unless we set threads:
-T, --threads=NUM use at most NUM threads; the default is 1; set to 0
to use as many threads as there are processor cores

Quoted 13 lines

---
Changes v1 -> v2:
- use PARALLEL_JOBS to determine how many threads to compress with
- disable multithreaded compression for reproducible builds

---
 fs/common.mk | 4 ++++
 1 file changed, 4 insertions(+)
fs/common.mk +0 -0
index a560417..76da6d8 100644

fs/common.mk endif · line 112

Quoted 7 lines
fs/common.mk +3 -0
@@ -106,7 +106,11 @@ endif
 ifeq ($$(BR2_TARGET_ROOTFS_$(2)_XZ),y)
 ROOTFS_$(2)_DEPENDENCIES += host-xz
 ROOTFS_$(2)_COMPRESS_EXT = .xz
+ifeq ($(BR2_REPRODUCIBLE),y)
 ROOTFS_$(2)_COMPRESS_CMD = xz -9 -C crc32 -c
+else
+ROOTFS_$(2)_COMPRESS_CMD = xz -T $(PARALLEL_JOBS) -9 -C crc32 -c

I double checked this logic below in package/Makefile.in for creating PARALLEL_JOBS still made sense for the use of the value above.

# If BR2_JLEVEL is 0, scale the maximum concurrency with the number of # CPUs. An additional job is used in order to keep processors busy

Quoted 9 lines
# while waiting on I/O.
# If the number of processors is not available, assume one.
ifeq ($(BR2_JLEVEL),0)
PARALLEL_JOBS := $(shell echo \
        $$((1 + `getconf _NPROCESSORS_ONLN 2>/dev/null || echo 1`)))
else
PARALLEL_JOBS := $(BR2_JLEVEL)
endif

When we eventually do filesystem creation in a top level parallel build, if we add xz multithreaded compression based on parallel jobs variable, I assume we'll create a problem with cpu loading? Maybe this isn't a concern at this point or is there a previous assumption/discussions on how this was going to be handled with different build systems (GO, ninja based, etc) which may be called in parallel while building pkgs.

Matt

#3 lore raw

PATCH v2 1/1 fs/common.mk: enable multithreaded xz compression

Quoted 12 lines · On Mon, 4 Feb 2019 04:36:56 -0600 Matthew Weber <[email protected]>…
Quoted 11 lines

# If BR2_JLEVEL is 0, scale the maximum concurrency with the number of # CPUs. An additional job is used in order to keep processors busy

# while waiting on I/O.
# If the number of processors is not available, assume one.
ifeq ($(BR2_JLEVEL),0)
PARALLEL_JOBS := $(shell echo \
        $$((1 + `getconf _NPROCESSORS_ONLN 2>/dev/null || echo 1`)))
else
PARALLEL_JOBS := $(BR2_JLEVEL)
endif
Quoted 7 lines

When we eventually do filesystem creation in a top level parallel build, if we add xz multithreaded compression based on parallel jobs variable, I assume we'll create a problem with cpu loading? Maybe this isn't a concern at this point or is there a previous assumption/discussions on how this was going to be handled with different build systems (GO, ninja based, etc) which may be called in parallel while building pkgs.

We have no real solution for non-make based build systems. For ninja specifically, there is this open issue:
https://github.com/ninja-build/ninja/issues/1139. For other build systems, I don't think there's a good solution.

For the filesystem generation, I don't think there will really be an actual problem. Indeed, by the time you generate the filesystem images, all packages have completed building, so the system load should be quite low. And most reasonable configurations don't have a lot of different filesystem formats enabled. Even if you have 3 or 4 of them, I don't think the parallel xz used for each instance will really cause a CPU load that isn't manageable. Of course, testing needed to verify those claims.

Best regards,

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#4 lore raw

PATCH v2 1/1 fs/common.mk: enable multithreaded xz compression

In reply to james.hilliard1 at gmail.com
Quoted 7 lines · On 24/01/2019 09:27, james.hilliard1 at gmail.com wrote:

From: James Hilliard <[email protected]>

xz help indicates only 1 thread is used unless we set threads:
-T, --threads=NUM use at most NUM threads; the default is 1; set to 0
to use as many threads as there are processor cores

Applied to master, thanks.

Quoted 14 lines

---
Changes v1 -> v2:
- use PARALLEL_JOBS to determine how many threads to compress with
- disable multithreaded compression for reproducible builds

---
 fs/common.mk | 4 ++++
 1 file changed, 4 insertions(+)
fs/common.mk +0 -0
index a560417..76da6d8 100644

fs/common.mk endif · line 112

Quoted 7 lines
fs/common.mk +3 -0
@@ -106,7 +106,11 @@ endif
 ifeq ($$(BR2_TARGET_ROOTFS_$(2)_XZ),y)
 ROOTFS_$(2)_DEPENDENCIES += host-xz
 ROOTFS_$(2)_COMPRESS_EXT = .xz
+ifeq ($(BR2_REPRODUCIBLE),y)
 ROOTFS_$(2)_COMPRESS_CMD = xz -9 -C crc32 -c
+else
+ROOTFS_$(2)_COMPRESS_CMD = xz -T $(PARALLEL_JOBS) -9 -C crc32 -c

I rewrote this to append the -T option instead of repeating it.

Regards,
Arnout

Quoted 5 lines
fs/common.mk +1 -0
+endif
 endif
 
 $$(BINARIES_DIR)/$$(ROOTFS_$(2)_FINAL_IMAGE_NAME): ROOTFS=$(2)