Skip to contentSkip to conversation

kbuild: modinst: remove the multithread option from zstd compression

PATCH In review Masahiro Yamada 3 participants 5 messages Aug 4, 2024 – Aug 24, 2024 Kbuild

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 kbuild: modinst: remove the multithread option from zstd compression

Parallel execution is supported by GNU Make:

  $ make -j<N> modules_install

It is questionable to enable multithreading within each zstd process by default.

If you still want to do it, you can use the environment variable:

  $ ZSTD_NBTHREADS=<N> make modules_install

---

 scripts/Makefile.modinst | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
scripts/Makefile.modinst +1 -1
index 0afd75472679..04f5229efa6b 100644
@@ -146,7 +146,7 @@ quiet_cmd_gzip = GZIP    $@
 quiet_cmd_xz = XZ      $@
       cmd_xz = $(XZ) --check=crc32 --lzma2=dict=1MiB -f $<
 quiet_cmd_zstd = ZSTD    $@
-      cmd_zstd = $(ZSTD) -T0 --rm -f -q $<
+      cmd_zstd = $(ZSTD) --rm -f -q $<
 
 $(dst)/%.ko.gz: $(dst)/%.ko FORCE
 	$(call cmd,gzip)
Signature · 1 line
2.43.0
#1 lore raw
Quoted 12 lines · Masahiro Yamada

Parallel execution is supported by GNU Make:

  $ make -j<N> modules_install

It is questionable to enable multithreading within each zstd process by default.

If you still want to do it, you can use the environment variable:

  $ ZSTD_NBTHREADS=<N> make modules_install

Hi Masahiro,

I have some understanding problems.

[ start-build.txt ]
dileks 24225 24217 0 17:55 tty2 00:00:00 /usr/bin/perf stat
make V=1 -k -j4 ARCH=x86_64 LLVM=1 LLVM=/opt/llvm/bin/ PAHOLE=/opt/pahole/bin/pahole KBUILD_BUILD_HOST=iniza [email protected] KBUILD_BUILD_TIMESTAMP=2024-08-03 KDEB_PKGVERSION=6.10.3-1~trixie+dileks1 LOCALVERSION=-1-amd64-clang18-kcfi olddefconfig bindeb-pkg

^^ How shall someone pass so that ... ZSTD_NBTHREADS=<N> make modules_install ... is used?

As far as I understood, each kernel-module file is taken - in the Debian build-process - sequentially file for file - ZSTD compressed and afterwards deleted.
Is there a benefit when 'make -j<N>' is used?

[ EXAMPLE - with my patch from [1] ]
zstd -T0 -19 --rm -f -q debian/linux-image-6.10.3-1-amd64-clang18-kcfi/lib/modules/6.10.3-1-amd64-clang18-kcfi/kernel/drivers/gpu/drm/amd/amdgpu/amdgpu.ko

^^ How should ZSTD_NBTHREADS=<N> replace 'zstd -T0'?

When I pass '-T0 -L19' to ZSTD two cores are used on my CPU and a better compress-level produces smaller kernel-module files. I would like to benefit from both.

Thanks.

Best regards,
-Sedat-

Quoted 23 lines

---

 scripts/Makefile.modinst | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
scripts/Makefile.modinst +1 -2
index 0afd75472679..04f5229efa6b 100644
@@ -146,7 +146,7 @@ quiet_cmd_gzip = GZIP    $@
 quiet_cmd_xz = XZ      $@
       cmd_xz = $(XZ) --check=crc32 --lzma2=dict=1MiB -f $<
 quiet_cmd_zstd = ZSTD    $@
-      cmd_zstd = $(ZSTD) -T0 --rm -f -q $<
+      cmd_zstd = $(ZSTD) --rm -f -q $<
 
 $(dst)/%.ko.gz: $(dst)/%.ko FORCE
        $(call cmd,gzip)
--

2.43.0

#2 lore raw
Quoted 28 lines · Sedat Dilek
Quoted 12 lines · Masahiro Yamada

Parallel execution is supported by GNU Make:

  $ make -j<N> modules_install

It is questionable to enable multithreading within each zstd process by default.

If you still want to do it, you can use the environment variable:

  $ ZSTD_NBTHREADS=<N> make modules_install

Hi Masahiro,

I have some understanding problems.

[ start-build.txt ]
dileks 24225 24217 0 17:55 tty2 00:00:00 /usr/bin/perf stat
make V=1 -k -j4 ARCH=x86_64 LLVM=1 LLVM=/opt/llvm/bin/ PAHOLE=/opt/pahole/bin/pahole KBUILD_BUILD_HOST=iniza [email protected] KBUILD_BUILD_TIMESTAMP=2024-08-03 KDEB_PKGVERSION=6.10.3-1~trixie+dileks1 LOCALVERSION=-1-amd64-clang18-kcfi olddefconfig bindeb-pkg

Quoted 2 lines

^^ How shall someone pass so that ... ZSTD_NBTHREADS=<N> make modules_install ... is used?

Option 1

$ export ZSTD_NBTHREADS=<N>
$ make V=1 -k  [snip] olddefconfig bindeb-pkg

Option 2

$ ZSTD_NBTHREADS=<N> make V=1 -k [snip] olddefconfig bindeb-pkg
Quoted 5 lines

As far as I understood, each kernel-module file is taken - in the Debian build-process - sequentially file for file - ZSTD compressed and afterwards deleted.
Is there a benefit when 'make -j<N>' is used?

zstd --rm -f -q foo.ko
zstd --rm -f -q bar.ko
zstd --rm -f -q baz.ko

will run in parallel with Make's -j<N> option.

Signature · 2 lines
Best Regards
Masahiro Yamada
Reviewed-by on v1 1/1
#3 lore raw
In reply to Masahiro Yamada
Quoted 33 lines · Masahiro Yamada

Parallel execution is supported by GNU Make:

  $ make -j<N> modules_install

It is questionable to enable multithreading within each zstd process by default.

If you still want to do it, you can use the environment variable:

  $ ZSTD_NBTHREADS=<N> make modules_install

---

 scripts/Makefile.modinst | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
scripts/Makefile.modinst +1 -1
index 0afd75472679..04f5229efa6b 100644
@@ -146,7 +146,7 @@ quiet_cmd_gzip = GZIP    $@
 quiet_cmd_xz = XZ      $@
       cmd_xz = $(XZ) --check=crc32 --lzma2=dict=1MiB -f $<
 quiet_cmd_zstd = ZSTD    $@
-      cmd_zstd = $(ZSTD) -T0 --rm -f -q $<
+      cmd_zstd = $(ZSTD) --rm -f -q $<
 
 $(dst)/%.ko.gz: $(dst)/%.ko FORCE
 	$(call cmd,gzip)
Signature · 1 line
2.43.0
Tested-by
#4 lore raw
Quoted 30 lines · Nicolas Schier
Quoted 28 lines · Masahiro Yamada

Parallel execution is supported by GNU Make:

  $ make -j<N> modules_install

It is questionable to enable multithreading within each zstd process by default.

If you still want to do it, you can use the environment variable:

  $ ZSTD_NBTHREADS=<N> make modules_install

---

 scripts/Makefile.modinst | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
scripts/Makefile.modinst +1 -1
index 0afd75472679..04f5229efa6b 100644
@@ -146,7 +146,7 @@ quiet_cmd_gzip = GZIP    $@
 quiet_cmd_xz = XZ      $@
       cmd_xz = $(XZ) --check=crc32 --lzma2=dict=1MiB -f $<
 quiet_cmd_zstd = ZSTD    $@
-      cmd_zstd = $(ZSTD) -T0 --rm -f -q $<
+      cmd_zstd = $(ZSTD) --rm -f -q $<
 
Quoted 8 lines
Quoted 5 lines
scripts/Makefile.modinst +0 -1
 $(dst)/%.ko.gz: $(dst)/%.ko FORCE
      $(call cmd,gzip)
--

2.43.0

-Sedat-