Skip to contentSkip to conversation

Enable compressed files in EXTRA_FIRMWARE

PATCH Superseded Kevin Martin 3 participants 9 messages Dec 20, 2023 – Jan 4, 2024

Subsystems (3)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

cover Enable compressed files in EXTRA_FIRMWARE

The linux-firmware packages on Gentoo, Fedora, Arch, and others compress the firmware files. This works well with CONFIG_FW_LOADER_COMPRESS, but does not work with CONFIG_EXTRA_FIRMWARE. This patch allows the build system to decompress firmware files specified by CONFIG_EXTRA_FIRMWARE.

PATCH 1/2 adds decompression routines next to the compression routines in scripts/Makefile.lib. That patch is then used by PATCH 2/2 to decompress files before compiling them into the kernel.

The patch works by copying or decompressing the specified firmware files into the build directory, then compiling them in from there. I would prefer to not copy any uncompressed files, but I have not found a clean way to do that.

Kevin Martin (2):
kbuild: Enable decompression for use by EXTRA_FIRMWARE The build
system can currently only compress files. This patch adds the functionality to decompress files. Decompression is needed for building firmware files into the kernel if those files are compressed on the filesystem. Compressed firmware files are in use by Gentoo, Fedora, Arch, and others.
firmware_loader: Enable compressed files with EXTRA_FIRMWARE The
linux-firmware packages on Gentoo, Fedora, Arch, and others compress the firmware files. This works well with CONFIG_FW_LOADER_COMPRESS but does not work with CONFIG_EXTRA_FIRMWARE. This patch allows the build system to decompress firmware files specified by CONFIG_EXTRA_FIRMWARE. Uncompressed files are used first, then the compressed files are used.

 drivers/base/firmware_loader/Kconfig          |  5 ++++-
 drivers/base/firmware_loader/builtin/Makefile | 16 ++++++++++++----
 scripts/Makefile.lib                          |  6 ++++++
 3 files changed, 22 insertions(+), 5 deletions(-)
Signature · 1 line
2.41.0
#1 lore raw

PATCH 1/2 kbuild: Enable decompression for use by EXTRA_FIRMWARE The build system can currently only compress files. This patch adds the functionality to decompress files. Decompression is needed for building firmware files into the kernel if those files are compressed on the filesystem. Compressed firmware files are in use by Gentoo, Fedora, Arch, and others.

---
 scripts/Makefile.lib | 6 ++++++
 1 file changed, 6 insertions(+)
scripts/Makefile.lib +6 -0
index 1a965fe68..d043be3dc 100644
@@ -523,6 +523,9 @@ quiet_cmd_xzkern_with_size = XZKERN  $@
 quiet_cmd_xzmisc = XZMISC  $@
       cmd_xzmisc = cat $(real-prereqs) | $(XZ) --check=crc32 --lzma2=dict=1MiB > $@
 
+quiet_cmd_xzdec = XZDEC   $@
+      cmd_xzdec = cat $(real-prereqs) | $(XZ) --decompress > $@
+
 # ZSTD
 # ---------------------------------------------------------------------------
 # Appends the uncompressed size of the data using size_append. The .zst
@@ -548,6 +551,9 @@ quiet_cmd_zstd22 = ZSTD22  $@
 quiet_cmd_zstd22_with_size = ZSTD22  $@
       cmd_zstd22_with_size = { cat $(real-prereqs) | $(ZSTD) -22 --ultra; $(size_append); } > $@
 
+quiet_cmd_zstddec = ZSTDDEC $@
+      cmd_zstddec = cat $(real-prereqs) | $(ZSTD) --decompress > $@
+
 # ASM offsets
 # ---------------------------------------------------------------------------
 
Signature · 1 line
2.41.0
#6 lore raw

PATCH 2/2 firmware_loader: Enable compressed files in EXTRA_FIRMWARE

In reply to Kevin Martin

The linux-firmware packages on Gentoo, Fedora, Arch, and others compress the firmware files. This works well with CONFIG_FW_LOADER_COMPRESS, but does not work with CONFIG_EXTRA_FIRMWARE. This patch allows the build system to decompress firmware files specified by CONFIG_EXTRA_FIRMWARE. Uncompressed files are used first, then the compressed files are used.

---
 drivers/base/firmware_loader/Kconfig          |  5 ++++-
 drivers/base/firmware_loader/builtin/Makefile | 16 ++++++++++++----
 2 files changed, 16 insertions(+), 5 deletions(-)
drivers/base/firmware_loader/Kconfig +4 -1
index 5ca00e02f..b7a908bff 100644
@@ -76,7 +76,10 @@ config EXTRA_FIRMWARE
 	  image since it combines both GPL and non-GPL work. You should
 	  consult a lawyer of your own before distributing such an image.
 
-	  NOTE: Compressed files are not supported in EXTRA_FIRMWARE.
+	  NOTE: Compressed files are supported by EXTRA_FIRMWARE. The build
+	  system will look for uncompressed files first then fall back to
+	  searching for compressed files in a similar way to
+	  CONFIG_FW_LOADER_COMPRESS.
 
 config EXTRA_FIRMWARE_DIR
 	string "Firmware blobs root directory"
drivers/base/firmware_loader/builtin/Makefile +12 -4
index 6c067dedc..cc60eb441 100644
@@ -20,7 +20,7 @@ filechk_fwbin = \
 	echo "    .section .rodata"				;\
 	echo "    .p2align 4"					;\
 	echo "_fw_$(FWSTR)_bin:"				;\
-	echo "    .incbin \"$(fwdir)/$(FWNAME)\""		;\
+	echo "    .incbin \"$(obj)/$(FWNAME)\""		;\
 	echo "_fw_end:"						;\
 	echo "    .section .rodata.str,\"aMS\",$(PROGBITS),1"	;\
 	echo "    .p2align $(ASM_ALIGN)"			;\
@@ -36,7 +36,15 @@ $(obj)/%.gen.S: FORCE
 	$(call filechk,fwbin)
 
 # The .o files depend on the binaries directly; the .S files don't.
-$(addprefix $(obj)/, $(firmware)): $(obj)/%.gen.o: $(fwdir)/%
+$(addprefix $(obj)/, $(firmware)): $(obj)/%.gen.o: $(obj)/%
 
-targets := $(patsubst $(obj)/%,%, \
-                                $(shell find $(obj) -name \*.gen.S 2>/dev/null))
+$(obj)/% : $(fwdir)/% FORCE
+	$(call if_changed,copy)
+
+$(obj)/% : $(fwdir)/%.xz FORCE
+	$(call if_changed,xzdec)
+
+$(obj)/% : $(fwdir)/%.zst FORCE
+	$(call if_changed,zstddec)
+
+targets := $(patsubst %.gen.o, %.gen.S, $(firmware)) $(CONFIG_EXTRA_FIRMWARE)
Signature · 1 line
2.41.0
Tested-by on v1 1/2
#2 lore raw

Re: [PATCH 1/2] kbuild: Enable decompression for use by EXTRA_FIRMWARE The build system can currently only compress files. This patch adds the functionality to decompress files. Decompression is needed for building firmware files into the kernel if those files are compressed on the filesystem. Compressed firmware files are in use by Gentoo, Fedora, Arch, and others.

In reply to Kevin Martin

Hi Kevin,

Quoted 6 lines

Subject: Re: [PATCH 1/2] kbuild: Enable decompression for use by
EXTRA_FIRMWARE The build system can currently only compress files. This patch adds the functionality to decompress files. Decompression is needed for building firmware files into the kernel if those files are compressed on the filesystem. Compressed firmware files are in use by Gentoo, Fedora, Arch, and others.

patch description is squashed into the subject. Did your tooling accidentially remove the empty line between?

The patch itself looks good to me.

Kind regards,
Nicolas

Quoted 32 lines · Kevin Martin
---
 scripts/Makefile.lib | 6 ++++++
 1 file changed, 6 insertions(+)
scripts/Makefile.lib +6 -0
index 1a965fe68..d043be3dc 100644
@@ -523,6 +523,9 @@ quiet_cmd_xzkern_with_size = XZKERN  $@
 quiet_cmd_xzmisc = XZMISC  $@
       cmd_xzmisc = cat $(real-prereqs) | $(XZ) --check=crc32 --lzma2=dict=1MiB > $@
 
+quiet_cmd_xzdec = XZDEC   $@
+      cmd_xzdec = cat $(real-prereqs) | $(XZ) --decompress > $@
+
 # ZSTD
 # ---------------------------------------------------------------------------
 # Appends the uncompressed size of the data using size_append. The .zst
@@ -548,6 +551,9 @@ quiet_cmd_zstd22 = ZSTD22  $@
 quiet_cmd_zstd22_with_size = ZSTD22  $@
       cmd_zstd22_with_size = { cat $(real-prereqs) | $(ZSTD) -22 --ultra; $(size_append); } > $@
 
+quiet_cmd_zstddec = ZSTDDEC $@
+      cmd_zstddec = cat $(real-prereqs) | $(ZSTD) --decompress > $@
+
 # ASM offsets
 # ---------------------------------------------------------------------------
 
Signature · 1 line
2.41.0
Tested-by on v1 2/2
#7 lore raw

Re: [PATCH 2/2] firmware_loader: Enable compressed files in EXTRA_FIRMWARE

In reply to Kevin Martin
Quoted 8 lines · Kevin Martin

The linux-firmware packages on Gentoo, Fedora, Arch, and others compress the firmware files. This works well with CONFIG_FW_LOADER_COMPRESS, but does not work with CONFIG_EXTRA_FIRMWARE. This patch allows the build system to decompress firmware files specified by CONFIG_EXTRA_FIRMWARE. Uncompressed files are used first, then the compressed files are used.

---

#3 lore raw

Re: [PATCH 1/2] kbuild: Enable decompression for use by EXTRA_FIRMWARE The build system can currently only compress files. This patch adds the functionality to decompress files. Decompression is needed for building firmware files into the kernel if those files are compressed on the filesystem. Compressed firmware files are in use by Gentoo, Fedora, Arch, and others.

In reply to Kevin Martin
Quoted 11 lines · Kevin Martin
---
 scripts/Makefile.lib | 6 ++++++
 1 file changed, 6 insertions(+)
scripts/Makefile.lib +0 -0
index 1a965fe68..d043be3dc 100644

scripts/Makefile.lib quiet_cmd_xzkern_with_size = XZKERN $@ · line 528

Quoted 6 lines
scripts/Makefile.lib +3 -0
@@ -523,6 +523,9 @@ quiet_cmd_xzkern_with_size = XZKERN  $@
 quiet_cmd_xzmisc = XZMISC  $@
       cmd_xzmisc = cat $(real-prereqs) | $(XZ) --check=crc32 --lzma2=dict=1MiB > $@
 
+quiet_cmd_xzdec = XZDEC   $@
+      cmd_xzdec = cat $(real-prereqs) | $(XZ) --decompress > $@
+

Please do not fork the meaningless 'cat' process.

This should be a single process to take just one input file.

    cmd_xzdec = $(XZ) --decompress --stdout $< > $@

Commit d3dd3b5a29bb9582957451531fed461628dfc834
was a very bad commit.

The 'cat' and compression/decompression must be
separate rules.

We should not repeat the mistake in the past.

scripts/Makefile.lib quiet_cmd_zstd22 = ZSTD22 $@ · line 556

Quoted 10 lines
scripts/Makefile.lib +3 -0
 # ZSTD
 # ---------------------------------------------------------------------------
 # Appends the uncompressed size of the data using size_append. The .zst
@@ -548,6 +551,9 @@ quiet_cmd_zstd22 = ZSTD22  $@
 quiet_cmd_zstd22_with_size = ZSTD22  $@
       cmd_zstd22_with_size = { cat $(real-prereqs) | $(ZSTD) -22 --ultra; $(size_append); } > $@
 
+quiet_cmd_zstddec = ZSTDDEC $@
+      cmd_zstddec = cat $(real-prereqs) | $(ZSTD) --decompress > $@
+

Same here.
Please make this a single process:

   cmd_zstddec = $(ZSTD) --decompress --force --output=$@ $<

One small concern in the future is, if we end up with adding quiet_cmd_bzip2dec, we will run out of the 7-column of the short log.

quiet_cmd_bzip2dec = BZIP2DEC$@

We can increase the column size if needed, so I do not think it is a big issue.

Quoted 6 lines
scripts/Makefile.lib +0 -1
 # ASM offsets
 # ---------------------------------------------------------------------------
 
--

2.41.0

Signature · 2 lines
Best Regards
Masahiro Yamada
#8 lore raw

Re: [PATCH 2/2] firmware_loader: Enable compressed files in EXTRA_FIRMWARE

In reply to Kevin Martin
Quoted 41 lines · Kevin Martin

The linux-firmware packages on Gentoo, Fedora, Arch, and others compress the firmware files. This works well with CONFIG_FW_LOADER_COMPRESS, but does not work with CONFIG_EXTRA_FIRMWARE. This patch allows the build system to decompress firmware files specified by CONFIG_EXTRA_FIRMWARE. Uncompressed files are used first, then the compressed files are used.

---
 drivers/base/firmware_loader/Kconfig          |  5 ++++-
 drivers/base/firmware_loader/builtin/Makefile | 16 ++++++++++++----
 2 files changed, 16 insertions(+), 5 deletions(-)
drivers/base/firmware_loader/Kconfig +4 -1
index 5ca00e02f..b7a908bff 100644
@@ -76,7 +76,10 @@ config EXTRA_FIRMWARE
          image since it combines both GPL and non-GPL work. You should
          consult a lawyer of your own before distributing such an image.
 
-         NOTE: Compressed files are not supported in EXTRA_FIRMWARE.
+         NOTE: Compressed files are supported by EXTRA_FIRMWARE. The build
+         system will look for uncompressed files first then fall back to
+         searching for compressed files in a similar way to
+         CONFIG_FW_LOADER_COMPRESS.
 
 config EXTRA_FIRMWARE_DIR
        string "Firmware blobs root directory"
drivers/base/firmware_loader/builtin/Makefile +0 -0
index 6c067dedc..cc60eb441 100644

drivers/base/firmware_loader/builtin/Makefile filechk_fwbin = \ · line 23

Quoted 5 lines
drivers/base/firmware_loader/builtin/Makefile +1 -1
@@ -20,7 +20,7 @@ filechk_fwbin = \
        echo "    .section .rodata"                             ;\
        echo "    .p2align 4"                                   ;\
        echo "_fw_$(FWSTR)_bin:"                                ;\
-       echo "    .incbin \"$(fwdir)/$(FWNAME)\""               ;\
+       echo "    .incbin \"$(obj)/$(FWNAME)\""         ;\

Insert a tab to keep the alignment of ";\"

Also, please update the .gitignore because
any arbitrary firmware file might be copied to
this directory.
All files except the 3 check-in files must be
ignored.

@@ -1,2 +1,5 @@
# SPDX-License-Identifier: GPL-2.0
-*.gen.S
+*
+!.gitignore
+!Makefile
+!main.c

--
Best Regards
Masahiro Yamada

#4 lore raw

Re: [PATCH 1/2] kbuild: Enable decompression for use by EXTRA_FIRMWARE The build system can currently only compress files. This patch adds the functionality to decompress files. Decompression is needed for building firmware files into the kernel if those files are compressed on the filesystem. Compressed firmware files are in use by Gentoo, Fedora, Arch, and others.

In reply to Masahiro Yamada
Quoted 30 lines · Masahiro Yamada
Quoted 11 lines · Kevin Martin
---
 scripts/Makefile.lib | 6 ++++++
 1 file changed, 6 insertions(+)
scripts/Makefile.lib +0 -0
index 1a965fe68..d043be3dc 100644

scripts/Makefile.lib quiet_cmd_xzkern_with_size = XZKERN $@ · line 528

Quoted 6 lines
scripts/Makefile.lib +3 -0
@@ -523,6 +523,9 @@ quiet_cmd_xzkern_with_size = XZKERN  $@
 quiet_cmd_xzmisc = XZMISC  $@
       cmd_xzmisc = cat $(real-prereqs) | $(XZ) --check=crc32 --lzma2=dict=1MiB > $@
 
+quiet_cmd_xzdec = XZDEC   $@
+      cmd_xzdec = cat $(real-prereqs) | $(XZ) --decompress > $@
+

Please do not fork the meaningless 'cat' process.

This should be a single process to take just one input file.

    cmd_xzdec = $(XZ) --decompress --stdout $< > $@
Quoted 8 lines

Commit d3dd3b5a29bb9582957451531fed461628dfc834
was a very bad commit.

The 'cat' and compression/decompression must be
separate rules.

We should not repeat the mistake in the past.

Would it be preferable to change all of the compression rules or just the new decompression rules?

I could change just the new ones and then begin working on a different patch to clean up the 'cat' processes in the compression rules.

Quoted 53 lines

scripts/Makefile.lib quiet_cmd_zstd22 = ZSTD22 $@ · line 556

Quoted 12 lines
scripts/Makefile.lib +3 -0
 # ZSTD
 # ---------------------------------------------------------------------------
 # Appends the uncompressed size of the data using size_append. The .zst
@@ -548,6 +551,9 @@ quiet_cmd_zstd22 = ZSTD22  $@
 quiet_cmd_zstd22_with_size = ZSTD22  $@
       cmd_zstd22_with_size = { cat $(real-prereqs) | $(ZSTD) -22 --ultra; $(size_append); } > $@
 
+quiet_cmd_zstddec = ZSTDDEC $@
+      cmd_zstddec = cat $(real-prereqs) | $(ZSTD) --decompress > $@
+

Same here.
Please make this a single process:

   cmd_zstddec = $(ZSTD) --decompress --force --output=$@ $<

One small concern in the future is, if we end up with adding quiet_cmd_bzip2dec, we will run out of the 7-column of the short log.

quiet_cmd_bzip2dec = BZIP2DEC$@

We can increase the column size if needed, so I do not think it is a big issue.

Quoted 6 lines
scripts/Makefile.lib +0 -1
 # ASM offsets
 # ---------------------------------------------------------------------------
 
--

2.41.0

Signature · 2 lines
Best Regards
Masahiro Yamada
#5 lore raw

Re: [PATCH 1/2] kbuild: Enable decompression for use by EXTRA_FIRMWARE The build system can currently only compress files. This patch adds the functionality to decompress files. Decompression is needed for building firmware files into the kernel if those files are compressed on the filesystem. Compressed firmware files are in use by Gentoo, Fedora, Arch, and others.

Quoted 37 lines · Kevin Martin
Quoted 33 lines · Masahiro Yamada
Quoted 11 lines · Kevin Martin
---
 scripts/Makefile.lib | 6 ++++++
 1 file changed, 6 insertions(+)
scripts/Makefile.lib +0 -0
index 1a965fe68..d043be3dc 100644

scripts/Makefile.lib quiet_cmd_xzkern_with_size = XZKERN $@ · line 528

Quoted 6 lines
scripts/Makefile.lib +3 -0
@@ -523,6 +523,9 @@ quiet_cmd_xzkern_with_size = XZKERN  $@
 quiet_cmd_xzmisc = XZMISC  $@
       cmd_xzmisc = cat $(real-prereqs) | $(XZ) --check=crc32 --lzma2=dict=1MiB > $@
 
+quiet_cmd_xzdec = XZDEC   $@
+      cmd_xzdec = cat $(real-prereqs) | $(XZ) --decompress > $@
+

Please do not fork the meaningless 'cat' process.

This should be a single process to take just one input file.

    cmd_xzdec = $(XZ) --decompress --stdout $< > $@

Commit d3dd3b5a29bb9582957451531fed461628dfc834
was a very bad commit.

Quoted 8 lines
Quoted 5 lines

The 'cat' and compression/decompression must be
separate rules.

We should not repeat the mistake in the past.

Would it be preferable to change all of the compression rules or just the new decompression rules?

I do not require you to change the existing code.

For decompression, it is unlikely that the recipe
takes multiple input files.
So, 'cat' is unneeded.

Quoted 2 lines

I could change just the new ones and then begin working on a different patch to clean up the 'cat' processes in the compression rules.

If you get rid of the 'cat', you need to refactor the user code. arch/x86/boot/compressed/Makefile relies on 'cat' and 'compress', but please double-check no other Makefile uses it.

Also, you might need some research about the potential impact onto the reproducible builds.
Without 'cat |', the compressed archive might encode the timestamp of the original file.
GZIP records the timestamp in the header.

Quoted 54 lines
Quoted 54 lines

scripts/Makefile.lib quiet_cmd_zstd22 = ZSTD22 $@ · line 556

Quoted 13 lines
scripts/Makefile.lib +3 -0
 # ZSTD
 # ---------------------------------------------------------------------------
 # Appends the uncompressed size of the data using size_append. The .zst
@@ -548,6 +551,9 @@ quiet_cmd_zstd22 = ZSTD22  $@
 quiet_cmd_zstd22_with_size = ZSTD22  $@
       cmd_zstd22_with_size = { cat $(real-prereqs) | $(ZSTD) -22 --ultra; $(size_append); } > $@
 
+quiet_cmd_zstddec = ZSTDDEC $@
+      cmd_zstddec = cat $(real-prereqs) | $(ZSTD) --decompress > $@
+

Same here.
Please make this a single process:

   cmd_zstddec = $(ZSTD) --decompress --force --output=$@ $<

One small concern in the future is, if we end up with adding quiet_cmd_bzip2dec, we will run out of the 7-column of the short log.

quiet_cmd_bzip2dec = BZIP2DEC$@

We can increase the column size if needed, so I do not think it is a big issue.

Quoted 6 lines
scripts/Makefile.lib +0 -1
 # ASM offsets
 # ---------------------------------------------------------------------------
 
--

2.41.0

--
Best Regards
Masahiro Yamada

Signature · 2 lines
Best Regards
Masahiro Yamada