Skip to contentSkip to conversation

Enable compressed files in EXTRA_FIRMWARE

PATCH In review Kevin Martin 2 participants 6 messages Jan 5, 2024 – Jan 9, 2024

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

v2 cover Enable compressed files in EXTRA_FIRMWARE

The linux-firmware packages on Gentoo, Fedora, Arch, and others compress the firmware files. This works well with CONFIG_FW_LOADER_COMPRESS, but does not work with CONFIG_EXTRA_FIRMWARE. This patch allows the build system to decompress firmware files specified by CONFIG_EXTRA_FIRMWARE.

PATCH 1/2 adds decompression routines next to the compression routines in scripts/Makefile.lib. That patch is then used by PATCH 2/2 to decompress files before compiling them into the kernel.

The patch works by copying or decompressing the specified firmware files into the build directory, then compiling them in from there. I would prefer to not copy any uncompressed files, but I have not found a clean way to do that.

 drivers/base/firmware_loader/Kconfig            |  5 ++++-
 drivers/base/firmware_loader/builtin/.gitignore |  5 ++++-
 drivers/base/firmware_loader/builtin/Makefile   | 16 ++++++++++++----
 scripts/Makefile.lib                            |  6 ++++++
 4 files changed, 26 insertions(+), 6 deletions(-)
Signature · 1 line
2.41.0
#1 lore raw

PATCH v2 1/2 kbuild: Enable decompression for use by EXTRA_FIRMWARE

The build system can currently only compress files. This patch adds the functionality to decompress files. Decompression is needed for building firmware files into the kernel if those files are compressed on the filesystem. Compressed firmware files are in use by Gentoo, Fedora, Arch, and others.

---
Changes in v2:
- Skipped running 'cat' and now just pass the file names directly.
- Added '--quiet' since 'zstd' started printing the status of each file now that it knows the file names.

 scripts/Makefile.lib | 6 ++++++
 1 file changed, 6 insertions(+)
scripts/Makefile.lib +6 -0
index 1a965fe68..d043be3dc 100644
@@ -523,6 +523,9 @@ quiet_cmd_xzkern_with_size = XZKERN  $@
 quiet_cmd_xzmisc = XZMISC  $@
       cmd_xzmisc = cat $(real-prereqs) | $(XZ) --check=crc32 --lzma2=dict=1MiB > $@
 
+quiet_cmd_xzdec = XZDEC   $@
+      cmd_xzdec = $(XZ) --decompress --stdout $< > $@
+
 # ZSTD
 # ---------------------------------------------------------------------------
 # Appends the uncompressed size of the data using size_append. The .zst
@@ -548,6 +551,9 @@ quiet_cmd_zstd22 = ZSTD22  $@
 quiet_cmd_zstd22_with_size = ZSTD22  $@
       cmd_zstd22_with_size = { cat $(real-prereqs) | $(ZSTD) -22 --ultra; $(size_append); } > $@
 
+quiet_cmd_zstddec = ZSTDDEC $@
+      cmd_zstddec = $(ZSTD) --decompress --force --quiet -o $@ $<
+
 # ASM offsets
 # ---------------------------------------------------------------------------
 
Signature · 1 line
2.41.0
Acked-by on v2 1/2
#2 lore raw

Re: [PATCH v2 1/2] kbuild: Enable decompression for use by EXTRA_FIRMWARE

Quoted 7 lines · Kevin Martin

The build system can currently only compress files. This patch adds the functionality to decompress files. Decompression is needed for building firmware files into the kernel if those files are compressed on the filesystem. Compressed firmware files are in use by Gentoo, Fedora, Arch, and others.

Quoted 7 lines

---
Changes in v2:
- Skipped running 'cat' and now just pass the file names directly.
- Added '--quiet' since 'zstd' started printing the status of each file now that it knows the file names.

Quoted 30 lines
 scripts/Makefile.lib | 6 ++++++
 1 file changed, 6 insertions(+)
scripts/Makefile.lib +6 -1
index 1a965fe68..d043be3dc 100644
@@ -523,6 +523,9 @@ quiet_cmd_xzkern_with_size = XZKERN  $@
 quiet_cmd_xzmisc = XZMISC  $@
       cmd_xzmisc = cat $(real-prereqs) | $(XZ) --check=crc32 --lzma2=dict=1MiB > $@
 
+quiet_cmd_xzdec = XZDEC   $@
+      cmd_xzdec = $(XZ) --decompress --stdout $< > $@
+
 # ZSTD
 # ---------------------------------------------------------------------------
 # Appends the uncompressed size of the data using size_append. The .zst
@@ -548,6 +551,9 @@ quiet_cmd_zstd22 = ZSTD22  $@
 quiet_cmd_zstd22_with_size = ZSTD22  $@
       cmd_zstd22_with_size = { cat $(real-prereqs) | $(ZSTD) -22 --ultra; $(size_append); } > $@
 
+quiet_cmd_zstddec = ZSTDDEC $@
+      cmd_zstddec = $(ZSTD) --decompress --force --quiet -o $@ $<
+
 # ASM offsets
 # ---------------------------------------------------------------------------
 
--

2.41.0

Signature · 2 lines
Best Regards
Masahiro Yamada
#3 lore raw

PATCH v2 2/2 firmware_loader: Enable compressed files with EXTRA_FIRMWARE

In reply to Kevin Martin

The linux-firmware packages on Gentoo, Fedora, Arch, and others compress the firmware files. This works well with CONFIG_FW_LOADER_COMPRESS but does not work with CONFIG_EXTRA_FIRMWARE. This patch allows the build system to decompress firmware files specified by CONFIG_EXTRA_FIRMWARE. Uncompressed files are used first, then the compressed files are used.

The patch works by copying or decompressing the specified firmware files into the build directory, then compiling them in from there. I would prefer to not copy any uncompressed files, but I have not found a clean way to do that.

---
Changes in v2:
- Changed .gitignore to ignore all firmware files copied into the directory.
- Fixed a tab.

 drivers/base/firmware_loader/Kconfig            |  5 ++++-
 drivers/base/firmware_loader/builtin/.gitignore |  5 ++++-
 drivers/base/firmware_loader/builtin/Makefile   | 16 ++++++++++++----
 3 files changed, 20 insertions(+), 6 deletions(-)
drivers/base/firmware_loader/Kconfig +4 -1
index 5ca00e02f..b7a908bff 100644
@@ -76,7 +76,10 @@ config EXTRA_FIRMWARE
 	  image since it combines both GPL and non-GPL work. You should
 	  consult a lawyer of your own before distributing such an image.
 
-	  NOTE: Compressed files are not supported in EXTRA_FIRMWARE.
+	  NOTE: Compressed files are supported by EXTRA_FIRMWARE. The build
+	  system will look for uncompressed files first then fall back to
+	  searching for compressed files in a similar way to
+	  CONFIG_FW_LOADER_COMPRESS.
 
 config EXTRA_FIRMWARE_DIR
 	string "Firmware blobs root directory"
drivers/base/firmware_loader/builtin/.gitignore +4 -1
index 166f76b43..b3124ee78 100644
@@ -1,2 +1,5 @@
 # SPDX-License-Identifier: GPL-2.0
-*.gen.S
+*
+!.gitignore
+!Makefile
+!main.c
drivers/base/firmware_loader/builtin/Makefile +12 -4
index 6c067dedc..cc60eb441 100644
@@ -20,7 +20,7 @@ filechk_fwbin = \
 	echo "    .section .rodata"				;\
 	echo "    .p2align 4"					;\
 	echo "_fw_$(FWSTR)_bin:"				;\
-	echo "    .incbin \"$(fwdir)/$(FWNAME)\""		;\
+	echo "    .incbin \"$(obj)/$(FWNAME)\""			;\
 	echo "_fw_end:"						;\
 	echo "    .section .rodata.str,\"aMS\",$(PROGBITS),1"	;\
 	echo "    .p2align $(ASM_ALIGN)"			;\
@@ -36,7 +36,15 @@ $(obj)/%.gen.S: FORCE
 	$(call filechk,fwbin)
 
 # The .o files depend on the binaries directly; the .S files don't.
-$(addprefix $(obj)/, $(firmware)): $(obj)/%.gen.o: $(fwdir)/%
+$(addprefix $(obj)/, $(firmware)): $(obj)/%.gen.o: $(obj)/%
 
-targets := $(patsubst $(obj)/%,%, \
-                                $(shell find $(obj) -name \*.gen.S 2>/dev/null))
+$(obj)/% : $(fwdir)/% FORCE
+	$(call if_changed,copy)
+
+$(obj)/% : $(fwdir)/%.xz FORCE
+	$(call if_changed,xzdec)
+
+$(obj)/% : $(fwdir)/%.zst FORCE
+	$(call if_changed,zstddec)
+
+targets := $(patsubst %.gen.o, %.gen.S, $(firmware)) $(CONFIG_EXTRA_FIRMWARE)
Signature · 1 line
2.41.0
#4 lore raw

Re: [PATCH v2 2/2] firmware_loader: Enable compressed files with EXTRA_FIRMWARE

Quoted 74 lines · Kevin Martin

The linux-firmware packages on Gentoo, Fedora, Arch, and others compress the firmware files. This works well with CONFIG_FW_LOADER_COMPRESS but does not work with CONFIG_EXTRA_FIRMWARE. This patch allows the build system to decompress firmware files specified by CONFIG_EXTRA_FIRMWARE. Uncompressed files are used first, then the compressed files are used.

The patch works by copying or decompressing the specified firmware files into the build directory, then compiling them in from there. I would prefer to not copy any uncompressed files, but I have not found a clean way to do that.

---
Changes in v2:
- Changed .gitignore to ignore all firmware files copied into the directory.
- Fixed a tab.

 drivers/base/firmware_loader/Kconfig            |  5 ++++-
 drivers/base/firmware_loader/builtin/.gitignore |  5 ++++-
 drivers/base/firmware_loader/builtin/Makefile   | 16 ++++++++++++----
 3 files changed, 20 insertions(+), 6 deletions(-)
drivers/base/firmware_loader/Kconfig +4 -1
index 5ca00e02f..b7a908bff 100644
@@ -76,7 +76,10 @@ config EXTRA_FIRMWARE
          image since it combines both GPL and non-GPL work. You should
          consult a lawyer of your own before distributing such an image.
 
-         NOTE: Compressed files are not supported in EXTRA_FIRMWARE.
+         NOTE: Compressed files are supported by EXTRA_FIRMWARE. The build
+         system will look for uncompressed files first then fall back to
+         searching for compressed files in a similar way to
+         CONFIG_FW_LOADER_COMPRESS.
 
 config EXTRA_FIRMWARE_DIR
        string "Firmware blobs root directory"
drivers/base/firmware_loader/builtin/.gitignore +4 -1
index 166f76b43..b3124ee78 100644
@@ -1,2 +1,5 @@
 # SPDX-License-Identifier: GPL-2.0
-*.gen.S
+*
+!.gitignore
+!Makefile
+!main.c
drivers/base/firmware_loader/builtin/Makefile +2 -4
index 6c067dedc..cc60eb441 100644
@@ -20,7 +20,7 @@ filechk_fwbin = \
        echo "    .section .rodata"                             ;\
        echo "    .p2align 4"                                   ;\
        echo "_fw_$(FWSTR)_bin:"                                ;\
-       echo "    .incbin \"$(fwdir)/$(FWNAME)\""               ;\
+       echo "    .incbin \"$(obj)/$(FWNAME)\""                 ;\
        echo "_fw_end:"                                         ;\
        echo "    .section .rodata.str,\"aMS\",$(PROGBITS),1"   ;\
        echo "    .p2align $(ASM_ALIGN)"                        ;\
@@ -36,7 +36,15 @@ $(obj)/%.gen.S: FORCE
        $(call filechk,fwbin)
 
 # The .o files depend on the binaries directly; the .S files don't.
-$(addprefix $(obj)/, $(firmware)): $(obj)/%.gen.o: $(fwdir)/%
+$(addprefix $(obj)/, $(firmware)): $(obj)/%.gen.o: $(obj)/%
 
-targets := $(patsubst $(obj)/%,%, \
-                                $(shell find $(obj) -name \*.gen.S 2>/dev/null))

drivers/base/firmware_loader/builtin/Makefile $(obj)/%.gen.S: FORCE · line 50

Quoted 10 lines
drivers/base/firmware_loader/builtin/Makefile +10 -0
+$(obj)/% : $(fwdir)/% FORCE
+       $(call if_changed,copy)
+
+$(obj)/% : $(fwdir)/%.xz FORCE
+       $(call if_changed,xzdec)
+
+$(obj)/% : $(fwdir)/%.zst FORCE
+       $(call if_changed,zstddec)
+
+targets := $(patsubst %.gen.o, %.gen.S, $(firmware)) $(CONFIG_EXTRA_FIRMWARE)

I noticed that "make clean" leaves copied firmware files in drivers/base/firmware_loader/builtin/.

You need to clean up all files in
drivers/base/firmware_loader/builtin/
except Makefile, main.c.

The following worked for me.

(quoted diff) +0 -0

b/drivers/base/firmware_loader/builtin/Makefile
index bcac1723dc32..4d62ee9f06f6 100644

drivers/base/firmware_loader/builtin/Makefile +2 -0
@@ -48,3 +48,5 @@ $(obj)/% : $(fwdir)/%.zst FORCE
        $(call if_changed,zstddec)
 
 targets := $(patsubst %.gen.o, %.gen.S, $(firmware)) $(CONFIG_EXTRA_FIRMWARE)
+
+clean-files := $(filter-out Makefile main.c, $(patsubst $(obj)/%,%,

$(wildcard $(obj)/*)))

Signature · 2 lines
Best Regards
Masahiro Yamada
#5 lore raw

Re: [PATCH v2 2/2] firmware_loader: Enable compressed files with EXTRA_FIRMWARE

Quoted 109 lines · Masahiro Yamada
Quoted 74 lines · Kevin Martin

The linux-firmware packages on Gentoo, Fedora, Arch, and others compress the firmware files. This works well with CONFIG_FW_LOADER_COMPRESS but does not work with CONFIG_EXTRA_FIRMWARE. This patch allows the build system to decompress firmware files specified by CONFIG_EXTRA_FIRMWARE. Uncompressed files are used first, then the compressed files are used.

The patch works by copying or decompressing the specified firmware files into the build directory, then compiling them in from there. I would prefer to not copy any uncompressed files, but I have not found a clean way to do that.

---
Changes in v2:
- Changed .gitignore to ignore all firmware files copied into the directory.
- Fixed a tab.

 drivers/base/firmware_loader/Kconfig            |  5 ++++-
 drivers/base/firmware_loader/builtin/.gitignore |  5 ++++-
 drivers/base/firmware_loader/builtin/Makefile   | 16 ++++++++++++----
 3 files changed, 20 insertions(+), 6 deletions(-)
drivers/base/firmware_loader/Kconfig +4 -1
index 5ca00e02f..b7a908bff 100644
@@ -76,7 +76,10 @@ config EXTRA_FIRMWARE
          image since it combines both GPL and non-GPL work. You should
          consult a lawyer of your own before distributing such an image.
 
-         NOTE: Compressed files are not supported in EXTRA_FIRMWARE.
+         NOTE: Compressed files are supported by EXTRA_FIRMWARE. The build
+         system will look for uncompressed files first then fall back to
+         searching for compressed files in a similar way to
+         CONFIG_FW_LOADER_COMPRESS.
 
 config EXTRA_FIRMWARE_DIR
        string "Firmware blobs root directory"
drivers/base/firmware_loader/builtin/.gitignore +4 -1
index 166f76b43..b3124ee78 100644
@@ -1,2 +1,5 @@
 # SPDX-License-Identifier: GPL-2.0
-*.gen.S
+*
+!.gitignore
+!Makefile
+!main.c
drivers/base/firmware_loader/builtin/Makefile +2 -4
index 6c067dedc..cc60eb441 100644
@@ -20,7 +20,7 @@ filechk_fwbin = \
        echo "    .section .rodata"                             ;\
        echo "    .p2align 4"                                   ;\
        echo "_fw_$(FWSTR)_bin:"                                ;\
-       echo "    .incbin \"$(fwdir)/$(FWNAME)\""               ;\
+       echo "    .incbin \"$(obj)/$(FWNAME)\""                 ;\
        echo "_fw_end:"                                         ;\
        echo "    .section .rodata.str,\"aMS\",$(PROGBITS),1"   ;\
        echo "    .p2align $(ASM_ALIGN)"                        ;\
@@ -36,7 +36,15 @@ $(obj)/%.gen.S: FORCE
        $(call filechk,fwbin)
 
 # The .o files depend on the binaries directly; the .S files don't.
-$(addprefix $(obj)/, $(firmware)): $(obj)/%.gen.o: $(fwdir)/%
+$(addprefix $(obj)/, $(firmware)): $(obj)/%.gen.o: $(obj)/%
 
-targets := $(patsubst $(obj)/%,%, \
-                                $(shell find $(obj) -name \*.gen.S 2>/dev/null))

drivers/base/firmware_loader/builtin/Makefile $(obj)/%.gen.S: FORCE · line 50

Quoted 10 lines
drivers/base/firmware_loader/builtin/Makefile +10 -0
+$(obj)/% : $(fwdir)/% FORCE
+       $(call if_changed,copy)
+
+$(obj)/% : $(fwdir)/%.xz FORCE
+       $(call if_changed,xzdec)
+
+$(obj)/% : $(fwdir)/%.zst FORCE
+       $(call if_changed,zstddec)
+
+targets := $(patsubst %.gen.o, %.gen.S, $(firmware)) $(CONFIG_EXTRA_FIRMWARE)

I noticed that "make clean" leaves copied firmware files in drivers/base/firmware_loader/builtin/.

You need to clean up all files in
drivers/base/firmware_loader/builtin/
except Makefile, main.c.

The following worked for me.

drivers/base/firmware_loader/builtin/Makefile +2 -0
index bcac1723dc32..4d62ee9f06f6 100644
@@ -48,3 +48,5 @@ $(obj)/% : $(fwdir)/%.zst FORCE
        $(call if_changed,zstddec)
 
 targets := $(patsubst %.gen.o, %.gen.S, $(firmware)) $(CONFIG_EXTRA_FIRMWARE)
+
+clean-files := $(filter-out Makefile main.c, $(patsubst $(obj)/%,%,
Quoted 2 lines

$(wildcard $(obj)/*)))

This explains why the "shell find" command was used. I was attempting to generate the list from $(CONFIG_EXTRA_FIRMWARE), but that is not defined during "make clean" as I am just now learning. While I would prefer to use a whitelist instead of a blacklist, I do not know a way to accomplish that. If everyone is ok with wiping out all files other than Makefile and main.c, then I will add the wildcard to "targets" and submit a new patch revision.

Quoted 16 lines
Signature · 3 lines
-- 
Best Regards
Masahiro Yamada