Enable compressed files in EXTRA_FIRMWARE
Subsystems (3)
v2 cover Enable compressed files in EXTRA_FIRMWARE
The linux-firmware packages on Gentoo, Fedora, Arch, and others compress the firmware files. This works well with CONFIG_FW_LOADER_COMPRESS, but does not work with CONFIG_EXTRA_FIRMWARE. This patch allows the build system to decompress firmware files specified by CONFIG_EXTRA_FIRMWARE.
PATCH 1/2 adds decompression routines next to the compression routines in scripts/Makefile.lib. That patch is then used by PATCH 2/2 to decompress files before compiling them into the kernel.
The patch works by copying or decompressing the specified firmware files into the build directory, then compiling them in from there. I would prefer to not copy any uncompressed files, but I have not found a clean way to do that.
drivers/base/firmware_loader/Kconfig | 5 ++++- drivers/base/firmware_loader/builtin/.gitignore | 5 ++++- drivers/base/firmware_loader/builtin/Makefile | 16 ++++++++++++---- scripts/Makefile.lib | 6 ++++++ 4 files changed, 26 insertions(+), 6 deletions(-)
Signature · 1 line
2.41.0
PATCH v2 1/2 kbuild: Enable decompression for use by EXTRA_FIRMWARE
The build system can currently only compress files. This patch adds the functionality to decompress files. Decompression is needed for building firmware files into the kernel if those files are compressed on the filesystem. Compressed firmware files are in use by Gentoo, Fedora, Arch, and others.
---
Changes in v2:
- Skipped running 'cat' and now just pass the file names directly.
- Added '--quiet' since 'zstd' started printing the status of each file now that it knows the file names.
scripts/Makefile.lib | 6 ++++++ 1 file changed, 6 insertions(+)
scripts/Makefile.lib +6 -0
@@ -523,6 +523,9 @@ quiet_cmd_xzkern_with_size = XZKERN $@ quiet_cmd_xzmisc = XZMISC $@ cmd_xzmisc = cat $(real-prereqs) | $(XZ) --check=crc32 --lzma2=dict=1MiB > $@ +quiet_cmd_xzdec = XZDEC $@ + cmd_xzdec = $(XZ) --decompress --stdout $< > $@ + # ZSTD # --------------------------------------------------------------------------- # Appends the uncompressed size of the data using size_append. The .zst
@@ -548,6 +551,9 @@ quiet_cmd_zstd22 = ZSTD22 $@ quiet_cmd_zstd22_with_size = ZSTD22 $@ cmd_zstd22_with_size = { cat $(real-prereqs) | $(ZSTD) -22 --ultra; $(size_append); } > $@ +quiet_cmd_zstddec = ZSTDDEC $@ + cmd_zstddec = $(ZSTD) --decompress --force --quiet -o $@ $< + # ASM offsets # ---------------------------------------------------------------------------
Signature · 1 line
2.41.0
PATCH v2 2/2 firmware_loader: Enable compressed files with EXTRA_FIRMWARE
In reply to Kevin MartinThe linux-firmware packages on Gentoo, Fedora, Arch, and others compress the firmware files. This works well with CONFIG_FW_LOADER_COMPRESS but does not work with CONFIG_EXTRA_FIRMWARE. This patch allows the build system to decompress firmware files specified by CONFIG_EXTRA_FIRMWARE. Uncompressed files are used first, then the compressed files are used.
The patch works by copying or decompressing the specified firmware files into the build directory, then compiling them in from there. I would prefer to not copy any uncompressed files, but I have not found a clean way to do that.
---
Changes in v2:
- Changed .gitignore to ignore all firmware files copied into the directory.
- Fixed a tab.
drivers/base/firmware_loader/Kconfig | 5 ++++- drivers/base/firmware_loader/builtin/.gitignore | 5 ++++- drivers/base/firmware_loader/builtin/Makefile | 16 ++++++++++++---- 3 files changed, 20 insertions(+), 6 deletions(-)
drivers/base/firmware_loader/Kconfig +4 -1
@@ -76,7 +76,10 @@ config EXTRA_FIRMWARE image since it combines both GPL and non-GPL work. You should consult a lawyer of your own before distributing such an image. - NOTE: Compressed files are not supported in EXTRA_FIRMWARE. + NOTE: Compressed files are supported by EXTRA_FIRMWARE. The build + system will look for uncompressed files first then fall back to + searching for compressed files in a similar way to + CONFIG_FW_LOADER_COMPRESS. config EXTRA_FIRMWARE_DIR string "Firmware blobs root directory"
drivers/base/firmware_loader/builtin/.gitignore +4 -1
@@ -1,2 +1,5 @@ # SPDX-License-Identifier: GPL-2.0 -*.gen.S +* +!.gitignore +!Makefile +!main.c
drivers/base/firmware_loader/builtin/Makefile +12 -4
@@ -20,7 +20,7 @@ filechk_fwbin = \ echo " .section .rodata" ;\ echo " .p2align 4" ;\ echo "_fw_$(FWSTR)_bin:" ;\ - echo " .incbin \"$(fwdir)/$(FWNAME)\"" ;\ + echo " .incbin \"$(obj)/$(FWNAME)\"" ;\ echo "_fw_end:" ;\ echo " .section .rodata.str,\"aMS\",$(PROGBITS),1" ;\ echo " .p2align $(ASM_ALIGN)" ;\
@@ -36,7 +36,15 @@ $(obj)/%.gen.S: FORCE $(call filechk,fwbin) # The .o files depend on the binaries directly; the .S files don't. -$(addprefix $(obj)/, $(firmware)): $(obj)/%.gen.o: $(fwdir)/% +$(addprefix $(obj)/, $(firmware)): $(obj)/%.gen.o: $(obj)/% -targets := $(patsubst $(obj)/%,%, \ - $(shell find $(obj) -name \*.gen.S 2>/dev/null)) +$(obj)/% : $(fwdir)/% FORCE + $(call if_changed,copy) + +$(obj)/% : $(fwdir)/%.xz FORCE + $(call if_changed,xzdec) + +$(obj)/% : $(fwdir)/%.zst FORCE + $(call if_changed,zstddec) + +targets := $(patsubst %.gen.o, %.gen.S, $(firmware)) $(CONFIG_EXTRA_FIRMWARE)
Signature · 1 line
2.41.0
Re: [PATCH v2 2/2] firmware_loader: Enable compressed files with EXTRA_FIRMWARE
Quoted 74 lines · Kevin Martin
The linux-firmware packages on Gentoo, Fedora, Arch, and others compress the firmware files. This works well with
CONFIG_FW_LOADER_COMPRESSbut does not work withCONFIG_EXTRA_FIRMWARE. This patch allows the build system to decompress firmware files specified byCONFIG_EXTRA_FIRMWARE. Uncompressed files are used first, then the compressed files are used.The patch works by copying or decompressing the specified firmware files into the build directory, then compiling them in from there. I would prefer to not copy any uncompressed files, but I have not found a clean way to do that.
Signed-off-by Kevin Martin <[email protected]>---
Changes in v2:
- Changed .gitignore to ignore all firmware files copied into the directory.
- Fixed a tab.drivers/base/firmware_loader/Kconfig | 5 ++++- drivers/base/firmware_loader/builtin/.gitignore | 5 ++++- drivers/base/firmware_loader/builtin/Makefile | 16 ++++++++++++---- 3 files changed, 20 insertions(+), 6 deletions(-)drivers/base/firmware_loader/Kconfig +4
-1@@ -76,7 +76,10 @@ config EXTRA_FIRMWARE image since it combines both GPL and non-GPL work. You should consult a lawyer of your own before distributing such an image. - NOTE: Compressed files are not supported in EXTRA_FIRMWARE. + NOTE: Compressed files are supported by EXTRA_FIRMWARE. The build + system will look for uncompressed files first then fall back to + searching for compressed files in a similar way to + CONFIG_FW_LOADER_COMPRESS. config EXTRA_FIRMWARE_DIR string "Firmware blobs root directory"drivers/base/firmware_loader/builtin/.gitignore +4
-1@@ -1,2 +1,5 @@ # SPDX-License-Identifier: GPL-2.0 -*.gen.S +* +!.gitignore +!Makefile +!main.cdrivers/base/firmware_loader/builtin/Makefile +2
-4@@ -20,7 +20,7 @@ filechk_fwbin = \ echo " .section .rodata" ;\ echo " .p2align 4" ;\ echo "_fw_$(FWSTR)_bin:" ;\ - echo " .incbin \"$(fwdir)/$(FWNAME)\"" ;\ + echo " .incbin \"$(obj)/$(FWNAME)\"" ;\ echo "_fw_end:" ;\ echo " .section .rodata.str,\"aMS\",$(PROGBITS),1" ;\ echo " .p2align $(ASM_ALIGN)" ;\@@ -36,7 +36,15 @@ $(obj)/%.gen.S: FORCE $(call filechk,fwbin) # The .o files depend on the binaries directly; the .S files don't. -$(addprefix $(obj)/, $(firmware)): $(obj)/%.gen.o: $(fwdir)/% +$(addprefix $(obj)/, $(firmware)): $(obj)/%.gen.o: $(obj)/% -targets := $(patsubst $(obj)/%,%, \ - $(shell find $(obj) -name \*.gen.S 2>/dev/null))
drivers/base/firmware_loader/builtin/Makefile $(obj)/%.gen.S: FORCE · line 50
Quoted 10 lines
drivers/base/firmware_loader/builtin/Makefile +10
-0+$(obj)/% : $(fwdir)/% FORCE + $(call if_changed,copy) + +$(obj)/% : $(fwdir)/%.xz FORCE + $(call if_changed,xzdec) + +$(obj)/% : $(fwdir)/%.zst FORCE + $(call if_changed,zstddec) + +targets := $(patsubst %.gen.o, %.gen.S, $(firmware)) $(CONFIG_EXTRA_FIRMWARE)
I noticed that "make clean" leaves copied firmware files in drivers/base/firmware_loader/builtin/.
You need to clean up all files indrivers/base/firmware_loader/builtin/
except Makefile, main.c.
The following worked for me.
(quoted diff) +0 -0
b/drivers/base/firmware_loader/builtin/Makefile
index bcac1723dc32..4d62ee9f06f6 100644
drivers/base/firmware_loader/builtin/Makefile +2 -0
@@ -48,3 +48,5 @@ $(obj)/% : $(fwdir)/%.zst FORCE $(call if_changed,zstddec) targets := $(patsubst %.gen.o, %.gen.S, $(firmware)) $(CONFIG_EXTRA_FIRMWARE) + +clean-files := $(filter-out Makefile main.c, $(patsubst $(obj)/%,%,
$(wildcard $(obj)/*)))
Signature · 2 lines
Best Regards Masahiro Yamada
Re: [PATCH v2 1/2] kbuild: Enable decompression for use by EXTRA_FIRMWARE
In reply to Kevin MartinQuoted 7 lines · Kevin Martin
The build system can currently only compress files. This patch adds the functionality to decompress files. Decompression is needed for building firmware files into the kernel if those files are compressed on the filesystem. Compressed firmware files are in use by Gentoo, Fedora, Arch, and others.
Quoted 7 lines
Signed-off-by Kevin Martin <[email protected]>---
Changes in v2:
- Skipped running 'cat' and now just pass the file names directly.
- Added '--quiet' since 'zstd' started printing the status of each file now that it knows the file names.
Quoted 30 lines
scripts/Makefile.lib | 6 ++++++ 1 file changed, 6 insertions(+)scripts/Makefile.lib +6
-1@@ -523,6 +523,9 @@ quiet_cmd_xzkern_with_size = XZKERN $@ quiet_cmd_xzmisc = XZMISC $@ cmd_xzmisc = cat $(real-prereqs) | $(XZ) --check=crc32 --lzma2=dict=1MiB > $@ +quiet_cmd_xzdec = XZDEC $@ + cmd_xzdec = $(XZ) --decompress --stdout $< > $@ + # ZSTD # --------------------------------------------------------------------------- # Appends the uncompressed size of the data using size_append. The .zst@@ -548,6 +551,9 @@ quiet_cmd_zstd22 = ZSTD22 $@ quiet_cmd_zstd22_with_size = ZSTD22 $@ cmd_zstd22_with_size = { cat $(real-prereqs) | $(ZSTD) -22 --ultra; $(size_append); } > $@ +quiet_cmd_zstddec = ZSTDDEC $@ + cmd_zstddec = $(ZSTD) --decompress --force --quiet -o $@ $< + # ASM offsets # --------------------------------------------------------------------------- --2.41.0
Signature · 2 lines
Best Regards Masahiro Yamada
Re: [PATCH v2 2/2] firmware_loader: Enable compressed files with EXTRA_FIRMWARE
In reply to Masahiro YamadaQuoted 109 lines · Masahiro Yamada
Quoted 74 lines · Kevin Martin
The linux-firmware packages on Gentoo, Fedora, Arch, and others compress the firmware files. This works well with
CONFIG_FW_LOADER_COMPRESSbut does not work withCONFIG_EXTRA_FIRMWARE. This patch allows the build system to decompress firmware files specified byCONFIG_EXTRA_FIRMWARE. Uncompressed files are used first, then the compressed files are used.The patch works by copying or decompressing the specified firmware files into the build directory, then compiling them in from there. I would prefer to not copy any uncompressed files, but I have not found a clean way to do that.
Signed-off-by Kevin Martin <[email protected]>---
Changes in v2:
- Changed .gitignore to ignore all firmware files copied into the directory.
- Fixed a tab.drivers/base/firmware_loader/Kconfig | 5 ++++- drivers/base/firmware_loader/builtin/.gitignore | 5 ++++- drivers/base/firmware_loader/builtin/Makefile | 16 ++++++++++++---- 3 files changed, 20 insertions(+), 6 deletions(-)drivers/base/firmware_loader/Kconfig +4
-1@@ -76,7 +76,10 @@ config EXTRA_FIRMWARE image since it combines both GPL and non-GPL work. You should consult a lawyer of your own before distributing such an image. - NOTE: Compressed files are not supported in EXTRA_FIRMWARE. + NOTE: Compressed files are supported by EXTRA_FIRMWARE. The build + system will look for uncompressed files first then fall back to + searching for compressed files in a similar way to + CONFIG_FW_LOADER_COMPRESS. config EXTRA_FIRMWARE_DIR string "Firmware blobs root directory"drivers/base/firmware_loader/builtin/.gitignore +4
-1@@ -1,2 +1,5 @@ # SPDX-License-Identifier: GPL-2.0 -*.gen.S +* +!.gitignore +!Makefile +!main.cdrivers/base/firmware_loader/builtin/Makefile +2
-4@@ -20,7 +20,7 @@ filechk_fwbin = \ echo " .section .rodata" ;\ echo " .p2align 4" ;\ echo "_fw_$(FWSTR)_bin:" ;\ - echo " .incbin \"$(fwdir)/$(FWNAME)\"" ;\ + echo " .incbin \"$(obj)/$(FWNAME)\"" ;\ echo "_fw_end:" ;\ echo " .section .rodata.str,\"aMS\",$(PROGBITS),1" ;\ echo " .p2align $(ASM_ALIGN)" ;\@@ -36,7 +36,15 @@ $(obj)/%.gen.S: FORCE $(call filechk,fwbin) # The .o files depend on the binaries directly; the .S files don't. -$(addprefix $(obj)/, $(firmware)): $(obj)/%.gen.o: $(fwdir)/% +$(addprefix $(obj)/, $(firmware)): $(obj)/%.gen.o: $(obj)/% -targets := $(patsubst $(obj)/%,%, \ - $(shell find $(obj) -name \*.gen.S 2>/dev/null))drivers/base/firmware_loader/builtin/Makefile $(obj)/%.gen.S: FORCE · line 50
Quoted 10 lines
drivers/base/firmware_loader/builtin/Makefile +10
-0+$(obj)/% : $(fwdir)/% FORCE + $(call if_changed,copy) + +$(obj)/% : $(fwdir)/%.xz FORCE + $(call if_changed,xzdec) + +$(obj)/% : $(fwdir)/%.zst FORCE + $(call if_changed,zstddec) + +targets := $(patsubst %.gen.o, %.gen.S, $(firmware)) $(CONFIG_EXTRA_FIRMWARE)I noticed that "make clean" leaves copied firmware files in
drivers/base/firmware_loader/builtin/.You need to clean up all files in
drivers/base/firmware_loader/builtin/
exceptMakefile,main.c.The following worked for me.
drivers/base/firmware_loader/builtin/Makefile +2
-0@@ -48,3 +48,5 @@ $(obj)/% : $(fwdir)/%.zst FORCE $(call if_changed,zstddec) targets := $(patsubst %.gen.o, %.gen.S, $(firmware)) $(CONFIG_EXTRA_FIRMWARE) + +clean-files := $(filter-out Makefile main.c, $(patsubst $(obj)/%,%,
Quoted 2 lines
$(wildcard $(obj)/*)))
This explains why the "shell find" command was used. I was attempting to generate the list from $(CONFIG_EXTRA_FIRMWARE), but that is not defined during "make clean" as I am just now learning. While I would prefer to use a whitelist instead of a blacklist, I do not know a way to accomplish that. If everyone is ok with wiping out all files other than Makefile and main.c, then I will add the wildcard to "targets" and submit a new patch revision.
Quoted 16 lines
Signature · 3 lines
-- Best Regards Masahiro Yamada