package/linux-firmware: install into images for early loading support
cover package/linux-firmware: install into images for early loading support
Some drivers request their firmware very early when built into the kernel, even before the initramfs is mounted - So the only way to provide firmware for those drivers is to include them directly in the kernel with the CONFIG_EXTRA_FIRMWARE option.
To support this, let linux-firmware install its files into BINARIES_DIR in addition to TARGET_DIR, similar to how we do it for intel-microcode.
Peter Korsgaard (4):
package/linux-firmware: make target install macros accept a
destination parameter
package/linux-firmware.mk: get rid of temporary tarball for file
installation
package/linux-firmware: also install into images for early loading
support
linux: build after linux-firmware if enabled for early loading support
linux/linux.mk | 3 ++- package/linux-firmware/linux-firmware.mk | 26 ++++++++++++++++-------- 2 files changed, 19 insertions(+), 10 deletions(-)
Signature · 1 line
2.20.1
PATCH 1/4 package/linux-firmware: make target install macros accept a destination parameter
So we can reuse them for also installing into the images directory
--- package/linux-firmware/linux-firmware.mk | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-)
package/linux-firmware/linux-firmware.mk +8 -8
@@ -642,7 +642,7 @@ ifneq ($(LINUX_FIRMWARE_FILES),) define LINUX_FIRMWARE_INSTALL_FILES cd $(@D) && \ $(TAR) cf install.tar $(sort $(LINUX_FIRMWARE_FILES)) && \ - $(TAR) xf install.tar -C $(TARGET_DIR)/lib/firmware + $(TAR) xf install.tar -C $(1) endef endif
@@ -651,9 +651,9 @@ ifneq ($(LINUX_FIRMWARE_DIRS),) # into it in itself, should we re-install the package. define LINUX_FIRMWARE_INSTALL_DIRS $(foreach d,$(LINUX_FIRMWARE_DIRS), \ - rm -rf $(TARGET_DIR)/lib/firmware/$(d); \ - mkdir -p $(dir $(TARGET_DIR)/lib/firmware/$(d)); \ - cp -a $(@D)/$(d) $(TARGET_DIR)/lib/firmware/$(d)$(sep)) + rm -rf $(1)/$(d); \ + mkdir -p $(dir $(1)/$(d)); \ + cp -a $(@D)/$(d) $(1)/$(d)$(sep)) endef endif
@@ -687,7 +687,7 @@ endif # a/foo -> ../b/foo where a/ (the directory where to put the symlink) does # not yet exist. define LINUX_FIRMWARE_CREATE_SYMLINKS - cd $(TARGET_DIR)/lib/firmware/ ; \ + cd $(1) ; \ sed -r -e '/^Link: (.+) -> (.+)$$/!d; s//\1 \2/' $(@D)/WHENCE | \ while read f d; do \ if test -f $$(readlink -m $$(dirname $$f)/$$d); then \
@@ -699,9 +699,9 @@ endef define LINUX_FIRMWARE_INSTALL_TARGET_CMDS mkdir -p $(TARGET_DIR)/lib/firmware - $(LINUX_FIRMWARE_INSTALL_FILES) - $(LINUX_FIRMWARE_INSTALL_DIRS) - $(LINUX_FIRMWARE_CREATE_SYMLINKS) + $(call LINUX_FIRMWARE_INSTALL_FILES,$(TARGET_DIR)/lib/firmware) + $(call LINUX_FIRMWARE_INSTALL_DIRS,$(TARGET_DIR)/lib/firmware) + $(call LINUX_FIRMWARE_CREATE_SYMLINKS,$(TARGET_DIR)/lib/firmware) endef $(eval $(generic-package))
Signature · 1 line
2.20.1
PATCH 2/4 package/linux-firmware.mk: get rid of temporary tarball for file installation
In reply to Peter KorsgaardWith the upcoming addition of the images-install step, LINUX_FIRMWARE_INSTALl_FILES is called twice, for installing to the target and to the images directory - Which may race with each other and cause problems with the temporary install.tar tarball.
There is no specific reason to use a temporary file, we can just as well pipe the two tar invocations together, so do that instead.
--- package/linux-firmware/linux-firmware.mk | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
package/linux-firmware/linux-firmware.mk +2 -2
@@ -641,8 +641,8 @@ endif ifneq ($(LINUX_FIRMWARE_FILES),) define LINUX_FIRMWARE_INSTALL_FILES cd $(@D) && \ - $(TAR) cf install.tar $(sort $(LINUX_FIRMWARE_FILES)) && \ - $(TAR) xf install.tar -C $(1) + $(TAR) c $(sort $(LINUX_FIRMWARE_FILES)) | \ + $(TAR) x -C $(1) endef endif
Signature · 1 line
2.20.1
PATCH 2/4 package/linux-firmware.mk: get rid of temporary tarball for file installation
Peter, All,
On 2021-02-12 19:40 +0100, Peter Korsgaard spake thusly:
Quoted 7 lines
With the upcoming addition of the images-install step,
LINUX_FIRMWARE_INSTALl_FILESis called twice, for installing to the target and to the images directory - Which may race with each other and cause problems with the temporary install.tar tarball.There is no specific reason to use a temporary file, we can just as well pipe the two tar invocations together, so do that instead.
Actually, there *is* a reason why an intermediate tarball is used, see commit 21a283ffb0d (linux-firmware: fail build for missing file).
The idea is that, when we bump the version a firmware file may get removed, and we may not notice right when updating (because of the many sub-options), and that file is still isted.
If that is the case, then the first tar will bail out on that missing file, and stop right away, thus not including the followign files. But because it is on the LHS of a pipe, its return code is ignored.
Then the second tar is still happy with what it gets, because it is a proper tar archive (if the last file in a tarball was extracted successfully, there is no way to know if the tarball was truncated or not).
As a consequence, the copy is only partial, and the build does not fail.
Regards,
Yann E. MORIN.
Quoted 23 lines
Signed-off-by Peter Korsgaard <[email protected]>--- package/linux-firmware/linux-firmware.mk | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)package/linux-firmware/linux-firmware.mk +2
-2@@ -641,8 +641,8 @@ endif ifneq ($(LINUX_FIRMWARE_FILES),) define LINUX_FIRMWARE_INSTALL_FILES cd $(@D) && \ - $(TAR) cf install.tar $(sort $(LINUX_FIRMWARE_FILES)) && \ - $(TAR) xf install.tar -C $(1) + $(TAR) c $(sort $(LINUX_FIRMWARE_FILES)) | \ + $(TAR) x -C $(1) endef endifSignature · 1 line
2.20.1
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 2/4 package/linux-firmware.mk: get rid of temporary tarball for file installation
Peter, All,
On 2021-02-12 21:39 +0100, Yann E. MORIN spake thusly:
Quoted 5 lines
On 2021-02-12 19:40 +0100, Peter Korsgaard spake thusly:
Quoted 2 lines
There is no specific reason to use a temporary file, we can just as well pipe the two tar invocations together, so do that instead.
Actually, there *is* a reason why an intermediate tarball is used, see commit
21a283ffb0d(linux-firmware: fail build for missing file).
Of course, I missed providing a suggestion, so here it is, see below...
[--SNIP--]
Quoted 5 lines
Quoted 5 lines
Signed-off-by Peter Korsgaard <[email protected]>--- package/linux-firmware/linux-firmware.mk | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
Quoted 12 lines
package/linux-firmware/linux-firmware.mk endif · line 645
Quoted 12 lines
package/linux-firmware/linux-firmware.mk +2
-2@@ -641,8 +641,8 @@ endif ifneq ($(LINUX_FIRMWARE_FILES),) define LINUX_FIRMWARE_INSTALL_FILES cd $(@D) && \ - $(TAR) cf install.tar $(sort $(LINUX_FIRMWARE_FILES)) && \ - $(TAR) xf install.tar -C $(1) + $(TAR) c $(sort $(LINUX_FIRMWARE_FILES)) | \ + $(TAR) x -C $(1)
First, I think we should merge INSTALL_FILES and INSTALL_DIRS together in a single command; there is no reason why we could not put directories in a tarball...
Seconf, we should =create the archive once in _BUILD_CMDS. Then we can extract it in _INSTALL_TARGET_CMDS and _INSTALL_IMAGES_CMDS.
define LINUX_FIRMWARE_BUILD_CMDS
cd $(@D) && \
$(TAR) cf br-firmware.tar $(sort $(LINUX_FIRMWARE_FILES) $(LINUX_FIRMWARE_DIRS))
endef
define LINUX_FIRMWARE_INSTALL
mkdir -p $(1)
$(TAR) xf $(@D)/br-firmware.tar -C $(1)
sed -r -e '/^Link: (.+) -> (.+)$$/!d; s//\1 \2/' $(@D)/WHENCE |
blabla actual code to create symlinks
endef
define LINUX_FIRMWARE_INSTALL_TARGET_CMDS
$(call LINUX_FIRMWARE_INSTALL, $(TARGET_DIR)/lib/firmware)
endef
define LINUX_FIRMWARE_INSTALL_IMAGES_CMDS
$(call LINUX_FIRMWARE_INSTALL, $(IMAGES_DIR)/linux-firmware)
endefTotally untested, written directly in the mail...
Regards,
Yann E. MORIN.
Quoted 18 lines
Quoted 6 lines
package/linux-firmware/linux-firmware.mk +0
-0endef endifSignature · 1 line
2.20.1Signature · 10 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' _______________________________________________ buildroot mailing list buildroot at busybox.net http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 2/4 package/linux-firmware.mk: get rid of temporary tarball for file installation
Peter, All,
On 2021-02-12 22:07 +0100, Yann E. MORIN spake thusly:
Quoted 32 lines
On 2021-02-12 21:39 +0100, Yann E. MORIN spake thusly:
Quoted 5 lines
On 2021-02-12 19:40 +0100, Peter Korsgaard spake thusly:
Quoted 2 lines
There is no specific reason to use a temporary file, we can just as well pipe the two tar invocations together, so do that instead.
Actually, there *is* a reason why an intermediate tarball is used, see commit
21a283ffb0d(linux-firmware: fail build for missing file).Of course, I missed providing a suggestion, so here it is, see below...
[--SNIP--]
Quoted 5 lines
Quoted 5 lines
Signed-off-by Peter Korsgaard <[email protected]>--- package/linux-firmware/linux-firmware.mk | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)Quoted 12 lines
package/linux-firmware/linux-firmware.mk endif · line 645
Quoted 12 lines
package/linux-firmware/linux-firmware.mk +2
-2@@ -641,8 +641,8 @@ endif ifneq ($(LINUX_FIRMWARE_FILES),) define LINUX_FIRMWARE_INSTALL_FILES cd $(@D) && \ - $(TAR) cf install.tar $(sort $(LINUX_FIRMWARE_FILES)) && \ - $(TAR) xf install.tar -C $(1) + $(TAR) c $(sort $(LINUX_FIRMWARE_FILES)) | \ + $(TAR) x -C $(1)First, I think we should merge
INSTALL_FILESandINSTALL_DIRStogether in a single command; there is no reason why we could not put directories in a tarball...
Quoted 2 lines
Seconf, we should =create the archive once in
_BUILD_CMDS. Then we can extract it in_INSTALL_TARGET_CMDSand_INSTALL_IMAGES_CMDS.
I whipped up something quick during my morning coffee:
Lightly tested by enabling a random set of firmwares, and comparing the lists of installed files before/after are identical. Then tested by enabling all firmwares. No delta in either case: it installed the same set of files, and there was no missing/extra ones.
Regards,
Yann E. MORIN.
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 2/4 package/linux-firmware.mk: get rid of temporary tarball for file installation
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Yann" == Yann E MORIN <[email protected]> writes:
Hi,
Quoted 8 lines
I whipped up something quick during my morning coffee:
Lightly tested by enabling a random set of firmwares, and comparing the lists of installed files before/after are identical. Then tested by enabling all firmwares. No delta in either case: it installed the same set of files, and there was no missing/extra ones.
Ok, that looks good. I see that you dropped the installation macros, so how do you suggest to handle the installation to $BINARIES_DIR? Copy the entire installation logic?
Signature · 1 line
Bye, Peter Korsgaard
PATCH 2/4 package/linux-firmware.mk: get rid of temporary tarball for file installation
Peter, All,
On 2021-02-13 10:46 +0100, Peter Korsgaard spake thusly:
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Yann" == Yann E MORIN <[email protected]> writes:
I whipped up something quick during my morning coffee:
https://git.buildroot.org/~ymorin/git/buildroot/commit/?h=yem/linux-fw-rework&id=8dd299681fdf80c284680833175387ac6a428a11Ok, that looks good. I see that you dropped the installation macros, so how do you suggest to handle the installation to $BINARIES_DIR? Copy the entire installation logic?
Ah, this patch is juste a cleanup. It replaces your patch 2.
The idea was that, after this cleanup, you'd adapt your patch 1 n top of it, with something like:
diff --git a/package/linux-firmware/linux-firmware.mk b/package/linux-firmware/linux-firmware.mk
index 9bc59435ef..8defedd309 100644
--- a/package/linux-firmware/linux-firmware.mk
+++ b/package/linux-firmware/linux-firmware.mk
@@ -674,10 +674,10 @@ endif
# sure we canonicalize the pointed-to file, to cover the symlinks of the form
# a/foo -> ../b/foo where a/ (the directory where to put the symlink) does
# not yet exist.
-define LINUX_FIRMWARE_INSTALL_TARGET_CMDS
- mkdir -p $(TARGET_DIR)/lib/firmware
- $(TAR) xf $(@D)/br-firmware.tar -C $(TARGET_DIR)/lib/firmware/
- cd $(TARGET_DIR)/lib/firmware/ ; \
+define LINUX_FIRMWARE_INSTALL_FW
+ mkdir -p $(1)
+ $(TAR) xf $(@D)/br-firmware.tar -C $(1)
+ cd $(1) ; \
sed -r -e '/^Link: (.+) -> (.+)$$/!d; s//\1 \2/' $(@D)/WHENCE | \
while read f d; do \
if test -f $$(readlink -m $$(dirname $$f)/$$d); then \
@@ -687,4 +687,8 @@ define LINUX_FIRMWARE_INSTALL_TARGET_CMDS
done
endef
+define LINUX_FIRMWARE_INSTALL_TARGET_CMDS
+ $(call LINUX_FIRMWARE_INSTALL_FW, $(TARGET_DIR)/lib/firmware)
+endef
+
$(eval $(generic-package))And then your patch 3 look like:
diff --git a/package/linux-firmware/linux-firmware.mk b/package/linux-firmware/linux-firmware.mk
index 9bc59435ef..8defedd309 100644
--- a/package/linux-firmware/linux-firmware.mk
+++ b/package/linux-firmware/linux-firmware.mk
@@ -691,4 +691,8 @@ define LINUX_FIRMWARE_INSTALL_TARGET_CMDS
$(call LINUX_FIRMWARE_INSTALL_FW, $(TARGET_DIR)/lib/firmware)
endef
+define LINUX_FIRMWARE_INSTALL_IMAGES_CMDS
+ $(call LINUX_FIRMWARE_INSTALL_FW, $(BINARIES_DIR)/linux-firmware)
+endef
+
$(eval $(generic-package))And then your final patch 4.
Regards,
Yann E. MORIN.
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 2/4 package/linux-firmware.mk: get rid of temporary tarball for file installation
In reply to Yann E. MORINQuoted 23 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Yann" == Yann E MORIN <[email protected]> writes:
Peter, All,
On 2021-02-12 19:40 +0100, Peter Korsgaard spake thusly:Quoted 7 lines
With the upcoming addition of the images-install step,
LINUX_FIRMWARE_INSTALl_FILESis called twice, for installing to the target and to the images directory - Which may race with each other and cause problems with the temporary install.tar tarball.There is no specific reason to use a temporary file, we can just as well pipe the two tar invocations together, so do that instead.
Actually, there *is* a reason why an intermediate tarball is used, see commit
21a283ffb0d(linux-firmware: fail build for missing file).The idea is that, when we bump the version a firmware file may get removed, and we may not notice right when updating (because of the many sub-options), and that file is still isted.
If that is the case, then the first tar will bail out on that missing file, and stop right away, thus not including the followign files. But because it is on the LHS of a pipe, its return code is ignored.
Quoted 6 lines
Then the second tar is still happy with what it gets, because it is a proper tar archive (if the last file in a tarball was extracted successfully, there is no way to know if the tarball was truncated or not).
As a consequence, the copy is only partial, and the build does not fail.
Ok, so we need to stick a set -o pipefail before?
Signature · 1 line
Bye, Peter Korsgaard
PATCH 2/4 package/linux-firmware.mk: get rid of temporary tarball for file installation
Peter, All,
On 2021-02-12 22:37 +0100, Peter Korsgaard spake thusly:
Quoted 8 lines
Quoted 7 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Yann" == Yann E MORIN <[email protected]> writes:
Peter, All,
On 2021-02-12 19:40 +0100, Peter Korsgaard spake thusly:Quoted 2 lines
There is no specific reason to use a temporary file, we can just as well pipe the two tar invocations together, so do that instead.
Actually, there *is* a reason why an intermediate tarball is used, see commit
21a283ffb0d(linux-firmware: fail build for missing file).Ok, so we need to stick a set -o pipefail before?
That wqe'd have to use set -o pipefail is ugly ;-)
I prefer if we could cleanup the install sequence, as I suggested and proposed in another reply.
Regards,
Yann E. MORIN.
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 2/4 package/linux-firmware.mk: get rid of temporary tarball for file installation
Quoted 9 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Yann" == Yann E MORIN <[email protected]> writes:
Peter, All,
On 2021-02-12 22:37 +0100, Peter Korsgaard spake thusly:Quoted 5 lines
Quoted 5 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Yann" == Yann E MORIN <[email protected]> writes:
Peter, All,
On 2021-02-12 19:40 +0100, Peter Korsgaard spake thusly:Quoted 2 lines
There is no specific reason to use a temporary file, we can just as well pipe the two tar invocations together, so do that instead.
Quoted 8 lines
Quoted 3 lines
Quoted 2 lines
Actually, there *is* a reason why an intermediate tarball is used, see commit
21a283ffb0d(linux-firmware: fail build for missing file).Ok, so we need to stick a set -o pipefail before?
That wqe'd have to use set -o pipefail is ugly ;-)
I prefer if we could cleanup the install sequence, as I suggested and proposed in another reply.
Well, having our temporary tarball in the build directory is also kind of ugly, but ok - Fine by me.
Signature · 1 line
Bye, Peter Korsgaard
PATCH 2/4 package/linux-firmware.mk: get rid of temporary tarball for file installation
Peter, All,
On 2021-02-13 10:47 +0100, Peter Korsgaard spake thusly:
Quoted 9 lines
Quoted 9 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Yann" == Yann E MORIN <[email protected]> writes:
Peter, All,
On 2021-02-12 22:37 +0100, Peter Korsgaard spake thusly:Quoted 5 lines
Quoted 5 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Yann" == Yann E MORIN <[email protected]> writes:
Peter, All,
On 2021-02-12 19:40 +0100, Peter Korsgaard spake thusly:Quoted 2 lines
There is no specific reason to use a temporary file, we can just as well pipe the two tar invocations together, so do that instead.
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Quoted 2 lines
Actually, there *is* a reason why an intermediate tarball is used, see commit
21a283ffb0d(linux-firmware: fail build for missing file).Ok, so we need to stick a set -o pipefail before?
I prefer if we could cleanup the install sequence, as I suggested and proposed in another reply.
Well, having our temporary tarball in the build directory is also kind of ugly, but ok - Fine by me.
That will not be worse than now: we already leave the temporary tarball in the build directory...
Regards,
Yann E. MORIN.
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 2/4 package/linux-firmware.mk: get rid of temporary tarball for file installation
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Yann" == Yann E MORIN <[email protected]> writes:
Hi,
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
Ok, so we need to stick a set -o pipefail before?
I prefer if we could cleanup the install sequence, as I suggested and proposed in another reply.
Well, having our temporary tarball in the build directory is also kind of ugly, but ok - Fine by me.
That will not be worse than now: we already leave the temporary tarball in the build directory...
Yes, I know - Which was part of the reason for me wanting to change it. But oh well, it is not a big deal.
Signature · 1 line
Bye, Peter Korsgaard
PATCH 2/4 package/linux-firmware.mk: get rid of temporary tarball for file installation
Peter, All,
On 2021-02-13 11:14 +0100, Peter Korsgaard spake thusly:
Quoted 10 lines
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
Quoted 1 line
"Yann" == Yann E MORIN <[email protected]> writes:
Ok, so we need to stick a set -o pipefail before?
I prefer if we could cleanup the install sequence, as I suggested and proposed in another reply.
Well, having our temporary tarball in the build directory is also kind of ugly, but ok - Fine by me.
That will not be worse than now: we already leave the temporary tarball in the build directory...
Yes, I know - Which was part of the reason for me wanting to change it. But oh well, it is not a big deal.
Ah, sorry, I miss-read that part, then...
But as I said in the commit log of that proposal of mine, I don't see how we can do it easily without a tarball... But if you think of a better alternative, I'm all ears^Weyes. ;-)
Regards,
Yann E. MORIN.
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 2/4 package/linux-firmware.mk: get rid of temporary tarball for file installation
Quoted 12 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Yann" == Yann E MORIN <[email protected]> writes:
Peter, All,
On 2021-02-13 11:14 +0100, Peter Korsgaard spake thusly:Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
Quoted 1 line
"Yann" == Yann E MORIN <[email protected]> writes:
Ok, so we need to stick a set -o pipefail before?
I prefer if we could cleanup the install sequence, as I suggested and proposed in another reply.
Well, having our temporary tarball in the build directory is also kind of ugly, but ok - Fine by me.
That will not be worse than now: we already leave the temporary tarball in the build directory...
Quoted 8 lines
Quoted 2 lines
Yes, I know - Which was part of the reason for me wanting to change it. But oh well, it is not a big deal.
Ah, sorry, I miss-read that part, then...
But as I said in the commit log of that proposal of mine, I don't see how we can do it easily without a tarball... But if you think of a better alternative, I'm all ears^Weyes. ;-)
Well, it is either a tarball or using set -o pipefail. Anyway, the tarball is fine by me.
Signature · 1 line
Bye, Peter Korsgaard
PATCH 3/4 package/linux-firmware: also install into images for early loading support
In reply to Peter KorsgaardSome drivers request their firmware very early when built into the kernel, even before the initramfs is mounted - So the only way to provide firmware for those drivers is to include them directly in the kernel with the CONFIG_EXTRA_FIRMWARE option.
An example of this is the uC firmware for modern Intel GPUs.
Conceptually you can point CONFIG_EXTRA_FIRMWARE to ${TARGET_DIR}/lib/firmware, but then you cannot remove the firmware from the initramfs and pay the size twice (inside the kernel + in initramfs), so instead also install linux-firmware to the image dir, similar to how we do it for intel-microcode.
--- package/linux-firmware/linux-firmware.mk | 8 ++++++++ 1 file changed, 8 insertions(+)
package/linux-firmware/linux-firmware.mk +8 -0
@@ -7,6 +7,7 @@ LINUX_FIRMWARE_VERSION = 20201022 LINUX_FIRMWARE_SITE = http://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git LINUX_FIRMWARE_SITE_METHOD = git +LINUX_FIRMWARE_INSTALL_IMAGES = YES LINUX_FIRMWARE_CPE_ID_VENDOR = kernel
@@ -704,4 +705,11 @@ define LINUX_FIRMWARE_INSTALL_TARGET_CMDS $(call LINUX_FIRMWARE_CREATE_SYMLINKS,$(TARGET_DIR)/lib/firmware) endef +define LINUX_FIRMWARE_INSTALL_IMAGES_CMDS + mkdir -p $(BINARIES_DIR) + $(call LINUX_FIRMWARE_INSTALL_FILES,$(BINARIES_DIR)) + $(call LINUX_FIRMWARE_INSTALL_DIRS,$(BINARIES_DIR)) + $(call LINUX_FIRMWARE_CREATE_SYMLINKS,$(BINARIES_DIR)) +endef + $(eval $(generic-package))
Signature · 1 line
2.20.1
PATCH 4/4 linux: build after linux-firmware if enabled for early loading support
In reply to Peter KorsgaardTo support building in (a subset of) the linux-firmware files into the kernel using the CONFIG_EXTRA_FIRMWARE option, we need to ensure that the firmware files are installed before the Linux kernel is built, similar to how it is done for intel-microcode.
--- linux/linux.mk | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
linux/linux.mk +2 -1
@@ -78,7 +78,8 @@ LINUX_MAKE_ENV = \ LINUX_INSTALL_IMAGES = YES LINUX_DEPENDENCIES = host-kmod \ - $(if $(BR2_PACKAGE_INTEL_MICROCODE),intel-microcode) + $(if $(BR2_PACKAGE_INTEL_MICROCODE),intel-microcode) \ + $(if $(BR2_PACKAGE_LINUX_FIRMWARE),linux-firmware) # Starting with 4.16, the generated kconfig paser code is no longer # shipped with the kernel sources, so we need flex and bison, but
Signature · 1 line
2.20.1
PATCH 4/4 linux: build after linux-firmware if enabled for early loading support
On 2021-02-12 19:40 +0100, Peter Korsgaard spake thusly:
Quoted 15 lines
To support building in (a subset of) the linux-firmware files into the kernel using the
CONFIG_EXTRA_FIRMWAREoption, we need to ensure that the firmware files are installed before the Linux kernel is built, similar to how it is done for intel-microcode.Signed-off-by Peter Korsgaard <[email protected]>--- linux/linux.mk | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)linux/linux.mk +0
-0
linux/linux.mk LINUX_MAKE_ENV = \ · line 82
Quoted 6 lines
linux/linux.mk +2
-1@@ -78,7 +78,8 @@ LINUX_MAKE_ENV = \ LINUX_INSTALL_IMAGES = YES LINUX_DEPENDENCIES = host-kmod \ - $(if $(BR2_PACKAGE_INTEL_MICROCODE),intel-microcode) + $(if $(BR2_PACKAGE_INTEL_MICROCODE),intel-microcode) \ + $(if $(BR2_PACKAGE_LINUX_FIRMWARE),linux-firmware)
You also need to tell the kernel where to find those firmware files, with CONFIG_EXTRA_FIRMWARE_DIR, otherwise it will look in /lib/firmware.
So we'd need something like:
$(if $(BR2_PACKAGE_LINUX_FIRMWARE),
$(call KCONFIG_SET_OPT,CONFIG_EXTRA_FIRMWARE_DIR,"$${BR_BINARIES_DIR}/linux-firmware"))(note the $$ escaping and the use of a shell-level variable, like is done for CONFIG_INITRAMFS_SOURCE).
Regards,
Yann E. MORIN.
Quoted 9 lines
linux/linux.mk +0
-0# Starting with 4.16, the generated kconfig paser code is no longer # shipped with the kernel sources, so we need flex and bison, butSignature · 6 lines
2.20.1 _______________________________________________ buildroot mailing list buildroot at busybox.net http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 4/4 linux: build after linux-firmware if enabled for early loading support
Quoted 19 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Yann" == Yann E MORIN <[email protected]> writes:
On 2021-02-12 19:40 +0100, Peter Korsgaard spake thusly:
Quoted 16 lines
To support building in (a subset of) the linux-firmware files into the kernel using the
CONFIG_EXTRA_FIRMWAREoption, we need to ensure that the firmware files are installed before the Linux kernel is built, similar to how it is done for intel-microcode.Signed-off-by Peter Korsgaard <[email protected]>---
linux/linux.mk| 3 ++-1 file changed, 2 insertions(+), 1 deletion(-)linux/linux.mk +0
-0@@ -78,7 +78,8 @@ LINUX_MAKE_ENV = \
Quoted 8 lines
linux/linux.mk LINUX_MAKE_ENV = \ · line 82
Quoted 5 lines
LINUX_INSTALL_IMAGES= YESLINUX_DEPENDENCIES= host-kmod \linux/linux.mk +2
-1- $(if $(BR2_PACKAGE_INTEL_MICROCODE),intel-microcode) + $(if $(BR2_PACKAGE_INTEL_MICROCODE),intel-microcode) \ + $(if $(BR2_PACKAGE_LINUX_FIRMWARE),linux-firmware)You also need to tell the kernel where to find those firmware files, with
CONFIG_EXTRA_FIRMWARE_DIR, otherwise it will look in/lib/firmware.
Yes, but that you can take care of in linux config file/fragment. We already expose BR2_BINARIES_DIR in the environment for this, E.G. you can do:
CONFIG_EXTRA_FIRMWARE_DIR="${BR_BINARIES_DIR}" CONFIG_EXTRA_FIRMWARE="intel-ucode/06-5e-03 i915/kbl_dmc_ver1_04.bin"
(or whatever files you want to include).
Quoted 4 lines
So we'd need something like:
$(if $(BR2_PACKAGE_LINUX_FIRMWARE), $(call KCONFIG_SET_OPT,CONFIG_EXTRA_FIRMWARE_DIR,"$${BR_BINARIES_DIR}/linux-firmware"))
NIT: Only ${BR_BINARIES_DIR}, not a linux-firmware subdir, otherwise you cannot include both microcode and linux-firmware files.
Quoted 2 lines
(note the $$ escaping and the use of a shell-level variable, like is done for
CONFIG_INITRAMFS_SOURCE).
Given that the user already has to use a custom kernel config file/fragment to specify what files to include, I don't think it makes sense to override the CONFIG_EXTRA_FIRMWARE_DIR setting. We don't do it for intel-microcode either.
Signature · 1 line
Bye, Peter Korsgaard
PATCH 4/4 linux: build after linux-firmware if enabled for early loading support
On 2021-02-13 11:32 +0100, Peter Korsgaard spake thusly:
Quoted 28 lines
Quoted 19 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Yann" == Yann E MORIN <[email protected]> writes:
On 2021-02-12 19:40 +0100, Peter Korsgaard spake thusly:
Quoted 16 lines
To support building in (a subset of) the linux-firmware files into the kernel using the
CONFIG_EXTRA_FIRMWAREoption, we need to ensure that the firmware files are installed before the Linux kernel is built, similar to how it is done for intel-microcode.Signed-off-by Peter Korsgaard <[email protected]>---
linux/linux.mk| 3 ++-1 file changed, 2 insertions(+), 1 deletion(-)linux/linux.mk +0
-0@@ -78,7 +78,8 @@ LINUX_MAKE_ENV = \Quoted 8 lines
linux/linux.mk LINUX_MAKE_ENV = \ · line 82
Quoted 5 lines
LINUX_INSTALL_IMAGES= YESLINUX_DEPENDENCIES= host-kmod \linux/linux.mk +2
-1- $(if $(BR2_PACKAGE_INTEL_MICROCODE),intel-microcode) + $(if $(BR2_PACKAGE_INTEL_MICROCODE),intel-microcode) \ + $(if $(BR2_PACKAGE_LINUX_FIRMWARE),linux-firmware)You also need to tell the kernel where to find those firmware files, with
CONFIG_EXTRA_FIRMWARE_DIR, otherwise it will look in/lib/firmware.
Quoted 8 lines
Yes, but that you can take care of in linux config file/fragment. We already expose
BR2_BINARIES_DIRin the environment for this, E.G. you can do:
CONFIG_EXTRA_FIRMWARE_DIR="${BR_BINARIES_DIR}"CONFIG_EXTRA_FIRMWARE="intel-ucode/06-5e-03i915/kbl_dmc_ver1_04.bin"(or whatever files you want to include).
But don't we want to make that seamless for the user? If they don't have CONFIG_EXTRA_FIRMWARE=y, then setting CONFIG_EXTRA_FIRMWARE_DIR is a no-op.
I think it is not obvious for a user to know that they can use ${BR_BINARIES_DIR} in their kernel defconfig/fragment: this is documented nowhere. Also, with the BR_ (not BR2_) prefix, it is in that namespace which we intedned to be reserved (but that we never enforced) for internal, non-public variables...
Quoted 6 lines
Quoted 3 lines
So we'd need something like:
$(if $(BR2_PACKAGE_LINUX_FIRMWARE), $(call KCONFIG_SET_OPT,CONFIG_EXTRA_FIRMWARE_DIR,"$${BR_BINARIES_DIR}/linux-firmware"))NIT: Only ${BR_BINARIES_DIR}, not a linux-firmware subdir, otherwise you cannot include both microcode and linux-firmware files.
Meh.. That's not nice, because then it means BINARIES_DIR is clutterred by all those firmware files.
Also, intel-microcode is in a sub-directory, and will make it less easy for users to find the file(s) they will actually have to flash on their devices...
Can we move those so that we can use: CONFIG_EXTRA_FIRMWARE_DIR="$(BINARIES_DIR)/firmware/" (for example)?
Quoted 6 lines
Quoted 2 lines
(note the $$ escaping and the use of a shell-level variable, like is done for
CONFIG_INITRAMFS_SOURCE).Given that the user already has to use a custom kernel config file/fragment to specify what files to include, I don't think it makes sense to override the
CONFIG_EXTRA_FIRMWARE_DIRsetting. We don't do it for intel-microcode either.
I think we should, too.
Regards,
Yann E. MORIN.
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 4/4 linux: build after linux-firmware if enabled for early loading support
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Yann" == Yann E MORIN <[email protected]> writes:
Hi,
Quoted 4 lines
Quoted 4 lines
Yes, but that you can take care of in linux config file/fragment. We already expose
BR2_BINARIES_DIRin the environment for this, E.G. you can do:
Quoted 8 lines
Quoted 4 lines
CONFIG_EXTRA_FIRMWARE_DIR="${BR_BINARIES_DIR}"CONFIG_EXTRA_FIRMWARE="intel-ucode/06-5e-03i915/kbl_dmc_ver1_04.bin"(or whatever files you want to include).
But don't we want to make that seamless for the user? If they don't have
CONFIG_EXTRA_FIRMWARE=y, then settingCONFIG_EXTRA_FIRMWARE_DIRis a no-op.
I would argue not. There is a cost to every kernel config override we make (E.G. maintenance but also the confusion it leads to when you have a config saying A but you end up with B because Buildroot tries to be clever). It is not because you have linux-firmware enabled that there couldn't be a use case for wanting to point CONFIG_EXTRA_FIRMWARE_DIR to other firmware files (E.G. some custom firmwares in your board directory or similar).
Quoted 5 lines
I think it is not obvious for a user to know that they can use ${BR_BINARIES_DIR} in their kernel defconfig/fragment: this is documented nowhere. Also, with the
BR_(notBR2_) prefix, it is in that namespace which we intedned to be reserved (but that we never enforced) for internal, non-public variables...
True. It dates back to 2015:
Author: Thomas De Schampheleire <[email protected]> Date: Tue Feb 3 15:21:48 2015 +0100
linux: avoid unnecessary changes in defconfig for INITRAMFS_SOURCE
Quoted 5 lines
Quoted 2 lines
NIT: Only ${BR_BINARIES_DIR}, not a linux-firmware subdir, otherwise you cannot include both microcode and linux-firmware files.
Meh.. That's not nice, because then it means
BINARIES_DIRis clutterred by all those firmware files.
True, but BINARIES_DIR is already like that today, only a few packages use a subdir.
Quoted 3 lines
Also, intel-microcode is in a sub-directory, and will make it less easy for users to find the file(s) they will actually have to flash on their devices...
Yes and no. intel-microcode indeed installs its (flat tree of) files under intel-ucode, but that directly matches what the kernel looks for (intel-ucode/<file>). Likewise, the kernel looks for the i915 firmware under i915/<file>, so to match the expectations of the kernel we will need to install the linux-firmware files (E.G. the i915 directory) directly in BINARIES_DIR.
Or alternatively move the intel-microcode AND linux-firmware files into a sub directory like $BINARIES_DIR/firmware/intel-ucode/<file>), but I am not sure if that is any nicer and it is perhaps not clear exactly what belongs under firmware/ and what doesn't.
Signature · 1 line
Bye, Peter Korsgaard