Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.
Linus, please pull from the for-2.6.27 branch of:
git://git.infradead.org/users/dwmw2/firmware-2.6.git for-2.6.27
This does the following (not quite in this order):
- Provides a method for firmware files to be built into the kernel, so that drivers can be updated to use request_firmware() without forcing people to use an initrd.
- In doing that, mark fw->data as 'const', and fix a few drivers to cope.
- Update a bunch of drivers to use request_firmware(), and move their existing firmware, previously hardcoded in all kinds of interesting ways, into the firmware/ directory of the kernel tree.
- Add a FIRMWARE_IN_KERNEL config option which allows the firmware for the above (and below) drivers to continue to be built into the kernel.
- Add a simple method for representing Intel HEX records in binary form, since a bunch of drivers need that kind of thing. And a tool to convert HEX files into that binary form, for the kernel to load.
- Drop special cases from a few drivers which could either use request_firmware() or build their firmware in; they don't need to do that for themselves any more.
- Implement 'make firmware_install' target, which installs all firmware shipped with the kernel.
- Modify 'make modules_install' to install any firmware required by the modules it installs (where that firmware is shipped in the kernel tree).
There are more drivers to be converted to request_firmware() -- notably drivers/net and drivers/scsi, but it's probably best to ensure that the infrastructure is all in place and working correctly before we update those. (Although since modern drivers have been using request_firmware() for some time, I don't think there's likely to be a problem there; we'll probably clean up the stragglers in time for 2.6.28.)
David Howells (5):
Fix a const pointer usage warning in the Digigram VX soundcard driver Fix a const assignment in moxa_load_fw()
Fix a const pointer error in the Conexant cx23418 MPEG encoder driver Fix a const pointer usage warning in the Digigram pcxhr soundcard driver Fix a const pointer usage warning in the Digigram miXart soundcard driver
Full shortlog and diffstat below, although the really interesting part is probably between commits fd4f80de..59890f74 -- seen at
http://git.infradead.org/users/dwmw2/firmware-2.6.git?a=shortlog;h=59890f74
David Woodhouse (52):
libertas: treat firmware data as const
bluetooth: treat firmware data as const
cyclades: treat firmware data as const
cx25840: treat firmware data as const
myri10ge: treat firmware data as const
vx222: treat firmware data as const
riptide: treat firmware data as const
pcxhr: treat firmware data as const
vx: treat firmware data as const
ueagle-atm: treat firmware data as const
cxacru: treat firmware data as const
aic94xx: treat firmware data as const
zd1201: treat firmware data as const
rt2x00: treat firmware data as const
p54: treat firmware data as const
atmel: treat firmware data as const
irda-usb: treat firmware data as const
cxgb3: treat firmware data as const
bt8xx: treat firmware data as const
ttusb-dec: treat firmware data as const
dvb frontends: treat firmware data as const
cxusb: treat firmware data as const
gp8psk: treat firmware data as const
tuners: treat firmware data as const
maestro3: treat firmware data as const
ymfpci: treat firmware data as const
firmware: make fw->data const
firmware: allow firmware files to be built into kernel image
firmware: Add CONFIG_EXTRA_FIRMWARE option
firmware: Add CONFIG_FIRMWARE_IN_KERNEL option.
firmware: Add firmware installation to modules_install, add firmware_install ihex.h: binary representation of ihex records
ihex: request_ihex_firmware() function to load and validate firmware
ihex: add ihex2fw tool for converting HEX files into firmware images
ihex: Add support for long records to ihex2fw.c
firmware: convert korg1212 driver to use firmware loader exclusively
firmware: convert maestro3 driver to use firmware loader exclusively
firmware: convert ymfpci driver to use firmware loader exclusively
smctr: use request_firmware()
kaweth: use request_firmware()
ttusb-budget: use request_firmware()
keyspan: use request_firmware()
keyspan_pda: use request_firmware()
emi26: use request_firmware()
emi62: use request_firmware()
ti_usb_3410_5052: use request_firmware()
whiteheat: use request_firmware()
firmware: convert Ambassador ATM driver to request_firmware()
ip2: use request_firmware()
cpia2: use request_firmware()
dabusb: use request_firmware()
Merge git://git.kernel.org/.../torvalds/linux-2.6.git
Greg Kroah-Hartman (3):
dell_rbu: firmware data is const
x86 microcode: firmware data is const
isight: treat firmware data as const
Jaswinder Singh (5):
vicam: use request_firmware()
edgeport: use request_firmware()
edgeport-ti: use request_firmware()
dsp56k: use request_firmware
firmware: convert sb16_csp driver to use firmware loader exclusively
Makefile | 15 +- arch/x86/kernel/microcode.c | 4 +- drivers/atm/Makefile | 6 +- drivers/atm/ambassador.c | 140 +- drivers/atm/ambassador.h | 11 - drivers/atm/atmsar11.data | 2063 ----- drivers/atm/atmsar11.regions | 6 - drivers/atm/atmsar11.start | 4 - drivers/base/Kconfig | 64 + drivers/base/firmware_class.c | 35 +- drivers/bluetooth/bfusb.c | 3 +- drivers/bluetooth/bt3c_cs.c | 3 +- drivers/char/cyclades.c | 10 +- drivers/char/dsp56k.c | 84 +- drivers/char/ip2/fip_firm.h | 2149 ----- drivers/char/ip2/ip2base.c | 5 +- drivers/char/ip2/ip2main.c | 47 +- drivers/char/moxa.c | 4 +- drivers/firmware/dell_rbu.c | 2 +- drivers/media/common/tuners/tuner-xc2028.c | 2 +- drivers/media/common/tuners/xc5000.c | 2 +- drivers/media/dvb/dvb-usb/cxusb.c | 21 +- drivers/media/dvb/dvb-usb/gp8psk.c | 3 +- drivers/media/dvb/frontends/bcm3510.c | 5 +- drivers/media/dvb/frontends/nxt200x.c | 3 +- drivers/media/dvb/frontends/or51211.c | 4 +- drivers/media/dvb/frontends/sp8870.c | 2 +- drivers/media/dvb/frontends/sp887x.c | 2 +- drivers/media/dvb/frontends/tda10048.c | 2 +- drivers/media/dvb/frontends/tda1004x.c | 2 +- drivers/media/dvb/ttusb-budget/dvb-ttusb-budget.c | 16 +- .../media/dvb/ttusb-budget/dvb-ttusb-dspbootcode.h | 1644 ---- drivers/media/dvb/ttusb-dec/ttusb_dec.c | 2 +- drivers/media/video/bt8xx/bttv-cards.c | 3 +- drivers/media/video/cpia2/cpia2_core.c | 46 +- drivers/media/video/cpia2/cpia2patch.h | 233 - drivers/media/video/cx18/cx18-av-firmware.c | 2 +- drivers/media/video/cx25840/cx25840-firmware.c | 27 +- drivers/media/video/dabfirmware.h | 1415 ---- drivers/media/video/dabusb.c | 44 +- drivers/media/video/usbvideo/vicam.c | 317 +- drivers/net/cxgb3/common.h | 5 +- drivers/net/cxgb3/t3_hw.c | 7 +- drivers/net/irda/irda-usb.c | 12 +- drivers/net/myri10ge/myri10ge.c | 11 +- drivers/net/tokenring/smctr.c | 56 +- drivers/net/tokenring/smctr.h | 2 - drivers/net/tokenring/smctr_firmware.h | 978 --- drivers/net/usb/kaweth.c | 43 +- drivers/net/usb/kawethfw.h | 557 -- drivers/net/wireless/atmel.c | 6 +- drivers/net/wireless/libertas/if_cs.c | 2 +- drivers/net/wireless/libertas/if_sdio.c | 4 +- drivers/net/wireless/libertas/if_usb.c | 4 +- drivers/net/wireless/p54/p54usb.c | 5 +- drivers/net/wireless/rt2x00/rt2x00.h | 4 +- drivers/net/wireless/rt2x00/rt2x00pci.h | 2 +- drivers/net/wireless/rt2x00/rt61pci.c | 4 +- drivers/net/wireless/rt2x00/rt73usb.c | 6 +- drivers/net/wireless/zd1201.c | 2 +- drivers/scsi/aic94xx/aic94xx_sds.c | 12 +- drivers/scsi/aic94xx/aic94xx_sds.h | 4 +- drivers/scsi/aic94xx/aic94xx_seq.c | 7 +- drivers/usb/atm/cxacru.c | 2 +- drivers/usb/atm/ueagle-atm.c | 20 +- drivers/usb/misc/emi26.c | 96 +- drivers/usb/misc/emi26_fw.h | 5779 ------------- drivers/usb/misc/emi62.c | 130 +- drivers/usb/misc/emi62_fw_m.h | 8853 -------------------- drivers/usb/misc/emi62_fw_s.h | 8837 ------------------- drivers/usb/misc/isight_firmware.c | 2 +- drivers/usb/serial/Kconfig | 24 +- drivers/usb/serial/io_edgeport.c | 183 +- drivers/usb/serial/io_fw_boot.h | 556 -- drivers/usb/serial/io_fw_boot2.h | 546 -- drivers/usb/serial/io_fw_down.h | 1229 --- drivers/usb/serial/io_fw_down2.h | 1133 --- drivers/usb/serial/io_fw_down3.h | 847 -- drivers/usb/serial/io_ti.c | 73 +- drivers/usb/serial/keyspan.c | 78 +- drivers/usb/serial/keyspan.h | 84 - drivers/usb/serial/keyspan_mpr_fw.h | 286 - drivers/usb/serial/keyspan_pda.S | 1124 --- drivers/usb/serial/keyspan_pda.c | 51 +- drivers/usb/serial/keyspan_pda_fw.h | 99 - drivers/usb/serial/keyspan_usa18x_fw.h | 447 - drivers/usb/serial/keyspan_usa19_fw.h | 285 - drivers/usb/serial/keyspan_usa19qi_fw.h | 284 - drivers/usb/serial/keyspan_usa19qw_fw.h | 448 - drivers/usb/serial/keyspan_usa19w_fw.h | 446 - drivers/usb/serial/keyspan_usa28_fw.h | 466 - drivers/usb/serial/keyspan_usa28x_fw.h | 447 - drivers/usb/serial/keyspan_usa28xa_fw.h | 449 - drivers/usb/serial/keyspan_usa28xb_fw.h | 448 - drivers/usb/serial/keyspan_usa49w_fw.h | 464 - drivers/usb/serial/keyspan_usa49wlc_fw.h | 476 -- drivers/usb/serial/ti_fw_3410.h | 885 -- drivers/usb/serial/ti_fw_5052.h | 885 -- drivers/usb/serial/ti_usb_3410_5052.c | 40 +- drivers/usb/serial/whiteheat.c | 77 +- drivers/usb/serial/whiteheat_fw.h | 1669 ---- drivers/usb/serial/xircom_pgs.S | 1192 --- drivers/usb/serial/xircom_pgs_fw.h | 103 - firmware/Makefile | 177 + firmware/WHENCE | 341 + firmware/atmsar11.HEX | 204 + firmware/cpia2/stv0672_vp4.bin.ihex | 73 + firmware/dabusb/bitstream.bin.ihex | 761 ++ firmware/dabusb/firmware.HEX | 649 ++ firmware/dsp56k/bootstrap.asm | 98 + firmware/dsp56k/bootstrap.bin.ihex | 26 + firmware/edgeport/boot.H16 | 29 + firmware/edgeport/boot2.H16 | 28 + firmware/edgeport/down.H16 | 29 + firmware/edgeport/down2.H16 | 29 + firmware/edgeport/down3.bin.ihex | 815 ++ firmware/emi26/bitstream.HEX | 4391 ++++++++++ firmware/emi26/firmware.HEX | 1261 +++ firmware/emi26/loader.HEX | 116 + firmware/emi62/bitstream.HEX | 4372 ++++++++++ firmware/emi62/loader.HEX | 107 + firmware/emi62/midi.HEX | 1266 +++ firmware/emi62/spdif.HEX | 1257 +++ firmware/ess/maestro3_assp_kernel.fw.ihex | 120 + firmware/ess/maestro3_assp_minisrc.fw.ihex | 51 + firmware/ihex2fw.c | 268 + firmware/intelliport2.bin.ihex | 2147 +++++ firmware/kaweth/new_code.bin.ihex | 206 + firmware/kaweth/new_code_fix.bin.ihex | 40 + firmware/kaweth/trigger_code.bin.ihex | 13 + firmware/kaweth/trigger_code_fix.bin.ihex | 3 + firmware/keyspan/mpr.HEX | 104 + firmware/keyspan/usa18x.HEX | 141 + firmware/keyspan/usa19.HEX | 101 + firmware/keyspan/usa19qi.HEX | 101 + firmware/keyspan/usa19qw.HEX | 142 + firmware/keyspan/usa19w.HEX | 141 + firmware/keyspan/usa28.HEX | 148 + firmware/keyspan/usa28x.HEX | 141 + firmware/keyspan/usa28xa.HEX | 141 + firmware/keyspan/usa28xb.HEX | 142 + firmware/keyspan/usa49w.HEX | 145 + firmware/keyspan/usa49wlc.HEX | 153 + firmware/keyspan_pda/keyspan_pda.HEX | 83 + firmware/keyspan_pda/keyspan_pda.S | 1124 +++ firmware/keyspan_pda/xircom_pgs.HEX | 87 + firmware/keyspan_pda/xircom_pgs.S | 1192 +++ firmware/korg/k1212.dsp.ihex | 987 +++ firmware/sb16/alaw_main.csp.ihex | 87 + firmware/sb16/ima_adpcm_capture.csp.ihex | 121 + firmware/sb16/ima_adpcm_init.csp.ihex | 70 + firmware/sb16/ima_adpcm_playback.csp.ihex | 122 + firmware/sb16/mulaw_main.csp.ihex | 84 + firmware/ti_3410.fw.ihex | 862 ++ firmware/ti_5052.fw.ihex | 862 ++ firmware/tr_smctr.bin.ihex | 477 ++ firmware/ttusb-budget/dspbootcode.bin.ihex | 820 ++ firmware/vicam/firmware.H16 | 7 + firmware/whiteheat.HEX | 1097 +++ firmware/whiteheat_loader.HEX | 314 + firmware/whiteheat_loader_debug.HEX | 403 + firmware/yamaha/ds1_ctrl.fw.ihex | 769 ++ firmware/yamaha/ds1_dsp.fw.ihex | 9 + firmware/yamaha/ds1e_ctrl.fw.ihex | 769 ++ include/asm-generic/vmlinux.lds.h | 7 + include/linux/firmware.h | 23 +- include/linux/ihex.h | 74 + scripts/Makefile.fwinst | 45 + sound/drivers/vx/vx_core.c | 4 +- sound/isa/Kconfig | 9 - sound/isa/sb/sb16_csp.c | 22 - sound/isa/sb/sb16_csp_codecs.h | 949 --- sound/pci/Kconfig | 30 - sound/pci/korg1212/korg1212-firmware.h | 987 --- sound/pci/korg1212/korg1212.c | 18 - sound/pci/maestro3.c | 186 +- sound/pci/mixart/mixart_hwdep.c | 2 +- sound/pci/pcxhr/pcxhr_core.c | 4 +- sound/pci/pcxhr/pcxhr_hwdep.c | 2 +- sound/pci/riptide/riptide.c | 10 +- sound/pci/vx222/vx222_ops.c | 2 +- sound/pci/ymfpci/ymfpci_image.h | 1565 ---- sound/pci/ymfpci/ymfpci_main.c | 74 +- 183 files changed, 31965 insertions(+), 52599 deletions(-)
Signature · 1 line
dwmw2 @linux.intel.com
Quoted 2 lines · On Mon, 14 Jul 2008 16:23:26 -0700 David Woodhouse <[email protected]> wrote:
Linus, please pull from the for-2.6.27 branch of:
git://git.infradead.org/users/dwmw2/firmware-2.6.git for-2.6.27
The firmware flamewars seem to have subsided lately. Is everyone happy with this now?
Quoted 8 lines · Andrew Morton
Quoted 2 lines · On Mon, 14 Jul 2008 16:23:26 -0700 David Woodhouse <[email protected]> wrote:
Linus, please pull from the for-2.6.27 branch of:
git://git.infradead.org/users/dwmw2/firmware-2.6.git for-2.6.27The firmware flamewars seem to have subsided lately. Is everyone happy with this now?
I've deliberately left drivers/net out of this pull request... all _other_ maintainers seemed quite happy to see their drivers updated to use request_firmware().
Let's see how it pans out. I don't expect any problems -- after all, modern drivers have been using request_firmware() for a _long_ time now, and the distributions cope quite happily with it already. We're ready to ship the firmware in a separate package in Fedora as soon as Linus pulls from my tree. Once the dust has settled and it's all working fine, we can look at converting drivers/net and drivers/scsi too.
Signature · 1 line
dwmw2
Quoted 14 lines · David Woodhouse
Quoted 8 lines · Andrew Morton
Quoted 2 lines · On Mon, 14 Jul 2008 16:23:26 -0700 David Woodhouse <[email protected]> wrote:
Linus, please pull from the for-2.6.27 branch of:
git://git.infradead.org/users/dwmw2/firmware-2.6.git for-2.6.27The firmware flamewars seem to have subsided lately. Is everyone happy with this now?
I've deliberately left
drivers/netout of this pull request... all _other_ maintainers seemed quite happy to see their drivers updated to userequest_firmware().
Quoted 6 lines
Let's see how it pans out. I don't expect any problems -- after all, modern drivers have been using
request_firmware()for a _long_ time now, and the distributions cope quite happily with it already. We're ready to ship the firmware in a separate package in Fedora as soon as Linus pulls from my tree. Once the dust has settled and it's all working fine, we can look at convertingdrivers/netanddrivers/scsitoo.
so after this patchset, if i put qllogic fw in firmware/, that fw will work or not...
YH
Quoted 2 lines · Yinghai Lu
so after this patchset, if i put qllogic fw in
firmware/, that fw will work or not...
The qlogic driver already uses request_firmware(), like most modern drivers do. If you want to include that firmware in your kernel image (assuming it's in /lib/firmware already), do something like this:
CONFIG_EXTRA_FIRMWARE="ql2500_fw.bin" CONFIG_EXTRA_FIRMWARE_DIR=/lib/firmware
Then it'll find that firmware file and pull it in to your kernel, and you can set CONFIG_SCSI_QLA_FC=y and expect it to work. Select the right firmware for your device, of course.
Signature · 1 line
dwmw2
Quoted 8 lines · David Woodhouse
Quoted 2 lines · Yinghai Lu
so after this patchset, if i put qllogic fw in
firmware/, that fw will work or not...The qlogic driver already uses
request_firmware(), like most modern drivers do. If you want to include that firmware in your kernel image (assuming it's in/lib/firmwarealready), do something like this:
Quoted 7 lines
CONFIG_EXTRA_FIRMWARE="ql2500_fw.bin" CONFIG_EXTRA_FIRMWARE_DIR=/lib/firmwareThen it'll find that firmware file and pull it in to your kernel, and you can set
CONFIG_SCSI_QLA_FC=yand expect it to work. Select the right firmware for your device, of course.
CONFIG_EXTRA_FIRMWARE="ql2100_fw.bin ql2300_fw.bin ql2400_fw.bin ql2200_fw.bin ql2322_fw.bin ql2500_fw.bin" CONFIG_EXTRA_FIRMWARE_DIR="firmware"
make[1]: *** No rule to make target include/config/builtin/firmware/dir.h', needed by firmware/ql2100_fw.bin.gen.S'. Stop.
make: *** [firmware] Error 2
YH
Quoted 4 lines · Yinghai Lu
make[1]: *** No rule to make target
include/config/builtin/firmware/dir.h', needed byfirmware/ql2100_fw.bin.gen.S'. Stop.
make: *** [firmware] Error 2
Oh, pants. Sorry, I renamed the CONFIG_BUILTIN_FIRMWARE option to CONFIG_EXTRA_FIRMWARE some time back, but all my own test builds had the include/config/builtin/firmware/dir.h file still lying around so I didn't notice that I'd forgotten to change that manual dependency. And evidently nobody using linux-next actually tried to include arbitrary _other_ firmware into their tree. This patch should fix it...
(Why _do_ we have stray files left in include/config/ when config options go away, btw? Shouldn't they get deleted?)
----
Subject: firmware: Correct dependency on CONFIG_EXTRA_FIRMWARE_DIR
When CONFIG_EXTRA_FIRMWARE_DIR gets changed, the filename in the .S file (which uses .incbin to include the binary) needs to change. When we renamed the BUILTIN_FIRMWARE_DIR option to EXTRA_FIRMWARE_DIR, we forgot to update the manual dependency in firmware/Makefile, so it was depending on a non-existent file in include/config/
firmware/Makefile +1 -1
@@ -146,7 +146,7 @@ $(patsubst %,$(obj)/%.gen.S, $(fw-shipped-y)): %: $(wordsize_deps) \ | $(objtree)/$$(dir %) $(call cmd,fwbin,$(patsubst %.gen.S,%,$@)) $(patsubst %,$(obj)/%.gen.S, $(fw-external-y)): %: $(wordsize_deps) \ - include/config/builtin/firmware/dir.h | $(objtree)/$$(dir %) + include/config/extra/firmware/dir.h | $(objtree)/$$(dir %) $(call cmd,fwbin,$(fwabs)/$(patsubst $(obj)/%.gen.S,%,$@)) # The .o files depend on the binaries directly; the .S files don't.
Signature · 1 line
dwmw2
Quoted 43 lines · David Woodhouse
Quoted 4 lines · Yinghai Lu
make[1]: *** No rule to make target
include/config/builtin/firmware/dir.h', needed byfirmware/ql2100_fw.bin.gen.S'. Stop.
make: *** [firmware] Error 2Oh, pants. Sorry, I renamed the
CONFIG_BUILTIN_FIRMWAREoption toCONFIG_EXTRA_FIRMWAREsome time back, but all my own test builds had theinclude/config/builtin/firmware/dir.hfile still lying around so I didn't notice that I'd forgotten to change that manual dependency. And evidently nobody using linux-next actually tried to include arbitrary _other_ firmware into their tree. This patch should fix it...(Why _do_ we have stray files left in
include/config/when config options go away, btw? Shouldn't they get deleted?)----
Subject: firmware: Correct dependency onCONFIG_EXTRA_FIRMWARE_DIRWhen
CONFIG_EXTRA_FIRMWARE_DIRgets changed, the filename in the .S file (which uses .incbin to include the binary) needs to change. When we renamed theBUILTIN_FIRMWARE_DIRoption toEXTRA_FIRMWARE_DIR, we forgot to update the manual dependency infirmware/Makefile, so it was depending on a non-existent file ininclude/config/Signed-off-by David Woodhouse <[email protected]>firmware/Makefile +1
-2@@ -146,7 +146,7 @@ $(patsubst %,$(obj)/%.gen.S, $(fw-shipped-y)): %: $(wordsize_deps) \ | $(objtree)/$$(dir %) $(call cmd,fwbin,$(patsubst %.gen.S,%,$@)) $(patsubst %,$(obj)/%.gen.S, $(fw-external-y)): %: $(wordsize_deps) \ - include/config/builtin/firmware/dir.h | $(objtree)/$$(dir %) + include/config/extra/firmware/dir.h | $(objtree)/$$(dir %) $(call cmd,fwbin,$(fwabs)/$(patsubst $(obj)/%.gen.S,%,$@)) # The .o files depend on the binaries directly; the .S files don't. --
Quoted 2 lines
dwmw2
thanks. it works
MK_FW firmware/ql2100_fw.bin.gen.S MK_FW firmware/ql2300_fw.bin.gen.S MK_FW firmware/ql2400_fw.bin.gen.S MK_FW firmware/ql2200_fw.bin.gen.S MK_FW firmware/ql2322_fw.bin.gen.S MK_FW firmware/ql2500_fw.bin.gen.S AS firmware/ql2100_fw.bin.gen.o AS firmware/ql2300_fw.bin.gen.o AS firmware/ql2400_fw.bin.gen.o AS firmware/ql2200_fw.bin.gen.o AS firmware/ql2322_fw.bin.gen.o AS firmware/ql2500_fw.bin.gen.o LD firmware/built-in.o
calling qla2x00_module_init+0x0/0x131
QLogic Fibre Channel HBA Driver: 8.02.01-k4
ACPI: PCI Interrupt 0000:0c:02.0[A] -> GSI 57 (level, low) -> IRQ 57 qla2xxx 0000:0c:02.0: Found an ISP2312, irq 57, iobase 0xffffc2002d00c000 qla2xxx 0000:0c:02.0: Configuring PCI space...
qla2xxx 0000:0c:02.0: Configure NVRAM parameters...
qla2xxx 0000:0c:02.0: Verifying loaded RISC code...
firmware: using built-in firmware ql2300_fw.bin
qla2xxx 0000:0c:02.0: Allocated (412 KB) for firmware dump...
YH
Quoted 6 lines · David Woodhouse
Quoted 4 lines · Yinghai Lu
make[1]: *** No rule to make target
include/config/builtin/firmware/dir.h', needed byfirmware/ql2100_fw.bin.gen.S'. Stop.
make: *** [firmware] Error 2
Quoted 6 lines
Oh, pants. Sorry, I renamed the
CONFIG_BUILTIN_FIRMWAREoption toCONFIG_EXTRA_FIRMWAREsome time back, but all my own test builds had theinclude/config/builtin/firmware/dir.hfile still lying around so I didn't notice that I'd forgotten to change that manual dependency. And evidently nobody using linux-next actually tried to include arbitrary _other_ firmware into their tree. This patch should fix it...
here's another bugreport, i get this build failure with latest -git (which includes your fix above):
CALL scripts/checksyscalls.sh
CHK include/linux/compile.h
make[1]: *** No rule to make target firmware/tigon/tg3.bin', needed by firmware/tigon/tg3.bin.gen.o'. Stop.
make: *** [firmware] Error 2
with:
http://redhat.com/~mingo/misc/config-Tue_Jul_15_09_38_58_CEST_2008.bad
such configs worked before.
(i'd exclude TIGON3 from my qa builds but i've got tg3 hardware.)
Ingo
Quoted 8 lines · David Woodhouse
Quoted 2 lines · Yinghai Lu
so after this patchset, if i put qllogic fw in
firmware/, that fw will work or not...The qlogic driver already uses
request_firmware(), like most modern drivers do. If you want to include that firmware in your kernel image (assuming it's in/lib/firmwarealready), do something like this:
Quoted 6 lines
CONFIG_EXTRA_FIRMWARE="ql2500_fw.bin" CONFIG_EXTRA_FIRMWARE_DIR=/lib/firmwareThen it'll find that firmware file and pull it in to your kernel, and you can set
CONFIG_SCSI_QLA_FC=yand expect it to work. Select the right firmware for your device, of course.
MK_FW firmware/tigon/tg3.bin.gen.S
make[1]: *** No rule to make target firmware/tigon/tg3.bin', needed by firmware/tigon/tg3.bin.gen.o'. Stop.
make: *** [firmware] Error 2
YH
Quoted 4 lines · Yinghai Lu
MK_FWfirmware/tigon/tg3.bin.gen.S
make[1]: *** No rule to make targetfirmware/tigon/tg3.bin', needed byfirmware/tigon/tg3.bin.gen.o'. Stop.
make: *** [firmware] Error 2
I didn't include the tg3 patch in what I just asked Linus to pull. Are you working from linux-next?
Signature · 1 line
dwmw2
Quoted 8 lines · David Woodhouse
Quoted 4 lines · Yinghai Lu
MK_FWfirmware/tigon/tg3.bin.gen.S
make[1]: *** No rule to make targetfirmware/tigon/tg3.bin', needed byfirmware/tigon/tg3.bin.gen.o'. Stop.
make: *** [firmware] Error 2I didn't include the tg3 patch in what I just asked Linus to pull. Are you working from linux-next?
linus tree.
YH
Quoted 4 lines · Yinghai Lu
MK_FWfirmware/tigon/tg3.bin.gen.S
make[1]: *** No rule to make targetfirmware/tigon/tg3.bin', needed byfirmware/tigon/tg3.bin.gen.o'. Stop.
make: *** [firmware] Error 2
I apologise; a line intended for the tg3 conversion accidentally slipped into firmware/Makefile along with the dabusb patch, when I cherry-picked from the old tree to omit the drivers/net patches. Again, it didn't show up in my own testing because I already _had_ the file in question.
(Sorry, Linus. This and the previous fix are now in the for-2.6.27 branch of the same repo as before:
git://git.infradead.org/users/dwmw2/firmware-2.6.git for-2.6.27
)
----
Subject: Fix accidental reference to tg3 firmware
We're not updating the tg3 driver to use request_firmware() yet, but a reference to its firmware accidentally slipped in as part of commit c4667746 ("dabusb: use request_firmware()"). Remove it again.
firmware/Makefile +0 -2
@@ -34,8 +34,6 @@ fw-shipped-$(CONFIG_SND_SB16_CSP) += sb16/mulaw_main.csp sb16/alaw_main.csp \ sb16/ima_adpcm_capture.csp fw-shipped-$(CONFIG_SND_YMFPCI) += yamaha/ds1_ctrl.fw yamaha/ds1_dsp.fw \ yamaha/ds1e_ctrl.fw -fw-shipped-$(CONFIG_TIGON3) += tigon/tg3.bin tigon/tg3_tso.bin \ - tigon/tg3_tso5.bin fw-shipped-$(CONFIG_USB_DABUSB) += dabusb/firmware.fw dabusb/bitstream.bin fw-shipped-$(CONFIG_USB_EMI26) += emi26/loader.fw emi26/firmware.fw \ emi26/bitstream.fw
Signature · 1 line
dwmw2
Quoted 7 lines · David Woodhouse
Quoted 6 lines · Andrew Morton
Quoted 2 lines · On Mon, 14 Jul 2008 16:23:26 -0700 David Woodhouse <[email protected]> wrote:
Linus, please pull from the for-2.6.27 branch of:
git://git.infradead.org/users/dwmw2/firmware-2.6.git for-2.6.27
Quoted 6 lines
Quoted 2 lines
The firmware flamewars seem to have subsided lately. Is everyone happy with this now?
I've deliberately left
drivers/netout of this pull request... all _other_ maintainers seemed quite happy to see their drivers updated to userequest_firmware().
Have you fixed the issue with FORCE in line 117 of scripts/Makefile.host?
Rafael
Quoted 2 lines · Rafael J. Wysocki
Have you fixed the issue with FORCE in line 117 of
scripts/Makefile.host?
I need to hassle Sam about that.
Sam, why do we have FORCE on the rules for $(hostprogs-y)? It forces the ihex2fw tool to be rebuilt, which in turn forces all .HEX->.fw conversions to happen again, which means that 'make modules_install' wants to write to the source tree. That's a problem in some strange situations like Rafael's, where he builds on another host and exports the tree read-only by NFS.
Can we do this? ...
scripts/Makefile.host +1 -1
@@ -114,7 +114,7 @@ hostcxx_flags = -Wp,-MD,$(depfile) $(__hostcxx_flags) quiet_cmd_host-csingle = HOSTCC $@ cmd_host-csingle = $(HOSTCC) $(hostc_flags) -o $@ $< \ $(HOST_LOADLIBES) $(HOSTLOADLIBES_$(@F)) -$(host-csingle): $(obj)/%: $(src)/%.c FORCE +$(host-csingle): $(obj)/%: $(src)/%.c $(call if_changed_dep,host-csingle) # Link an executable based on list of .o files, all plain c
Signature · 1 line
dwmw2
Quoted 8 lines · David Woodhouse
Sam, why do we have FORCE on the rules for $(hostprogs-y)? It forces the ihex2fw tool to be rebuilt, which in turn forces all .HEX->.fw conversions to happen again, which means that 'make
modules_install' wants to write to the source tree. That's a problem in some strange situations like Rafael's, where he builds on another host and exports the tree read-only by NFS.Can we do this? ...
Alternatively, Rafael, does this do it for you?
firmware/Makefile +2 -2
@@ -150,11 +150,11 @@ $(obj)/%: $(obj)/%.ihex | $(objtree)/$(obj)/$$(dir %) # is actually meaningful, because the firmware has to be loaded in a certain # order rather than as a single binary blob. Thus, we convert them into our # more compact binary representation of ihex records (<linux/ihex.h>) -$(obj)/%.fw: $(obj)/%.HEX $(obj)/ihex2fw | $(objtree)/$(obj)/$$(dir %) +$(obj)/%.fw: $(obj)/%.HEX | $(obj)/ihex2fw $(objtree)/$(obj)/$$(dir %) $(call cmd,ihex2fw) # .H16 is our own modified form of Intel HEX, with 16-bit length for records. -$(obj)/%.fw: $(obj)/%.H16 $(obj)/ihex2fw | $(objtree)/$(obj)/$$(dir %) +$(obj)/%.fw: $(obj)/%.H16 | $(obj)/ihex2fw $(objtree)/$(obj)/$$(dir %) $(call cmd,h16tofw) $(firmware-dirs):
Signature · 1 line
dwmw2
Quoted 11 lines · David Woodhouse
Quoted 8 lines · David Woodhouse
Sam, why do we have FORCE on the rules for $(hostprogs-y)? It forces the ihex2fw tool to be rebuilt, which in turn forces all .HEX->.fw conversions to happen again, which means that 'make
modules_install' wants to write to the source tree. That's a problem in some strange situations like Rafael's, where he builds on another host and exports the tree read-only by NFS.Can we do this? ...
Alternatively, Rafael, does this do it for you?
No, it doesn't.
Quoted 7 lines
firmware/Makefile +0
-0@@ -150,11 +150,11 @@ $(obj)/%: $(obj)/%.ihex | $(objtree)/$(obj)/$$(dir %) # is actually meaningful, because the firmware has to be loaded in a certain # order rather than as a single binary blob. Thus, we convert them into our # more compact binary representation of ihex records (<linux/ihex.h>)
firmware/Makefile $(obj)/%: $(obj)/%.ihex | $(objtree)/$(obj)/$$(dir %) · line 160
Quoted 10 lines
firmware/Makefile +2
-2-$(obj)/%.fw: $(obj)/%.HEX $(obj)/ihex2fw | $(objtree)/$(obj)/$$(dir %) +$(obj)/%.fw: $(obj)/%.HEX | $(obj)/ihex2fw $(objtree)/$(obj)/$$(dir %) $(call cmd,ihex2fw) # .H16 is our own modified form of Intel HEX, with 16-bit length for records. -$(obj)/%.fw: $(obj)/%.H16 $(obj)/ihex2fw | $(objtree)/$(obj)/$$(dir %) +$(obj)/%.fw: $(obj)/%.H16 | $(obj)/ihex2fw $(objtree)/$(obj)/$$(dir %) $(call cmd,h16tofw) $(firmware-dirs):
Rafael
Quoted 1 line · David Woodhouse
I need to hassle Sam about that.
I think I have it. Rafael, does _this_ fix it for you?
scripts/Makefile.fwinst +12 -0
@@ -43,3 +43,15 @@ __fw_install: $(installed-fw) __fw_modinst: $(mod-fw) FORCE: + +# Read all saved command lines and dependencies for the $(targets) we +# may be building using $(if_changed{,_dep}). As an optimization, we +# don't need to read them if the target does not exist; we will rebuild +# anyway in that case. + +targets := $(wildcard $(sort $(targets))) +cmd_files := $(wildcard $(foreach f,$(targets),$(dir $(f)).$(notdir $(f)).cmd)) + +ifneq ($(cmd_files),) + include $(cmd_files) +endif
Signature · 1 line
dwmw2
Quoted 4 lines · David Woodhouse
Quoted 1 line · David Woodhouse
I need to hassle Sam about that.
I think I have it. Rafael, does _this_ fix it for you?
No, but the error message seems to be different:
HOSTCC firmware/ihex2fw/home/rafael/src/linux-next/firmware/ihex2fw.c:268: fatal error: opening dependency file firmware/.ihex2fw.d: Read-only file system compilation terminated.
Removing the FORCE from line 117 of scripts/Makefile.host makes it work.
Of course, I still have to run "make ... firmware_install" locally on the server.
Quoted 21 lines
scripts/Makefile.fwinst +12
-0@@ -43,3 +43,15 @@ __fw_install: $(installed-fw) __fw_modinst: $(mod-fw) FORCE: + +# Read all saved command lines and dependencies for the $(targets) we +# may be building using $(if_changed{,_dep}). As an optimization, we +# don't need to read them if the target does not exist; we will rebuild +# anyway in that case. + +targets := $(wildcard $(sort $(targets))) +cmd_files := $(wildcard $(foreach f,$(targets),$(dir $(f)).$(notdir $(f)).cmd)) + +ifneq ($(cmd_files),) + include $(cmd_files) +endif
Quoted 11 lines · Rafael J. Wysocki
Quoted 4 lines · David Woodhouse
Quoted 1 line · David Woodhouse
I need to hassle Sam about that.
I think I have it. Rafael, does _this_ fix it for you?
No, but the error message seems to be different:
HOSTCC
firmware/ihex2fw/home/rafael/src/linux-next/firmware/ihex2fw.c:268: fatal error: opening dependency filefirmware/.ihex2fw.d: Read-only file system compilation terminated.
Hm, can you do that again with V=2?
Quoted 4 lines
Removing the FORCE from line 117 of
scripts/Makefile.hostmakes it work.Of course, I still have to run "make ...
firmware_install" locally on the server.
Yeah, that part is easy enough to fix, once the unwanted rebuilding is fixed.
Signature · 1 line
dwmw2
Quoted 7 lines · David Woodhouse
Quoted 6 lines · Rafael J. Wysocki
Quoted 4 lines · David Woodhouse
Quoted 1 line · David Woodhouse
I need to hassle Sam about that.
I think I have it. Rafael, does _this_ fix it for you?
Quoted 7 lines
Quoted 5 lines
No, but the error message seems to be different:
HOSTCC
firmware/ihex2fw/home/rafael/src/linux-next/firmware/ihex2fw.c:268: fatal error: opening dependency filefirmware/.ihex2fw.d: Read-only file system compilation terminated.Hm, can you do that again with V=2?
HOSTCC firmware/ihex2fw - due to: FORCE/home/rafael/src/linux-next/firmware/ihex2fw.c:268: fatal error: opening dependency file firmware/.ihex2fw.d: Read-only file system compilation terminated.
Thanks,
Rafael
Quoted 5 lines · Rafael J. Wysocki
HOSTCC
firmware/ihex2fw- due to: FORCE/home/rafael/src/linux-next/firmware/ihex2fw.c:268: fatal error:
opening dependency filefirmware/.ihex2fw.d: Read-only file system compilation terminated.
I'm very confused. I can do this...
make INSTALL_FW_PATH=/tmp/foo firmware_install su otheruser
touch firmware/.ihex2fw.cmd (gives EPERM as expected) make INSTALL_FW_PATH=/tmp/foo2 firmware_install
And it _works_ for me, after I've made that change in scripts/Makefile.fwinst. But I don't have .ihex2fw.d -- I have .ihex2fw.cmd. What is in your .ihex2fw.d? And does it work if you include _that_ from firmware/Makefile?
/me wonders where Sam is hiding...
Signature · 1 line
dwmw2
Quoted 9 lines · David Woodhouse
Quoted 5 lines · Rafael J. Wysocki
HOSTCC
firmware/ihex2fw- due to: FORCE/home/rafael/src/linux-next/firmware/ihex2fw.c:268: fatal error:
opening dependency filefirmware/.ihex2fw.d: Read-only file system compilation terminated.I'm very confused. I can do this...
Quoted 8 lines
make INSTALL_FW_PATH=/tmp/foo
firmware_installsu otheruser
touchfirmware/.ihex2fw.cmd(gives EPERM as expected) make INSTALL_FW_PATH=/tmp/foo2firmware_installAnd it _works_ for me, after I've made that change in
scripts/Makefile.fwinst. But I don't have .ihex2fw.d -- I have .ihex2fw.cmd. What is in your .ihex2fw.d?
There's no '.ihex2fw.d' under 'firmware' in my O= path.
I just do 'make modules_install' on the NFS client, after I have run 'make O=... ... firmware_install' on the server and it prints the above error message.
Should I add anything to the 'make modules_install' command line?
Rafael
Quoted 4 lines · Rafael J. Wysocki
HOSTCC
firmware/ihex2fw- due to: FORCE/home/rafael/src/linux-next/firmware/ihex2fw.c:268: fatal error:
opening dependency filefirmware/.ihex2fw.d: Read-only file system compilation terminated.
I believe you can't reproduce this with Linus' current HEAD? Please poke me if it happens again.
Signature · 1 line
dwmw2
Quoted 10 lines · On Mon, 14 Jul 2008 16:41:19 -0700 Andrew Morton <[email protected]>…
Quoted 3 lines · On Mon, 14 Jul 2008 16:23:26 -0700 David Woodhouse <[email protected]> wrote:
Linus, please pull from the for-2.6.27 branch of:
git://git.infradead.org/users/dwmw2/firmware-2.6.gitfor-2.6.27
The firmware flamewars seem to have subsided lately. Is everyone happy with this now?
this seems to have left the contentious pieces out....
Signature · 3 lines
If you want to reach me at my work email, use [email protected] For development, discussion and tips for power savings, visit http://www.lesswatts.org
Quoted 10 lines · Arjan van de Ven
Quoted 7 lines · On Mon, 14 Jul 2008 16:41:19 -0700 Andrew Morton <[email protected]>…
Quoted 3 lines · On Mon, 14 Jul 2008 16:23:26 -0700 David Woodhouse <[email protected]> wrote:
Linus, please pull from the for-2.6.27 branch of:
git://git.infradead.org/users/dwmw2/firmware-2.6.gitfor-2.6.27
Quoted 5 lines
Quoted 3 lines
The firmware flamewars seem to have subsided lately. Is everyone happy with this now?
this seems to have left the contentious pieces out....
almost, the one thing that this could have done would be to offer an option to bundle the firmware with the module. currently it offers the option to load the firmware at the same time as the module, not _quite_ the same thing.
there was also the issue that was raised about how to handle firmware during suspend/resume. I don't remember seeing a happy solution to that one.
David Lang
Quoted 3 lines · On Mon, 14 Jul 2008, [email protected] wrote:
there was also the issue that was raised about how to handle firmware during suspend/resume. I don't remember seeing a happy solution to that one.
I don't understand why this even keeps coming up.
The only reliable way to handle it is to load the firmware before the suspend happens - preferably by simply never unloading it in the first place.
Anything else is fraught with problems.
Stuff that expects to eject itself anyway(*) over a suspend event can obviously decide to throw it away and require a reload at resume time (by virtue of simply re-discovering the whole device!).
Everybody else should just keep the firmware in memory in the driver.
Does it waste some ram? Sure. Tough.
Linus
(*) eg some random USB device that really _really_ doesn't care and isn't expected to be involved in any filesystem access - say a webcam.
Quoted 13 lines · Linus Torvalds
Quoted 3 lines · On Mon, 14 Jul 2008, [email protected] wrote:
there was also the issue that was raised about how to handle firmware during suspend/resume. I don't remember seeing a happy solution to that one.
I don't understand why this even keeps coming up.
The only reliable way to handle it is to load the firmware before the suspend happens - preferably by simply never unloading it in the first place.
Anything else is fraught with problems.
Quoted 7 lines
Stuff that expects to eject itself
anyway(*)over a suspend event can obviously decide to throw it away and require a reload at resume time (by virtue of simply re-discovering the whole device!).Everybody else should just keep the firmware in memory in the driver.
Does it waste some ram? Sure. Tough.
I agree with this, but the proponents of the seperate firmware are listing the fact that the firmware doesn't tie up ram as one of the big reasons for making the change.
David Lang
Quoted 6 lines · On Mon, 14 Jul 2008, [email protected] wrote:
Quoted 2 lines
Does it waste some ram? Sure. Tough.
I agree with this, but the proponents of the seperate firmware are listing the fact that the firmware doesn't tie up ram as one of the big reasons for making the change.
That's a totally bogus argument.
The fact is, if you build it into your module, you'll waste at _least_ as much ram as if you just load it once at module load time.
So there is no actual valid reason to object to "request_firmware()".
I don't know why people get confused about this. I suspect that people kind of expect that since they need to reload the firmware when resuming the device, they should also do the "request_firmware()" at resume time.
Maybe it's worth explicitly documenting that request_firmware()/release() should be done as a module init/exit (or a device detect-eject) time option. Quite frankly, I think the current firmware docs are actually actively misleading, because they link the request_firmware() with the copying to device: quoting from Documentation/firmware_class/README:
High level behavior (driver code): ================================== if(request_firmware(&fw_entry, $FIRMWARE, device) == 0) copy_fw_to_device(fw_entry->data, fw_entry->size); release(fw_entry);
and that is a fundamentally broken world-view.
The logic _should_ be that the firmware is requested at module init or device discovery, and the release is done at module exit or device eject.
The "request_firmware()" should absolutely *not* be mentally tied to "copy_fw_to_device" at all. They are very distinct issues, and in fact must be totally separate for any driver that supports hotplug.
Linus
Quoted 8 lines · Linus Torvalds
Quoted 6 lines · On Mon, 14 Jul 2008, [email protected] wrote:
Quoted 2 lines
Does it waste some ram? Sure. Tough.
I agree with this, but the proponents of the seperate firmware are listing the fact that the firmware doesn't tie up ram as one of the big reasons for making the change.
Quoted 6 lines
That's a totally bogus argument.
The fact is, if you build it into your module, you'll waste at _least_ as much ram as if you just load it once at module load time.
So there is no actual valid reason to object to "
request_firmware()".
you misunderstood me. the people pushing request_firmware() are doing so on the basis that they won't have to use kernel ram to hold the firmware. the people pushing for having the option of building the firmware into the module are acknowleding that this may use a little more ram, but they see it as being more reliable.
Quoted 3 lines
I don't know why people get confused about this. I suspect that people kind of expect that since they need to reload the firmware when resuming the device, they should also do the "
request_firmware()" at resume time.
according to David W they would, becouse the driver would not keep the firmware in kernel memory after it's initialized, so if you need to reinitialize the hardware after a resume you would need to do request_firmware() at resume time (the option of doing request_firmware() just before going to sleep has been floated as an option to avoid this problem)
Quoted 6 lines
Maybe it's worth explicitly documenting that request_firmware()/release() should be done as a module init/exit (or a device detect-eject) time option. Quite frankly, I think the current firmware docs are actually actively misleading, because they link the
request_firmware()with the copying to device: quoting fromDocumentation/firmware_class/README:
Quoted 8 lines
High level behavior (driver code): ================================== if(request_firmware(&fw_entry, $FIRMWARE, device) == 0) copy_fw_to_device(fw_entry->data, fw_entry->size); release(fw_entry);and that is a fundamentally broken world-view.
but that is what David W has been telling everyone is one of the big benifits of using request_firmware(), that the memory gets released once the device is initialized. (as always, it's possible I have misunderstood the argument, but I can't see any other way that it would save memory)
Quoted 2 lines
The logic _should_ be that the firmware is requested at module init or device discovery, and the release is done at module exit or device eject.
this would go a long way to satisfying the people who have been objecting to the request_firmware() push. It would solve the various reinitialization problems (at resume, and at other times for some drivers).
the only thing it doesn't satisfy is the people who don't want to be forced to have seperate firmware files out on the filesystem.
As I understand the arguments, the reason to not force the firmeware files to be seperate is that it adds complexity and introduces the possibility that the firmware may be incompatible with the driver, forcing the creation of some sort of userspace/filesystem versioning.
the reson to force the firmware files to be seperate is that it avoids using kernel memory to hold them and it allows for the firmware to be removed from the kernel tree entirely and shipped seperately (apparently Fedora is eager to do the latter for perceived licensing reasons)
Quoted 3 lines
The "
request_firmware()" should absolutely *not* be mentally tied to "copy_fw_to_device" at all. They are very distinct issues, and in fact must be totally separate for any driver that supports hotplug.
in the last flamewar it seemed pretty clear that the expectation was that a hotplug event would trigger a new request_firmware() call out to userspace.
David Lang
Quoted 9 lines · On Mon, 14 Jul 2008, [email protected] wrote:
Quoted 2 lines · Linus Torvalds
That's a totally bogus argument.
you misunderstood me. the people pushing
request_firmware()are doing so on the basis that they won't have to use kernel ram to hold the firmware. the people pushing for having the option of building the firmware into the module are acknowleding that this may use a little more ram, but they see it as being more reliable.
I'm just saying that it's a totally bogus argument to claim that it takes less memory - Either way.
As to reliability, I don't buy that, especially with a generic interface, and with a way to link the thing in-kernel anyway. Using common infrastructure is going to be more reliable.
Quoted 6 lines
Quoted 3 lines
I don't know why people get confused about this. I suspect that people kind of expect that since they need to reload the firmware when resuming the device, they should also do the "
request_firmware()" at resume time.according to David W they would, becouse the driver would not keep the firmware in kernel memory after it's initialized
And if so, David W is a total moran, and shouldn't have been doing this.
The fact is, there _are_ good arguments for request_firmware(), but they have nothing what-so-ever to do with memory use or anything like that.
The argument for request_firmware() is that it's a good _single_ interface to the whole firmware issue, allowing us to split up the driver from the firmware without every driver having to do some hack of its own.
Not memory use.
So next time you see somebody arguing about memory use (either way), just slap them, and tell them Linus told you to.
Linus
Quoted 26 lines · Linus Torvalds
Quoted 9 lines · On Mon, 14 Jul 2008, [email protected] wrote:
Quoted 2 lines · Linus Torvalds
That's a totally bogus argument.
you misunderstood me. the people pushing
request_firmware()are doing so on the basis that they won't have to use kernel ram to hold the firmware. the people pushing for having the option of building the firmware into the module are acknowleding that this may use a little more ram, but they see it as being more reliable.I'm just saying that it's a totally bogus argument to claim that it takes less memory - Either way.
As to reliability, I don't buy that, especially with a generic interface, and with a way to link the thing in-kernel anyway. Using common infrastructure is going to be more reliable.
Quoted 6 lines
Quoted 3 lines
I don't know why people get confused about this. I suspect that people kind of expect that since they need to reload the firmware when resuming the device, they should also do the "
request_firmware()" at resume time.according to David W they would, becouse the driver would not keep the firmware in kernel memory after it's initialized
Quoted 8 lines
And if so, David W is a total moran, and shouldn't have been doing this.
The fact is, there _are_ good arguments for
request_firmware(), but they have nothing what-so-ever to do with memory use or anything like that.The argument for
request_firmware()is that it's a good _single_ interface to the whole firmware issue, allowing us to split up the driver from the firmware without every driver having to do some hack of its own.
I don't think anyone objects to request_firmware() as a common interface. they object to forcing the firmware to be seperate.
in the monolithic kernel case the people pushing this bowed to preasure and improved request_firmware() so that the firmware could be embedded in the kernel image as opposed to the initial approach of requireing an initramfs.
Jeff and David M are arguing that it should be possible to also embed the firmware into the module .ko file. If that option was available I don't think there would be opposition to having the one interface. the lack of that option is causing them to object and point out cases where it's not as good as previously.
David Lang
Quoted 7 lines
Not memory use.
So next time you see somebody arguing about memory use (either way), just slap them, and tell them Linus told you to.
Linus
Quoted 10 lines · Linus Torvalds
Quoted 7 lines · On Mon, 14 Jul 2008, [email protected] wrote:
Quoted 1 line · Linus Torvalds
That's a totally bogus argument.
you misunderstood me. the people pushing
request_firmware()are doing so on the basis that they won't have to use kernel ram to hold the firmware. the people pushing for having the option of building the firmware into the module are acknowleding that this may use a little more ram, but they see it as being more reliable.I'm just saying that it's a totally bogus argument to claim that it takes less memory - Either way.
Agreed.
Quoted 1 line
As to reliability, I don't buy that, especially with a generic interface, and with a way to link the thing in-kernel anyway. Using common infrastructure is going to be more reliable.
A more complex, multi-file inter-dependent system is more reliable than a single-file driver with built-in firmware, doing the same thing?
Come _on_.
There are good arguments for request_firmware(), but that ain't it.
Quoted 1 line
The argument for
request_firmware()is that it's a good _single_ interface to the whole firmware issue, allowing us to split up the driver from the firmware without every driver having to do some hack of its own.
Agreed.
Jeff
Quoted 3 lines · Jeff Garzik
A more complex, multi-file inter-dependent system is more reliable than a single-file driver with built-in firmware, doing the same thing?
I'm not interested in your made-up arguments.
Please just build everything statically. Don't use modules. They are clearly too complex for you, adn everybody is happy.
Linus
Quoted 6 lines · Linus Torvalds
Quoted 2 lines · Jeff Garzik
A more complex, multi-file inter-dependent system is more reliable than a single-file driver with built-in firmware, doing the same thing?
I'm not interested in your made-up arguments.
It's a simple question... you claimed A was more reliable than B, even though A is a more complex system than B.
I'm curious how that works, especially given that you have claimed the _exact opposite_ in years past, by pointing out how firmware separation could mean no-boot.
Quoted 1 line
Please just build everything statically. Don't use modules. They are clearly too complex for you, adn everybody is happy.
Can't you make a simple, objective, technical comparison between two systems?
firmware-outside-driver has more utility and flexibility, but more things can go wrong, like driver present/firmware absent case that happens every day with today's wireless drivers.
firmware-inside-driver guarantees without _any_ doubt that the firmware is present, if the driver is guarantee. There is no guarantee more solid than that.
Jeff
Quoted 4 lines · Jeff Garzik
I'm curious how that works, especially given that you have claimed the
_exactopposite_in years past, by pointing out how firmware separation could mean no-boot.
Umm. Firmware separation _could_ mean non-boot.
DavidW &co fixed it. You can now build it in.
You seem to ignore that things change, and _have_ changed.
Linus
Quoted 11 lines · Linus Torvalds
Quoted 3 lines · Jeff Garzik
I'm curious how that works, especially given that you have claimed the
_exactopposite_in years past, by pointing out how firmware separation could mean no-boot.Umm. Firmware separation _could_ mean non-boot.
DavidW &co fixed it. You can now build it in.
You seem to ignore that things change, and _have_ changed.
Already addressed -- you and David seem to be ignoring the common case where you may update the driver but not the kernel, making that feature far less useful than it appears.
Inability to update the kernel image (vmlinuz) is the case with just about every enterprise distro out there, including ones to be based on >= 2.6.27.
Driver disks and our own in-tree build-module-out-of-tree support are built precisely for this purpose: to build a new driver against an existing kernel, because so often you cannot update the core vmlinuz image.
(rather, you _can_ update vmlinuz, but that would fail on the next automated update, so its not a realistic option)
Jeff
Quoted 4 lines · Jeff Garzik
Already addressed -- you and David seem to be ignoring the common case where you may update the driver but not the kernel, making that feature far less useful than it appears.
This is the "I'm too stupid to live" argument. I've already answered that. What's so hard with "make firmware_install"?
Linus
Quoted 3 lines · Linus Torvalds
This is the "I'm too stupid to live" argument. I've already answered that. What's so hard with "make
firmware_install"?
Btw, that is by no means the only way to handle it either. As I said in other emails - about trying to find _productive_ questions to answer - maybe we could also just install it in the modules directory or some similar per-kernel location, and just teach request_firmware() to load binary blobs directly.
Then - horror of horrors - we could even make "make modules_install" do things like that automatically for you with some Makefile rule trickery.
See? There are _productive_ things you can do or ask others to do.
But singing the same "I'm so stupid that I cannot install the firmware" song over and over again doesn't make me think you are worth listening to.
Linus
Quoted 11 lines · Linus Torvalds
Quoted 1 line · Linus Torvalds
This is the "I'm too stupid to live" argument. I've already answered that. What's so hard with "make
firmware_install"?Btw, that is by no means the only way to handle it either. As I said in other emails - about trying to find _productive_ questions to answer - maybe we could also just install it in the modules directory or some similar per-kernel location, and just teach
request_firmware()to load binary blobs directly.Then - horror of horrors - we could even make "make
modules_install" do things like that automatically for you with someMakefilerule trickery.See? There are _productive_ things you can do or ask others to do.
But singing the same "I'm so stupid that I cannot install the firmware" song over and over again doesn't make me think you are worth listening to.
Can't you see that changing build processes takes time? Fixing driver disk creation tools take time? Validating that each driver is packaged with its correct firmware takes time?
With each driver converted, it is entirely possible that all users are completely unaware that it even had a firmware, much less that it now needs 'make firmware_install' to continue working.
If one is completely unaware of a driver's firmware need, (a) you don't know you need to install it and (b) you don't find out you need it until you boot and the driver fails to work.
I am trying to work to minimize (a) and (b) here, because as deployed, there are large chances for failure here, across a random selection of drivers.
Jeff
P.S. AFAIK, 'make modules_install' installs firmware, unless that change was not pushed upstream.
Quoted 4 lines · Jeff Garzik
Can't you see that changing build processes takes time? Fixing driver disk creation tools take time? Validating that each driver is packaged with its correct firmware takes time?
Umm. Can't you see that if we don't start doing it, it will never get done AT ALL?
Yes, this will take time.
But that's not an argument for not merging it. Quite the reverse.
Linus
Quoted 9 lines · Linus Torvalds
Quoted 3 lines · Jeff Garzik
Can't you see that changing build processes takes time? Fixing driver disk creation tools take time? Validating that each driver is packaged with its correct firmware takes time?
Umm. Can't you see that if we don't start doing it, it will never get done AT ALL?
Yes, this will take time.
Which, by implication, means that userland is not -already- prepared.
Which means that if you don't have a recent userland of a mainstream distro, the result is non-working drivers.
Which is the type of regression I thought we did not want.
Quoted 1 line
But that's not an argument for not merging it. Quite the reverse.
This is not an either/or proposition, and that is what you are missing.
The change should go in _while at the same time_ not creating regressions.
The change is not the problem.
That the change exists without the ability to recreate similar outputs (i.e. firmware in module) is the problem.
Jeff
Quoted 7 lines · Linus Torvalds
Quoted 3 lines · Jeff Garzik
Already addressed -- you and David seem to be ignoring the common case where you may update the driver but not the kernel, making that feature far less useful than it appears.
This is the "I'm too stupid to live" argument. I've already answered that. What's so hard with "make
firmware_install"?
So hard that several kernel hackers wound up with non-working drivers (read the pre-push bug reports).
Which is why 'make modules_install' installs the firmware, or at least it did before David W pushed upstream.
That was a key build process regression fix (thanks David!).
Sorry Linus, the end change is a good idea, but you cannot hand-wave away breakage just by calling people stupid.
Jeff
Quoted 3 lines · Jeff Garzik
Which is why 'make
modules_install' installs the firmware, or at least it did before David W pushed upstream.
So you're literally just about making this be "make modules_install" rather than "make firmware_install"
Ok. Are you going to be happy if "make modules_install" just copies the firmware files of the affected modules too?
Or are you just whining because you want to whine, and will make up some new argument then?
Linus
Quoted 11 lines · Linus Torvalds
Quoted 3 lines · Jeff Garzik
Which is why 'make
modules_install' installs the firmware, or at least it did before David W pushed upstream.So you're literally just about making this be "make
modules_install" rather than "makefirmware_install"Ok. Are you going to be happy if "make
modules_install" just copies the firmware files of the affected modules too?
I did that already. I'm not _entirely_ sure it was the right thing to do, but it does help to reduce surprises when people don't realise that a driver they're using now has separate firmware.
I don't believe it was the result of a constructive request from Jeff; I think someone else suggested it in response to his whining. It may even have been my idea, but I don't recall.
This is the current situation:
'make modules_install' will install the firmware required by the modules it installs. It'll install that to $(INSTALL_MOD_PATH)/lib/firmware/ unless you override that by setting $(INSTALL_FW_PATH). So you can put it into a kernel-specific directory if you want to.
'make firmware_install' will install _all_ available firmware files to /lib/firmware (or $(INSTALL_FW_PATH)), regardless of architecture or configuration. That's what Fedora is using for its 'kernel-firmware' package, which is arch-independent and required by the various kernel binary packages.
Signature · 1 line
dwmw2
Hi David,
Quoted 24 lines
Quoted 8 lines
Quoted 2 lines
Which is why 'make
modules_install' installs the firmware, or at least it did before David W pushed upstream.So you're literally just about making this be "make
modules_install" rather than "makefirmware_install"Ok. Are you going to be happy if "make
modules_install" just copies the firmware files of the affected modules too?I did that already. I'm not _entirely_ sure it was the right thing to do, but it does help to reduce surprises when people don't realise that a driver they're using now has separate firmware.
I don't believe it was the result of a constructive request from Jeff; I think someone else suggested it in response to his whining. It may even have been my idea, but I don't recall.
This is the current situation:
'make
modules_install' will install the firmware required by the modules it installs. It'll install that to $(INSTALL_MOD_PATH)/lib/firmware/ unless you override that by setting $(INSTALL_FW_PATH). So you can put it into a kernel-specific directory if you want to.
Quoted 5 lines
'make
firmware_install' will install _all_ available firmware files to/lib/firmware(or $(INSTALL_FW_PATH)), regardless of architecture or configuration. That's what Fedora is using for its 'kernel-firmware' package, which is arch-independent and required by the various kernel binary packages.
I really think we should use /lib/firmware/uname -r/. I do see the point here that I don't wanna overwrite existing firmware from other installed kernels. Especially if modules_install will install the firmware files.
So in case of non-Ubuntu distros we have to push a fix to udev, but that is not a big deal. It should be a one-line change if I am not mistaken.
Regards
Marcel
Quoted 7 lines · Marcel Holtmann
I really think we should use
/lib/firmware/uname -r/. I do see the point here that I don't wanna overwrite existing firmware from other installed kernels. Especially ifmodules_installwill install the firmware files.So in case of non-Ubuntu distros we have to push a fix to udev, but that is not a big deal. It should be a one-line change if I am not mistaken.
My concern with that is that even though it's a one-line change, it's a one-line change which people don't already _have_. We really do need to remain compatible with existing setups.
For _now_, I think it's much better to leave it in /lib/firmware where existing upstream udev will find it. Let's change udev ASAP and then we can talk about when it makes sense to change the default setting of $(INSTALL_FW_PATH) to match.
I'm not too worried about overwriting existing firmware
in /lib/firmware. We very rarely change firmware anyway, when we _do_ change it it's almost always compatible rather than having ABI changes. On the rare occasions that remain, the incompatibly changed firmware really ought to have a new filename so that older drivers don't pick it up when they want the old one.
It's not a _complete_ non-problem, but it's fairly close. And the incompatibility with existing upstream udev would be much more of an issue, I think.
Signature · 1 line
dwmw2
Hi David,
Quoted 8 lines
Quoted 7 lines
I really think we should use
/lib/firmware/uname -r/. I do see the point here that I don't wanna overwrite existing firmware from other installed kernels. Especially ifmodules_installwill install the firmware files.So in case of non-Ubuntu distros we have to push a fix to udev, but that is not a big deal. It should be a one-line change if I am not mistaken.
Quoted 8 lines
My concern with that is that even though it's a one-line change, it's a one-line change which people don't already _have_. We really do need to remain compatible with existing setups.
For _now_, I think it's much better to leave it in
/lib/firmwarewhere existing upstream udev will find it. Let's change udev ASAP and then we can talk about when it makes sense to change the default setting of $(INSTALL_FW_PATH) to match.
fair enough. What about keeping symlink in /lib/firmware, but actually doing the install in a versioned directory?
Quoted 6 lines
I'm not too worried about overwriting existing firmware
in/lib/firmware. We very rarely change firmware anyway, when we _do_ change it it's almost always compatible rather than having ABI changes. On the rare occasions that remain, the incompatibly changed firmware really ought to have a new filename so that older drivers don't pick it up when they want the old one.
Actually in some case we do. I think about the case where I have an two years old kernel that came with the distro and would need that firmware to make the driver work. However now we have the new firmware and it falls over.
Breaking a distro install is worse then breaking a new kernel. At least this is how I see it. I do like to have the distro setup as fallback or rescue when I mess up.
Regards
Marcel
Quoted 6 lines · David Woodhouse
Quoted 5 lines · Marcel Holtmann
I really think we should use
/lib/firmware/uname -r/. I do see the point here that I don't wanna overwrite existing firmware from other installed kernels. Especially ifmodules_installwill install the firmware files.
Quoted 6 lines
Quoted 2 lines
So in case of non-Ubuntu distros we have to push a fix to udev, but that is not a big deal. It should be a one-line change if I am not mistaken.
My concern with that is that even though it's a one-line change, it's a one-line change which people don't already _have_. We really do need to remain compatible with existing setups.
when I read this I had to double check who the mail was from. this sounds like exactly the statement that Jeff has been making in his arguments to allow for the firmware-in-the-modules approach.
David Lang
Quoted 8 lines · Linus Torvalds
Quoted 2 lines · Jeff Garzik
Which is why 'make
modules_install' installs the firmware, or at least it did before David W pushed upstream.So you're literally just about making this be "make
modules_install" rather than "makefirmware_install"Ok. Are you going to be happy if "make
modules_install" just copies the firmware files of the affected modules too?
Re-read the above, and what you snipped. My statement was
* upstream _already_ does this (installs firmware via modules_install)
* it is a good thing, because it means existing build scripts and processes will Do The Right Thing and produce a working driver
* thus eliminating the most common problem likely to be encountered
That does not eliminate all problems, because build processes still have hardcoded assumptions about outputs.
Restoring firmware-in-module ability closes the last logical hole, last fundamental flaw in the scheme, by permitting distros to reproduce precisely what they built in 2.6.26.
All of the regressions examples I am citing are cured by firmware-in-module, because they are all examples of problems that occur when you remove the ability to reproduce the same functional pieces as 2.6.26.
Jeff
Quoted 3 lines · Jeff Garzik
That does not eliminate all problems, because build processes still have hardcoded assumptions about outputs.
Tough. They need to get fixed.
What's the downside of fixing them, again?
Linus
Quoted 8 lines · Linus Torvalds
Quoted 2 lines · Jeff Garzik
That does not eliminate all problems, because build processes still have hardcoded assumptions about outputs.
Tough. They need to get fixed.
What's the downside of fixing them, again?
Yes, they need fixing.
But without firmware-in-module this would be akin to adding kernel module support in version 1.0.1, and forcing all distros to fix their stuff immediately in order to upgrade from version 1.0.0.
Permitting firmware-in-module means you do not force each distro, build script, and user immediately into the new regime, with the associated long list of breakage examples.
It's just sad because it feels "not the Linux way".
The way this was rolled out just feels "harder", far less friendly on the distros -- our customers -- than most other transitions. Not as hard as vmlinuz->kmodule and IDE->libata transitions, I grant, but still
not the way we usually do things.
Nobody was ever objecting to request_firmware(). It was always about the removal of firmware-in-module and associated forced breakage on distros and build processes. The fallout of _that_ is the rich source of breakage examples I've been pulling from.
Jeff
Quoted 4 lines · Jeff Garzik
But without firmware-in-module this would be akin to adding kernel module support in version 1.0.1, and forcing all distros to fix their stuff immediately in order to upgrade from version 1.0.0.
You're just making that up. The distro's can fix it long before they use the kernel.
And the users we can just fix by making request_firmware() look up the files itself, if it doesn't already.
You're just making up your bogus arguments.
Linus
Quoted 4 lines · Jeff Garzik
All of the regressions examples I am citing are cured by firmware-in-module, because they are all examples of problems that occur when you remove the ability to reproduce the same functional pieces as 2.6.26.
Jeff,
I think you've said this before, but let me try to help you
cut to the chase. You are willing to *implement* firmware-in-the-module for request_firmware(), but you want a commitment that David Woodhouse and Linus will accept such a patch before you go off and write it. Is that right?
If so, may I suggest you start implementing, instead of
sending e-mails? For all the time people have spent arguing about it, maybe it could have been implemented already.
Once it has been implemented, do you have any further
objections aka ideas about how request_firmware() could be improved?
- Ted
(who is hoping we can guide
this thread onto slightly more productive avenues)
Quoted 11 lines · Theodore Tso
Quoted 1 line · Jeff Garzik
All of the regressions examples I am citing are cured by firmware-in-module, because they are all examples of problems that occur when you remove the ability to reproduce the same functional pieces as 2.6.26.
Jeff,
I think you've said this before, but let me try to help you
cut to the chase. You are willing to *implement* firmware-in-the-module forrequest_firmware(), but you want a commitment that David Woodhouse and Linus will accept such a patch before you go off and write it. Is that right?
Quoted 3 lines
If so, may I suggest you start implementing, instead of
sending e-mails? For all the time people have spent arguing about it, maybe it could have been implemented already.
Already started, in fact, since Linus said he would not reject it out of hands.
Obviously that is not acceptance; I know as well as any that acceptance does not occur until the moment of upstream merge.
Quoted 2 lines
Once it has been implemented, do you have any further
objections aka ideas about howrequest_firmware()could be improved?
Further objections? None major.
The two other minor problems I feel need solving, but are not related to breakage or regressions, are:
1) firmware should be able to live alongside the driver. We don't need to start growing firmware/net alongside drivers/net, firmware/scsi alongside drivers/scsi, firmware/char alongside drivers/char, etc.
2) firmware should be able to be shipped in final format (binary), rather than ihex.
I feel strongly that 2.6.27 should not be released without firmware-in-module, for all the reasons mentioned in these threads. It's a major regression, IMO, without firmware-in-module ability.
The other stuff (#1, #2 in list above) is small potatoes.
Jeff
Quoted 3 lines · Jeff Garzik
Already started, in fact, since Linus said he would not reject it out of hands.
Btw, I need to see it actually solve a real problem. As it is, I really think you've just painted yourself into a corner where you can't just admit that we can just load the firmware directly.
Could you _please_ entertain at least thinking about that option?
Linus
Quoted 2 lines · Linus Torvalds
Could you _please_ entertain at least thinking about that option?
.. and the reason I ask you to do that is quite frankly, you seem to want to just hide the thing under a rug - and _encouraging_ people to not even try to work with a new and more flexible model.
For example, there's nothing wrong with having people replace their firmware when they want to. Equally, there's nothing wrong in having distros put stable firmware into another package, so that you don't need to have <n> different versions of the firmware in <n> different kernel module directories just because you have <n> different versions of the module itself.
So why do you want to work so hard at not just encouraging people to get the firmware loaded dynamically from the filesystem?
Linus
Quoted 6 lines · Linus Torvalds
Quoted 2 lines · Jeff Garzik
Already started, in fact, since Linus said he would not reject it out of hands.
Btw, I need to see it actually solve a real problem. As it is, I really think you've just painted yourself into a corner where you can't just admit that we can just load the firmware directly.
Kernel 2.6.26. Older userland w/ initrd requirement, plus module w/ compiled-in firmware.
Simply dropping 2.6.27 into that situation will produce an unbootable system, since the firmware would not get copied into the initrd.
That's not just a kernel hacker situation. Plenty of sites, even a few large Red Hat deployments, run newer kernels on a known-stable, older userland. RH does not actively support swap-in-your-own-kernel, preferring our own tested kernel, but new-kernel/older-userland is not uncommon.
With upstream as it is today, there are just too many little ways the firmware can fail to follow the driver it once lived inside. And the consequences of failure are delayed (until next boot), silent (until next boot), and notable (may produce non-booting system, certainly a non-working driver).
Jeff
Quoted 10 lines · Jeff Garzik
Quoted 8 lines · Linus Torvalds
Quoted 2 lines · Jeff Garzik
Already started, in fact, since Linus said he would not reject it out of hands.
Btw, I need to see it actually solve a real problem. As it is, I really think you've just painted yourself into a corner where you can't just admit that we can just load the firmware directly.
Quoted 5 lines
Kernel 2.6.26. Older userland w/ initrd requirement, plus module w/ compiled-in firmware.
Simply dropping 2.6.27 into that situation will produce an unbootable system, since the firmware would not get copied into the initrd.
Kernel 2.6.15 with ancient userland. You update to 2.6.16, which includes commit 5433383e ('convert qla2xxx to request_firmware()').
Same problem? Only three years ago. And not really a problem back then either, because _even_ then you had to have a fairly ancient userland for it not to work.
You're far too late to start whining about it now, Jeff. We're just cleaning up the last drivers now. The rest have been where we want them to be for _years_.
Signature · 1 line
dwmw2
From: David Woodhouse <[email protected]>
Date: Tue, 15 Jul 2008 14:24:21 -0700
Quoted 4 lines
Kernel 2.6.15 with ancient userland. You update to 2.6.16, which includes commit
5433383e('convert qla2xxx torequest_firmware()').Same problem?
For the record I did get burnt by that one and I did make a huge stink about it.
I used to be able to avoid initrd entirely on every single one of my boxes, even my machines with qla2xxx based root filesystems, and that changeset basically meant the end of that.
I simply don't do much hacking on that particular machine any more, it's not worth the pain just to get a kernel that boots on it.
Yes, I know I can do stupid things like use the dist's initrd image and just ignore the module version warnings, which is what Linus happens to do, but that's just too gross for my tastes.
Quoted 6 lines · David Miller
From: David Woodhouse <[email protected]>
Date: Tue, 15 Jul 2008 14:24:21 -0700Quoted 3 lines
Kernel 2.6.15 with ancient userland. You update to 2.6.16, which includes commit
5433383e('convert qla2xxx torequest_firmware()').
Quoted 8 lines
Quoted 1 line
Same problem?
For the record I did get burnt by that one and I did make a huge stink about it.
I used to be able to avoid initrd entirely on every single one of my boxes, even my machines with qla2xxx based root filesystems, and that changeset basically meant the end of that.
Until now. The only significant _generic_ change that just went into Linus' kernel is the fact that you can now build that firmware _into_ your kernel, and you can boot qla2xxx without an initrd again.
HTH.
Signature · 1 line
dwmw2
From: David Woodhouse <[email protected]>
Date: Tue, 15 Jul 2008 15:42:32 -0700
Quoted 12 lines · David Miller
Quoted 11 lines
From: David Woodhouse <[email protected]>
Date: Tue, 15 Jul 2008 14:24:21 -0700Quoted 4 lines
Kernel 2.6.15 with ancient userland. You update to 2.6.16, which includes commit
5433383e('convert qla2xxx torequest_firmware()').Same problem?
For the record I did get burnt by that one and I did make a huge stink about it.
Quoted 7 lines
Quoted 3 lines
I used to be able to avoid initrd entirely on every single one of my boxes, even my machines with qla2xxx based root filesystems, and that changeset basically meant the end of that.
Until now. The only significant _generic_ change that just went into Linus' kernel is the fact that you can now build that firmware _into_ your kernel, and you can boot qla2xxx without an initrd again.
That's great.
Even better would be if the qla2xxx firmware was the tree like it used to be too. Then it truly would be fully back to how things used to be.
Quoted 3 lines · David Woodhouse
Until now. The only significant _generic_ change that just went into Linus' kernel is the fact that you can now build that firmware _into_ your kernel, and you can boot qla2xxx without an initrd again.
Yes, that is quite nice...
It will be nice to be able to do the same with modules too :)
Jeff
Quoted 6 lines · Jeff Garzik
Kernel 2.6.26. Older userland w/ initrd requirement, plus module w/ compiled-in firmware.
Simply dropping 2.6.27 into that situation will produce an unbootable system, since the firmware would not get copied into the initrd.
Umm. Neither would the module.
So I assume you are doing the module load from the disk, not the initrd?
Or are you rebuilding the module into the initrd? If so, why didn't you do the firmware too and my "load directly from file" idea which is pretty simple? IOW, what's the fundamnetal problem here?
But hey, if you _do_ the "link it together", go ahead. I don't see why it's so magically superior, but I wanted it for the compiled-in driver case, so it's not fundamentally different from that either. And I can't really find the energy to care any more.
Linus
Quoted 9 lines · Linus Torvalds
Quoted 5 lines · Jeff Garzik
Kernel 2.6.26. Older userland w/ initrd requirement, plus module w/ compiled-in firmware.
Simply dropping 2.6.27 into that situation will produce an unbootable system, since the firmware would not get copied into the initrd.
Umm. Neither would the module.
Sure it would.
Normal kernel build with modules, older userland (with older initrd).
'make modules_install && make install' will
* install modules in /lib
* install firmware in /lib [2.6.27 only]
* install kernel
* update grub
* rebuild initrd
Do this, under both 2.6.26 and 2.6.27.
2.6.26 will put a working driver into initrd.
2.6.27 will put a dead driver into initrd (because no firmware got copied into initrd image).
Quoted 1 line
Or are you rebuilding the module into the initrd? If so, why didn't you do the firmware too and my "load directly from file" idea which is pretty simple? IOW, what's the fundamnetal problem here?
The firmware doesn't get copied into the initrd, because it is an automated process (older mkinitrd) that is unaware of the need to copy the firmware into the initrd.
Do the same build process on 2.6.26 and 2.6.27, and you WILL get different results on older userlands, for the same simple reason: the scripts do not know to copy the firmware when they copy the kernel module.
Quoted 1 line
But hey, if you _do_ the "link it together", go ahead. I don't see why
Great! I will :)
Quoted 1 line
it's so magically superior, but I wanted it for the compiled-in driver case, so it's not fundamentally different from that either.
It's not magically superior, just a necessary tool that helps avoid a whole class of regressions, keeps us working on older userland, etc., etc.
Jeff
Quoted 8 lines · Jeff Garzik
'make
modules_install&& make install' will
* install modules in /lib
* install firmware in /lib [2.6.27 only]
* install kernel
* update grub
* rebuild initrd
Quoted 6 lines
Do this, under both 2.6.26 and 2.6.27.
2.6.26 will put a working driver into initrd.
2.6.27 will put a dead driver into initrd (because no firmware got copied into initrd image).
I believe your claim is untrue. It uses the distribution's tool to create the initrd, doesn't it? And those tools _do_ handle the MODULE_FIRMWARE() tags and pull in appropriate firmware. They've had to get that right for _years_ already.
Signature · 1 line
dwmw2
Quoted 10 lines · David Woodhouse
Quoted 9 lines · Jeff Garzik
'make
modules_install&& make install' will
* install modules in /lib
* install firmware in /lib [2.6.27 only]
* install kernel
* update grub
* rebuild initrdDo this, under both 2.6.26 and 2.6.27.
Quoted 7 lines
Quoted 3 lines
2.6.26 will put a working driver into initrd.
2.6.27 will put a dead driver into initrd (because no firmware got copied into initrd image).
I believe your claim is untrue. It uses the distribution's tool to create the initrd, doesn't it? And those tools _do_ handle the
MODULE_FIRMWARE()tags and pull in appropriate firmware.
slackware's CURRENT mkinitrd and kernel-modules scripts appear to disprove your claims. And yes, they use a modern kernel, 2.6.24.
Quoted 2 lines
They've had to
get that right for _years_ already.
You should not just /assume/ that everything is ok, simply because time passed since request_firmware() was added. Or let impatience blind you to the breakage that will result from your change.
Further, there are several factors that clearly differentiate a driver that has been using request_firmware() for a while, and newly converting a large block of drivers en masse.
Jeff
Quoted 13 lines · David Woodhouse
Quoted 12 lines · Jeff Garzik
'make
modules_install&& make install' will
* install modules in /lib
* install firmware in /lib [2.6.27 only]
* install kernel
* update grub
* rebuild initrdDo this, under both 2.6.26 and 2.6.27.
2.6.26 will put a working driver into initrd.
Quoted 7 lines
Quoted 2 lines
2.6.27 will put a dead driver into initrd (because no firmware got copied into initrd image).
I believe your claim is untrue. It uses the distribution's tool to create the initrd, doesn't it? And those tools _do_ handle the
MODULE_FIRMWARE()tags and pull in appropriate firmware. They've had to get that right for _years_ already.
Something from the non-development world:
At least the initramfs-tools (which contains mkinitrd) of Debian Etch do not do this (the current testing/development version Lenny does) - no firmware support in the initrd at all - and not needed for booting until now.
(It has /lib/firmware support after the initrd for the not-so-important stuff like usb/sound/video/wireless)
Yes, it may be nearly ancient by now, but for servers it contains all we need, and most certainly would not want to use something that changes and needs checking/hand-fixing every month - I expect this version to be still in use a year from now, then just gradually being updated to the next debian version.
Currently we use 2.6.24, and I expect to use some newer kernel to support newer replacement hardware before all installs are upgraded. So having the firmware-inside-module support would be a help for us when modules are starting to be needed for disk or network access, keeping from the need to try backporting huge initrd/make-kpkg changes.
c'ya
sven
Signature · 2 lines
The Internet treats censorship as a routing problem, and routes around it. (John Gilmore on http://www.cygnus.com/~gnu/)
Quoted 15 lines · Sven-Haegar Koch
Something from the non-development world:
At least the initramfs-tools (which contains mkinitrd) of Debian Etch do not do this (the current testing/development version Lenny does) - no firmware support in the initrd at all - and not needed for booting until now.
(It has/lib/firmwaresupport after the initrd for the not-so-important stuff like usb/sound/video/wireless)Yes, it may be nearly ancient by now, but for servers it contains all we need, and most certainly would not want to use something that changes and needs checking/hand-fixing every month - I expect this version to be still in use a year from now, then just gradually being updated to the next debian version.
Quoted 5 lines
Currently we use 2.6.24, and I expect to use some newer kernel to support newer replacement hardware before all installs are upgraded. So having the firmware-inside-module support would be a help for us when modules are starting to be needed for disk or network access, keeping from the need to try backporting huge initrd/make-kpkg changes.
You can easyly install initramfs-tools from lenny on an etch box. It only depends on a newer klibc, so it won't disturb any other services.
Btw., the Debian-provided 2.6.25 kernel already lacks the firmware for some popular modules (like bnx2). So if you're using Debian, you as a sysadmin already have to handle the "firmware is in a separate package" issue.
Gabor
Signature · 4 lines
---------------------------------------------------------
MTA SZTAKI Computer and Automation Research Institute
Hungarian Academy of Sciences
---------------------------------------------------------Quoted 7 lines · Jeff Garzik
Quoted 2 lines · Theodore Tso
If so, may I suggest you start implementing, instead of sending
e-mails? For all the time people have spent arguing about it, maybe it could have been implemented already.Already started, in fact, since Linus said he would not reject it out of hands.
Great, as I really hope this gets in. Was looking but am very scared of the linker goo...
I'm going to like the feature in itself once it has grown the ability for firmware in the module and really do not right now. One of the drivers I care about (old ISA SB16 CSP code) was also affected and my old 486 system with the hardware in it is _still_ lauging its little balls of at the people stating that "if you have userspace to load the module you have userspace to support at the hotplug/udev goo to load the firmware".
Given that one needs to be rather irrelevant to care about the SB16 CSP in the first place it wasn't enough to complain really, but yes, I'm really quite sure that I'm going to use the feature of building it into the module more than once while running around with floppies trying to support that old stuff.
Rene.
Quoted 6 lines · Theodore Tso
Quoted 4 lines · Jeff Garzik
All of the regressions examples I am citing are cured by firmware-in-module, because they are all examples of problems that occur when you remove the ability to reproduce the same functional pieces as 2.6.26.
Quoted 7 lines
Jeff,
I think you've said this before, but let me try to help you
cut to the chase. You are willing to *implement* firmware-in-the-module forrequest_firmware(), but you want a commitment that David Woodhouse and Linus will accept such a patch before you go off and write it. Is that right?
The more important question is whether _Linus_ will take it, and of course I can't answer for him.
Personally I think it's pointless. But since it's only _mildly_ counterproductive, if you really want to do it then I wouldn't actively oppose it unless it turns out to be a mess to implement, or unless it's limited to working only with the in-tree firmware.
If you implement it cleanly and optionally (and defaulting off), and if it works properly with out-of-tree firmware too (so I can build a libertas usb8xxx.ko module with the firmware included, for example), then I certainly wouldn't throw my toys out of the pram over it. Ideally, it would work for out-of-tree drivers (make M=... modules) too.
I was thinking that we could maybe post-process .ko files (which we do already for other reasons), and add a special section in them containing the firmware which is mentioned in MODULE_FIRMWARE() tags. The special section is how we do it for the kernel too. That would take a little bit of code in module.c to keep track of those sections and add them to a list somewhere that firmware_class.c can see them, and then you have to think about lifetime and locking issues.
But those were just preliminary thoughts -- I wasn't really planning to implement it so I didn't go through it in detail.
An alternative approach might be to turn firmware blobs into actual modules, which just register themselves with the firmware loader in module_init(). I'm not sure I like that approach though, and depmod wouldn't 'notice' them automatically so it wouldn't work with current userspace. I've tried to be very careful to ensure that anything we do here is entirely compatible with existing userspace.
Signature · 1 line
dwmw2
From: [email protected]
Date: Mon, 14 Jul 2008 17:51:37 -0700 (PDT)
Quoted 3 lines
I agree with this, but the proponents of the seperate firmware are listing the fact that the firmware doesn't tie up ram as one of the big reasons for making the change.
Exactly.
Otherwise these firmware changes are utterly pointless.
Quoted 9 lines · David Miller
From: [email protected] Date: Mon, 14 Jul 2008 17:51:37 -0700 (PDT)
Quoted 3 lines
I agree with this, but the proponents of the seperate firmware are listing the fact that the firmware doesn't tie up ram as one of the big reasons for making the change.
Exactly.
Otherwise these firmware changes are utterly pointless.
The point of them is legal. Which everyone knows so could perhaps that particular dance be done away with?
The current patches allow to
1) load the firmware from userspace
2) load the firmware from the kernel image (vmlinux)
They lack the ability to load the firmware from the module image.
From a technical standpoint that seems an odd setup. As a driver author you'd either be satisfied with loading the firmware from userspace, or if for example you feel you're very much tied to your specific firmware, you want it to be as close to you as possible -- in the kernel image while you yourself are a module makes little sense. You couldn't for example just copy the single module around.
That seems a clear technical point to fix but after that, the point is legal. At least for existing drivers, this consists of distributions worrying about GPL consequences and wanting to seperate out the existing firmware blobs.
They do worry -- whatever those worries might be worth -- so there's the point. You may feel it not a good point but that's something other than no point existing.
I followed along on these flamewars and I'd say to reject these current patches on technical grounds until the ability to build the firmware into the module exists and merge it after -- even though people might feel there's little or no upside to them, I'd say there's little of no downside either once that ability is in.
Rene.
From: Rene Herman <[email protected]>
Date: Tue, 15 Jul 2008 04:23:14 +0200
Quoted 8 lines · David Miller
Quoted 7 lines
From: [email protected] Date: Mon, 14 Jul 2008 17:51:37 -0700 (PDT)
Quoted 3 lines
I agree with this, but the proponents of the seperate firmware are listing the fact that the firmware doesn't tie up ram as one of the big reasons for making the change.
Quoted 5 lines
Quoted 3 lines
Exactly.
Otherwise these firmware changes are utterly pointless.
The point of them is legal.
Thanks for proving something I tried to establish for weeks but which Alan Cox, David W., and others vehemently denied.
They states that it was being done on a technical basis rather than being predominantly a legal one.
Quoted 12 lines · David Miller
From: Rene Herman <[email protected]>
Date: Tue, 15 Jul 2008 04:23:14 +0200Quoted 9 lines · David Miller
Quoted 8 lines
From: [email protected] Date: Mon, 14 Jul 2008 17:51:37 -0700 (PDT)
Quoted 3 lines
I agree with this, but the proponents of the seperate firmware are listing the fact that the firmware doesn't tie up ram as one of the big reasons for making the change.
Exactly.
Quoted 8 lines
Quoted 2 lines
Quoted 1 line
Otherwise these firmware changes are utterly pointless.
The point of them is legal.
Thanks for proving something I tried to establish for weeks but which Alan Cox, David W., and others vehemently denied.
They states that it was being done on a technical basis rather than being predominantly a legal one.
Yes, they were obstinate or dishonest (I won't say "respectively"). As to "proving" though, I cannot prove anything, being a mere observer.
At this point I really believe this discussion should be about the other part of my reply -- the point mostly put forward by Jef Garzik about the firmware inside the module image. Without that ability, I don't believe these are good patches.
_With_ that ability, I myself do. Let's allow allow everyone their own level of fear, uncertainty and doubt.
Rene
Quoted 3 lines · David Miller
They states that it was being done on a technical basis rather than being predominantly a legal one.
No.
Yes, the original reason for request_firmware() was obviously very much partly legal.
HOWEVER.
Once you have a model that is required (for whatever reasons) for some drivers, we're much better off using the _same_ model for all drivers, whether it is necessary for legal reasons for those other drivers.
Put this way: if you do a distro, you _need_ to support firmware loading anyway. And once you do that, it's just annoying how some drivers then do something odd and special for the same thing, for no real good reason.
It's like having each wireless driver do their own 802.11 stack. It works, and all those driver writers seem to be always convinced that it's the right thing to do. But as a network maintainer, do you like it?
Now, replace "802.11 stack" with "firmware loader", and then replace "network maintainer" with "kernel maintainer".
And maybe you can see why I like just making everybody do request_firmware().
Do you call that a legal reason? Or a technical one?
Linus
From: Linus Torvalds <[email protected]>
Date: Mon, 14 Jul 2008 19:39:03 -0700 (PDT)
Quoted 3 lines
Put this way: if you do a distro, you _need_ to support firmware loading anyway. And once you do that, it's just annoying how some drivers then do something odd and special for the same thing, for no real good reason.
In what way is it annoying?
Most distribution people aren't even aware that drivers like tg3 and bnx2 even have firmware. In fact it's self contained and less for them to worry about.
The correct firmware is right there in the driver, it's what the driver maintainer tested as a unit, and everything.
It neither gets in the distro guys way nor causes any functional problems.
It may cause potential legal issues, but that is another matter.
Quoted 6 lines
It's like having each wireless driver do their own 802.11 stack. It works, and all those driver writers seem to be always convinced that it's the right thing to do. But as a network maintainer, do you like it?
Now, replace "802.11 stack" with "firmware loader", and then replace "network maintainer" with "kernel maintainer".
Distro folks do ship external drivers that have their own 802.11 stack, so this is a very good example in fact :-)
I think the modular vs. static kernel image is a better analogy. At least with that I have a real choice, with this stuff I suddently won't.
Regardless of what you or I think about these changes, the fact is that there were definitely some areas of contention and even some level of misrepresentation of these changes. And given that, some of us feel that merging this in right now was on the premature side.
Quoted 8 lines · On Mon, 14 Jul 2008 19:45:57 -0700 (PDT) David Miller <[email protected]>…
From: Linus Torvalds <[email protected]>
Date: Mon, 14 Jul 2008 19:39:03 -0700 (PDT)Quoted 4 lines
Put this way: if you do a distro, you _need_ to support firmware loading anyway. And once you do that, it's just annoying how some drivers then do something odd and special for the same thing, for no real good reason.
Quoted 5 lines
In what way is it annoying?
Most distribution people aren't even aware that drivers like tg3 and bnx2 even have firmware. In fact it's self contained and less for them to worry about.
and.. after this patch that still seems to be the case, unless I'm looking at it really cross eyed.
Nothing in this patch makes it impossible to do so.. or changes the drivers you mention. in fact it's the opposite almost: it improves request_firmware() usability significantly. (by allowing vmlinux build in etc).
Signature · 3 lines
If you want to reach me at my work email, use [email protected] For development, discussion and tips for power savings, visit http://www.lesswatts.org
Quoted 13 lines · Arjan van de Ven
Quoted 10 lines · On Mon, 14 Jul 2008 19:45:57 -0700 (PDT) David Miller <[email protected]>…
From: Linus Torvalds <[email protected]>
Date: Mon, 14 Jul 2008 19:39:03 -0700 (PDT)Quoted 4 lines
Put this way: if you do a distro, you _need_ to support firmware loading anyway. And once you do that, it's just annoying how some drivers then do something odd and special for the same thing, for no real good reason.
In what way is it annoying?
Quoted 8 lines
Quoted 3 lines
Most distribution people aren't even aware that drivers like tg3 and bnx2 even have firmware. In fact it's self contained and less for them to worry about.
and.. after this patch that still seems to be the case, unless I'm looking at it really cross eyed.
Nothing in this patch makes it impossible to do so.. or changes the drivers you mention.
Well, the HEAD of my tree (and linux-next) does change tg3. I just haven't asked Linus to pull that part, because I wanted to keep the contentious part separate from the simple and obvious parts.
(Even though the tg3 fix _is_ simple and obvious, to me. But that evidently doesn't stop it from being contentious too. So it can wait.)
Signature · 1 line
dwmw2
Quoted 23 lines · David Woodhouse
Quoted 13 lines · Arjan van de Ven
Quoted 10 lines · On Mon, 14 Jul 2008 19:45:57 -0700 (PDT) David Miller <[email protected]>…
From: Linus Torvalds <[email protected]>
Date: Mon, 14 Jul 2008 19:39:03 -0700 (PDT)Quoted 4 lines
Put this way: if you do a distro, you _need_ to support firmware loading anyway. And once you do that, it's just annoying how some drivers then do something odd and special for the same thing, for no real good reason.
In what way is it annoying?
Quoted 8 lines
Quoted 3 lines
Most distribution people aren't even aware that drivers like tg3 and bnx2 even have firmware. In fact it's self contained and less for them to worry about.
and.. after this patch that still seems to be the case, unless I'm looking at it really cross eyed.
Nothing in this patch makes it impossible to do so.. or changes the drivers you mention.
Quoted 6 lines
Well, the HEAD of my tree (and linux-next) does change tg3. I just haven't asked Linus to pull that part, because I wanted to keep the contentious part separate from the simple and obvious parts.
(Even though the tg3 fix _is_ simple and obvious, to me. But that evidently doesn't stop it from being contentious too. So it can wait.)
a techinical question here (I know, how dare I ask a technical question in the middle of the flamefest ;-)
linus pointed out that the documentation reccomended the
request_firmware();load_firmware();release() approach and stated that that approach was the wrong way to do things, instead doing a request_firmware early and release when the module is unloaded.
does this patch series follow the documented reccomendation? or does it follow the more concervative approach Linus pointed out? (it's far faster to ask then to search Internet archives for the patches)
David Lang
Quoted 8 lines · On Mon, 2008-07-14 at 22:45 -0700, [email protected] wrote:
linus pointed out that the documentation reccomended the request_firmware();load_firmware();release() approach and stated that that approach was the wrong way to do things, instead doing a
request_firmwareearly and release when the module is unloaded.does this patch series follow the documented reccomendation? or does it follow the more concervative approach Linus pointed out? (it's far faster to ask then to search Internet archives for the patches)
Mostly it follows the documented recommendation, since most of the touched drivers are USB drivers and you end up re-enumerating and starting from scratch on resume anyway. And the remainder are so old that they don't have suspend/resume support anyway. Remember, we're only really updating the older drivers; newer drivers tend to use request_firmware() already, and have done for years.
The only one I remember offhand that loads the firmware and keeps it around is tg3 -- and I didn't ask Linus to pull that one. In fact, we only did it that way for tg3 since that driver seems to doing _all_ its chip reset and firmware reload (including a boatload of udelays) within a spinlock. That in itself is probably not optimal, but it wasn't really within the scope of what we were doing to fix it.
Signature · 1 line
dwmw2
Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel , use it in more drivers.
Am Dienstag 15 Juli 2008 08:03:15 schrieb David Woodhouse:
Quoted 6 lines
Mostly it follows the documented recommendation, since most of the touched drivers are USB drivers and you end up re-enumerating and starting from scratch on resume anyway. And the remainder are so old that they don't have suspend/resume support anyway. Remember, we're only really updating the older drivers; newer drivers tend to use
request_firmware()already, and have done for years.
As for USB that was before persistance. It now allows you to even support a loss of power without reprobing, provided
a) the devices does not change IDs
b) you have firmware around when reset_resume() is called
Regards
Oliver
Quoted 8 lines · Oliver Neukum
Am Dienstag 15 Juli 2008 08:03:15 schrieb David Woodhouse:
Quoted 6 lines
Mostly it follows the documented recommendation, since most of the touched drivers are USB drivers and you end up re-enumerating and starting from scratch on resume anyway. And the remainder are so old that they don't have suspend/resume support anyway. Remember, we're only really updating the older drivers; newer drivers tend to use
request_firmware()already, and have done for years.
Quoted 5 lines
As for USB that was before persistance. It now allows you to even support a loss of power without reprobing, provided
a) the devices does not change IDs
b) you have firmware around whenreset_resume()is called
c) the driver in question actually _has_ a reset_resume() method
The old, unloved drivers which I've touched as part of this patch set did not. In general the newer, loved drivers use request_firmware() already.
Signature · 1 line
dwmw2
Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel , use it in more drivers.
Am Dienstag 15 Juli 2008 16:40:54 schrieb David Woodhouse:
Quoted 12 lines · Oliver Neukum
Quoted 11 lines
Am Dienstag 15 Juli 2008 08:03:15 schrieb David Woodhouse:
Quoted 6 lines
Mostly it follows the documented recommendation, since most of the touched drivers are USB drivers and you end up re-enumerating and starting from scratch on resume anyway. And the remainder are so old that they don't have suspend/resume support anyway. Remember, we're only really updating the older drivers; newer drivers tend to use
request_firmware()already, and have done for years.As for USB that was before persistance. It now allows you to even support a loss of power without reprobing, provided
Quoted 9 lines
Quoted 2 lines
a) the devices does not change IDs
b) you have firmware around whenreset_resume()is calledc) the driver in question actually _has_ a
reset_resume()methodThe old, unloved drivers which I've touched as part of this patch set did not. In general the newer, loved drivers use
request_firmware()already.
Generally we love all drivers and welcome new additions. Which did you touch? I lost track.
Regards
Oliver
Quoted 18 lines · Oliver Neukum
Am Dienstag 15 Juli 2008 16:40:54 schrieb David Woodhouse:
Quoted 17 lines · Oliver Neukum
Quoted 8 lines
Am Dienstag 15 Juli 2008 08:03:15 schrieb David Woodhouse:
Quoted 6 lines
Mostly it follows the documented recommendation, since most of the touched drivers are USB drivers and you end up re-enumerating and starting from scratch on resume anyway. And the remainder are so old that they don't have suspend/resume support anyway. Remember, we're only really updating the older drivers; newer drivers tend to use
request_firmware()already, and have done for years.Quoted 5 lines
As for USB that was before persistance. It now allows you to even support a loss of power without reprobing, provided
a) the devices does not change IDs
b) you have firmware around whenreset_resume()is calledc) the driver in question actually _has_ a
reset_resume()method
Quoted 7 lines
Quoted 4 lines
The old, unloved drivers which I've touched as part of this patch set did not. In general the newer, loved drivers use
request_firmware()already.Generally we love all drivers and welcome new additions. Which did you touch? I lost track.
Ignoring the simple bits to make fw->data const? See
http://git.infradead.org/users/dwmw2/firmware-2.6.git?a=shortlog&h=d71792ac
or git-diff --stat 59890f74..d71792ac | grep -v firmware/
21 drivers, mostly older USB drivers and including four sound drivers which were already using request_firmware() optionally. The HEAD of my tree contains a few more updates in drivers/scsi and drivers/net, but out of respect for Dave's and Jeff's objections I left those out of the pull request for now. Hopefully the noise will die down when the sky doesn't _actually_ fall on our heads, and we can merge the remaining changes into 2.6.28.
drivers/atm/Makefile | 6 +- drivers/atm/ambassador.c | 140 +- drivers/atm/ambassador.h | 11 - drivers/atm/atmsar11.data | 2063 ----- drivers/atm/atmsar11.regions | 6 - drivers/atm/atmsar11.start | 4 - drivers/char/dsp56k.c | 84 +- drivers/char/ip2/fip_firm.h | 2149 ----- drivers/char/ip2/ip2base.c | 5 +- drivers/char/ip2/ip2main.c | 47 +- drivers/media/dvb/ttusb-budget/dvb-ttusb-budget.c | 16 +- .../media/dvb/ttusb-budget/dvb-ttusb-dspbootcode.h | 1644 ---- drivers/media/video/cpia2/cpia2_core.c | 46 +- drivers/media/video/cpia2/cpia2patch.h | 233 - drivers/media/video/dabfirmware.h | 1415 ---- drivers/media/video/dabusb.c | 44 +- drivers/media/video/usbvideo/vicam.c | 317 +- drivers/net/tokenring/smctr.c | 56 +- drivers/net/tokenring/smctr.h | 2 - drivers/net/tokenring/smctr_firmware.h | 978 --- drivers/net/usb/kaweth.c | 43 +- drivers/net/usb/kawethfw.h | 557 -- drivers/usb/misc/emi26.c | 96 +- drivers/usb/misc/emi26_fw.h | 5779 ------------- drivers/usb/misc/emi62.c | 130 +- drivers/usb/misc/emi62_fw_m.h | 8853 -------------------- drivers/usb/misc/emi62_fw_s.h | 8837 ------------------- drivers/usb/serial/Kconfig | 24 +- drivers/usb/serial/io_edgeport.c | 183 +- drivers/usb/serial/io_fw_boot.h | 556 -- drivers/usb/serial/io_fw_boot2.h | 546 -- drivers/usb/serial/io_fw_down.h | 1229 --- drivers/usb/serial/io_fw_down2.h | 1133 --- drivers/usb/serial/io_fw_down3.h | 847 -- drivers/usb/serial/io_ti.c | 73 +- drivers/usb/serial/keyspan.c | 78 +- drivers/usb/serial/keyspan.h | 84 - drivers/usb/serial/keyspan_mpr_fw.h | 286 - drivers/usb/serial/keyspan_pda.S | 1124 --- drivers/usb/serial/keyspan_pda.c | 51 +- drivers/usb/serial/keyspan_pda_fw.h | 99 - drivers/usb/serial/keyspan_usa18x_fw.h | 447 - drivers/usb/serial/keyspan_usa19_fw.h | 285 - drivers/usb/serial/keyspan_usa19qi_fw.h | 284 - drivers/usb/serial/keyspan_usa19qw_fw.h | 448 - drivers/usb/serial/keyspan_usa19w_fw.h | 446 - drivers/usb/serial/keyspan_usa28_fw.h | 466 - drivers/usb/serial/keyspan_usa28x_fw.h | 447 - drivers/usb/serial/keyspan_usa28xa_fw.h | 449 - drivers/usb/serial/keyspan_usa28xb_fw.h | 448 - drivers/usb/serial/keyspan_usa49w_fw.h | 464 - drivers/usb/serial/keyspan_usa49wlc_fw.h | 476 -- drivers/usb/serial/ti_fw_3410.h | 885 -- drivers/usb/serial/ti_fw_5052.h | 885 -- drivers/usb/serial/ti_usb_3410_5052.c | 40 +- drivers/usb/serial/whiteheat.c | 77 +- drivers/usb/serial/whiteheat_fw.h | 1669 ---- drivers/usb/serial/xircom_pgs.S | 1192 --- drivers/usb/serial/xircom_pgs_fw.h | 103 - sound/isa/Kconfig | 9 - sound/isa/sb/sb16_csp.c | 22 - sound/isa/sb/sb16_csp_codecs.h | 949 --- sound/pci/Kconfig | 30 - sound/pci/korg1212/korg1212-firmware.h | 987 --- sound/pci/korg1212/korg1212.c | 18 - sound/pci/maestro3.c | 171 - sound/pci/ymfpci/ymfpci_image.h | 1565 ---- sound/pci/ymfpci/ymfpci_main.c | 63 -
Signature · 1 line
dwmw2
Quoted 16 lines · Arjan van de Ven
Quoted 7 lines · On Mon, 14 Jul 2008 19:45:57 -0700 (PDT) David Miller <[email protected]>…
From: Linus Torvalds <[email protected]>
Date: Mon, 14 Jul 2008 19:39:03 -0700 (PDT)Quoted 4 lines
Put this way: if you do a distro, you _need_ to support firmware loading anyway. And once you do that, it's just annoying how some drivers then do something odd and special for the same thing, for no real good reason.
Quoted 5 lines
In what way is it annoying?
Most distribution people aren't even aware that drivers like tg3 and bnx2 even have firmware. In fact it's self contained and less for them to worry about.
Quoted 6 lines
and.. after this patch that still seems to be the case, unless I'm looking at it really cross eyed.
Nothing in this patch makes it impossible to do so.. or changes the drivers you mention. in fact it's the opposite almost: it improvesrequest_firmware()usability significantly. (by allowing vmlinux build in etc).
Only by virtue of those specific drivers having been skipped for now. For those that weren't skipped, the only way to have them still be self contained is to have the kernel itself compile in all firmware for all modules after which the module will find it there. Distributions will of course not do this, not does it make any sense for own kernels.
Rene.
Quoted 6 lines · Linus Torvalds
Quoted 2 lines · David Miller
They states that it was being done on a technical basis rather than being predominantly a legal one.
No.
Quoted 7 lines
Yes, the original reason for
request_firmware()was obviously very much partly legal.HOWEVER.
Once you have a model that is required (for whatever reasons) for some drivers, we're much better off using the _same_ model for all drivers, whether it is necessary for legal reasons for those other drivers.
Put this way: if you do a distro, you _need_ to support firmware loading anyway. And once you do that, it's just annoying how some drivers then do something odd and special for the same thing, for no real good reason.
There's little alternative infrastructure in declaring a static array of unsigned char though.
Yes, it still makes sense to make everyone use request_firmware() if only because these other drivers now all of a sudden have a more accessible way of updating their firmwares, but David's objective is legal here.
Which is fine.
But the very clear _technical_ objection about this not providing for keeping modules and their firmware together for those that really do want it to be that way was not at all addressed. Note, this while still using request_firmware(), just at the option of the kernel builder with the firmware compiled into the module. Right now, this allows the firmware to be compiled into the vmlinux only which makes fairly little sense if the driver itself is a module.
Rene.
Quoted 3 lines · Linus Torvalds
[...] I like just making everybody do
request_firmware().Do you call that a legal reason? Or a technical one?
\o.
Hmm, let's se...
How about legal?
If they don't budge in, you can terminate their license to Linux, because they are distributing your GPLed work combined with software that is not under a GPL-compatible license.
:-P :-D
Signature · 4 lines
Alexandre Oliva http://www.lsd.ic.unicamp.br/~oliva/
Free Software Evangelist oliva@{lsd.ic.unicamp.br, gnu.org}
FSFLA Board Member ¡Sé Libre! => http://www.fsfla.org/
Red Hat Compiler Engineer aoliva@{redhat.com, gcc.gnu.org}Quoted 7 lines
Thanks for proving something I tried to establish for weeks but which Alan Cox, David W., and others vehemently denied.
They states that it was being done on a technical basis rather than being predominantly a legal one.
Despite the risk of going against the current here, I do like David patches for one major reason: I love my kernels with everything built-in, that's how I can netboot them etc... without having to bother with installing modules.
David's patches allow me to do that with drivers that would have -required- disk based firmwares beforehand. It allows things like rooting off a block device that needs a firmware without an initrd, etc....
As long as the firmwares are shipped with the kernel tree, I'm happpy, and I don't care much about building the firmware inside the .ko's. Now, if somebody care, I suppose it's a feature that can be added fairly transparently.
Cheers,
Ben.
Quoted 11 lines · Benjamin Herrenschmidt
Quoted 6 lines
Thanks for proving something I tried to establish for weeks but which Alan Cox, David W., and others vehemently denied.
They states that it was being done on a technical basis rather than being predominantly a legal one.
Despite the risk of going against the current here, I do like David patches for one major reason: I love my kernels with everything built-in, that's how I can netboot them etc... without having to bother with installing modules.
I think you're definitely _with_ the current on that one.
Quoted 6 lines
David's patches allow me to do that with drivers that would have -required- disk based firmwares beforehand. It allows things like rooting off a block device that needs a firmware without an initrd, etc....
As long as the firmwares are shipped with the kernel tree, I'm happpy,
They don't even need to be shipped with the kernel tree. You can set CONFIG_EXTRA_FIRMWARE_DIR to point to wherever you keep them (like /lib/firmware) and it'll pull them in from there. That's _why_ it's useful for drivers which would have required an initrd before, and which don't have their firmware shipped in the kernel tree.
Signature · 1 line
dwmw2
From: Linus Torvalds <[email protected]>
Date: Mon, 14 Jul 2008 17:33:58 -0700 (PDT)
Quoted 3 lines
The only reliable way to handle it is to load the firmware before the suspend happens - preferably by simply never unloading it in the first place.
That's what we were told was the huge benefit of moving some of these drivers over to this request_firmware() stuff, it would save memory.
To me, especially for the networking drivers for which I am familiar, this transformation seems completely pointless from a technical perspective. In fact it's a regression as-is.
Quoted 10 lines · David Miller
From: Linus Torvalds <[email protected]>
Date: Mon, 14 Jul 2008 17:33:58 -0700 (PDT)Quoted 3 lines
The only reliable way to handle it is to load the firmware before the suspend happens - preferably by simply never unloading it in the first place.
That's what we were told was the huge benefit of moving some of these drivers over to this
request_firmware()stuff, it would save memory.
There _are_ RAM benefits for a number of drivers, even when you load the firmware just once at initialisation time and keep it resident all the time the device is active (and no, I won't slap myself for saying it).
The tg3 driver is a prime example -- it contains three separate firmware images for different revisions of the card, but you'll almost never actually need all three. Mostly you'll only need _one_ of them.
Signature · 1 line
dwmw2
Quoted 12 lines · David Woodhouse
Quoted 10 lines · David Miller
From: Linus Torvalds <[email protected]>
Date: Mon, 14 Jul 2008 17:33:58 -0700 (PDT)Quoted 3 lines
The only reliable way to handle it is to load the firmware before the suspend happens - preferably by simply never unloading it in the first place.
That's what we were told was the huge benefit of moving some of these drivers over to this
request_firmware()stuff, it would save memory.
Quoted 7 lines
There _are_ RAM benefits for a number of drivers, even when you load the firmware just once at initialisation time and keep it resident all the time the device is active (and no, I won't slap myself for saying it).
The tg3 driver is a prime example -- it contains three separate firmware images for different revisions of the card, but you'll almost never actually need all three. Mostly you'll only need _one_ of them.
and when a card gets hotplugged into the system for a driver that's already in memory handling other cards you intend that it will have to go out to userspace to find the firmware?
how much memory are you expecting to save this way?
David Lang
Quoted 16 lines · On Mon, 2008-07-14 at 17:06 -0700, [email protected] wrote:
Quoted 14 lines · Arjan van de Ven
Quoted 10 lines · On Mon, 14 Jul 2008 16:41:19 -0700 Andrew Morton <[email protected]>…
Quoted 3 lines · On Mon, 14 Jul 2008 16:23:26 -0700 David Woodhouse <[email protected]> wrote:
Linus, please pull from the for-2.6.27 branch of:
git://git.infradead.org/users/dwmw2/firmware-2.6.gitfor-2.6.27
The firmware flamewars seem to have subsided lately. Is everyone happy with this now?
Quoted 6 lines
Quoted 1 line
this seems to have left the contentious pieces out....
almost, the one thing that this could have done would be to offer an option to bundle the firmware with the module. currently it offers the option to load the firmware at the same time as the module, not _quite_ the same thing.
I see no real point in that. If you have userspace to load modules, then you have userspace to load firmware.
We made 'make modules_install' install the firmware in /lib/firmware for you at the same time as it installs the modules, to avoid problems if people forgot to run 'make firmware_install'. That really ought to be enough.
I know Jeff complained that it wasn't, and insisted that he _needed_ to be able to scp a single .ko file around which contained both, and the world was broken if he could not -- but I disagree with him.
But since I wanted this tree to be uncontentious and obviously the correct thing to do, I've dropped the drivers/net changes for now anyway.
It's odd that this request has suddenly come out of the blue when we've been using request_firmware() from modules for years already.
Quoted 3 lines
there was also the issue that was raised about how to handle firmware during suspend/resume. I don't remember seeing a happy solution to that one.
My b43 seems to work fine on suspend/resume, as do most other modern drivers which use request_firmware(). There is certainly no fundamental, generic problem with suspend/resume vs. the firmware loader.
If you see a problem with a specific driver which we've converted, do please point it out.
Signature · 1 line
dwmw2
Quoted 3 lines · David Woodhouse
I see no real point in that. If you have userspace to load modules, then you have userspace to load firmware.
David, the case is different: the module is loaded _across_ the suspend/resume, but the device needs to have its firmware reloaded at resume time because it lost power.
And since the device may be resumed before the root filesystem disk device, it needs to do so _without_ calling request_firmware() at resume time.
The above is _normal_.
And the fact is, the only way to handle it sanely is to do the request_firmware() before suspending.
I _really_ don't understand why this keeps coming up all the time. It's a no-brainer, with a single possible solution, yet people continue to whine about it.
Linus
Quoted 9 lines · Linus Torvalds
Quoted 3 lines · David Woodhouse
I see no real point in that. If you have userspace to load modules, then you have userspace to load firmware.
David, the case is different: the module is loaded _across_ the suspend/resume, but the device needs to have its firmware reloaded at resume time because it lost power.
Either I put my reply in response to the wrong quotation, or you thought I did.
I intended the above to be a response to the "please put firmware _into_ the .ko file" request. Not to the suspend/resume thing.
Yes, I agree that suspend/resume is a non-issue.
Signature · 1 line
dwmw2
Quoted 11 lines · David Woodhouse
Quoted 9 lines · Linus Torvalds
Quoted 3 lines · David Woodhouse
I see no real point in that. If you have userspace to load modules, then you have userspace to load firmware.
David, the case is different: the module is loaded _across_ the suspend/resume, but the device needs to have its firmware reloaded at resume time because it lost power.
Quoted 7 lines
Either I put my reply in response to the wrong quotation, or you thought I did.
I intended the above to be a response to the "please put firmware _into_ the .ko file" request. Not to the suspend/resume thing.
Yes, I agree that suspend/resume is a non-issue.
Besides, request_firmware() from a resume() method is broken. Has always been and will probably always be.
Ben.
Quoted 12 lines · David Woodhouse
Quoted 11 lines · On Mon, 2008-07-14 at 17:06 -0700, [email protected] wrote:
Quoted 9 lines · Arjan van de Ven
Quoted 6 lines · On Mon, 14 Jul 2008 16:41:19 -0700 Andrew Morton <[email protected]>…
Quoted 3 lines · On Mon, 14 Jul 2008 16:23:26 -0700 David Woodhouse <[email protected]> wrote:
Linus, please pull from the for-2.6.27 branch of:
git://git.infradead.org/users/dwmw2/firmware-2.6.gitfor-2.6.27
Quoted 8 lines
Quoted 5 lines
Quoted 4 lines
Quoted 3 lines
The firmware flamewars seem to have subsided lately. Is everyone happy with this now?
this seems to have left the contentious pieces out....
almost, the one thing that this could have done would be to offer an option to bundle the firmware with the module. currently it offers the option to load the firmware at the same time as the module, not _quite_ the same thing.
I see no real point in that. If you have userspace to load modules, then you have userspace to load firmware.
You have been provided with several counter-examples here.
Driver disks, initrd, and other such constructs are not necessarily the land of happy, fat and healthy userspace that you believe it universally to be. Image build scripts need to be aware that they need to copy firmware. Some already know -- but many, particularly with networking -- were such simple one-driver affairs that nobody bothered to script in the extra smarts.
Quoted 4 lines
We made 'make
modules_install' install the firmware in/lib/firmwarefor you at the same time as it installs the modules, to avoid problems if people forgot to run 'makefirmware_install'. That really ought to be enough.
And it was like pulling teeth, just get that change in, ensuring the normal build process will not suddenly produce non-working drivers... which it did for several kernel hackers. As predicted.
Quoted 3 lines
I know Jeff complained that it wasn't, and insisted that he _needed_ to be able to scp a single .ko file around which contained both, and the world was broken if he could not -- but I disagree with him.
It's simple logic: existing systems DO copy around modules. You seem to have forgotten an example not so easily derided: driver disks and their build scripts, which are used all over the place.
Particularly in cases, like enterprise distros, where you are updating a driver but never touch the main kernel image.
That will continue to be true even when enterprise distros are based off of 2.6.27 or later.
Quoted 6 lines
But since I wanted this tree to be uncontentious and obviously the correct thing to do, I've dropped the
drivers/netchanges for now anyway.It's odd that this request has suddenly come out of the blue when we've been using
request_firmware()from modules for years already.
Why is it out of the blue to worry about a working driver suddenly ceasing to work?
This has nothing to do with request_firmware() itself -- its about having the infrastructure in place so that users do not notice the switch.
Jeff
From: Jeff Garzik <[email protected]>
Date: Mon, 14 Jul 2008 22:12:24 -0400
Quoted 5 lines
Quoted 4 lines · David Woodhouse
But since I wanted this tree to be uncontentious and obviously the correct thing to do, I've dropped the
drivers/netchanges for now anyway.
Quoted 8 lines
Quoted 2 lines
It's odd that this request has suddenly come out of the blue when we've been using
request_firmware()from modules for years already.Why is it out of the blue to worry about a working driver suddenly ceasing to work?
This has nothing to do with
request_firmware()itself -- its about having the infrastructure in place so that users do not notice the switch.
And I want to reiterate my adversion to the folks who keep saying that being opposed to request_firmware() is just like being opposed to modules.
That's is very far from the truth.
When module support was added, guess what? I could still build a completely static kernel image like I always could.
And in fact, to this day, that's what I personally do because that's how I like my kernels.
But this request_firmware() change does not allow one to get what he could get before, which is a completely self-contained driver module object file.
This is the difference between providing an option and making something mandatory. This firmware split up is now mandatory.
Quoted 5 lines · David Miller
When module support was added, guess what? I could still build a completely static kernel image like I always could.
And in fact, to this day, that's what I personally do because that's how I like my kernels.
Good. You can still do precisely that, and build the firmware into your kernel. You can have exactly what you like. Hey, you can build even _more_ firmware into your kernel now. You can have NFS-root on devices you previously had to use an initrd for. hth.
Quoted 6 lines
But this
request_firmware()change does not allow one to get what he could get before, which is a completely self-contained driver module object file.This is the difference between providing an option and making something mandatory. This firmware split up is now mandatory.
In all the years we've been using request_firmware(), nobody ever asked for a way to build the firmware _into_ the .ko file, until now. Why is it suddenly so important for a small handful of older network drivers, when nobody else has ever seen the need for it -- even in modern network drivers?
Signature · 1 line
dwmw2
Quoted 11 lines · David Woodhouse
Quoted 5 lines · David Miller
When module support was added, guess what? I could still build a completely static kernel image like I always could.
And in fact, to this day, that's what I personally do because that's how I like my kernels.
Good. You can still do precisely that, and build the firmware into your kernel. You can have exactly what you like. Hey, you can build even _more_ firmware into your kernel now. You can have NFS-root on devices you previously had to use an initrd for. hth.
we recognise that a monolithic kernel can have the firmware embedded in it (and I thank you for doing so, this will help me with my Intel wireless cards). the objection is that you can't embed firmware into modules.
Quoted 4 lines
Quoted 4 lines
But this
request_firmware()change does not allow one to get what he could get before, which is a completely self-contained driver module object file.
Quoted 8 lines
Quoted 2 lines
This is the difference between providing an option and making something mandatory. This firmware split up is now mandatory.
In all the years we've been using
request_firmware(), nobody ever asked for a way to build the firmware _into_ the .ko file, until now. Why is it suddenly so important for a small handful of older network drivers, when nobody else has ever seen the need for it -- even in modern network drivers?
the reason the objections are showing up now is that the drivers that previously had the firmware embedded in them (so that nobody but the driver author needed to even know that there _was_ firmware involved) are now being converted to use request_firmware(). that change is breaking the invisability of the firmware that they have cultivated. if they had the option to keep things as they are (even while you provide the option for others to move the firmware out) they could not object like they are.
David Lang
Hi,
just noticed this thread which frightens me...
Quoted 5 lines · David Woodhouse
In all the years we've been using
request_firmware(), nobody ever asked for a way to build the firmware _into_ the .ko file, until now. Why is it suddenly so important for a small handful of older network drivers, when nobody else has ever seen the need for it -- even in modern network drivers?
Because most often, you need to bring the network driver from any location to your machine by any method *before* you can transfer anything easily. It's already painful to have to load modules from userland, please do not complicate the installation process. I can assure you that it's painful to boot a system and see that neither network nor USB work. When you find a way to pass a network driver there through any method (included writing it onto a CD), you're finally safe. If now you have to iterate because once you modprobe, you notice that *some* firmware files are missing, it'll get really really annoying.
You say that it is suddenly important, but I disagree. The only network driver I have which needs a firmware is myri10ge. And guess what ? Getting it to work on test platforms was a real hassle. Upon modprobe, the system would wait 2 minutes (I never understood why we have to wait when a file is missing, but that's another story of crappy design). From the first second, I could think "oh sh*t, the firmware again...". The I had to get the firmware files, hotplug script and also the script to configure /proc and /sys entries just to be able to load this driver. Fortunately, I could transfer these files via the other NIC which did not require a firmware to be loaded.
It was so much hard to get it to work on test platforms that I finally found how to rebuild the myri10ge driver with the firmware included in it, and now it has changed my life. Simply modprobe and that's all. Firmwares are so much annoying that I think that those should just be optional, and be built into modules *by default*. There should be a really valid reason for wanting to extract them. Upgradability is not one, because there are more frequent changes in modules than in firmwares.
Just my 2 cents,
Willy
Quoted 6 lines · Arjan van de Ven
Quoted 3 lines · On Mon, 14 Jul 2008 16:41:19 -0700 Andrew Morton <[email protected]>…
On Mon, 14 Jul 2008 16:23:26 -0700
David Woodhouse <[email protected]> wrote:
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
Linus, please pull from the for-2.6.27 branch of:
git://git.infradead.org/users/dwmw2/firmware-2.6.gitfor-2.6.27
The firmware flamewars seem to have subsided lately. Is everyone happy with this now?
this seems to have left the contentious pieces out....
Correction: It seems to have completely ignored contentious issues, which is not the same thing.
It -removes- the ability to build firmware into the kernel module -- a currently deployed, highly reliable configuration.
I'm not surprised that fact was snipped in David's submission -- wouldn't want to mention that you are unconditionally replacing a reliable method of firmware loading with a less reliable one.
Building the firmware into the kernel is far less useful than it appears: often users lack the ability to maintain their own vmlinuz kernel image build, but do build updated drivers. e.g. driver disks for Red Hat Enterprise Linux.
IMO the newly added /inability/ to build firmware into kernel modules is a clear regression.
Jeff
Quoted 3 lines · Jeff Garzik
IMO the newly added /inability/ to build firmware into kernel modules is a clear regression.
IMO you're being stupid.
How about explainign why it makes any difference what-so-ever?
If you can load the module, you can load the firmware. Claiming anything else is just _stupid_.
And if you cannot see the advantage of having _one_ interface to firmware loading, instead of each driver picking a random approach, I don't know what to say.
The fact that some people have been confused and apparently blathered about memory savings is totally immaterial. Ignore the stupid people.
Linus
From: Linus Torvalds <[email protected]>
Date: Mon, 14 Jul 2008 19:13:16 -0700 (PDT)
Quoted 8 lines · Jeff Garzik
Quoted 3 lines
IMO the newly added /inability/ to build firmware into kernel modules is a clear regression.
IMO you're being stupid.
How about explainign why it makes any difference what-so-ever?
Just like Jeff, I think it does.
Quoted 2 lines
If you can load the module, you can load the firmware. Claiming anything else is just _stupid_.
It's about having one foo.ko file that is self contained.
We've always been able to do that.
Just like how when module support was added to the kernel, and we could still build everything statically into one kernel image if we wanted to, we likewise should still be able to get everything necessary for a driver module into a single foo.ko file.
Quoted 9 lines · Linus Torvalds
Quoted 2 lines · Jeff Garzik
IMO the newly added /inability/ to build firmware into kernel modules is a clear regression.
IMO you're being stupid.
How about explainign why it makes any difference what-so-ever?
If you can load the module, you can load the firmware. Claiming anything else is just _stupid_.
False -- for every router build script, driver disk build script, etc. that has not yet been updated to copy both a module and the /lib/firmware that the kernel tree is newly spitting out.
And for every case, the breakage is completely silent, until one boots into a firmware-less image and loads drivers.
Quoted 1 line
And if you cannot see the advantage of having _one_ interface to firmware loading, instead of each driver picking a random approach, I don't know what to say.
It's a great approach! As I've said over and over and over (and now over) again.
My complaints are about --not breaking stuff--, not request_firmware().
You have to look at the build process, second stage image builders, embedded system image creators, and other "hangers on" that do not look and behave like a mainstream Linux distro.
It is a mistake to assume that all systems are _already_ prepared to install and digest new /lib/firmware files that the kernel build now spits out.
And the consequences of that assumption is a non-working driver, often a non-booting system, not just a few quirks here and there.
Jeff
Quoted 2 lines · Jeff Garzik
My complaints are about --not breaking stuff--, not
request_firmware().
Guys, request_firmware() has been around for about five years now.
If this discussion had happened in 2003, I'd agree.
As it is, I think it's time to just face it - many people do want to have a unified interface (and yes, I obviously count myself in that group), and you can't just continue do nothing and ignore that.
Your argument seems to be that you don't want to break anything, but you've also clearly not wanted to _fix_ anything for the last five years.
At some point, somebody has to just do it. And five years is _way_ long enough.
Linus
From: Linus Torvalds <[email protected]>
Date: Mon, 14 Jul 2008 19:33:25 -0700 (PDT)
Quoted 7 lines · Jeff Garzik
Quoted 2 lines
My complaints are about --not breaking stuff--, not
request_firmware().Guys,
request_firmware()has been around for about five years now.
Jeff's not against request_firmware() in an of itself.
He's against the fact that it's not possible to build a self-contained foo.ko module file any longer, and that would definitely be possible with using request_firmware().
Quoted 6 lines · David Miller
Jeff's not against
request_firmware()in an of itself.He's against the fact that it's not possible to build a self-contained foo.ko module file any longer, and that would definitely be possible with using
request_firmware().
.. but is it really that big of a deal?
If it were about not being able to build a self-contained non-modular "vmlinux", I'd be upset too.
But once you can load a module, you can load the firmware. You just have to _remember_ to move it along with the module.
I mean really, you're the one who was complaining that there was no technical reason, but now you're apparently using the "alzheimer's defense".
Besides, it's not even true that foo.ko modules are self-contained. We've for years and years (pretty much since day one) had nesting module models, where in order to load foo.ko you need to load baz.ko first. Yes, the firmware file is named differently, but it _is_ different.
Linus
Quoted 8 lines · Linus Torvalds
Quoted 5 lines · David Miller
Jeff's not against
request_firmware()in an of itself.He's against the fact that it's not possible to build a self-contained foo.ko module file any longer, and that would definitely be possible with using
request_firmware().
Quoted 5 lines
.. but is it really that big of a deal?
If it were about not being able to build a self-contained non-modular "vmlinux", I'd be upset too.
But once you can load a module, you can load the firmware. You just have to _remember_ to move it along with the module.
And failure to "remember" to update associated build image scripts silently and without complaint producing a non-working driver. For drivers that, in many cases, have worked for 5 or 10 or more years without needing additional files.
So it's not about remembering. You can't just wish new userland and build processes upon the universe.
Heck, you yourself have in the past supported building current kernels on older userlands -- this flies in the face of that.
Quoted 1 line
Besides, it's not even true that foo.ko modules are self-contained. We've for years and years (pretty much since day one) had nesting module models, where in order to load foo.ko you need to load baz.ko first. Yes, the firmware file is named differently, but it _is_ different.
It's not only named differently, it is in a markedly different directory hierarchy that isn't automatically picked up alongside *.ko.
But fundamentally, why remove a capability that works, thereby depriving system builders of the ability to _choose_ when, if ever, to prefer /lib/firmware to built-in firmware?
Let's standardize on a request_firmware() sure, one that does not remove that choice.
Jeff
Quoted 10 lines · Linus Torvalds
Quoted 6 lines · David Miller
Jeff's not against
request_firmware()in an of itself.He's against the fact that it's not possible to build a self-contained foo.ko module file any longer, and that would definitely be possible with using
request_firmware().
Quoted 7 lines
.. but is it really that big of a deal?
If it were about not being able to build a self-contained non-modular "vmlinux", I'd be upset too.
But once you can load a module, you can load the firmware. You just have to _remember_ to move it along with the module.
In order to transfer it, right now you feed it through a working device. When that device itself requires firmware to work, you will suddenly discover that it becomes harder and harder to get any communication device to work on your target system.
Then, in order to load it, you need to have a properly set up system (hotplug scripts, firmware at the proper location, etc...). This is really not convenient on embedded systems or live CD systems. In fact, among all my machines, only my notebook is currently able to load a firmware.
Also, there's a versionning problem. It's easy to rename a driver "foo.ko" => "foo-1.ko" and just load the exact file or hold several versions there by simply moving a symlink. Having to fiddle around with firmware files depending on the driver version will get much harder.
Also, when you're cleaning up a system after installation, how do you know which firmware files you need to keep and which ones you can safely remove ?
Just like Jeff and Davem, I really want to be able to continue to use my drivers like I have done since modules were introduced. Switching a single-file to a multi-file/multi-dir installation would be a huge maintenance regression IMHO.
Willy
Quoted 8 lines · Willy Tarreau
Quoted 3 lines
But once you can load a module, you can load the firmware. You just have to _remember_ to move it along with the module.
In order to transfer it, right now you feed it through a working device. When that device itself requires firmware to work, you will suddenly discover that it becomes harder and harder to get any communication device to work on your target system.
Willy, stop this blathering.
I suspect I will have to just delete this thread unread because it's so full of total crap.
This whole "working device" argument is total bullshit.
If the driver was a module before, it needed a module load to become working. If you could load the module, you could load the firmware. Thus the transfer is non-issue.
And if the driver was built-in, you can still build in the firmware.
And by proof of induction - your "now you feed it through a working device" - this also holds for that "a working device" driver. The device you load the module off has exactly the same rules: you can build in the firmware.
So it didn't get any harder at all. Except for the fact that you need to remember to install the firmware.
Which is just about the same thing as asking people to remember to do "make modules_install" to get a working system. Yes, if you have a driver as a module, you need to install that module for the device to work. Yes, it's definitely "harder", but seriously..
Linus
Quoted 28 lines · Linus Torvalds
Quoted 8 lines · Willy Tarreau
Quoted 3 lines
But once you can load a module, you can load the firmware. You just have to _remember_ to move it along with the module.
In order to transfer it, right now you feed it through a working device. When that device itself requires firmware to work, you will suddenly discover that it becomes harder and harder to get any communication device to work on your target system.
Willy, stop this blathering.
I suspect I will have to just delete this thread unread because it's so full of total crap.
This whole "working device" argument is total bullshit.
If the driver was a module before, it needed a module load to become working. If you could load the module, you could load the firmware. Thus the transfer is non-issue.
And if the driver was built-in, you can still build in the firmware.
And by proof of induction - your "now you feed it through a working device" - this also holds for that "a working device" driver. The device you load the module off has exactly the same rules: you can build in the firmware.
Quoted 7 lines
So it didn't get any harder at all. Except for the fact that you need to remember to install the firmware.
Which is just about the same thing as asking people to remember to do "make
modules_install" to get a working system. Yes, if you have a driver as a module, you need to install that module for the device to work. Yes, it's definitely "harder", but seriously..
has a standard been defined for how to maintain different versions of firmware for different versions of drivers yet? driver maintainers do not test drivers with all different firmware versions, just the one(s) that ship with that kernel version. there are also reports from maintainers that some driver/firmware combos do not have full backwards compatability so you can't just use the new firmware with the old drivers (causing unexpected glitches when switching between kernel versions)
that was one of the things that was identified in the last flamewar that was handwaved away but nothing was defined.
yes it's something that other drivers that use request_firmware() have not run into problems with, but the maintainers of drivers that have embedded the firmware into the driver up until now have stated that they see this as a problem.
it's also not that hard to do (it's just someone making a statement of what the standard will be that others can agree with), but it needs to be done.
the trivial way out is to define a firmware tree per kernel version (similar to how modules are handled), but that would offend some people (I think it was Alan Cox who was arguing in the last flamewar about how much bandwidth it would save to not distribute firmware with each kernel version), and it would definantly be a waste of disk space since the firmware changes far less frequently then the modules do.
David Lang
Quoted 3 lines · On Tue, 15 Jul 2008, [email protected] wrote:
has a standard been defined for how to maintain different versions of firmware for different versions of drivers yet?
No. We haven't had that possibility, probably largely because we haven't even really had the ability (Davem and Jeff talk about "choice" and how request_firmware() supposedly removes it, but they ignore the fact that it very much cuts the other way too - request_firmware() _adds_ choice and makes it easier for people to try to upgrade firmware without upgrading the kernel).
Quoted 2 lines
that was one of the things that was identified in the last flamewar that was handwaved away but nothing was defined.
I don't think we can - or want to - define things until we've seen the problems and advantages in practice over a longer time.
Do we want a versioning scheme? Probably not - exactly because people may well want to update (or downgrade) firmware from vendors independently of the kernel.
But maybe people will eventually want some kind of blacklisting scheme where the driver can say things like "I know I won't work with versions that match this checksum" and cause a nice big warning at load-time.
And I think we can all _imagine_ scenarios in all kinds of possible ways for things to work better or fail spectaclarly, and even come up with anecdotal evidence where scheme X (out of any number of random variations) is the "only possible one to work". I'd argue that we simply don't have any realistic way to decide which ones make sense.
For example, we can be in the situation where we have some case where "only this particular firmware is known to work, and others are known to break", and people can still take two totally different mental approaches to the problem.
One approach is the "strict control one", where people argue that the above problem means that we absolutely *must* have very strict versioning, and we just always pair things up exactly.
But the other argument is that firmware is part of hardware, and the driver should strive to work with different versions of vendor firmware, the same way it should strive to work with different versions of chip hardware.
Which one is right? Probably neither. Sometimes one, sometimes the other, and in both cases it should probably not be black-and-white.
IOW, I suspect that in the long run we want to say
"Ok, there's no way we can support _all_ firmware versions, but on the
other hand, we also don't want to force particular versions and not let people experiment.
So drivers should strive to be flexible, and driver writers should not always just say 'there is only one "blessed" firmware', but on the other hand we cannot require drivers to work around horrible known bugs in firmware when a fixed firmware is available"
but the exact details will have to depend on particular drivers.
And we should obviously have _a_ version of the firmware available with the kernel when that is possible. But I'd hate for it to be 1:1 with a particular driver version - because at that point it smells of being a single work, and if it is more than mere aggregation it's no longer viable with most of our firmware (I don't think we have source for more than one or two cases).
Linus
Quoted 1 line · Linus Torvalds
(I don't think we have source for more than one or two cases).
We don't. As part of the cleanup I've been trying to document what we know about the origins of the firmware, in the firmware/WHENCE file. Unfortunately, mostly that's just "unknown".
We have managed to dig out the original source for one or two hex arrays though.
Signature · 1 line
dwmw2
Quoted 11 lines · Linus Torvalds
Quoted 3 lines · On Tue, 15 Jul 2008, [email protected] wrote:
has a standard been defined for how to maintain different versions of firmware for different versions of drivers yet?
No. We haven't had that possibility, probably largely because we haven't even really had the ability (Davem and Jeff talk about "choice" and how
request_firmware()supposedly removes it, but they ignore the fact that it very much cuts the other way too -request_firmware()_adds_ choice and makes it easier for people to try to upgrade firmware without upgrading the kernel).
and they aren't objecting to this choice being added. they just want to keep the choice of working the other way (although they implicitly are saying that they won't be doing a lot of testing of different firmware versions becouse they do want to work the other way)
David Lang
Quoted 10 lines · Linus Torvalds
Willy, stop this blathering.
I suspect I will have to just delete this thread unread because it's so full of total crap.
This whole "working device" argument is total bullshit.
If the driver was a module before, it needed a module load to become working. If you could load the module, you could load the firmware. Thus the transfer is non-issue.
Sometimes your responses leave me speechless! It's not only a problem of transfer, but also a problem of selection of what to copy overthere. It has happened to me more than once when installing a machine to proceed this way :
- leave the data center
- go back to the office to find the driver somewhere
- burn a CD with it (I once even had to feed it through a 9600 bauds Annex port concentrator).
- go back to the DC and copy it from the CD. I must say that at this stage when you see warnings because it's not the exact same kernel version, you don't mind anymore and force the load in order to be able to stick an IP address on your machine.
Now when you load it and you see it requires a firmware, you're back to #1, without even the certainty that the firmware you get will :
a) be the right one
b) work
c) be enough
So please don't say that the transfer is a non-issue. It is what magnifies the added complexity.
Instead, I would have loved to see multiboot-compatible module loading. When your kernel does not support your fresh new RAID card and panics at boot, you simply put the driver on a floppy (or USB stick), pre-load the driver from GRUB and boot the kernel. I don't see how this would be possible anymore with separate firmwares.
Don't get me wrong: I'm not against the ability to separate firmware from modules, I'm against making that mandatory when it obviously complexifies a lot of things without any obvious advantage in return.
Quoted 4 lines
Which is just about the same thing as asking people to remember to do "make
modules_install" to get a working system. Yes, if you have a driver as a module, you need to install that module for the device to work. Yes, it's definitely "harder", but seriously..
Please don't compare an upgrade of a working desktop or developer machine with an installation from scratch of a server in an uncomfortable location. Those are two completely different things.
Willy
Quoted 2 lines · Willy Tarreau
It's not only a problem of transfer, but also a problem of selection of what to copy overthere.
This is why the MODULE_FIRMWARE() tag exists.
Most scripts tend to get this right already -- they _have_ to, since modern drivers have been using request_firmware() for some time already.
Quoted 3 lines
Please don't compare an upgrade of a working desktop or developer machine with an installation from scratch of a server in an uncomfortable location. Those are two completely different things.
If your setup doesn't cope with MODULE_FIRMWARE(), it's likely to break when you next deploy a new server anyway -- because your new server will have new hardware, and your new hardware will have a modern driver. And modern drivers use request_firmware() _anyway_.
Signature · 1 line
dwmw2
Quoted 6 lines · Linus Torvalds
Quoted 1 line · Jeff Garzik
My complaints are about --not breaking stuff--, not
request_firmware().Guys,
request_firmware()has been around for about five years now.
Quoted 7 lines
If this discussion had happened in 2003, I'd agree.
As it is, I think it's time to just face it - many people do want to have a unified interface (and yes, I obviously count myself in that group), and you can't just continue do nothing and ignore that.
Your argument seems to be that you don't want to break anything, but you've also clearly not wanted to _fix_ anything for the last five years.
At some point, somebody has to just do it. And five years is _way_ long enough.
In threads of years past, you've agreed with the plainly obvious -- built-in firmware is obviously more reliable, and is more likely to produce a working driver.
It doesn't break anything to keep this feature.
Keeping this feature doesn't prevent any other fixes.
Keeping this feature closes the window for potential regressions.
And keeping this feature means Linux users can make a _choice_, rather than forcing /lib/firmware on them.
Do you really think embedded system and micro distro authors are in 100% agreement that /lib/firmware is best for their situation? Really?
Jeff
Quoted 3 lines · Jeff Garzik
Do you really think embedded system and micro distro authors are in 100% agreement that
/lib/firmwareis best for their situation? Really?
Umm. Why are you bringing up these total red herrings?
Those embedded systems can just build everything into one image, and are much better off using no modules at all.
But how about instead of complaining, help add the infrastructure then, so that you can insert those firmware things into modules and still use just one single interface?
Move _forward_, not backwards.
Linus
Quoted 8 lines · Linus Torvalds
Quoted 3 lines · Jeff Garzik
Do you really think embedded system and micro distro authors are in 100% agreement that
/lib/firmwareis best for their situation? Really?Umm. Why are you bringing up these total red herrings?
It's not red herring, Linus. I (as a user, as an embedded system developer and as a micro distro maintainer) do really care.
Quoted 2 lines
Those embedded systems can just build everything into one image, and are much better off using no modules at all.
While it is sometimes acceptable to have everything built into one image, sometimes your embedded system is modular in that the same image works on several hardware variants. In this case, it's largely preferable to work with modules.
Quoted 3 lines
But how about instead of complaining, help add the infrastructure then, so that you can insert those firmware things into modules and still use just one single interface?
I think that would make everyone happy. It would be worth taking a look at myri10ge version 1.4.2 to see how they managed to support both mechanisms.
Quoted 1 line
Move _forward_, not backwards.
This is just a matter of point of view. From my POV, having to feed several unrelated files into a system to be able to get it to support any device is moving backwards, compared to the currently clean all-in-1 driver model.
BTW, I still did not understand what existing problem we are trying to solve by this move "forward".
Willy
Quoted 4 lines · Willy Tarreau
This is just a matter of point of view. From my POV, having to feed several unrelated files into a system to be able to get it to support any device is moving backwards, compared to the currently clean all-in-1 driver model.
So you disapprove of having to load mac80211.ko before your wireless NIC driver? Or sungem_phy.ko before sungem.ko? Or libphy.ko before various other drivers?
Or having to install /sbin/modprobe before you can load modules?
Signature · 1 line
dwmw2
Quoted 11 lines · David Woodhouse
Quoted 4 lines · Willy Tarreau
This is just a matter of point of view. From my POV, having to feed several unrelated files into a system to be able to get it to support any device is moving backwards, compared to the currently clean all-in-1 driver model.
So you disapprove of having to load mac80211.ko before your wireless NIC driver? Or sungem_phy.ko before sungem.ko? Or libphy.ko before various other drivers?
Or having to install
/sbin/modprobebefore you can load modules?
I actually have systems that don't have modprobe (Tivo series 1 boxes, they have the insmod/rmmod binary , but not modproble)
Quoted 8 lines · David Woodhouse
Quoted 4 lines · Willy Tarreau
This is just a matter of point of view. From my POV, having to feed several unrelated files into a system to be able to get it to support any device is moving backwards, compared to the currently clean all-in-1 driver model.
So you disapprove of having to load mac80211.ko before your wireless NIC driver?
for this one, I don't know, as it is typically something which might be installed without me noticing.
Quoted 1 line
Or sungem_phy.ko before sungem.ko?
that one might be more annoying, however, having the same name and being in the same directory as its brother makes it quite obvious. The ones which had already caught me were mii, crc32, etc...
Quoted 1 line
Or libphy.ko before various other drivers?
cf above.
Quoted 1 line
Or having to install
/sbin/modprobebefore you can load modules?
David, please do not exagerate. I hoped you were able to see the difference between being lazy and encountering difficulties to set up a system.
Willy
Quoted 9 lines · David Woodhouse
Quoted 4 lines · Willy Tarreau
This is just a matter of point of view. From my POV, having to feed several unrelated files into a system to be able to get it to support any device is moving backwards, compared to the currently clean all-in-1 driver model.
So you disapprove of having to load mac80211.ko before your wireless NIC driver? Or sungem_phy.ko before sungem.ko? Or libphy.ko before various other drivers?
Hey, how about a config option that turns firwmare files into stand-alone kernel modules that register the blob into the request_firmware internal list, and MODULE_FIRMWARE into a module dependency?
Signature · 4 lines
Alexandre Oliva http://www.lsd.ic.unicamp.br/~oliva/
Free Software Evangelist oliva@{lsd.ic.unicamp.br, gnu.org}
FSFLA Board Member ¡Sé Libre! => http://www.fsfla.org/
Red Hat Compiler Engineer aoliva@{redhat.com, gcc.gnu.org}Quoted 5 lines · Alexandre Oliva
Hey, how about a config option that turns firwmare files into stand-alone kernel modules that register the blob into the
request_firmwareinternal list, andMODULE_FIRMWAREinto a module dependency?
I think that should work fine from a technical angle, and I don't disagree per se, but it really is very ugly.
Wouldn't it be much simpler to just require people to have a reasonably up-to-date mkinitrd. Apparently it's not available _everywhere_, but if we check for this up-front (even if we can apparently only do so at "make install" time due to the root requirements), wouldn't that really solve the issue with much less work?
Linus
Quoted 8 lines · Linus Torvalds
Quoted 2 lines · Jeff Garzik
Do you really think embedded system and micro distro authors are in 100% agreement that
/lib/firmwareis best for their situation? Really?Umm. Why are you bringing up these total red herrings?
Those embedded systems can just build everything into one image, and are much better off using no modules at all.
Oh come on, embedded systems use modules all the time, and surely you know that. My router running OpenWRT says
root@gw:~# ipkg list | grep -c kmod 91
Therefore, simple kernel replacement can create a regression by doing
1) build 2.6.26 kernel, including all associated kernel module packages ("kmod-*")
2) install new kernel and associated kmod packages
3) watch system work as expected
4) build 2.6.27 kernel, including all associated kernel module packages
5) install new kernel and associated kmod packages
6) watch system fail
Why fail? The package file lists for kmod-* ipkg's are unaware of any firmware needs newly added in 2.6.27.
So the driver gets packaged, but ipkg build process is completely unaware of the additional firmware requirement until a manifest is updated. The normal build process appears to succeed -- yet at next boot you see that it failed.
Similar breakage for driver disks.
Similar breakage for older versions of mainstream distros (won't get copied into initrd, with obvious results).
Quoted 3 lines
But how about instead of complaining, help add the infrastructure then, so that you can insert those firmware things into modules and still use just one single interface?
Move _forward_, not backwards.
Does that mean you agree it's a regression? :) Sure, I'd be happy to help fix the damage.
I am still surprised this was merged at all in its state, and am a bit disappointed. This one
* removed choice, by removing ability to build firmware into modules
* forced a flag day build process change upon all distros/builders who switch to >= 2.6.27. no build script updates == non-working drivers.
* the legal subtext of these changes was not mentioned at all
Generally, we don't do that. Generally, we give distros the ability to choose between old-way and new-way during a time of transition. Generally we make it easy for newer kernels to work on existing systems, maximizing the [test] audience.
Jeff
Quoted 2 lines · Jeff Garzik
* forced a flag day build process change upon all distros/builders who switch to >= 2.6.27. no build script updates == non-working drivers.
Jeff, we don't usually call it a 'flag day' when it's just the normal and uninteresting case that you update the upstream version of the package, and you need minor tweaks to how you package it. That happens all the time. A general rule of thumb is to stop drinking at least 4 hours before undertaking a full version upgrade of _any_ package you maintain.
You could argue that converting a wireless driver to use mac80211 would break ipkg distributions too. And call that a 'flag day'.
But that would be silly. Wouldn't it?
Signature · 1 line
dwmw2
Quoted 10 lines · David Woodhouse
Quoted 1 line · Jeff Garzik
* forced a flag day build process change upon all distros/builders who switch to >= 2.6.27. no build script updates == non-working drivers.
Jeff, we don't usually call it a 'flag day' when it's just the normal and uninteresting case that you update the upstream version of the package, and you need minor tweaks to how you package it. That happens all the time. A general rule of thumb is to stop drinking at least 4 hours before undertaking a full version upgrade of _any_ package you maintain.
Quoted 4 lines
You could argue that converting a wireless driver to use mac80211 would break ipkg distributions too. And call that a 'flag day'.
But that would be silly. Wouldn't it?
Normally, one can upgrade the kernel, not upgrade userland, and happily continue working.
With this change, there are several exceptions where this is not the case, across all driver types -- media, wan, char, net, scsi, ...
Package manifests, build scripts and other details MUST be updated, or these drivers will not work as they did in kernels <= 2.6.26.
That is not an opinion, that is a simple fact of life with the kernel as of 50515af207d410c9f228380e529c56f43c3de0bd.
And yes, for the people who have to clean up the mess, I call that a flag day.
Jeff
Quoted 6 lines · Jeff Garzik
With this change, there are several exceptions where this is not the case, across all driver types -- media, wan, char, net, scsi, ...
Package manifests, build scripts and other details MUST be updated, or these drivers will not work as they did in kernels <= 2.6.26.
Umm. I complained six months ago when the e1000e driver took over, and the e1000 driver stopped working for me, and you suddenly had to know which version of the e1000 you had - even though there was not any sane way to tell them apart.
Ingo _still_ complains about that occasionally.
What did you do? You didn't care.
Now you suddenly do care that some driver needs additional love and learning.
I call bullshit. You don't like the change - ok. But stop making bogus arguments about it.
Linus
Quoted 5 lines · Linus Torvalds
Quoted 3 lines · Jeff Garzik
With this change, there are several exceptions where this is not the case, across all driver types -- media, wan, char, net, scsi, ...
Quoted 8 lines
Quoted 2 lines
Package manifests, build scripts and other details MUST be updated, or these drivers will not work as they did in kernels <= 2.6.26.
Umm. I complained six months ago when the e1000e driver took over, and the e1000 driver stopped working for me, and you suddenly had to know which version of the e1000 you had - even though there was not any sane way to tell them apart.
Ingo _still_ complains about that occasionally.
What did you do? You didn't care.
Completely false -- what was the outcome? The breakage complained about was fixed, we made sure Ingo had proper defconfig entries, the changes to e1000e were pushed back.
That's a lot of taking-feedback-from-others that you define as "didn't care."
Jeff
Quoted 1 line · Linus Torvalds
I call bullshit. You don't like the change - ok.
SIGH. Wrong, wrong, wrong. I have always thought request_firmware() was a good idea, even if it decreased reliability. Because we need that ability and that API.
The changes are good,
They just don't need to break stuff.
Which they do.
Quoted 1 line
But stop making bogus arguments about it.
How many more real world examples of breakage do you need?
* This breaks newer kernel / older userland
* This breaks anywhere with a file list for modules' contents (ipkg kernel module packages)
* This breaks driver disk build packages
* This breaks for the oddball, not just me, who
* This breaks for an unknown amount of people who have been assuming that their driver will always work when modprobe'd, even though they are completely unaware of its new firmware needs
All these are cases in use today. Real users, Linus. Real distros.
Jeff
P.S. And as I stated in another email, I'm happy to help fix this regression.
Quoted 6 lines · Jeff Garzik
The changes are good,
They just don't need to break stuff.
Which they do.
If you were serious about that, then you'd be suggesting _fixes_.
I'm not seeing you do that. I'm just seeing non-productive "we should never have changed" emails.
See my emails about what can be _productive_ avenues to explore. As it is, I have yet to see a _single_ productive email from you.
Really, Jeff.
Linus
Quoted 9 lines · Linus Torvalds
Quoted 5 lines · Jeff Garzik
The changes are good,
They just don't need to break stuff.
Which they do.
If you were serious about that, then you'd be suggesting _fixes_.
The fix was suggested from day one: permit firmware-in-module, which I've already said I will work on... provided its not dead on arrival.
Other fixes I've stated in these threads I want to tackle:
* newly separated firmware should live alongside the driver
* driver author should be able to deliver binary firmware direct from vendor, without any ihex or C-source-code mangling.
Quoted 1 line
I'm not seeing you do that. I'm just seeing non-productive "we should never have changed" emails.
The emails are "this change should not contain obvious breakages"
To review the parts you've apparently missed, the problems and regressions I have found during review were
- Kconfig produced non-working drivers. 10+ skilled kernel hackers hit this. I complained, was called silly, pointless, and having a tantrum. Then, the issue was fixed, and kernel hackers could boot again.
- the normal build process produced non-working drivers. A couple more skilled kernel hackers hit this one. Build process regression was OBVIOUS. Again, fixing this was called pointless and silly. Again, it got fixed. Again, kernel hackers started working again.
Now we have one last major regression hole to fix -- firmware-in-module.
Quoted 1 line
See my emails about what can be _productive_ avenues to explore. As it is, I have yet to see a _single_ productive email from you.
Strange, considering that it has been me who pointed out problems that would -- and did -- hit other kernel hackers, and those problems after much complaining and insulting got fixed.
Jeff
Quoted 8 lines · David Woodhouse
Quoted 3 lines · Jeff Garzik
* forced a flag day build process change upon all distros/builders who switch to >= 2.6.27. no build script updates == non-working drivers.
Jeff, we don't usually call it a 'flag day' when it's just the normal and uninteresting case that you update the upstream version of the package, and you need minor tweaks to how you package it. [...]
well, i have no opinion one way or another about the firmware issue, but breaking the build and breaking previously working setups is not a "minor tweak".
Quoted 1 line
[...] That happens all the time. [...]
No, you are quite wrong, it does not happen all the time.
It happens occasionally out of 10,000 changes per kernel cycle. It happens unintentionally most of the time and it's called a regression.
The difference here is that you seem to have broken my setup fully intentionally.
I have no problem with going forward, but i have a problem with breaking working setups and i have a problem with forced migration to something that doesnt work by default.
If there's a fix i can apply and test i'm a happy camper, but all this arguing in these threads with zero patches posted makes me quite nervous.
Ingo
Quoted 3 lines · Ingo Molnar
If there's a fix i can apply and test i'm a happy camper, but all this arguing in these threads with zero patches posted makes me quite nervous.
ah, Yinghai forwarded me the fix below - i am a happy camper :-)
btw.:
| from the old tree to omit the drivers/net patches. Again, it didn't | show up in my own testing because I already _had_ the file in | question.
6 out of 7 test-distros i have didnt have the tg3.bin file.
Ingo
---------- Forwarded message ----------
From: David Woodhouse <[email protected]>
Date: Mon, Jul 14, 2008 at 6:13 PM
Subject: Re: [GIT *] Allow request_firmware() to be satisfied from in-kernel, use it in more drivers.
To: Yinghai Lu <[email protected]>
Cc: Andrew Morton <[email protected]>, [email protected], [email protected], [email protected]
Quoted 4 lines · Yinghai Lu
MK_FWfirmware/tigon/tg3.bin.gen.S
make[1]: *** No rule to make targetfirmware/tigon/tg3.bin', needed byfirmware/tigon/tg3.bin.gen.o'. Stop.
make: *** [firmware] Error 2
I apologise; a line intended for the tg3 conversion accidentally slipped into firmware/Makefile along with the dabusb patch, when I cherry-picked from the old tree to omit the drivers/net patches. Again, it didn't show up in my own testing because I already _had_ the file in question.
(Sorry, Linus. This and the previous fix are now in the for-2.6.27 branch of the same repo as before:
git://git.infradead.org/users/dwmw2/firmware-2.6.git for-2.6.27
)
----
Subject: Fix accidental reference to tg3 firmware
We're not updating the tg3 driver to use request_firmware() yet, but a reference to its firmware accidentally slipped in as part of commit c4667746 ("dabusb: use request_firmware()"). Remove it again.
firmware/Makefile +0 -0
@@ -34,8 +34,6 @@ fw-shipped-$(CONFIG_SND_SB16_CSP) +=
sb16/mulaw_main.csp sb16/alaw_main.csp \
sb16/ima_adpcm_capture.csp
fw-shipped-$(CONFIG_SND_YMFPCI) += yamaha/ds1_ctrl.fw yamaha/ds1_dsp.fw \
yamaha/ds1e_ctrl.fw
-fw-shipped-$(CONFIG_TIGON3) += tigon/tg3.bin tigon/tg3_tso.bin \
- tigon/tg3_tso5.bin
fw-shipped-$(CONFIG_USB_DABUSB) += dabusb/firmware.fw dabusb/bitstream.bin fw-shipped-$(CONFIG_USB_EMI26) += emi26/loader.fw emi26/firmware.fw \
emi26/bitstream.fw
--
dwmw2
Quoted 1 line
* the legal subtext of these changes was not mentioned at all
The only legal subtext has been some mad fantasy proposed by Dave Miller that David Woodhouse current employer and previous one were both in some conspiracy to push firmware elsewhere. It would be very helpful if you stuck to the real world.
The only legal agenda people I know actually don't like request_firmware because it makes it easier to have non-free firmware files. In their strange world they'd rather delete the driver
From: Alan Cox <[email protected]>
Date: Tue, 15 Jul 2008 13:42:46 +0100
Quoted 6 lines
Quoted 1 line
* the legal subtext of these changes was not mentioned at all
The only legal subtext has been some mad fantasy proposed by Dave Miller that David Woodhouse current employer and previous one were both in some conspiracy to push firmware elsewhere. It would be very helpful if you stuck to the real world.
That was wrong, but there have been other legal subtexts.
request_firmware() itself, although partially done for technical reasons, had some very real legal impetus.
Linus has even said that in this thread.
Quoted 7 lines · Jeff Garzik
You have to look at the build process, second stage image builders, embedded system image creators, and other "hangers on" that do not look and behave like a mainstream Linux distro.
It is a mistake to assume that all systems are _already_ prepared to install and digest new
/lib/firmwarefiles that the kernel build now spits out.
I didn't just assume it. I checked the major distributions -- and they _do_ check the MODULE_FIRMWARE() tags in the modules they include, and include the corresponding firmware. The firmware loader has been around for a _long_ time now, and they've had to support this for almost as long.
Yes, there are occasional bugs -- and I'm sure there _are_ some minor distributions or 'image creators' out there which still don't cope.
But that isn't a new problem -- and it's certainly not _caused_ by us updating a small handful of the older drivers to match what modern drivers do.
Signature · 1 line
dwmw2
Quoted 9 lines · David Woodhouse
Quoted 7 lines · Jeff Garzik
You have to look at the build process, second stage image builders, embedded system image creators, and other "hangers on" that do not look and behave like a mainstream Linux distro.
It is a mistake to assume that all systems are _already_ prepared to install and digest new
/lib/firmwarefiles that the kernel build now spits out.
Quoted 5 lines
I didn't just assume it. I checked the major distributions -- and they _do_ check the
MODULE_FIRMWARE()tags in the modules they include, and include the corresponding firmware. The firmware loader has been around for a _long_ time now, and they've had to support this for almost as long.
David, "being around" and "being heavily used" are not the same. My experience with firmware is limited to WiFi drivers. I've never seen that annoying thing on a server yet. Not that it does not exist there, just that fortunately I've not being facing it.
Quoted 6 lines
Yes, there are occasional bugs -- and I'm sure there _are_ some minor distributions or 'image creators' out there which still don't cope.
But that isn't a new problem -- and it's certainly not _caused_ by us updating a small handful of the older drivers to match what modern drivers do.
Pardon? If by "updating" you mean you keep backwards compatibility, OK. But if by "updating" you mean "breaking existing behaviour", yes, it is your change which will cause the problems.
Willy
Quoted 10 lines · Linus Torvalds
Quoted 2 lines · Jeff Garzik
IMO the newly added /inability/ to build firmware into kernel modules is a clear regression.
IMO you're being stupid.
How about explainign why it makes any difference what-so-ever?
If you can load the module, you can load the firmware. Claiming anything else is just _stupid_.
Sure, in theory it's that simple. Here's a concrete example that shows how things are harder in practice.
I use the 'make deb-pkg' target (from scripts/package) to build my Debian kernel packages from git. So that needs to be adapted to include /lib/firmware. No real problem so far.
So then I build 2.6.27-rc1 and install it. Great.
You release 2.6.27-rc2 and I build it. Ouch! It fails to install, at least if I want to install it _alongside_ 2.6.27-rc1 or other kernels (which I do!). Why does it fail? Because dpkg's package management does not allow one package to overwrite files already "owned" by another package.
So, how is this solved by Debian for already existing firmware packages? Basically by making a separate package for each firmware file (or driver). This works because there are not too many of them, but having a huge number of tiny packages is a nightmare by itself.
But anyway, the dep-pkg target will have to be made smarter than it is now if it's to deal with this [1]. And at least currently it is broken.
If I were able to compile firmware into the modules, the problem would be solved in one go.
I don't know how the Debian kernel team plans to deal with this for distro kernel packages. They probably _do_ want to keep them separate [2]. Maybe by grouping firmware for really common drivers into firmware-basic-drivers or something along those lines.
Cheers,
FJP
[1] Only quick solution I see is to have it install the firmware in a versioned directory and have the postinst copy things from there to /lib/firmware.
[2] As one of the developers for Debian Installer I'm not looking forward to the complications that is going to cause for us and users.
Quoted 19 lines · On Tue, 15 Jul 2008 17:57:09 +0200 Frans Pop <[email protected]> wrote:
Quoted 10 lines · Linus Torvalds
Quoted 2 lines · Jeff Garzik
IMO the newly added /inability/ to build firmware into kernel modules is a clear regression.
IMO you're being stupid.
How about explainign why it makes any difference what-so-ever?
If you can load the module, you can load the firmware. Claiming anything else is just _stupid_.
Sure, in theory it's that simple. Here's a concrete example that shows how things are harder in practice.
I use the 'make deb-pkg' target (from
scripts/package) to build my Debian kernel packages from git. So that needs to be adapted to include/lib/firmware. No real problem so far.
Quoted 8 lines
So then I build 2.6.27-rc1 and install it. Great.
You release 2.6.27-rc2 and I build it. Ouch! It fails to install, at least if I want to install it _alongside_ 2.6.27-rc1 or other kernels (which I do!). Why does it fail? Because dpkg's package management does not allow one package to overwrite files already "owned" by another package.
question: how do you deal with this with the dozens of drivers that use request_firmware() even before yesterday ?
the answer is critical in how to deal with this "new" situation as well ;-)
Quoted 11 lines · Arjan van de Ven
Quoted 5 lines · Frans Pop
You release 2.6.27-rc2 and I build it. Ouch! It fails to install, at least if I want to install it _alongside_ 2.6.27-rc1 or other kernels (which I do!). Why does it fail? Because dpkg's package management does not allow one package to overwrite files already "owned" by another package.
question: how do you deal with this with the dozens of drivers that use
request_firmware()even before yesterday ?
the answer is critical in how to deal with this "new" situation as well ;-)
For Debian packages that was in the part of the mail you snipped.
The simple answer for my current use of dep-pkg for custom kernels is:
I don't _have_ any hardware that requires firmware that's split out with current kernels.
Maybe I'll be lucky and will find that I don't have any with 2.6.27, but I'm not betting on that.
Quoted 10 lines · you
Quoted 5 lines
You release 2.6.27-rc2 and I build it. Ouch! It fails to install, at least if I want to install it _alongside_ 2.6.27-rc1 or other kernels (which I do!). Why does it fail? Because dpkg's package management does not allow one package to overwrite files already "owned" by another package.
question: how do you deal with this with the dozens of drivers that use
request_firmware()even before yesterday ?
the answer is critical in how to deal with this "new" situation as well ;-)
For Debian packages that's in the part of the mail you snipped.
The simple answer for my current use of dep-pkg is: I don't _have_ any hardware that requires firmware that's split out with current kernels.
Quoted 12 lines · On Tue, 15 Jul 2008 18:36:56 +0200 Frans Pop <[email protected]> wrote:
Quoted 10 lines · you
Quoted 5 lines
You release 2.6.27-rc2 and I build it. Ouch! It fails to install, at least if I want to install it _alongside_ 2.6.27-rc1 or other kernels (which I do!). Why does it fail? Because dpkg's package management does not allow one package to overwrite files already "owned" by another package.
question: how do you deal with this with the dozens of drivers that use
request_firmware()even before yesterday ?
the answer is critical in how to deal with this "new" situation as well ;-)
Quoted 5 lines
For Debian packages that's in the part of the mail you snipped.
The simple answer for my current use of dep-pkg is: I don't _have_ any hardware that requires firmware that's split out with current kernels.
... and do you have any with current -git ?
Signature · 3 lines
If you want to reach me at my work email, use [email protected] For development, discussion and tips for power savings, visit http://www.lesswatts.org
Quoted 5 lines · Arjan van de Ven
Quoted 3 lines
The simple answer for my current use of dep-pkg is: I don't _have_ any hardware that requires firmware that's split out with current kernels.
... and do you have any with current -git ?
Haven't tried that yet. Only just finished building .26 for my boxes and have other priorities ATM anyway.
You'll hear about it on the list when I do :-)
Quoted 5 lines · Frans Pop
So, how is this solved by Debian for already existing firmware packages? Basically by making a separate package for each firmware file (or driver). This works because there are not too many of them, but having a huge number of tiny packages is a nightmare by itself.
Why don't you just take the kernel-supplied firmware and make it part of the kernel package? The same way the kernel-supplied modules are part of it?
You're sounding like
"Mommy, mommy, it hurts when I repeatedly hit my head against the wall
and stick forks in my eyes!"
and quite frankly, it doesn't make me go "Oh, poor baby" at all.
Linus
Quoted 10 lines · Linus Torvalds
Quoted 5 lines · Frans Pop
So, how is this solved by Debian for already existing firmware packages? Basically by making a separate package for each firmware file (or driver). This works because there are not too many of them, but having a huge number of tiny packages is a nightmare by itself.
Why don't you just take the kernel-supplied firmware and make it part of the kernel package? The same way the kernel-supplied modules are part of it?
David W has made it very clear that the eventual intent of this patch series is to remove all firmware from the kernel tree and have it shipped as a seperate package.
David Lang
Quoted 14 lines
You're sounding like
"Mommy, mommy, it hurts when I repeatedly hit my head against the wall
and stick forks in my eyes!"and quite frankly, it doesn't make me go "Oh, poor baby" at all.
Linus
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 4 lines · On Tue, 15 Jul 2008, [email protected] wrote:
David W has made it very clear that the eventual intent of this patch series is to remove all firmware from the kernel tree and have it shipped as a seperate package.
I don't think that kind of black-and-white model is necessarily attainable. Nor do I think it's even _good_.
It's probably the right thign for _some_ firmware, assuming we have models that just make it easy enough - for drivers where it's generally better for distros to get the firmware from vendors rather than the kernel.
But there is certainly nothing inherently good in forcing a split.
I think both sides here need to calm down and stop being so extreme.
I do want request_firmware() to be the way to load firmware (and possibly other configuration data too, for that matter - I don't think it's necessarily wrong for people to "misuse" it for other long-term data despite the name), and I do want people to be able to update firmware easily without having to force a kernel bump, but I also don't think it means that firmware has to go away from the kernel sources.
So I think the _infrastructure_ is important. But the extreme "you have to split it off entirely" mindset is equally as stupid as the "you must compile it into the module" mindset.
Linus
Quoted 3 lines · you
So I think the _infrastructure_ is important. But the extreme "you have to split it off entirely" mindset is equally as stupid as the "you must compile it into the module" mindset.
Agreed. All I've wanted to say is that having the _option_ to continue to compile it into the module can simplify certain use cases, including my current use of the kernel's own deb-pkg target for private builds of the kernel for testing purposes.
One thing I would dearly like to see is the ability to choose a different location for firmware for my own custom built kernels than the location used for firmware from official Debian kernel packages so that at least those two don't clash.
Doesn't look like that is possible with current git head.
Quoted 3 lines · Linus Torvalds
So I think the _infrastructure_ is important. But the extreme "you have to split it off entirely" mindset is equally as stupid as the "you must compile it into the module" mindset.
I wouldn't necessarily argue that we _have_ to split it off, but I'd certainly like our infrastructure to be mature enough that it could cope with doing so.
I plan to make a 'linux-firmware-from-linus' git tree which is generated automatically from yours by a script, much like the kernel-headers git tree is, commit by commit with an automatic transform on the tree objects.
Once we have that tree being created, we could pull from it into a canonical 'linux-firmware' tree, and _that_ tree could also start to include firmware belonging to companies who have said that we may distribute their firmware, but _not_ as part of the Linux kernel.
We can see how that tree works out in practice, and how/whether people start using it. And then maybe we'll be in a position to have a _discussion_ about how best to ship the firmware. But for now it's premature.
Signature · 1 line
dwmw2
Hi Linus,
Quoted 13 lines
Quoted 3 lines
David W has made it very clear that the eventual intent of this patch series is to remove all firmware from the kernel tree and have it shipped as a seperate package.
I don't think that kind of black-and-white model is necessarily attainable. Nor do I think it's even _good_.
It's probably the right thign for _some_ firmware, assuming we have models that just make it easy enough - for drivers where it's generally better for distros to get the firmware from vendors rather than the kernel.
But there is certainly nothing inherently good in forcing a split.
Quoted 8 lines
I think both sides here need to calm down and stop being so extreme.
I do want
request_firmware()to be the way to load firmware (and possibly other configuration data too, for that matter - I don't think it's necessarily wrong for people to "misuse" it for other long-term data despite the name), and I do want people to be able to update firmware easily without having to force a kernel bump, but I also don't think it means that firmware has to go away from the kernel sources.
actually the PCMCIA subsystem is already using request_firmware() to load external CIS data to overwrite broken ones.
Regards
Marcel
Quoted 4 lines · Linus Torvalds
Why don't you just take the kernel-supplied firmware and make it part of the kernel package? The same way the kernel-supplied modules are part of it?
And, btw, if there is something we can do to help, we obviously should.
So ask yourself (and DavidW) things like
- Is "make firmware_install" not enough for the package build system?
- Does it need some other way to figure out what firmware is needed for just the configured set of drivers, eg "make firmware_list"?
- Do we need to make the firmware loader have multiple directories it looks at? Do we want first a kernel-version-specific one, then a generic fallback one, so that we can easily have per-kernel firmware, the same way we have per-kernel modules?
- Add your interesting question here.
There are obviously all these kinds of _productive_ questions you can ask. But people seem to be mostly in an unproductive "Mommy, it hurts when I do something stupid" mode.
Linus
Quoted 3 lines · Linus Torvalds
There are obviously all these kinds of _productive_ questions you can ask. But people seem to be mostly in an unproductive "Mommy, it hurts when I do something stupid" mode.
Of course, but the main reason I replied in the first place was because your "there are no real problems" attitude in this discussion was annoying me.
And even in your first reply to me you basically ignored that fact that at least one kernel packaging tool and one that is *part of the kernel tree itself* is now broken.
There *are* real problems and some of them *could* be solved by allowing firmware to be compiled into modules. Sure, other solutions are possible too and if I actually run into the problems I foresee with dpk-deb you can be certain I will try to help solve them.
But if you want to invite people to hold a productive discussion, it helps if you don't completely ignore their concerns out of hand.
Quoted 4 lines · Frans Pop
Of course, but the main reason I replied in the first place was because your "there are no real problems" attitude in this discussion was annoying me.
I really don't think there are any real problems.
The problems are all due to inertia - people (and scripts) are used to one way of doing things. Those aren't really technical issues, and more importantly, they are not issues that _can_ be solved any other way than just doing it, and fixing the fallout.
Trying to be "proactive" doesn't work. Waiting for people (and scripts) to get used to a new model before merging it also doesn't work. And complaining about a change that pretty obviously is needed also doesn't work.
So let's just do it. And then, when specific scripts are encountered, fix them. And yes, fixing the scripts may _well_ imply adding more support for it in the kernel. It could be either in build and installation scripts, or in udev infrastructure, or in request_firmware() capability itself.
For example, some people hate udev, and they may have some really good reason why they really don't want to rely on that in their setup. So maybe we should teach request_firmware() to try to look up (binary) firmware files directly from the filesystem before it does a udev callback (in fact, maybe it even already does - I really didn't check very closely).
None of these are "problems".
But people who don't even want to look for solutions - _that_ is a problem.
Linus
Hi Linus,
Quoted 10 lines
Quoted 3 lines
Why don't you just take the kernel-supplied firmware and make it part of the kernel package? The same way the kernel-supplied modules are part of it?
And, btw, if there is something we can do to help, we obviously should.
So ask yourself (and DavidW) things like
- Is "make
firmware_install" not enough for the package build system?
Quoted 7 lines
- Does it need some other way to figure out what firmware is needed for just the configured set of drivers, eg "make
firmware_list"?- Do we need to make the firmware loader have multiple directories it looks at? Do we want first a kernel-version-specific one, then a generic fallback one, so that we can easily have per-kernel firmware, the same way we have per-kernel modules?
as just mentioned in another post. If you look at Ubuntu, their udev helper that actually looks for the firmware and loads it into the kernel does exactly this. Historically we have also old paths since the agreement on /lib/firmware came later, but that is userspace policy anyway and not part of the kernel.
Regards
Marcel
Quoted 9 lines · you
Quoted 4 lines · Frans Pop
So, how is this solved by Debian for already existing firmware packages? Basically by making a separate package for each firmware file (or driver). This works because there are not too many of them, but having a huge number of tiny packages is a nightmare by itself.
Why don't you just take the kernel-supplied firmware and make it part of the kernel package? The same way the kernel-supplied modules are part of it?
Important note: this is not me, this is the Debian kernel team based on Debian Free Software Guidelines (DFSG). If it were purely up to me I'd be a lot more pragmatic.
1) Because Debian _wants_ them split out for DFSG compliance reasons. Most of the current firmware packages are kept in the "non-free" section of the Debian archive while the kernel itself lives in "main". As long as firmware could not be split out, the compliance problem was the source of many discussions, but for the most part ignored in practice, with the exception of a few drivers with really problematic firmware licences.
2) Because of the overwrite and version management problems. /lib/firmware as a single dumping ground for firmware for all kernel versions really sucks from that PoV. One of the huge strengths of Debian is its ability to clean up after itself when packages are removed. This is partially solved by giving each firmware file it's own package because then you can use the versioning of the firmware itself in the package versions, which allows proper file management by the packaging system. But as I said, I'm not sure that still works if their number suddenly explodes.
Quoted 6 lines · Frans Pop
So then I build 2.6.27-rc1 and install it. Great.
You release 2.6.27-rc2 and I build it. Ouch! It fails to install, at least if I want to install it _alongside_ 2.6.27-rc1 or other kernels (which I do!). Why does it fail? Because dpkg's package management does not allow one package to overwrite files already "owned" by another package.
I believe I read in the previous thread that some distros are already using /lib/firmware/<kernel version>/.
There was also the suggestion of moving the entire set of kernel-packaged firmware to /lib/modules/<kernel version>/firmware, probably while keeping /lib/firmware as a second place to look for firmware so that we don't hose any system.
Quoted 2 lines
If I were able to compile firmware into the modules, the problem would be solved in one go.
And this thread would have been shorter, even. I hope someone decides to write that support instead of complaining ;-)
But I do feel we still need a smarter userspace firmware loader to make firmware packaging less insane. The "current aproach" you described, with one firmware per package, is not good. It doesn't allow for multiple versions of the firmware to be installed should you need it.
Quoted 4 lines
I don't know how the Debian kernel team plans to deal with this for distro kernel packages. They probably _do_ want to keep them separate [2]. Maybe by grouping firmware for really common drivers into firmware-basic-drivers or something along those lines.
I sure hope we go with the more proper fix (a version-enabled firmware loader that can do the unversioned /lib/firmware as well). It is far more resilient in the long run.
Quoted 3 lines
[1] Only quick solution I see is to have it install the firmware in a versioned directory and have the postinst copy things from there to
/lib/firmware.
Yuck. That would work only if you never needed two active copies of the kernel [with different firmware files] active at the same time.
Quoted 2 lines
[2] As one of the developers for Debian Installer I'm not looking forward to the complications that is going to cause for us and users.
That was my point. These firmware loading changes are good, but there is a lot of crap missing (most of it NOT in the kernel) before it can be exposed to ordinary users.
And without firmware-in-the-module support (which is the ONLY scenario where the entire userspace will not notice anything different), this WILL cause problem to distros, we will need to scramble up and fix it all before we can package 2.6.26. I don't know if this is a big problem, though. The work will need to be done sooner or later anyway, and we should have enough time to do so as long as we don't care for packaging the early -rc.
Signature · 4 lines
"One disk to rule them all, One disk to find them. One disk to bring them all and in the darkness grind them. In the Land of Redmond where the shadows lie." -- The Silicon Valley Tarot Henrique Holschuh
Quoted 5 lines · Henrique de Moraes Holschuh
Quoted 1 line · Frans Pop
If I were able to compile firmware into the modules, the problem would be solved in one go.
And this thread would have been shorter, even. I hope someone decides to write that support instead of complaining ;-)
I've volunteered -- but if Linus and David will reject my code out of hand, there is not much point in writing it.
Quoted 5 lines
Quoted 1 line
[2] As one of the developers for Debian Installer I'm not looking forward to the complications that is going to cause for us and users.
That was my point. These firmware loading changes are good, but there is a lot of crap missing (most of it NOT in the kernel) before it can be exposed to ordinary users.
100% agreed.
That's the point I've been trying to make all along.
The end goal is good, but it's not at all ready for normal users.
Quoted 6 lines
And without firmware-in-the-module support (which is the ONLY scenario where the entire userspace will not notice anything different), this WILL cause problem to distros, we will need to scramble up and fix it all before we can package 2.6.26. I don't know if this is a big problem, though. The work will need to be done sooner or later anyway, and we should have enough time to do so as long as we don't care for packaging the early -rc.
Agreed.
Jeff
Quoted 10 lines · Jeff Garzik
Quoted 6 lines · Henrique de Moraes Holschuh
Quoted 2 lines · Frans Pop
If I were able to compile firmware into the modules, the problem would be solved in one go.
And this thread would have been shorter, even. I hope someone decides to write that support instead of complaining ;-)
I've volunteered -- but if Linus and David will reject my code out of hand, there is not much point in writing it.
Quite frankly, I don't think it's something to reject out of hand, but it does sound pretty damn stupid.
Why compile them into the module, when that's just the _inflexible_ model?
Why not just install them with the module (perhaps in the module directory, perhaps not), and just load them?
Really. You _do_ have do to "make modules_install" or similar. That has _already_ got inter-file dependencies (ie some modules simply depend on other modules). I'm not seeing why you cannot accept change.
Linus
Quoted 5 lines · Linus Torvalds
Why compile them into the module, when that's just the _inflexible_ model?
Why not just install them with the module (perhaps in the module directory, perhaps not), and just load them?
Really. You _do_ have do to "make
modules_install" or similar. That has _already_ got inter-file dependencies (ie some modules simply depend on other modules). I'm not seeing why you cannot accept change.
Even if the end goal is to support more flexible firmware loading, it would seem to make sense to allow building the firmware into the module--if only as an interim step to avoid breaking existing setups.
Chris
Quoted 7 lines · Linus Torvalds
Quoted 4 lines · Jeff Garzik
Quoted 3 lines · Henrique de Moraes Holschuh
Quoted 2 lines · Frans Pop
If I were able to compile firmware into the modules, the problem would be solved in one go.
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
And this thread would have been shorter, even. I hope someone decides to write that support instead of complaining ;-)
I've volunteered -- but if Linus and David will reject my code out of hand, there is not much point in writing it.
Quite frankly, I don't think it's something to reject out of hand, but it does sound pretty damn stupid.
Why compile them into the module, when that's just the _inflexible_ model?
Because it works today.
Because that is what people and their scripts expect, today.
Because people should not be forced to fix all their firmware-related breakage immediately, just to boot 2.6.27.
Quoted 1 line
I'm not seeing why you cannot accept change.
It is highly feasible to do the change without breaking stuff and removing choice.
Beer can be great tasting AND less filling.
Jeff
Quoted 3 lines · Jeff Garzik
Because people should not be forced to fix all their firmware-related breakage immediately, just to boot 2.6.27.
You're continuing to make an argument that doesn't seem to backed up with any actual real problems.
Can you point to real breakage? For real people?
It sounds like your whole argument literally boils down to "one or two people doing something really stupid or odd cannot just fix their setup".
What is the real-life situation where copying the firmware with the modules (but still as separate files) actually breaks?
IOW, what _is_ this theoretical breakage? And why is it so deadly suddenly? Give us real examples that somehow cannot be fixed?
Linus
Quoted 9 lines · Linus Torvalds
Quoted 3 lines · Jeff Garzik
Because people should not be forced to fix all their firmware-related breakage immediately, just to boot 2.6.27.
You're continuing to make an argument that doesn't seem to backed up with any actual real problems.
Can you point to real breakage? For real people?
several kernel devs ran into this with -next (Jeff can name names if needed)
Quoted 8 lines
It sounds like your whole argument literally boils down to "one or two people doing something really stupid or odd cannot just fix their setup".
What is the real-life situation where copying the firmware with the modules (but still as separate files) actually breaks?
IOW, what _is_ this theoretical breakage? And why is it so deadly suddenly? Give us real examples that somehow cannot be fixed?
there are no cases that cannot somehow be fixed. the desire is to make it possible to avoid having to find and fix everything before booting 2.6.27, while still making it possible to compile a kernel with all the new capabilities for those who want to fix everything now.
David Lang
Quoted 8 lines · Linus Torvalds
Quoted 2 lines · Jeff Garzik
Because people should not be forced to fix all their firmware-related breakage immediately, just to boot 2.6.27.
You're continuing to make an argument that doesn't seem to backed up with any actual real problems.
Can you point to real breakage? For real people?
Already posted a list.
Quoted 3 lines
It sounds like your whole argument literally boils down to "one or two people doing something really stupid or odd cannot just fix their setup".
What is the real-life situation where copying the firmware with the modules (but still as separate files) actually breaks?
The breakage comes from all the existing-at-this-point-today processes that will not copy the firmware with the module.
We're not talking about one or two people, we are talking about projects being actively used.
Which in turn means not only those projects need to immediately fix stuff to make 2.6.27 work, but users who build and test kernels are left to pick up the pieces when their drivers break too.
Quoted 1 line
IOW, what _is_ this theoretical breakage? And why is it so deadly suddenly? Give us real examples that somehow cannot be fixed?
It's not theoretical, we have already run into non-working drivers due to build process changes. And those were the smart guys, the kernel hackers.
All of the examples I have listed can certainly be fixed -- well, except the "new kernel, old system" case -- sometimes easily.
* the consequences of the breakage -- 100% non-working driver, possibly unbootable system
* the likelihood of breakage -- anything that is not a recent version of a mainstream distro WILL have problems, because it simply does not know about the firmware that must follow the module whereever it goes.
* the need to immediately add/fix userland and build processes, just to keep the same driver set working.
* the need for time, to figure out if you are even affected by this change
* the need for time, to plan the best method to implement firmware deployment in your distro
Without firmware-in-module, it is a basic fact that you are forcing everyone to fix their stuff, simply to be able to use 2.6.27 like they did 2.6.26.
That's unfriendly to distros, users, and kernel testers.
Jeff
Quoted 2 lines · Jeff Garzik
Already posted a list.
No you didn't.
You posted a list of problems you have from doing odd and stupid things. Not the reports, not the explanations of what those odd and stupid things actually were in practice, and why they had to be that stupid.
Quite frankly, I suspect that it's all a matter of "people compiled their own kernels, and didn't copy the new files, because they just didn't realize they needed to".
Which we really don't even need kernel fixes fot, it just needs some education. BUT. If we knew what the details were, we could make the need for education even less, so that it would just work with whatever they do.
But yes, if it fundamentally boils down do "you had a script that copied named files by hand", then that script needs fixing. But that's somethign that somebody who compiles his own kernel (especially in an odd environment, apparently) just needs to do.
Sometimes build updates need people to fix things. We find gcc bugs, we require people to update. The whole "new kernels should work with old distributions" does not mean that people building the kernels might not have to do _something_ special.
Linus
Quoted 9 lines · Linus Torvalds
Quoted 1 line · Jeff Garzik
Already posted a list.
No you didn't.
You posted a list of problems you have from doing odd and stupid things. Not the reports, not the explanations of what those odd and stupid things actually were in practice, and why they had to be that stupid.
Quite frankly, I suspect that it's all a matter of "people compiled their own kernels, and didn't copy the new files, because they just didn't realize they needed to".
No, I spelled out a list of projects, what would break, and how it would break.
Again,
* Red Hat driver disks. The build process is unaware of the need to bundle firmware. Any use with Fedora (or, later, RHEL) will result in a non-working driver, for the simple reason that firmware is not copied onto the image.
* Embedded projects like OpenWRT, that divide up kernel modules into multiple packages. Since it's not just one big package, you have to update each package's list of files to include the newly split-out firmwares. Failure to do so results in a successful build set of packages... of drivers that lack their firmware.
* Router and embedded system image builders like tomsrtbt, with similar problem to RH driver disks: they simply do not know that a firmware needs to be copied into the image, alongside the tg3 driver.
* Older userlands where, axiomatically, the mkinitrd and other firmware fixes have not been made, and may never be made.
Yes, it is a matter of education.
Yes, it is a matter of fixes.
That does not change the fact that 2.6.27 will mean non-working drivers for these situations, and there is a simple remedy for this entire class of problems: retain ability to compile firmware into the module.
That does not change the fact that useful testers may be turned away from 2.6.2[78] while waiting for Red Hat or OpenWRT or whomever to fix their stuff.
Jeff
Quoted 9 lines · Jeff Garzik
No, I spelled out a list of projects, what would break, and how it would break.
Again,
* Red Hat driver disks. The build process is unaware of the need to bundle firmware. Any use with Fedora (or, later, RHEL) will result in a non-working driver, for the simple reason that firmware is not copied onto the image.
For old RHEL - they won't support 2.6.27, so moot. For next-gen RHEL - they need fixing anyway. For Fedora, they don't (and hopefully won't) exist.
Dave
Signature · 1 line
http://www.codemonkey.org.uk
Quoted 7 lines · Dave Jones
Quoted 5 lines · Jeff Garzik
No, I spelled out a list of projects, what would break, and how it would break.
Again,
Quoted 8 lines
Quoted 4 lines
* Red Hat driver disks. The build process is unaware of the need to bundle firmware. Any use with Fedora (or, later, RHEL) will result in a non-working driver, for the simple reason that firmware is not copied onto the image.
For old RHEL - they won't support 2.6.27, so moot. For next-gen RHEL - they need fixing anyway. For Fedora, they don't (and hopefully won't) exist.
Grin,
http://docs.fedoraproject.org/fedora-install-guide-en/fc5/sn-bootoptions-hardware.html
Or DKMS, which should package firmware:
http://linux.dell.com/projects.shtml
Quoted 13 lines · Jeff Garzik
Quoted 12 lines · Dave Jones
Quoted 9 lines · Jeff Garzik
No, I spelled out a list of projects, what would break, and how it would break.
Again,
* Red Hat driver disks. The build process is unaware of the need to bundle firmware. Any use with Fedora (or, later, RHEL) will result in a non-working driver, for the simple reason that firmware is not copied onto the image.
Quoted 6 lines
Quoted 3 lines
For old RHEL - they won't support 2.6.27, so moot. For next-gen RHEL - they need fixing anyway. For Fedora, they don't (and hopefully won't) exist.
Grin,
http://docs.fedoraproject.org/fedora-install-guide-en/fc5/sn-bootoptions-hardware.html
I don't think any of this stuff has ever been actually used in a Fedora context. We certainly haven't released any official ones ttbomk.
Quoted 2 lines
Or DKMS, which should package firmware:
http://linux.dell.com/projects.shtml
Not quite the same thing as a driver disk, but I'll humour you.. That handles MODULE_FIRMWARE fine afaik. Though again, this doesn't see much actual use in a Fedora context.
Dave
Signature · 1 line
http://www.codemonkey.org.uk
Quoted 4 lines · Jeff Garzik
* Red Hat driver disks. The build process is unaware of the need to bundle firmware. Any use with Fedora (or, later, RHEL) will result in a non-working driver, for the simple reason that firmware is not copied onto the image.
Then Red Hat driver disks are already broken, as there are current modules (e.g. qla2xxx) that require external firmware.
Quoted 5 lines
* Embedded projects like OpenWRT, that divide up kernel modules into multiple packages. Since it's not just one big package, you have to update each package's list of files to include the newly split-out firmwares. Failure to do so results in a successful build set of packages... of drivers that lack their firmware.
AFAIK OpenWRT and the like don't just throw in random new kernel releases without at least reading release notes and testing.
Signature · 3 lines
Chris Adams <[email protected]> Systems and Network Administrator - HiWAAY Internet Services I don't speak for anybody but myself - that's enough trouble.
Quoted 5 lines · Chris Adams
Quoted 1 line · Jeff Garzik
* Red Hat driver disks. The build process is unaware of the need to bundle firmware. Any use with Fedora (or, later, RHEL) will result in a non-working driver, for the simple reason that firmware is not copied onto the image.
Then Red Hat driver disks are already broken, as there are current modules (e.g. qla2xxx) that require external firmware.
That doesn't change the problem any. There are still users of this thing, regardless of what appellations are attached.
Quoted 4 lines
Quoted 1 line
* Embedded projects like OpenWRT, that divide up kernel modules into multiple packages. Since it's not just one big package, you have to update each package's list of files to include the newly split-out firmwares. Failure to do so results in a successful build set of packages... of drivers that lack their firmware.
AFAIK OpenWRT and the like don't just throw in random new kernel releases without at least reading release notes and testing.
Sure. The difference here is that we usually take care to make sure there is some way for the distributor to continue configuring their system in the manner known to work, while developing and testing new stuff.
But in this case, that option was not available to distributors (but hopefully it will be, in another day or two).
Jeff
Quoted 16 lines · Jeff Garzik
Quoted 15 lines · Linus Torvalds
Quoted 8 lines · Jeff Garzik
Quoted 5 lines · Henrique de Moraes Holschuh
Quoted 2 lines · Frans Pop
If I were able to compile firmware into the modules, the problem would be solved in one go.
And this thread would have been shorter, even. I hope someone decides to write that support instead of complaining ;-)
I've volunteered -- but if Linus and David will reject my code out of hand, there is not much point in writing it.
Quite frankly, I don't think it's something to reject out of hand, but it does sound pretty damn stupid.
Quoted 8 lines
Quoted 1 line
Why compile them into the module, when that's just the _inflexible_ model?
Because it works today.
Because that is what people and their scripts expect, today.
Because people should not be forced to fix all their firmware-related breakage immediately, just to boot 2.6.27.
On one hand, they don't -- the drivers I've updated in 2.6.27 aren't the ones they need to boot. I deliberately left drivers/net and drivers/scsi till later. I don't _expect_ any problems, but there's no rush -- we can iron out any kinks with the drivers I've done already, then merge the remaining updates into 2.6.28.
One the other hand, they _do_ need to fix such breakage immediately, if they have it -- because what I'm doing here is nothing new. Most modern drivers use request_firmware() already.
So if your userspace has this hypothetical "firmware-related breakage" of which you speak, then it already won't boot on machines with qla2xxx, which was converted to use request_firmware() in 2005. To pick but one example out of many.
Signature · 1 line
dwmw2
Quoted 2 lines · In article <[email protected]>…
Why not just install them with the module (perhaps in the module directory, perhaps not), and just load them?
I also don't get why one can't put them in a module wrapper in some OTHER dir as
__init__ fwFOO = something_huge_made_from_a_string ; EXPORT_SYMBOL(fwFOO);
and have the real driver module just reference fwFOO in its init sequence. Modprobe can deal with where to look for it. No kernel source need be tainted.
Peter
Quoted 8 lines · Henrique de Moraes Holschuh
I believe I read in the previous thread that some distros are already using /lib/firmware/<kernel version>/.
There was also the suggestion of moving the entire set of kernel-packaged firmware to /lib/modules/<kernel version>/firmware, probably while keeping
/lib/firmwareas a second place to look for firmware so that we don't hose any system.
You can do that right now; I expect Ubuntu will, since their userspace already looks like that. You just need to set $(INSTALL_FW_PATH) when you install the firmware (or modules).
The Ubuntu folks have chosen not to share their changes with upstream, though -- and if we were to _impose_ that policy, then we would be requiring changes to userspace which are not yet widespread. That's a bad idea. What we're doing right now is _entirely_ compatible with what userspace already has to deal with for modern drivers, and that's very important.
There may be some merit in Ubuntu's policy, but we need userspace to change first, long before we can change the default in the kernel.
But certainly the kernel _allows_ that, if you want it.
Signature · 1 line
dwmw2
Quoted 4 lines · David Woodhouse
Quoted 3 lines · Henrique de Moraes Holschuh
I believe I read in the previous thread that some distros are already using /lib/firmware/<kernel version>/.
Quoted 8 lines
Quoted 4 lines
There was also the suggestion of moving the entire set of kernel-packaged firmware to /lib/modules/<kernel version>/firmware, probably while keeping
/lib/firmwareas a second place to look for firmware so that we don't hose any system.You can do that right now; I expect Ubuntu will, since their userspace already looks like that. You just need to set $(INSTALL_FW_PATH) when you install the firmware (or modules).
With the disclaimer that I really haven't researched this all that much I think we'll end up finding that a multi-level model is needed.
Something like:
- /lib/firmware: intended for externally provided firmware and local overrides; try to load from there first (version checks may fail)
- /lib/modules/firmware: not sure if this level is necessary, but could be useful for e.g. firmware for out-of-tree drivers
- /lib/modules/$kver/firmware or /lib/modules/$kver/$modulepath: firmware built (and possibly packaged together with the kernel
Some scheme like this would introduce at least some sanity when it comes to keeping control of what is installed and provided (and may therefore be removed) by what and to avoid the question "how the hell did that file get there and do I still actually need it with my current kernels?".
It sounds to me like $(INSTALL_FW_PATH) is a good start, but maybe even too flexible as it does not define a standard: it allows user spaces of different distros to diverge which at some point will create its own nightmares.
It's also not flexible enough as it only supports a single location for firmware while clearly there can be different sources of firmware.
Cheers,
FJP
Hi Henrique,
Quoted 5 lines
Quoted 4 lines
You release 2.6.27-rc2 and I build it. Ouch! It fails to install, at least if I want to install it _alongside_ 2.6.27-rc1 or other kernels (which I do!). Why does it fail? Because dpkg's package management does not allow one package to overwrite files already "owned" by another package.
Quoted 7 lines
I believe I read in the previous thread that some distros are already using /lib/firmware/<kernel version>/.
There was also the suggestion of moving the entire set of kernel-packaged firmware to /lib/modules/<kernel version>/firmware, probably while keeping
/lib/firmwareas a second place to look for firmware so that we don't hose any system.
can we please leave the /lib/modules/*/firmware idea out of it. While this thread is about the kernel drivers that need firmware loading, but it is not the whole story. We do have hardware that needs a firmware loading process that takes place entirely in userspace. In some USB and UART cases we can do the firmware loading step before any kernel driver takes control of that device and currently these firmware loading helpers look into /lib/firmware/ where every firmware file should be placed.
Also the /lib/firmware directory was chosen a long time ago and you find the discussion on the linux-hotplug mailing list archive if you are interested.
That said, using /lib/firmware/uname -r/ like Ubuntu supports right now and then fall back to /lib/firmware looks like a sane approach to me. However as David mentioned, this needs patches to udev first before we go any step further.
Regards
Marcel
Hi Frans,
Quoted 41 lines
Quoted 10 lines · Jeff Garzik
Quoted 2 lines
IMO the newly added /inability/ to build firmware into kernel modules is a clear regression.
IMO you're being stupid.
How about explainign why it makes any difference what-so-ever?
If you can load the module, you can load the firmware. Claiming anything else is just _stupid_.
Sure, in theory it's that simple. Here's a concrete example that shows how things are harder in practice.
I use the 'make deb-pkg' target (from
scripts/package) to build my Debian kernel packages from git. So that needs to be adapted to include/lib/firmware. No real problem so far.So then I build 2.6.27-rc1 and install it. Great.
You release 2.6.27-rc2 and I build it. Ouch! It fails to install, at least if I want to install it _alongside_ 2.6.27-rc1 or other kernels (which I do!). Why does it fail? Because dpkg's package management does not allow one package to overwrite files already "owned" by another package.
So, how is this solved by Debian for already existing firmware packages? Basically by making a separate package for each firmware file (or driver). This works because there are not too many of them, but having a huge number of tiny packages is a nightmare by itself.
But anyway, the dep-pkg target will have to be made smarter than it is now if it's to deal with this [1]. And at least currently it is broken.
If I were able to compile firmware into the modules, the problem would be solved in one go.
I don't know how the Debian kernel team plans to deal with this for distro kernel packages. They probably _do_ want to keep them separate [2]. Maybe by grouping firmware for really common drivers into firmware-basic-drivers or something along those lines.
Quoted 6 lines
Cheers,
FJP[1] Only quick solution I see is to have it install the firmware in a versioned directory and have the postinst copy things from there to
/lib/firmware.
using /lib/firmware/uname -r/ is actually not a bad idea. You only have to fix udev to actually include this in the list of directories to look for firmware files. Also Ubuntu is already doing this.
Regards
Marcel
Quoted 4 lines · Marcel Holtmann
using
/lib/firmware/uname -r/ is actually not a bad idea. You only have to fix udev to actually include this in the list of directories to look for firmware files. Also Ubuntu is already doing this.
I really don't think we need to even "fix udev".
Why don't we just load it ourselves? Esepcially as there are probably places that try to avoid udev entirely, or at least use a very cut-down-version.
We should be fairly trivially able to be _entirely_ backwards compatible with any sane setup (not the _sane_ part! It implies that people don't copy individual modules around by hand!), with no actual breakage or need for distros to even update anything at all - just make the kernel able to look up binary blobs in the same place it installed them.
That sounds like the RightThing(tm) to do _regardless_ of any other issues, doesn't it? If the kernel installs it in some known place, why should it not just read them from that known place?
Linus
Hi Linus,
Quoted 5 lines
Quoted 3 lines
using
/lib/firmware/uname -r/ is actually not a bad idea. You only have to fix udev to actually include this in the list of directories to look for firmware files. Also Ubuntu is already doing this.I really don't think we need to even "fix udev".
fixing udev means adding an extra path to it :)
Quoted 10 lines
Why don't we just load it ourselves? Esepcially as there are probably places that try to avoid udev entirely, or at least use a very cut-down-version.
We should be fairly trivially able to be _entirely_ backwards compatible with any sane setup (not the _sane_ part! It implies that people don't copy individual modules around by hand!), with no actual breakage or need for distros to even update anything at all - just make the kernel able to look up binary blobs in the same place it installed them.
Quoted 3 lines
That sounds like the RightThing(tm) to do _regardless_ of any other issues, doesn't it? If the kernel installs it in some known place, why should it not just read them from that known place?
Personally I am against hacking file loading from within the kernel. This is why we involved udev (or hotplug in the beginning). It allows you to have userspace policy for the search paths etc.
Regards
Marcel
Quoted 4 lines · Marcel Holtmann
Personally I am against hacking file loading from within the kernel. This is why we involved udev (or hotplug in the beginning). It allows you to have userspace policy for the search paths etc.
Well, I'm personally against _forcing_ people to use udev.
Yes, desktop distros generally will do so, since there are lots of complex issues etc. But I don't think we've generally forced it on people if they don't want it (ie I haven't tried it personally since the distro I used started using udev, but I think you can still just ignore udev events and set everything up statically).
And I don't think that's wrong. Which implies that setup things should still generally at least allow us to avoid udev.
(But maybe I'm wrong, and everybody already uses udev just because they couldn't be bothered not to).
Linus
Hi Linus,
Quoted 12 lines
Quoted 3 lines
Personally I am against hacking file loading from within the kernel. This is why we involved udev (or hotplug in the beginning). It allows you to have userspace policy for the search paths etc.
Well, I'm personally against _forcing_ people to use udev.
Yes, desktop distros generally will do so, since there are lots of complex issues etc. But I don't think we've generally forced it on people if they don't want it (ie I haven't tried it personally since the distro I used started using udev, but I think you can still just ignore udev events and set everything up statically).
Quoted 5 lines
And I don't think that's wrong. Which implies that setup things should still generally at least allow us to avoid udev.
(But maybe I'm wrong, and everybody already uses udev just because they couldn't be bothered not to).
I actually think that is the case here. udev is so lightweight and really slim nowadays that not using it is a big drawback. There might be some system out there that are still using a static /dev directory, but for sure there are not many and even the embedded world doesn't really mind.
Also I think the /sbin/hotplug thing is still available (not verified this) and thus calling a firmware loading helper script with a udev replacement would be possible. Someone correct me here if I am wrong.
Regards
Marcel
Quoted 20 lines · Marcel Holtmann
Hi Linus,
Quoted 12 lines
Quoted 3 lines
Personally I am against hacking file loading from within the kernel. This is why we involved udev (or hotplug in the beginning). It allows you to have userspace policy for the search paths etc.
Well, I'm personally against _forcing_ people to use udev.
Yes, desktop distros generally will do so, since there are lots of complex issues etc. But I don't think we've generally forced it on people if they don't want it (ie I haven't tried it personally since the distro I used started using udev, but I think you can still just ignore udev events and set everything up statically).
Quoted 5 lines
And I don't think that's wrong. Which implies that setup things should still generally at least allow us to avoid udev.
(But maybe I'm wrong, and everybody already uses udev just because they couldn't be bothered not to).
Quoted 6 lines
I actually think that is the case here. udev is so lightweight and really slim nowadays that not using it is a big drawback. There might be some system out there that are still using a static /dev directory, but for sure there are not many and even the embedded world doesn't really mind.
...
Depends on which embedded systems you look at.
For MID devices that's definitely true, but there are also today still cases where customers want fast startup times on < 200 MHz embedded devices, and when every second counts you don't want to add a "Waiting for /dev to be fully populated" from udev to the bootup.
(And on my computer (whose Linux installation predates the inclusion of
devfsd (sic) into the kernel) the static /dev also still works fine.)
Quoted 3 lines
Regards
Marcel
cu
Adrian
Signature · 4 lines
"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
"Only a promise," Lao Er said.
Pearl S. Buck - Dragon SeedQuoted 3 lines · Linus Torvalds
[..] just make the kernel able to look up binary blobs in the same place it installed them.
Obviously, we'd want to make it dynamically configurable, ie the same way we specify where to find 'modprobe', let people specify (in /proc/kernel/firmware-dir or something) where to find the firmware. Just default it to something sane (that obviously matches where we installed them).
I'm not a huge udevd fan, obviously. I think we can short-circuit this all in the name of expediency, with udevd as a fallback, not as the only way.
Linus
Quoted 19 lines · Linus Torvalds
Quoted 4 lines · Marcel Holtmann
using
/lib/firmware/uname -r/ is actually not a bad idea. You only have to fix udev to actually include this in the list of directories to look for firmware files. Also Ubuntu is already doing this.I really don't think we need to even "fix udev".
Why don't we just load it ourselves? Esepcially as there are probably places that try to avoid udev entirely, or at least use a very cut-down-version.
We should be fairly trivially able to be _entirely_ backwards compatible with any sane setup (not the _sane_ part! It implies that people don't copy individual modules around by hand!), with no actual breakage or need for distros to even update anything at all - just make the kernel able to look up binary blobs in the same place it installed them.
Quoted 3 lines
That sounds like the RightThing(tm) to do _regardless_ of any other issues, doesn't it? If the kernel installs it in some known place, why should it not just read them from that known place?
I'm unconvinced that the kernel should be setting this kind of policy.
But I suppose if you make it tunable in sysfs and just switch to calling do_filp_open() directly from firmware_class.c instead of punting to userspace, that might work.
It leaves you with less flexibility -- it would prevent stuff like the udev trick I posted a week or so back to fix the Intel microcode loader by automatically generating the needed binary blob on the fly from microcode.dat, for example.
We also have a long tradition of pointing and laughing at people who want to call open() from within the kernel. But it _could_ work, certainly.
I'm not really convinced it helps though. The script which creates an initrd still needs to look at the MODULE_FIRMWARE() tag and include the right firmware file. If that's broken, you're screwed anyway. And that was true in 2.6.25 too.
Signature · 1 line
dwmw2
Quoted 2 lines · David Woodhouse
I'm unconvinced that the kernel should be setting this kind of policy.
I do think it should probably be conditional, but I think that's true of udevd itself too, so hey, it cuts both ways.
Quoted 3 lines
But I suppose if you make it tunable in sysfs and just switch to calling
do_filp_open()directly fromfirmware_class.cinstead of punting to userspace, that might work.
Yup. And then you can disable it either statically (config option) or by writing an invalid path into /proc/sys/kernel/firmware-dir or whatever.
Or you can just decide that if you find something in the kernel-specific firmware directory, then it should always take precedence over whatever udev rules. Which sounds good to me anyway.
Maybe you really do have some very kernel-specific issue (ie you're trying a new driver that can handle a new experimental firmware, but you don't want your old fall-back kernels to use it because you just fixed the bug that makes it work again).
Requiring you to write udevd scripts for that sounds insane. I wouldn't even know where to start.
So kernel-specific directories do make sense. As does the whole "I don't want to handle the pain that is udev scripts".
Linus
Hi Linus,
Quoted 5 lines
Requiring you to write udevd scripts for that sounds insane. I wouldn't even know where to start.
So kernel-specific directories do make sense. As does the whole "I don't want to handle the pain that is udev scripts".
you don't have to. We extend udev once and then it will always work. As mentioned before, Ubuntu is already doing this. They never bothered to submit this upstream and I didn't really care since I have all my firmware files in /lib/firmware and had no need to add versions to it.
Regards
Marcel
Quoted 2 lines · Marcel Holtmann
you don't have to. We extend udev once and then it will always work.
Umm. The thing is, people running new kernels with old user land is not just supposed to work, it's _really_ supposed to work.
It's what I do. Something that breaks that has to have damn good reasons to break it.
So I do not disagree with Jeff on that point _at_all_. I'm in violent agreement with Jeff on the fact that we should not require system updates for the kernel to do the right thing.
The thing I disagree with Jeff on is that he then seems to turn that into something very negative ("let's not separate the firmware at all").
And I'd much rather just fix it. And that means that if people can point to udevd's that get confused - or lack of udevd's entirely - both of which sound very likely to me, then we should have a graceful fallback position.
And just supporting the notion of loading the firmware directly sounds like an obvious such case. It may not be the _only_ solution, for example, which is why I'd actually like to see people point to the _actual_ reported problems.
IOW, the problems shouldn't a "don't do this" thing. They should be a "ok, that problem happened, we can solve it by doing X".
Linus
Hi Linus,
Quoted 7 lines
Quoted 1 line
you don't have to. We extend udev once and then it will always work.
Umm. The thing is, people running new kernels with old user land is not just supposed to work, it's _really_ supposed to work.
It's what I do. Something that breaks that has to have damn good reasons to break it.
so request_firmware() is well established and works. It is used by a bunch of drivers at the moment.
If we do wanna make more heavy use of it (which seems everybody is in agreement) then we need to make some userspace changes like adding the kernel version to keep it sane. This can be done step by step to make sure we keep the current status quo.
And as David mentioned multiple times, he left drivers/{net,scsi} out of it for now.
Quoted 7 lines
So I do not disagree with Jeff on that point _at_all_. I'm in violent agreement with Jeff on the fact that we should not require system updates for the kernel to do the right thing.
The thing I disagree with Jeff on is that he then seems to turn that into something very negative ("let's not separate the firmware at all").
Quoted 8 lines
And I'd much rather just fix it. And that means that if people can point to udevd's that get confused - or lack of udevd's entirely - both of which sound very likely to me, then we should have a graceful fallback position.
And just supporting the notion of loading the firmware directly sounds like an obvious such case. It may not be the _only_ solution, for example, which is why I'd actually like to see people point to the _actual_ reported problems.
I really don't see that problem here since request_firmware() has been around for so long now and so many drivers are using it nowadays that I don't see your point here on how it will break.
However if you wanna make the kernel do the loading from the filesystem directly, I am actually fine with it. We were opposed to that idea, because it was always kinda hackish to have the kernel load files and it felt wrong to do it this way. Policy decisions on directory layouts belong into the userspace and not the kernel. At least that is my take on this.
Regards
Marcel
Quoted 3 lines · Marcel Holtmann
so
request_firmware()is well established and works. It is used by a bunch of drivers at the moment.
That's not the question.
The question is whether it works with the *new* users.
Which it would apparently NOT do for everybody if we put the new drivers in any other place than /lib/firmware, very much including a kernel- specific subdirectory there.
Linus
Quoted 3 lines · Marcel Holtmann
I really don't see that problem here since
request_firmware()has been around for so long now and so many drivers are using it nowadays that I don't see your point here on how it will break.
There is no fundamental breakage in the new request_firmware() system.
However, it is misleading to state
what I'm doing here is nothing new. Most modern drivers use request_firmware() already.
because we are not talking about new drivers, we are talking about existing drivers changing en masse from "not requiring firmware" state to "requiring firmware" state.
We are also talking about a totally new source of critically important data (the firmware, coming from the kernel tree), and scripts must be updated to take that into account.
Sure, in many cases the fix may even be a one-liner... but you still need that fix, in order to continue to use drivers that were working in 2.6.26.
Jeff
Quoted 2 lines · Marcel Holtmann
And as David mentioned multiple times, he left
drivers/{net,scsi}out of it for now.
True, but we obviously must consider them -- and booting-related implications -- since that is the obvious direction everybody is headed towards.
Jeff
Quoted 11 lines · Linus Torvalds
Quoted 1 line · Marcel Holtmann
you don't have to. We extend udev once and then it will always work.
Umm. The thing is, people running new kernels with old user land is not just supposed to work, it's _really_ supposed to work.
It's what I do. Something that breaks that has to have damn good reasons to break it.
So I do not disagree with Jeff on that point _at_all_. I'm in violent agreement with Jeff on the fact that we should not require system updates for the kernel to do the right thing.
The thing I disagree with Jeff on is that he then seems to turn that into something very negative ("let's not separate the firmware at all").
How many times do I have to say you misunderstand?
I'm happy with separating the firmware at the source code level. I never objected to that.
All I want is the _option_ to compile firmware into a module. We are talking about compiled output, here.
That _option_ has the desireable properties of
* working on older userland, even with DavidW's tg3 and scsi patches
* retaining an option several kernel hackers and users find useful
* permitting use of 2.6.27 without _immediately_ having to fix stuff
Quoted 5 lines
And I'd much rather just fix it. And that means that if people can point to udevd's that get confused - or lack of udevd's entirely - both of which sound very likely to me, then we should have a graceful fallback position.
And just supporting the notion of loading the firmware directly sounds like an obvious such case. It may not be the _only_ solution, for example, which is why I'd actually like to see people point to the _actual_ reported problems.
IOW, the problems shouldn't a "don't do this" thing. They should be a "ok, that problem happened, we can solve it by doing X".
Stop using strawmen, please.
I never said "don't do that". Ever.
I said "when you do that, don't remove ability to compile firmware into the driver."
And as has been stated repeatedly, the compiling firmware into the driver clearly DOES NOT exclude the more flexible, load-from-userland request_firmware() model.
To be extremely concrete, firmware-in-module is
* add Kconfig option (kernel-wide or per-driver, dunno) asking "build firmware into drivers, as before?"
* tweak build process to build firmware into foo.ko output, probably in a specially marked ELF section
* get request_firmware() to automatically notice that the MODULE_FIRMWARE() was built into this driver, and to look at the special ELF section for its data
See? That is not a "don't do that." That has never been "don't do that."
That is the one major request here, one that enables us to avoid all those potential breakages I've been listing, and the many more breakages of which I'm unaware. It enables use of older userlands.
And yes, it enables "weirdos" like me to continue building the firmware into the driver, because it has a proven track record of high reliability and simplicity.
All without stomping on the flexibility of request_firmware().
Jeff
Quoted 11 lines · Jeff Garzik
To be extremely concrete, firmware-in-module is
* add
Kconfigoption (kernel-wide or per-driver, dunno) asking "build firmware into drivers, as before?"* tweak build process to build firmware into foo.ko output, probably in a specially marked ELF section
* get
request_firmware()to automatically notice that theMODULE_FIRMWARE()was built into this driver, and to look at the special ELF section for its data
Jeff, just thinking, wouldn't it be slightly easier to move the firmware in a separate module on its own and just add a dependency, so that foo.ko automatically loads foo-fw.ko ? I know it will be slightly differente, but would not change in-site deployment workflows nor installed scripts.
Regards,
Willy
Quoted 9 lines · Willy Tarreau
Quoted 8 lines · Jeff Garzik
To be extremely concrete, firmware-in-module is
* add
Kconfigoption (kernel-wide or per-driver, dunno) asking "build firmware into drivers, as before?"* tweak build process to build firmware into foo.ko output, probably in a specially marked ELF section
Quoted 8 lines
Quoted 3 lines
* get
request_firmware()to automatically notice that theMODULE_FIRMWARE()was built into this driver, and to look at the special ELF section for its dataJeff, just thinking, wouldn't it be slightly easier to move the firmware in a separate module on its own and just add a dependency, so that foo.ko automatically loads foo-fw.ko ? I know it will be slightly differente, but would not change in-site deployment workflows nor installed scripts.
Quite true. That's definitely an option, but I feel that building the firmware into the driver module itself would be about the same level of difficulty, but carry with it additional benefits:
Users and distros can be _certain_ their driver setup will not break due to these changes, if you have Kconfig options available to reproduce exactly what 2.6.26 produced [firmware compiled into the driver itself].
If Kconfig options are set such that the outputs are the same in 2.6.26 and 2.6.27 -- vmlinuz and kernel modules -- then that should close windows of regression both real and theoretical, by producing _exactly_ the same outputs.
Jeff
Quoted 3 lines · Jeff Garzik
And yes, it enables "weirdos" like me to continue building the firmware into the driver, because it has a proven track record of high reliability and simplicity.
I'm one of those "weirdos", and I'd like to explain from my user-POV (a user who usually compiles and tests -rcs) why I prefer firmware-in-module I don't find it "weird".
I use modules as much as I can. Apparently many people here uses non-modular .configs. Me, I learnt lot of time ago that we live in a hotplugged world were you can't predict what hardware you are going to need to plug, and you don't want to tell your girlfriend that you need to compile a "kernel module" because she brought a USB memory stick and you didn't compiled the support when you compiled your kernel. And I don't like doing "lspci" every time I want to know what network card I have in a box. So I compile everything as a module (except the support for all sata/ide controllers and ext3, to avoid the nightmares of initrds) and I let udev do all the job. I only use external firmware when it's strictly needed for legal reasons.
Yes, just like I can do "make modules_install", I can do "make firmware_install". It's not hard - just an extra command. But it's an extra one. And I'm lazy to the point that I'm _not_ willing to type that extra command. You could install the firmware automatically with "make modules_install". Still, I'd need to cleanup frequently /lib/firmware/<version> because of all the crap (and MB) that old -rcs and -next's versions will leave there.
Those are _extra_ steps. Steps that force me to learn things that I didn't need to know before. I didn't even know if my network card had firmware or not. I never cared, and I don't want to start caring about it now.
With firmware-in-module, I can switch a simple config option that will get me back to my ignorance of firmware related things. And Jeff will be my hero of the month for bringing back the simplicity of the old times, while the rest of the people that want to force me to learn about firmware (iow, the people who thinks that firmware-in-module is worthless) remember me of the microkernel people, who could write books and give speeches in universities about how insanely great is to have fulfilled one of the most important problems in the story of CS - separating firmware from the driver code. Despite of the fact that it does not bring ANYTHING new for me, excepting added complexity.
I'll continue using external firmware only when for legal reasons it can't be redistributed, of course, but fortunately for me that's an exception on my hardware devices, not the rule.
Quoted 10 lines · Linus Torvalds
Quoted 2 lines · Marcel Holtmann
you don't have to. We extend udev once and then it will always work.
Umm. The thing is, people running new kernels with old user land is not just supposed to work, it's _really_ supposed to work.
It's what I do. Something that breaks that has to have damn good reasons to break it.
Quoted 6 lines
So I do not disagree with Jeff on that point _at_all_. I'm in violent agreement with Jeff on the fact that we should not require system updates for the kernel to do the right thing.
The thing I disagree with Jeff on is that he then seems to turn that into something very negative ("let's not separate the firmware at all").
he's not saying not to seperate the firmware at all, he's asking for the _option_ of compiling a kernel that doesn't have the firmware seperate from the modules.
a kernel compiled with this option would just drop-in to an older distro with zero impact. newer distros that have updated their userspace tools could compile with different options and have the firmware seperate.
David Lang
Quoted 3 lines · [email protected] wrote:
he's not saying not to seperate the firmware at all, he's asking for the _option_ of compiling a kernel that doesn't have the firmware seperate from the modules.
a kernel compiled with this option would just drop-in to an older distro with zero impact. newer distros that have updated their userspace tools could compile with different options and have the firmware seperate.
Correct.
Thanks :)
Jeff
Quoted 4 lines · On Tue, 15 Jul 2008, [email protected] wrote:
a kernel compiled with this option would just drop-in to an older distro with zero impact. newer distros that have updated their userspace tools could compile with different options and have the firmware seperate.
The 'zero impact' is what doesn't make sense here.
You are supposed to be able to run ol distributions, yes.
But that doesn't mean that you can necessarily just plop things in the same way as you always did before.
For example, you have to rewrite your distro's initrd if you are using modules. You cannot just re-use the modules in the distro initrd. So doing a new kernel has _never_ been 'zero impact' in the sense that you could just switch vmlinux files around.
(Btw, I personally actually want my kernel to be _truly_ zero impact, but that also means that I don't use modules - because that way I really can avoid changing even the initrd image too. But that also already works)
Why is it suddenly so important that a kernel be 'zero impact' for that module case, when it's never been zero impact for that case before? You had to rewrite the initrd to begin with, but now you're not willing to do it again, just because you have to rewrite it slightly _differently_?
THAT is what I find so odd. The inability to accept just a slight change in kernel build.
But whatever. This really isn't worth it. The request_firmware() thing will clearly happen regardless, and as long as the backwards compat code is small and Jeff writes it, what do I care? Even if I think it looks largely pointless..
Linus
Quoted 5 lines · you
Why is it suddenly so important that a kernel be 'zero impact' for that module case, when it's never been zero impact for that case before? You had to rewrite the initrd to begin with, but now you're not willing to do it again, just because you have to rewrite it slightly _differently_?
Because in most cases users do not write and maintain the scripts used to build custom kernels themselves. They use (huge and way too) tools like Debian's 'make-kpkg' for that.
Because users may want to build a custom 2.6.27 in for example a Debian Etch environment using the make-kpkg package that is available in that release of Debian. Or even a Debian Lenny environment (Lenny is about to be frozen so is very unlikely to include whatever may be needed for this).
Why is it good to extend the lifetime of the i386 and amd64 symlinks for x86 images by an extra few years so that older versions of kernel packaging tools - like make-kpkg - don't break, but is breaking the exact same tools with this firmware change suddenly OK?
Users _want_ to be able to build new kernels in older environments. Why take that option away from them? Why force them through the pain of partial upgrades of their userspace if it can so easily be avoided?
Quoted 1 line
From my PoV adding this option is a no-brainer: if Jeff is willing to do
the work, then what harm can it do? The benefits to me are obvious. I really don't get what you have against it.
Cheers,
FJP
P.S. Sorry to keep coming up with Debian examples, but that's the environment I know.
Quoted 10 lines · Linus Torvalds
Quoted 4 lines · On Tue, 15 Jul 2008, [email protected] wrote:
a kernel compiled with this option would just drop-in to an older distro with zero impact. newer distros that have updated their userspace tools could compile with different options and have the firmware seperate.
The 'zero impact' is what doesn't make sense here.
You are supposed to be able to run ol distributions, yes.
Quoted 7 lines
But that doesn't mean that you can necessarily just plop things in the same way as you always did before.
For example, you have to rewrite your distro's initrd if you are using modules. You cannot just re-use the modules in the distro initrd. So doing a new kernel has _never_ been 'zero impact' in the sense that you could just switch vmlinux files around.
this time I will call 'strawman'. we've been required to put in the appropriate modules ever since modules were introduced. as Jeff gave examples of the userspace tools have been taking care of that for many years (and on distros where they don't take care of that you go with monolithic kernels), but they don't take care of firmware in many cases.
Quoted 3 lines
(Btw, I personally actually want my kernel to be _truly_ zero impact, but that also means that I don't use modules - because that way I really can avoid changing even the initrd image too. But that also already works)
I do the same thing 99% of the time.
Quoted 4 lines
Why is it suddenly so important that a kernel be 'zero impact' for that module case, when it's never been zero impact for that case before? You had to rewrite the initrd to begin with, but now you're not willing to do it again, just because you have to rewrite it slightly _differently_?
becouse the tools that wrote the initrd already put the modules in. I don't maintain those tools, they came with the distro. we're just asking to not require those tools to be updated immediatly.
David Lang
Quoted 12 lines
THAT is what I find so odd. The inability to accept just a slight change in kernel build.
But whatever. This really isn't worth it. The
request_firmware()thing will clearly happen regardless, and as long as the backwards compat code is small and Jeff writes it, what do I care? Even if I think it looks largely pointless..Linus
Quoted 9 lines · On Tue, 2008-07-15 at 15:24 -0700, [email protected] wrote:
Quoted 5 lines
Why is it suddenly so important that a kernel be 'zero impact' for that module case, when it's never been zero impact for that case before? You had to rewrite the initrd to begin with, but now you're not willing to do it again, just because you have to rewrite it slightly _differently_?
becouse the tools that wrote the initrd already put the modules in. I don't maintain those tools, they came with the distro. we're just asking to not require those tools to be updated immediatly.
The tools that write the initrd already put the firmware in too.
Signature · 1 line
dwmw2
Quoted 7 lines · David Woodhouse
Quoted 6 lines · On Tue, 2008-07-15 at 15:24 -0700, [email protected] wrote:
Quoted 5 lines
Why is it suddenly so important that a kernel be 'zero impact' for that module case, when it's never been zero impact for that case before? You had to rewrite the initrd to begin with, but now you're not willing to do it again, just because you have to rewrite it slightly _differently_?
Quoted 5 lines
Quoted 3 lines
becouse the tools that wrote the initrd already put the modules in. I don't maintain those tools, they came with the distro. we're just asking to not require those tools to be updated immediatly.
The tools that write the initrd already put the firmware in too.
not always, as Jeff showed.
David Lang
Quoted 4 lines · On Tue, 15 Jul 2008, [email protected] wrote:
becouse the tools that wrote the initrd already put the modules in. I don't maintain those tools, they came with the distro. we're just asking to not require those tools to be updated immediatly.
But mkinitrd (which is the _only_ thing that people tend to use to write initrd's - is there even anything else) has already been doing this for years, as has been pointed out several times.
This is why I harped on the fact that you already rebuilt your initrd image.
So this really isn't a "updated immediately" issue, afaik. Googling for
mkinitrd MODULE_FIRMWARE
shows a patch from two years ago as the #1 hit in order to make the aic94xx driver work.
Loking down a bit, there's a hotplug discussion from early 2005 (gmane says "3 years, 20 weeks, 3 days, 6 hours and 39 minutes ago" in the header).
Quite frankly, if it's still a problem, there's simply something _wrong_ with the distribution. And yeah, maybe people need to update their kernel building tools.
I already pointed out how the kernel development team quite often says "we will no longer build with gcc-.xyz because it's too old and buggy". The build tools requirements are simply *different* from the runtime tools.
If it's literally just an issue of an mkinitrd that is too damn old, why don't we just make that test at kernel build time. *EXACTLY* the same way we test for compilers that are too old, and refuse to build with them?
(And no, I have no idea which version we should test for. In fact, mkinitrd seems to be singularly hard to test versions for, in that it seems to want the user to be root even just to give the version output. Ooh).
Linus
Quoted 8 lines · Linus Torvalds
Quoted 3 lines · On Tue, 15 Jul 2008, [email protected] wrote:
becouse the tools that wrote the initrd already put the modules in. I don't maintain those tools, they came with the distro. we're just asking to not require those tools to be updated immediatly.
But mkinitrd (which is the _only_ thing that people tend to use to write initrd's - is there even anything else) has already been doing this for years, as has been pointed out several times.
Debian doe not have mkinitrd but initramfs-tools [1] and yaird [2]. The status of the last is uncertain (not currently included for Lenny).
Where did you get this strange notion that mkinitrd is the _only_ (your emphasis) thing people use?
Point is that I find it really strange that - in the extremely rich and varied world that is based on Linux - you seem so certain that every tool out there already does the right thing for all cases, or can be trivially updated to do so. I just don't believe that.
I even _know_ that yaird is broken wrt. firmware [3]. It's one of the reasons it's unlikely to be included in Lenny.
But hell, I'll see if I can give this a try tomorrow. Get out of this crazy make-belief discussion and provide some facts. Who knows, maybe I'll be surprised and the existing Debian toolsets will do the right thing. I seriously doubt it though.
[1] http://packages.debian.org/sid/initramfs-tools
[2] http://packages.debian.org/sid/yaird
[3] Only works if config files are edited manually which makes it useless during e.g. new installs.
Quoted 3 lines · Frans Pop
Where did you get this strange notion that mkinitrd is the _only_ (your emphasis) thing people use?
Heh. Maybe from the fact that I don't personally use modules or initrd at all, and have no interest in doing so ;)
Ok, so people use other things. Grr. I guess we can't just require updated tools, because we don't know what the tools are.
Linus
Quoted 7 lines · Linus Torvalds
Quoted 1 line · Frans Pop
Where did you get this strange notion that mkinitrd is the _only_ (your emphasis) thing people use?
Heh. Maybe from the fact that I don't personally use modules or initrd at all, and have no interest in doing so ;)
Ok, so people use other things. Grr. I guess we can't just require updated tools, because we don't know what the tools are.
Yes, that's part of the problem :/
Everybody seems to love rolling their own stuff in this area, partly necessitated I suppose by the wide variety of ways to boot on various platforms.
Jeff
Prosaic firmware issues
Could somebody from firmware people fix boring stuff, please:
1) after allyesconfig build, git-status show plenty of crap:
# firmware/atmsar11.fw
# firmware/atmsar11.fw.gen.S
# firmware/cpia2/stv0672_vp4.bin.gen.S
# firmware/dabusb/bitstream.bin.gen.S
# firmware/dabusb/firmware.fw
# firmware/dabusb/firmware.fw.gen.S
# firmware/edgeport/boot.fw
# firmware/edgeport/boot.fw.gen.S
# firmware/edgeport/boot2.fw
# firmware/edgeport/boot2.fw.gen.S
# firmware/edgeport/down.fw
# firmware/edgeport/down.fw.gen.S
# firmware/edgeport/down2.fw
# firmware/edgeport/down2.fw.gen.S
# firmware/edgeport/down3.bin.gen.S
# firmware/emi26/bitstream.fw
# firmware/emi26/bitstream.fw.gen.S
# firmware/emi26/firmware.fw
# firmware/emi26/firmware.fw.gen.S
# firmware/emi26/loader.fw
# firmware/emi26/loader.fw.gen.S
# firmware/emi62/bitstream.fw
# firmware/emi62/bitstream.fw.gen.S
# firmware/emi62/loader.fw
# firmware/emi62/loader.fw.gen.S
# firmware/emi62/midi.fw
# firmware/emi62/midi.fw.gen.S
# firmware/emi62/spdif.fw
# firmware/emi62/spdif.fw.gen.S
# firmware/ess/maestro3_assp_kernel.fw
# firmware/ess/maestro3_assp_kernel.fw.gen.S
# firmware/ess/maestro3_assp_minisrc.fw
# firmware/ess/maestro3_assp_minisrc.fw.gen.S
# firmware/ihex2fw
# firmware/intelliport2.bin.gen.S
# firmware/kaweth/new_code.bin.gen.S
# firmware/kaweth/new_code_fix.bin.gen.S
# firmware/kaweth/trigger_code.bin.gen.S
# firmware/kaweth/trigger_code_fix.bin.gen.S
# firmware/keyspan/mpr.fw
# firmware/keyspan/mpr.fw.gen.S
# firmware/keyspan/usa18x.fw
# firmware/keyspan/usa18x.fw.gen.S
# firmware/keyspan/usa19.fw
# firmware/keyspan/usa19.fw.gen.S
# firmware/keyspan/usa19qi.fw
# firmware/keyspan/usa19qi.fw.gen.S
# firmware/keyspan/usa19qw.fw
# firmware/keyspan/usa19qw.fw.gen.S
# firmware/keyspan/usa19w.fw
# firmware/keyspan/usa19w.fw.gen.S
# firmware/keyspan/usa28.fw
# firmware/keyspan/usa28.fw.gen.S
# firmware/keyspan/usa28x.fw
# firmware/keyspan/usa28x.fw.gen.S
# firmware/keyspan/usa28xa.fw
# firmware/keyspan/usa28xa.fw.gen.S
# firmware/keyspan/usa28xb.fw
# firmware/keyspan/usa28xb.fw.gen.S
# firmware/keyspan/usa49w.fw
# firmware/keyspan/usa49w.fw.gen.S
# firmware/keyspan/usa49wlc.fw
# firmware/keyspan/usa49wlc.fw.gen.S
# firmware/keyspan_pda/keyspan_pda.fw
# firmware/keyspan_pda/keyspan_pda.fw.gen.S
# firmware/keyspan_pda/xircom_pgs.fw
# firmware/keyspan_pda/xircom_pgs.fw.gen.S
# firmware/korg/k1212.dsp
# firmware/korg/k1212.dsp.gen.S
# firmware/ti_3410.fw
# firmware/ti_3410.fw.gen.S
# firmware/ti_5052.fw
# firmware/ti_5052.fw.gen.S
# firmware/ttusb-budget/dspbootcode.bin.gen.S
# firmware/vicam/firmware.fw
# firmware/vicam/firmware.fw.gen.S
# firmware/whiteheat.fw
# firmware/whiteheat.fw.gen.S
# firmware/whiteheat_loader.fw
# firmware/whiteheat_loader.fw.gen.S
# firmware/yamaha/ds1_ctrl.fw
# firmware/yamaha/ds1_ctrl.fw.gen.S
# firmware/yamaha/ds1_dsp.fw
# firmware/yamaha/ds1_dsp.fw.gen.S
# firmware/yamaha/ds1e_ctrl.fw
# firmware/yamaha/ds1e_ctrl.fw.gen.S
2) after "make mrproper" after allyesconfig, some files aren't removed, use "git-ls-files -o" to see them.
# firmware/atmsar11.fw
# firmware/dabusb/firmware.fw
# firmware/emi26/bitstream.fw
# firmware/emi26/firmware.fw
# firmware/emi26/loader.fw
# firmware/emi62/bitstream.fw
# firmware/emi62/loader.fw
# firmware/emi62/midi.fw
# firmware/emi62/spdif.fw
# firmware/ess/maestro3_assp_kernel.fw
# firmware/ess/maestro3_assp_minisrc.fw
# firmware/korg/k1212.dsp
# firmware/yamaha/ds1_ctrl.fw
# firmware/yamaha/ds1_dsp.fw
# firmware/yamaha/ds1e_ctrl.fw
Re: Prosaic firmware issues
Quoted 4 lines · Alexey Dobriyan
Could somebody from firmware people fix boring stuff, please:
1) after allyesconfig build, git-status show plenty of crap:
Something like the appended, perhaps?
(I'm not going to commit it, I want some firmware person to double-check it).
Quoted 2 lines
2) after "make mrproper" after allyesconfig, some files aren't removed, use "git-ls-files -o" to see them.
Ok, that needs some Makefile magic. git magic I can do.
Linus
--- firmware/.gitignore | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) create mode 100644 firmware/.gitignore
firmware/.gitignore new file +2 -0
@@ -0,0 +1,2 @@ +*.gen.S +*.fw
Re: Prosaic firmware issues
Quoted 11 lines · Linus Torvalds
Quoted 4 lines · Alexey Dobriyan
Could somebody from firmware people fix boring stuff, please:
1) after allyesconfig build, git-status show plenty of crap:
Something like the appended, perhaps?
(I'm not going to commit it, I want some firmware person to double-check it).
Generally sane, but wants a few more things added.
Quoted 4 lines
Quoted 2 lines
2) after "make mrproper" after allyesconfig, some files aren't removed, use "git-ls-files -o" to see them.
Ok, that needs some
Makefilemagic. git magic I can do.
Ah, that's caused by a stray := where there should be a +=, so the firmware files listed before that get dropped from the list of files to be cleaned.
Alexey, thanks for the report. Please could you verify that this fixes it for you?
----
Subject: Fix 'make clean' and .gitignore for firmware/ directory.
Provide a .gitignore file, and fix a typo which prevented some of the generated binary files from being removed on 'make clean'.
firmware/.gitignore new file +6 -0
@@ -0,0 +1,6 @@ +*.gen.S +*.fw +*.bin +*.csp +*.dsp +ihex2fw
firmware/Makefile +1 -1
@@ -73,7 +73,7 @@ fw-shipped-$(CONFIG_USB_SERIAL_KEYSPAN_USA28X) += keyspan/usa28x.fw fw-shipped-$(CONFIG_USB_SERIAL_KEYSPAN_USA49W) += keyspan/usa49w.fw fw-shipped-$(CONFIG_USB_SERIAL_KEYSPAN_USA49WLC) += keyspan/usa49wlc.fw else -fw-shipped- := keyspan/mpr.fw keyspan/usa18x.fw keyspan/usa19.fw \ +fw-shipped- += keyspan/mpr.fw keyspan/usa18x.fw keyspan/usa19.fw \ keyspan/usa19qi.fw keyspan/usa19qw.fw keyspan/usa19w.fw \ keyspan/usa28.fw keyspan/usa28xa.fw keyspan/usa28xb.fw \ keyspan/usa28x.fw keyspan/usa49w.fw keyspan/usa49wlc.fw
Signature · 1 line
dwmw2
Re: Prosaic firmware issues
Quoted 18 lines · David Woodhouse
Quoted 11 lines · Linus Torvalds
Quoted 4 lines · Alexey Dobriyan
Could somebody from firmware people fix boring stuff, please:
1) after allyesconfig build, git-status show plenty of crap:
Something like the appended, perhaps?
(I'm not going to commit it, I want some firmware person to double-check it).
Generally sane, but wants a few more things added.
Quoted 3 lines
Quoted 2 lines
2) after "make mrproper" after allyesconfig, some files aren't removed, use "git-ls-files -o" to see them.
Quoted 8 lines
Quoted 1 line
Ok, that needs some
Makefilemagic. git magic I can do.Ah, that's caused by a stray := where there should be a +=, so the firmware files listed before that get dropped from the list of files to be cleaned.
Alexey, thanks for the report. Please could you verify that this fixes it for you?
It does, thank you.
Quoted 9 lines · Jeff Garzik
Quoted 8 lines · Linus Torvalds
Quoted 2 lines · Frans Pop
Where did you get this strange notion that mkinitrd is the _only_ (your emphasis) thing people use?
Heh. Maybe from the fact that I don't personally use modules or initrd at all, and have no interest in doing so ;)
Quoted 8 lines
Quoted 2 lines
Ok, so people use other things. Grr. I guess we can't just require updated tools, because we don't know what the tools are.
Yes, that's part of the problem :/
Everybody seems to love rolling their own stuff in this area, partly necessitated I suppose by the wide variety of ways to boot on various platforms.
Exactly. To give you an idea, I have a lot of servers running on a "firmware" which consists in two parts :
- a bzImage containing an initramfs with modules and a few scripts
- an initrd which is in fact the rootfs
When the kernel boots, it mounts the initrd, does a pivot_root and mounts its modules into /boot/$(uname -r). All my kernels run modules from /boot and not from /lib/modules, because it makes them more convenient to add and remove. So /lib/modules is just a symlink to /boot.
The above process is very convenient, as it is compatible with a lot of boot methods : hard disk, CDROM, USB stick, PXE, etc...
And moreover, I can have multiple kernels with only one rootfs (SMP, etc). Introducing firmware there would mean a major thinking and rework of the build (and possibly boot) process. The least invasive would probably be to stuff them next to the modules in $(uname -r)/firmware, but even then, it will take a lot of time to switch to a new system.
And judging from what I see around, I'm far from being the only one not using mkinitrd.
Willy
Quoted 13 lines · Willy Tarreau
Exactly. To give you an idea, I have a lot of servers running on a "firmware" which consists in two parts :
- a bzImage containing an initramfs with modules and a few scripts
- an initrd which is in fact the rootfsWhen the kernel boots, it mounts the initrd, does a
pivot_rootand mounts its modules into /boot/$(uname -r). All my kernels run modules from /boot and not from/lib/modules, because it makes them more convenient to add and remove. So/lib/modulesis just a symlink to /boot.The above process is very convenient, as it is compatible with a lot of boot methods : hard disk, CDROM, USB stick, PXE, etc...
Quoted 8 lines
And moreover, I can have multiple kernels with only one rootfs (SMP, etc). Introducing firmware there would mean a major thinking and rework of the build (and possibly boot) process. The least invasive would probably be to stuff them next to the modules in $(uname -r)/firmware, but even then, it will take a lot of time to switch to a new system.
And judging from what I see around, I'm far from being the only one not using mkinitrd.
FWIW, due to both lack of time and lack of (apparent) interest, I didn't bother moving forward with the compile-firmware-into-module, even though I still think it would be useful to add [back].
Jeff
Quoted 20 lines · Jeff Garzik
Quoted 19 lines · Willy Tarreau
Exactly. To give you an idea, I have a lot of servers running on a "firmware" which consists in two parts :
- a bzImage containing an initramfs with modules and a few scripts
- an initrd which is in fact the rootfsWhen the kernel boots, it mounts the initrd, does a
pivot_rootand mounts its modules into /boot/$(uname -r). All my kernels run modules from /boot and not from/lib/modules, because it makes them more convenient to add and remove. So/lib/modulesis just a symlink to /boot.The above process is very convenient, as it is compatible with a lot of boot methods : hard disk, CDROM, USB stick, PXE, etc...
And moreover, I can have multiple kernels with only one rootfs (SMP, etc). Introducing firmware there would mean a major thinking and rework of the build (and possibly boot) process. The least invasive would probably be to stuff them next to the modules in $(uname -r)/firmware, but even then, it will take a lot of time to switch to a new system.
Quoted 6 lines
Quoted 2 lines
And judging from what I see around, I'm far from being the only one not using mkinitrd.
FWIW, due to both lack of time and lack of (apparent) interest, I didn't bother moving forward with the compile-firmware-into-module, even though
This is what I was assuming, but thanks for clarifying this, Jeff.
Quoted 1 line
I still think it would be useful to add [back].
So do I. I think it will be easier to find time and work efficiently on the subject when directly facing an issue.
Regards,
Willy
Quoted 9 lines · Linus Torvalds
Quoted 4 lines · On Tue, 15 Jul 2008, [email protected] wrote:
becouse the tools that wrote the initrd already put the modules in. I don't maintain those tools, they came with the distro. we're just asking to not require those tools to be updated immediatly.
But mkinitrd (which is the _only_ thing that people tend to use to write initrd's - is there even anything else) has already been doing this for years, as has been pointed out several times.
Actually, there is also mkinitramfs (which is what Debian and Ubuntu use), and Yaird (Yet Another mkinitrd). Also, the distro's tend to heavily customize mkinitrd in general, so there is no guarantee that just because two distro's use some tool that happens to be named mkinitrd that the two tools have much in common with each other....
This is highly unfortunate, and I've often argued that we probably would have been better off if we had included an mkinitrd tool as part of the kernel sources to make life easier instead of having every single distro do their own thing; it would also make it easier to run a newer kernel on an older enterprise distro --- I remember the pain trying to make a 2.6.16 kernel boot on a RHEL4 base system two years ago; I don't want to even think about what would happen with something more modern at this point.
Quoted 3 lines
If it's literally just an issue of an mkinitrd that is too damn old, why don't we just make that test at kernel build time. *EXACTLY* the same way we test for compilers that are too old, and refuse to build with them?
For one reason, because there's more than one mkinitrd. FC9 ships with mkinitrd 6.0.52; OpenSuSE ships with mkinitrd 2.1, and the sources don't look even vaguely similar to one another. For example, Red Hat's mkinitrd uses nash, which is a maddingly useless shell that made it near-impossible to debug why a 2.6.16 kernel wasn't booting on a RHEL-4 system on a LS-21 blade. OpenSuSE's mkinitrd ships with bash, and Debian/Ubuntu will give you a busybox shell, which makes life *much* easier to debug when you're trying to figure out why your latest bleeding edge kernel isn't able to mount the root partition out of initrd.
- Ted
P.S. For bonus points, it would be nice if initrd's included fsck, since it then it would be possible to safely fsck the root partition and not require a reboot if the root partition was modified. But given that every single distro seems to ship their own initrd, not to mention their own init scripts, it's hard to try to push for this sort of change. The amount of distro-specific engineering work probably makes it not worth it.
Quoted 3 lines · Theodore Tso
For one reason, because there's more than one mkinitrd. FC9 ships with mkinitrd 6.0.52; OpenSuSE ships with mkinitrd 2.1, and the sources don't look even vaguely similar to one another.
Right. Other than name, they've historically shared nothing. Both have grown through the evolution of multiple distros, requiring different workarounds in each due to differences in CONFIG_ options in the kernel between vendors for eg.
Whilst it would be great for unified development on the tools that create the early boot process, I think it's a non-starter due to the fact that you can't really do it without throwing out everything you already have today. The same reason imo, that hpa's klibc work hasn't gained mass-appeal from vendors.
Even if we had a 'reference' mkinitrd in the kernel, it would be pretty much useless until it reached feature parity with every distros existing tools, and if everyone uses those instead of furthering the reference implementation, it fails on the starting blocks.
Dave
Signature · 1 line
http://www.codemonkey.org.uk
Quoted 5 lines · Dave Jones
Whilst it would be great for unified development on the tools that create the early boot process, I think it's a non-starter due to the fact that you can't really do it without throwing out everything you already have today. The same reason imo, that hpa's klibc work hasn't gained mass-appeal from vendors.
Yeah, it would have been much better if we had included a reference mkinitrd from the beginning, instead of saying, "Kernel doesn't set policy", and hoping the distro's would converge on something similar. The reality though is to the extent that we want to make changes like request_firmware() we are actually better off setting policy for stuff which is this low-level.
Quoted 5 lines
Even if we had a 'reference' mkinitrd in the kernel, it would be pretty much useless until it reached feature parity with every distros existing tools, and if everyone uses those instead of furthering the reference implementation, it fails on the starting blocks.
In order to work now it would need to have a relatively simple way for distro's to hook in their own enhancements, and someone would have to do most of the work of migrating at least the major distro's to use the new reference "official" mkinitrd. It's something we should have done from the very beginning, and it's a lot harder to fix up that mistake years later.
- Ted
Quoted 5 lines · Dave Jones
Whilst it would be great for unified development on the tools that create the early boot process, I think it's a non-starter due to the fact that you can't really do it without throwing out everything you already have today. The same reason imo, that hpa's klibc work hasn't gained mass-appeal from vendors.
Quite true and IMO disappointing, because klibc would make great infrastructure for rolling out things just like this.
Jeff
Quoted 9 lines · Jeff Garzik
Quoted 5 lines · Dave Jones
Whilst it would be great for unified development on the tools that create the early boot process, I think it's a non-starter due to the fact that you can't really do it without throwing out everything you already have today. The same reason imo, that hpa's klibc work hasn't gained mass-appeal from vendors.
Quite true and IMO disappointing, because klibc would make great infrastructure for rolling out things just like this.
Indeed, for one i've been told it would enable cross compiling of initrds, without which I deem initrd useless junk - whoever still compiles their kernel on the machine it will run on? :-)
Quoted 2 lines · Linus Torvalds
So kernel-specific directories do make sense. As does the whole "I don't want to handle the pain that is udev scripts".
I'm not violently opposed to kernel-specific directories. Ubuntu does it that way already, and it may well make sense. I'm not sure it's really necessary either, but it's an option.
Unless we really are going to make the kernel open files for itself, though, I think it's too early for us to think about making that change in the kernel. We'd need userspace to cope with it that way _first_.
(When I say 'in the kernel', I think the only place this 'policy' is yet implemented is the default for $(INSTALL_FW_PATH) in modules_install and firmware_install make targets, right?)
Signature · 1 line
dwmw2
Hi David,
Quoted 6 lines
Quoted 2 lines
So kernel-specific directories do make sense. As does the whole "I don't want to handle the pain that is udev scripts".
I'm not violently opposed to kernel-specific directories. Ubuntu does it that way already, and it may well make sense. I'm not sure it's really necessary either, but it's an option.
then lets get the udev extension merged upstream (since it doesn't do any harm anyway) and then install everything in /lib/firmware/uname -r and see how that works out.
So Ubuntu will obviously not break since it does this already. For Fedora deploying new packages for 8 and 9 should not be a problem at all once the maintainer is convinced that it is a good idea. They update packages all the time anyway. For the other distros, I don't know.
Regards
Marcel
Quoted 5 lines · Marcel Holtmann
So Ubuntu will obviously not break since it does this already. For Fedora deploying new packages for 8 and 9 should not be a problem at all once the maintainer is convinced that it is a good idea. They update packages all the time anyway. For the other distros, I don't know.
Umm. Again, Marcel. We don't require new user land to work.
But just loading the thing on our own does solve this problem. But udev we simply cannot require people to magically have a new version of.
Linus
Hi Linus,
Quoted 9 lines
Quoted 4 lines
So Ubuntu will obviously not break since it does this already. For Fedora deploying new packages for 8 and 9 should not be a problem at all once the maintainer is convinced that it is a good idea. They update packages all the time anyway. For the other distros, I don't know.
Umm. Again, Marcel. We don't require new user land to work.
But just loading the thing on our own does solve this problem. But udev we simply cannot require people to magically have a new version of.
we might have to do a step by step upgrade for this. Something like install everything in /lib/firmware/uname -r/ and create symlinks for a grace period of the next sixt month or something like this.
These are details that have be sorted out, but should not stop us to get this right.
And before you say anything. You can expect that firmware loading from /lib/firmware works almost everywhere. At least this has been for the last 3 years or so. Remember that we do have request_firmware() even in the latest 2.4 kernels.
Regards
Marcel
Quoted 7 lines · David Woodhouse
Quoted 2 lines · Linus Torvalds
So kernel-specific directories do make sense. As does the whole "I don't want to handle the pain that is udev scripts".
I'm not violently opposed to kernel-specific directories. Ubuntu does it that way already, and it may well make sense. I'm not sure it's really necessary either, but it's an option.
Some of the wireless drivers have an issue where firmware x works with driver y, so it's useful for that case even if nothing else uses it.
Dave
Signature · 1 line
http://www.codemonkey.org.uk
From: Andrew Morton <[email protected]>
Date: Mon, 14 Jul 2008 16:41:19 -0700
Quoted 8 lines
Quoted 2 lines · On Mon, 14 Jul 2008 16:23:26 -0700 David Woodhouse <[email protected]> wrote:
Linus, please pull from the for-2.6.27 branch of:
git://git.infradead.org/users/dwmw2/firmware-2.6.git for-2.6.27The firmware flamewars seem to have subsided lately. Is everyone happy with this now?
I'm not happy for network drivers, although I'm happy to see that David respected out NACKs on tg3/bnx2/etc. and didn't include those bits.
I still haven't seen an answer to the chip reset issues. The argument has been that this stuff is going to save kernel memory when the driver isn't loading firmware, but that's not really possible.
When a lot of these network drivers reset, due to a transmit timeout or other exception, we need to load the firmware again.
So this means, that if request_firmware() dies or fails for some reason in these scenerios, we can't reset the network card. Something as simple as a memory or other allocation failure deep inside of request_firmware() means we lose the networking interface.
To me this seems like a very non-resilient setup. It makes sense to just keep the firmware in-core so we can load it without having to even think about possible failures.
And that to me basically makes this transformation pointless. It's in fact a regression.
Furthermore, the issue of suspend and resume was brought up. What if request_firmware() fails then? And can request_firmware() even be allowed in that context? What if the disk on which the firmware resides hasn't been resumed yet?
In response to the suspend/resume queries, a lot of special voodoo was mentioned that perhaps would allow request_firmware() to get invoked in the correct context and with the firmware partition's block device resumed already.
But who the heck is going to write all of that code and fix up all of these drivers so that they can resume reliably again?
And for what?
None of the people who wrote and maintain these effected networking drivers want these changes installed. And they are the ones who will have to diagnose strange request_firmware() failures that lead to non-functioning devices during resume or after a transmit timeout.
Hi David,
Quoted 11 lines
I'm not happy for network drivers, although I'm happy to see that David respected out NACKs on tg3/bnx2/etc. and didn't include those bits.
I still haven't seen an answer to the chip reset issues. The argument has been that this stuff is going to save kernel memory when the driver isn't loading firmware, but that's not really possible.
When a lot of these network drivers reset, due to a transmit timeout or other exception, we need to load the firmware again.
Quoted 8 lines
So this means, that if
request_firmware()dies or fails for some reason in these scenerios, we can't reset the network card. Something as simple as a memory or other allocation failure deep inside ofrequest_firmware()means we lose the networking interface.To me this seems like a very non-resilient setup. It makes sense to just keep the firmware in-core so we can load it without having to even think about possible failures.
actually when we wrote request_firmware() a long time ago, the idea was to support various kind of setups (although until now only the on-demand got used).
So one idea was to allow minimal inclusion of firmware as binary blobs in the driver and allow overwriting them by loading newer version from the filesystem. This was intended to ensure minimal working drivers. The main background was that some devices work with a small firmware and provide basic features and only need extra firmware for advanced features. Everything suppose to be hidden behind request_firmware() and a method to register binary blobs as firmware from within a driver. So the driver developer really doesn't have to invent anything by himself.
We are not there yet and at the moment, I am not sure if this is still a valid requirement, but David's work goes into the right direction and is a good start.
Regards
Marcel
Quoted 46 lines · David Woodhouse
Linus, please pull from the for-2.6.27 branch of:
git://git.infradead.org/users/dwmw2/firmware-2.6.git for-2.6.27This does the following (not quite in this order):
- Provides a method for firmware files to be built into the kernel, so that drivers can be updated to use
request_firmware()without forcing people to use an initrd.- In doing that, mark
fw->dataas 'const', and fix a few drivers to cope.- Update a bunch of drivers to use
request_firmware(), and move their existing firmware, previously hardcoded in all kinds of interesting ways, into thefirmware/directory of the kernel tree.- Add a
FIRMWARE_IN_KERNELconfig option which allows the firmware for the above (and below) drivers to continue to be built into the kernel.- Add a simple method for representing Intel HEX records in binary form, since a bunch of drivers need that kind of thing. And a tool to convert HEX files into that binary form, for the kernel to load.
- Drop special cases from a few drivers which could either use
request_firmware()or build their firmware in; they don't need to do that for themselves any more.- Implement 'make
firmware_install' target, which installs all firmware shipped with the kernel.- Modify 'make
modules_install' to install any firmware required by the modules it installs (where that firmware is shipped in the kernel tree).There are more drivers to be converted to
request_firmware()-- notablydrivers/netanddrivers/scsi, but it's probably best to ensure that the infrastructure is all in place and working correctly before we update those. (Although since modern drivers have been usingrequest_firmware()for some time, I don't think there's likely to be a problem there; we'll probably clean up the stragglers in time for 2.6.28.)David Howells (5):
Fix a const pointer usage warning in the Digigram VX soundcard driver Fix a const assignment inmoxa_load_fw()
Fix a const pointer error in the Conexant cx23418 MPEG encoder driver Fix a const pointer usage warning in the Digigram pcxhr soundcard driver Fix a const pointer usage warning in the Digigram miXart soundcard driver
Quoted 2 lines
Full shortlog and diffstat below, although the really interesting part
is probably between commitsfd4f80de..59890f74-- seen at http://git.infradead.org/users/dwmw2/firmware-2.6.git?a=shortlog;h=59890f74
After the long threads, you would think you would have at least mentioned controversy attached to these changes?
Or perhaps mention that you are specifically excluding the ability to build firmware into modules -- the most reliable method available for those who use modules, MORE reliable than a system where firmware and kernel module are separated.
Or some of the objections raised, that were not met?
Or even _acknowledge_ that these changes have a high probability of creating a non-working driver, and therefore advising distros with Big Fat Warnings and Be Carefuls sprinkled liberally?
Or perhaps describe the outside-the-kernel migration path for distros?
Isn't it silly to create firmware/ top-level dir, when the firmware more naturally lives in the same dir as the driver to which it is intimately tied? We'll just have to start creating firmware/net firmware/char firmware/media etc. after a while.
What about keeping binary objects as binary objects? Surely git can handle binary file. And you are already converting each firmware's data from format A to format B. Might as well convert it to unpacked, un-ASCII'd format that will be loaded directly, as that can be easier in some cases.
I think it's just downright shady to push this crap without mentioning any of the swirl of issues surrounding this, that are quite relevant to its inclusion.
It should be obvious that building firmware into a kernel module (or the kernel itself, if you don't use modules) is _the_ most reliable method of ensuring your driver will work [if it requires firmware].
Jeff