Skip to contentSkip to conversation

Intel Microcode loader, tg3 driver, and the -rc8-mmotd New World Order firmware...

DISCUSSION [email protected] 7 participants 24 messages Jul 3, 2008

lore
Share & tools
lore
#0 lore raw

I built the -rc8-mmotd kernel, and built it with 'CONFIG_FIRMWARE_IN_KERNEL=n'. Lo and behold, the microcode.ko was now doing a request_firmware for 'intel-ucode/06-0f-06' (which makes sense, the Core2 Duo in this laptop is family 6, model 15, stepping 6). However, what I had in /lib/firmware was the Intel-distributed 'microcode.dat' with updates for all the CPUs (which used to work in times past).

What's the magic incantation to take the microcode.dat and create something that the firmware driver is willing to use, or is this all borked up and I need to do a major rethink or fix my config?

Another minor annoyance - the tg3 driver, when builtin to the kernel, wouldn't load the microcode in this config. It complained it couldn't get 'tigon/tg3_tos.bin', but that's almost certainly an issue with Fedora's 'nash' firmware support and/or my understanding of it - I got *that* part working by dropping the file into /lib/firmware/tigon and building the driver as a module. Fortunately, I don't need the tg3 driver to boot far enough to get a full udev running.

#1 lore raw
Quoted 11 lines · On Wed, 02 Jul 2008 21:32:38 -0400 [email protected] wrote:

I built the -rc8-mmotd kernel, and built it with 'CONFIG_FIRMWARE_IN_KERNEL=n'. Lo and behold, the microcode.ko was now doing a request_firmware for 'intel-ucode/06-0f-06' (which makes sense, the Core2 Duo in this laptop is family 6, model 15, stepping 6). However, what I had in /lib/firmware was the Intel-distributed 'microcode.dat' with updates for all the CPUs (which used to work in times past).

What's the magic incantation to take the microcode.dat and create something that the firmware driver is willing to use, or is this all borked up and I need to do a major rethink or fix my config?

Quoted 7 lines

Another minor annoyance - the tg3 driver, when builtin to the kernel, wouldn't load the microcode in this config. It complained it couldn't get 'tigon/tg3_tos.bin', but that's almost certainly an issue with Fedora's 'nash' firmware support and/or my understanding of it - I got *that* part working by dropping the file into /lib/firmware/tigon and building the driver as a module. Fortunately, I don't need the tg3 driver to boot far enough to get a full udev running.

(cc dwmw2)

#2 lore raw
In reply to [email protected]

Hi Valdis,

Quoted 10 lines · On Wed, 2 Jul 2008 [email protected] wrote:

I built the -rc8-mmotd kernel, and built it with 'CONFIG_FIRMWARE_IN_KERNEL=n'. Lo and behold, the microcode.ko was now doing a request_firmware for 'intel-ucode/06-0f-06' (which makes sense, the Core2 Duo in this laptop is family 6, model 15, stepping 6). However, what I had in /lib/firmware was the Intel-distributed 'microcode.dat' with updates for all the CPUs (which used to work in times past).

What's the magic incantation to take the microcode.dat and create something that the firmware driver is willing to use, or is this all borked up and I need to do a major rethink or fix my config?

that's because it expects the Intel-supplied microcode data and you are using the old style microcode.dat data.

Kind regards
Tigran

Quoted 9 lines

Another minor annoyance - the tg3 driver, when builtin to the kernel, wouldn't load the microcode in this config. It complained it couldn't get 'tigon/tg3_tos.bin', but that's almost certainly an issue with Fedora's 'nash' firmware support and/or my understanding of it - I got *that* part working by dropping the file into /lib/firmware/tigon and building the driver as a module. Fortunately, I don't need the tg3 driver to boot far enough to get a full udev running.

#3 lore raw
Quoted 5 lines · Tigran Aivazian

Hi Valdis,

Quoted 1 line · On Wed, 2 Jul 2008 [email protected] wrote:

I built the -rc8-mmotd kernel, and built it with 'CONFIG_FIRMWARE_IN_KERNEL

=n'.

Quoted 6 lines
Quoted 6 lines

Lo and behold, the microcode.ko was now doing a request_firmware for 'intel-ucode/06-0f-06' (which makes sense, the Core2 Duo in this laptop is family 6, model 15, stepping 6). However, what I had in /lib/firmware was the Intel-distributed 'microcode.dat' with updates for all the CPUs (which used to work in times past).

Quoted 6 lines
Quoted 3 lines

What's the magic incantation to take the microcode.dat and create something that the firmware driver is willing to use, or is this all borked up and I need to do a major rethink or fix my config?

that's because it expects the Intel-supplied microcode data and you are using the old style microcode.dat data.

I fed it the stuff I downloaded today from this URL:

http://downloadcenter.intel.com/filter_results.aspx?strTypes=all&ProductID=2643&OSFullName=Linux*&lang=eng&strOSs=39&submit=Go!

which gets me a microcode-20080401.dat that does the same thing. Is there some *other* Intel-supplied microcode data I should be getting instead?

(If I should be looking elsewhere, can somebody fix http://www.urbanmyth.org/microcode/
to point somewhere other than http://downloadcenter.intel.com/default.aspx so people go to the right place?)

Quoted 6 lines

Kind regards
Tigran

Quoted 2 lines

Another minor annoyance - the tg3 driver, when builtin to the kernel, would

n't

Quoted 1 line
Quoted 1 line

load the microcode in this config. It complained it couldn't get 'tigon/tg3

_tos.bin',

Quoted 1 line
Quoted 1 line

but that's almost certainly an issue with Fedora's 'nash' firmware support

and/or

Quoted 1 line
Quoted 1 line

my understanding of it - I got *that* part working by dropping the file int

o

Quoted 1 line
Quoted 1 line

/lib/firmware/tigon and building the driver as a module. Fortunately, I don

't

Quoted 4 lines
Quoted 3 lines

need the tg3 driver to boot far enough to get a full udev running.

#4 lore raw
Quoted 20 lines · On Thu, 3 Jul 2008 [email protected] wrote:
Quoted 5 lines · Tigran Aivazian

Hi Valdis,

Quoted 1 line · On Wed, 2 Jul 2008 [email protected] wrote:

I built the -rc8-mmotd kernel, and built it with 'CONFIG_FIRMWARE_IN_KERNEL

=n'.

Quoted 6 lines
Quoted 6 lines

Lo and behold, the microcode.ko was now doing a request_firmware for 'intel-ucode/06-0f-06' (which makes sense, the Core2 Duo in this laptop is family 6, model 15, stepping 6). However, what I had in /lib/firmware was the Intel-distributed 'microcode.dat' with updates for all the CPUs (which used to work in times past).

Quoted 6 lines
Quoted 3 lines

What's the magic incantation to take the microcode.dat and create something that the firmware driver is willing to use, or is this all borked up and I need to do a major rethink or fix my config?

that's because it expects the Intel-supplied microcode data and you are using the old style microcode.dat data.

Quoted 6 lines

I fed it the stuff I downloaded today from this URL:

http://downloadcenter.intel.com/filter_results.aspx?strTypes=all&ProductID=2643&OSFullName=Linux*&lang=eng&strOSs=39&submit=Go!

which gets me a microcode-20080401.dat that does the same thing. Is there some *other* Intel-supplied microcode data I should be getting instead?

Oh, sorry, I assumed that Intel distribute the data in the format that driver expects.

Kind regards
Tigran

#5 lore raw
Quoted 24 lines · Tigran Aivazian
Quoted 22 lines · On Thu, 3 Jul 2008 [email protected] wrote:
Quoted 5 lines · Tigran Aivazian

Hi Valdis,

Quoted 1 line · On Wed, 2 Jul 2008 [email protected] wrote:

I built the -rc8-mmotd kernel, and built it with 'CONFIG_FIRMWARE_IN_KERNEL

=n'.

Quoted 6 lines
Quoted 6 lines

Lo and behold, the microcode.ko was now doing a request_firmware for 'intel-ucode/06-0f-06' (which makes sense, the Core2 Duo in this laptop is family 6, model 15, stepping 6). However, what I had in /lib/firmware was the Intel-distributed 'microcode.dat' with updates for all the CPUs (which used to work in times past).

Quoted 6 lines
Quoted 3 lines

What's the magic incantation to take the microcode.dat and create something that the firmware driver is willing to use, or is this all borked up and I need to do a major rethink or fix my config?

that's because it expects the Intel-supplied microcode data and you are using the old style microcode.dat data.

I fed it the stuff I downloaded today from this URL:

Quoted 7 lines
Quoted 4 lines

http://downloadcenter.intel.com/filter_results.aspx?strTypes=all&ProductID=2643&OSFullName=Linux*〈=eng&strOSs=39&submit=Go!

which gets me a microcode-20080401.dat that does the same thing. Is there some *other* Intel-supplied microcode data I should be getting instead?

Oh, sorry, I assumed that Intel distribute the data in the format that driver expects.

I think the kernel _does_ manage to extract just the part it wants from the data, but it expects the data in binary form. Drop the attached files in /lib/udev/microcode.sh and /etc/udev/rules.d/51-microcode.rules respectively. (There's probably a better way to handle this kind of thing by putting hooks in firmware.sh rather than using a special rule which overrides it?)

The recent firmware changes haven't modified this. The important change seems to have been here (in 2006):

http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=a30a6a2c

Signature · 1 line
dwmw2
# firmware requests for intel-ucode/* are satisfied by microcode_ctl
SUBSYSTEM=="firmware", ENV{FIRMWARE}=="intel-ucode/*", ACTION=="add", RUN="microcode.sh"
#6 lore raw
Quoted 40 lines · David Woodhouse
Quoted 24 lines · Tigran Aivazian
Quoted 22 lines · On Thu, 3 Jul 2008 [email protected] wrote:
Quoted 5 lines · Tigran Aivazian

Hi Valdis,

Quoted 1 line · On Wed, 2 Jul 2008 [email protected] wrote:

I built the -rc8-mmotd kernel, and built it with 'CONFIG_FIRMWARE_IN_KERNEL

=n'.

Quoted 6 lines
Quoted 6 lines

Lo and behold, the microcode.ko was now doing a request_firmware for 'intel-ucode/06-0f-06' (which makes sense, the Core2 Duo in this laptop is family 6, model 15, stepping 6). However, what I had in /lib/firmware was the Intel-distributed 'microcode.dat' with updates for all the CPUs (which used to work in times past).

Quoted 6 lines
Quoted 3 lines

What's the magic incantation to take the microcode.dat and create something that the firmware driver is willing to use, or is this all borked up and I need to do a major rethink or fix my config?

that's because it expects the Intel-supplied microcode data and you are using the old style microcode.dat data.

I fed it the stuff I downloaded today from this URL:

Quoted 7 lines
Quoted 4 lines

http://downloadcenter.intel.com/filter_results.aspx?strTypes=all&ProductID=2643&OSFullName=Linux*〈=eng&strOSs=39&submit=Go!

which gets me a microcode-20080401.dat that does the same thing. Is there some *other* Intel-supplied microcode data I should be getting instead?

Oh, sorry, I assumed that Intel distribute the data in the format that driver expects.

I think the kernel _does_ manage to extract just the part it wants from the data, but it expects the data in binary form. Drop the attached files in /lib/udev/microcode.sh and /etc/udev/rules.d/51-microcode.rules respectively. (There's probably a better way to handle this kind of thing by putting hooks in firmware.sh rather than using a special rule which overrides it?)

Quoted 4 lines

The recent firmware changes haven't modified this. The important change seems to have been here (in 2006):

http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=a30a6a2c

This commit states:
"so we don't need the application 'microcode_ctl' to assist".
Seems the kernel code extracts the right section, in the all-in-one file, for the actual CPU:
while ((offset = get_next_ucode_from_buffer(&mc, buf, size, offset))

So, maybe a:

  # rewrite firmware file name to all-in-one Intel CPU microcode data file
  SUBSYSTEM=="firmware", ENV{FIRMWARE}=="intel-ucode/*", ENV{FIRMWARE}="intel-ucode/microcode.dat"
would be enough?

Thanks,
Kay

#7 lore raw
Quoted 4 lines · Kay Sievers

So, maybe a:

  # rewrite firmware file name to all-in-one Intel CPU microcode data file
  SUBSYSTEM=="firmware", ENV{FIRMWARE}=="intel-ucode/*", ENV{FIRMWARE}="intel-ucode/microcode.dat"
would be enough?

Nah, it needs the binary form. The actual 'microcode.dat' just looks like this...

0x00000001,     0x00000013,     0x02062001,     0x00000683,
0x2f0da1b0,     0x00000001,     0x00000001,     0x00000000,
0x00000000,     0x00000000,     0x00000000,     0x00000000,
0xbf5ad468,     0xc79f5237,     0xbd53889e,     0x896bfd13,
0x7adc0c8f,     0x44e9e0bc,     0x1a331fc9,     0x00b0f479,

That's all microcode_ctl actually does; read in the text file and output the (complete) binary. Hence:
/sbin/microcode_ctl -f "$DIR/microcode.dat" -d /sys$DEVPATH/data

At a later date, we could make userspace output only the part for the desired CPU, and rip out the kernel-side code which searches for it within the big blob. That was presumably the intention in the commit which changed things. But then again, the kernel-side code still needs to do a certain amount of sanity checking on what it receives -- so I'm not sure we gain a huge amount by trying to remove that functionality from the kernel (not that I've looked hard at the line count).

If we're _not_ going to make userspace provide only what's required, and we keep the selection code in the kernel, then I don't see the point in having separate firmware filenames for different cpus. We could just change the driver to call request_firmware("intel-microcode.bin") and do this one-off conversion:
touch /lib/firmware/intel-microcode.bin microcode_ctl -d /lib/firmware/intel-microcode.bin

Signature · 1 line
dwmw2
#8 lore raw
Quoted 8 lines · David Woodhouse
Quoted 4 lines · Kay Sievers

So, maybe a:

  # rewrite firmware file name to all-in-one Intel CPU microcode data file
  SUBSYSTEM=="firmware", ENV{FIRMWARE}=="intel-ucode/*", ENV{FIRMWARE}="intel-ucode/microcode.dat"
would be enough?

Nah, it needs the binary form. The actual 'microcode.dat' just looks like this...

Ah, I see.

Quoted 10 lines
0x00000001,     0x00000013,     0x02062001,     0x00000683,
0x2f0da1b0,     0x00000001,     0x00000001,     0x00000000,
0x00000000,     0x00000000,     0x00000000,     0x00000000,
0xbf5ad468,     0xc79f5237,     0xbd53889e,     0x896bfd13,
0x7adc0c8f,     0x44e9e0bc,     0x1a331fc9,     0x00b0f479,

That's all microcode_ctl actually does; read in the text file and output the (complete) binary. Hence:
/sbin/microcode_ctl -f "$DIR/microcode.dat" -d /sys$DEVPATH/data

Quoted 7 lines

At a later date, we could make userspace output only the part for the desired CPU, and rip out the kernel-side code which searches for it within the big blob. That was presumably the intention in the commit which changed things. But then again, the kernel-side code still needs to do a certain amount of sanity checking on what it receives -- so I'm not sure we gain a huge amount by trying to remove that functionality from the kernel (not that I've looked hard at the line count).

I think, the in-kernel selection logic is nice, people can still provide a custom .bin file which contains only the needed code for the actual CPU, and it will not really do any "search", right?

Quoted 7 lines

If we're _not_ going to make userspace provide only what's required, and we keep the selection code in the kernel, then I don't see the point in having separate firmware filenames for different cpus. We could just change the driver to call request_firmware("intel-microcode.bin") and do this one-off conversion:
touch /lib/firmware/intel-microcode.bin microcode_ctl -d /lib/firmware/intel-microcode.bin

Sounds fine, yeah. With the in-kernel search, we should make the file just a name without the CPU id. This would also make it easier to find and include the fw into the kernel, right?

We could always export the CPU id somewhere else, so "advanced" tools could look it up in sysfs and upload in a "custom" way. I don't think anybody will really need that. :)

Thanks,
Kay

#9 lore raw
In reply to David Woodhouse
Quoted 4 lines · David Woodhouse

The recent firmware changes haven't modified this. The important change seems to have been here (in 2006):

http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=a30a6a2c

Aha. That's the part I was missing. :)

Quoted 1 line

From the changelog for that commit, Shaohua Li wrote:

"with the changes, we should put all intel-ucode/xx-xx-xx microcode files into the firmware dir (I had a tool to split previous big data file into small one and later we will release new style data file). The init script should be changed to ..."

And apparently I got stuck between the unreleased tool to split the file, and the release of the new style data file.

Anyhow, it appears the firmware_request() was just a bullet loaded in the chamber waiting for me to pull the trigger 2 years later by setting CONFIG_FIRMWARE_IN_KERNEL=n :)

The behavior is explained, and presumably Intel will eventually release a method of getting the new-format bits, and all will be right with the world (or at least this part of it.. ;)

#10 lore raw
Quoted 11 lines · On Thu, 2008-07-03 at 06:17 -0400, [email protected] wrote:
Quoted 4 lines · David Woodhouse

The recent firmware changes haven't modified this. The important change seems to have been here (in 2006):

http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=a30a6a2c

Aha. That's the part I was missing. :)

From the changelog for that commit, Shaohua Li wrote:

Quoted 7 lines

"with the changes, we should put all intel-ucode/xx-xx-xx microcode files into the firmware dir (I had a tool to split previous big data file into small one and later we will release new style data file). The init script should be changed to ..."

And apparently I got stuck between the unreleased tool to split the file, and the release of the new style data file.

That 'new style' data file is just the binary output of the microcode_ctl tool. The kernel is still capable of finding the section it requires, when fed the whole blob.

Quoted 3 lines

Anyhow, it appears the firmware_request() was just a bullet loaded in the chamber waiting for me to pull the trigger 2 years later by setting CONFIG_FIRMWARE_IN_KERNEL=n :)

No, the recent firmware changes haven't modified this. The FIRMWARE_IN_KERNEL option makes no difference here.

Quoted 3 lines

The behavior is explained, and presumably Intel will eventually release a method of getting the new-format bits, and all will be right with the world (or at least this part of it.. ;)

Some would say that Intel already released a method for making it work, in the form of the email to which you're replying... does that udev configuration not work for you?

Signature · 1 line
dwmw2
#11 lore raw
Quoted 6 lines · David Woodhouse
Quoted 3 lines

Anyhow, it appears the firmware_request() was just a bullet loaded in the chamber waiting for me to pull the trigger 2 years later by setting CONFIG_FIRMWARE_IN_KERNEL=n :)

No, the recent firmware changes haven't modified this. The FIRMWARE_IN_KERNEL option makes no difference here.

"D'Oh!" -- H. Simpson.
"Nevermind" - Emily Litella.

Actually, FIRMWARE_IN_KERNEL *did* make a difference - it busticated the tg3 driver and while fighting with the 'firmware: requesting tigon/tg3_tso.bin' errors I had just inflicted on myself, I found the 'requesting intel-ucode/060f06' messages as well - and totally failed to notice that the *previous* kernel and initscripts had *all along* been doing a modprobe, which generated 2 requests (one per CPU) which failed, and then the initscript ran microcode_ctl *anyhow*, papering over the two failed requests.. ;)

#12 lore raw
Quoted 3 lines · On Thu, 2008-07-03 at 07:34 -0400, [email protected] wrote:

Actually, FIRMWARE_IN_KERNEL *did* make a difference - it busticated the tg3 driver and while fighting with the 'firmware: requesting tigon/tg3_tso.bin' errors I had just inflicted on myself,

Just to confirm: that works either with CONFIG_FIRMWARE_IN_KERNEL=y or if you run 'make INSTALL_FW_PATH=/lib/firmware firmware_install', right?

I'm not entirely sure whether we should make 'make firmware_install' install to /lib/firmware by default, or to $(O)/usr/lib/firmware as it does at the moment.

The former is consistent with 'make modules_install', and the latter is consistent with 'make headers_install'. Both of which have good reasons for being the way they are.

Signature · 1 line
dwmw2
#13 lore raw
Quoted 8 lines · David Woodhouse
Quoted 3 lines · On Thu, 2008-07-03 at 07:34 -0400, [email protected] wrote:

Actually, FIRMWARE_IN_KERNEL *did* make a difference - it busticated the tg3 driver and while fighting with the 'firmware: requesting tigon/tg3_tso.bin' errors I had just inflicted on myself,

Just to confirm: that works either with CONFIG_FIRMWARE_IN_KERNEL=y or if you run 'make INSTALL_FW_PATH=/lib/firmware firmware_install', right?

Quoted 7 lines

I'm not entirely sure whether we should make 'make firmware_install' install to /lib/firmware by default, or to $(O)/usr/lib/firmware as it does at the moment.

The former is consistent with 'make modules_install', and the latter is consistent with 'make headers_install'. Both of which have good reasons for being the way they are.

Headers are not needed for bootup, firmware might be. :) But /usr might be on a different partition/disk/storage at the time we need it, right?

I would say /usr/lib/firmware should not even exist, udev does intentionally not even look there.

Thanks,
Kay

#14 lore raw
Quoted 6 lines · Kay Sievers

Headers are not needed for bootup, firmware might be. :) But /usr might be on a different partition/disk/storage at the time we need it, right?

I would say /usr/lib/firmware should not even exist, udev does intentionally not even look there.

Not /usr/lib/firmware. Currently, when you run 'make firmware_install', it gets installed to usr/lib/firmware _within_ the kernel build directory. And you're expected to copy it from there (or override with 'make INSTALL_FW_PATH=/lib/firmware firmware_install')

Signature · 1 line
dwmw2
#15 lore raw
Quoted 5 lines · David Woodhouse
Quoted 4 lines · Kay Sievers

Headers are not needed for bootup, firmware might be. :) But /usr might be on a different partition/disk/storage at the time we need it, right?

Quoted 7 lines
Quoted 2 lines

I would say /usr/lib/firmware should not even exist, udev does intentionally not even look there.

Not /usr/lib/firmware. Currently, when you run 'make firmware_install', it gets installed to usr/lib/firmware _within_ the kernel build directory. And you're expected to copy it from there (or override with 'make INSTALL_FW_PATH=/lib/firmware firmware_install')

Ah, ok. So by default "make modules_install" installs in the rootfs, but "make firmware_install" installs in the build directory? Hmm ...

Thanks,
Kay

#16 lore raw
Quoted 2 lines · Kay Sievers

Ah, ok. So by default "make modules_install" installs in the rootfs, but "make firmware_install" installs in the build directory?

Right. And 'make headers_install' installs in the build directory, as I said. And I'm not entirely _which_ one 'firmware_install' should do.

Signature · 1 line
dwmw2
#17 lore raw
In reply to Kay Sievers
Quoted 9 lines · Kay Sievers
Quoted 8 lines · David Woodhouse
Quoted 6 lines · Kay Sievers

Headers are not needed for bootup, firmware might be. :) But /usr might be on a different partition/disk/storage at the time we need it, right?

I would say /usr/lib/firmware should not even exist, udev does intentionally not even look there.

Quoted 7 lines
Quoted 4 lines

Not /usr/lib/firmware. Currently, when you run 'make firmware_install', it gets installed to usr/lib/firmware _within_ the kernel build directory. And you're expected to copy it from there (or override with 'make INSTALL_FW_PATH=/lib/firmware firmware_install')

Ah, ok. So by default "make modules_install" installs in the rootfs, but "make firmware_install" installs in the build directory? Hmm ...

The *odd* part is that make firmware_install isn't symmetric - it drops it in $O/usr/lib/firmware, but you don't want the final result in /usr/lib, you want it in /lib. Maybe it should be dropping it in $O/lib/firmware instead?

#18 lore raw
Quoted 4 lines · On Thu, 2008-07-03 at 09:54 -0400, [email protected] wrote:

The *odd* part is that make firmware_install isn't symmetric - it drops it in $O/usr/lib/firmware, but you don't want the final result in /usr/lib, you want it in /lib. Maybe it should be dropping it in $O/lib/firmware instead?

Not sure that makes sense. $(O)/usr is already used for userspace stuff. It doesn't necessarily correspond to /usr.

Signature · 1 line
dwmw2
#19 lore raw
In reply to David Woodhouse
Quoted 2 lines · David Woodhouse
Quoted 1 line · On Thu, 2008-07-03 at 07:34 -0400, [email protected] wrote:

Actually, FIRMWARE_IN_KERNEL *did* make a difference - it busticated the tg

3

Quoted 5 lines
Quoted 2 lines

driver and while fighting with the 'firmware: requesting tigon/tg3_tso.bin' errors I had just inflicted on myself,

Just to confirm: that works either with CONFIG_FIRMWARE_IN_KERNEL=y or if you run 'make INSTALL_FW_PATH=/lib/firmware firmware_install', right?

The old config was FIRMWARE_IN_KERNEL=y, and that worked fine because the microcode was compiled into tg3.o and All Was Good.

My current, *also* working config has FIRMWARE_IN_KERNEL=n, and CONFIG_TIGON3=m. System comes up, initrd runs, the real root gets loaded, udev gets started, and when the modprobe happens, /lib/firmware/tigon is there to handle the request_firmware that gets launched when the module loads.

The *broken* config was FIRMWARE_IN_KERNEL=n, CONFIG_TIGON3=y. System comes up, the request_firmware pops *very* early (as in while we're still on the initrd), and I couldn't figure out how to get Fedora/nash/etc to do the firmware load from the initrd (nash seems to want it in /firmware/tigon3, but putting it there didn't help any). By the time udev gets launched, that request is DOA and the interface doesn't come up. But as I said, that *seems* to be just a lack of clue on my part about what nash wants.

Quoted 7 lines

I'm not entirely sure whether we should make 'make firmware_install' install to /lib/firmware by default, or to $(O)/usr/lib/firmware as it does at the moment.

The former is consistent with 'make modules_install', and the latter is consistent with 'make headers_install'. Both of which have good reasons for being the way they are.

Somebody else can decide that one. Kay does make a point in another mail that you *really* want it to end up in /lib/firmware for good reasons.

#20 lore raw
In reply to Tigran Aivazian

[ Added Arjan and the relevant Intel contact]

Quoted 24 lines · Tigran Aivazian
Quoted 22 lines · On Thu, 3 Jul 2008 [email protected] wrote:
Quoted 5 lines · Tigran Aivazian

Hi Valdis,

Quoted 1 line · On Wed, 2 Jul 2008 [email protected] wrote:

I built the -rc8-mmotd kernel, and built it with 'CONFIG_FIRMWARE_IN_KERNEL

=n'.

Quoted 6 lines
Quoted 6 lines

Lo and behold, the microcode.ko was now doing a request_firmware for 'intel-ucode/06-0f-06' (which makes sense, the Core2 Duo in this laptop is family 6, model 15, stepping 6). However, what I had in /lib/firmware was the Intel-distributed 'microcode.dat' with updates for all the CPUs (which used to work in times past).

Quoted 6 lines
Quoted 3 lines

What's the magic incantation to take the microcode.dat and create something that the firmware driver is willing to use, or is this all borked up and I need to do a major rethink or fix my config?

that's because it expects the Intel-supplied microcode data and you are using the old style microcode.dat data.

I fed it the stuff I downloaded today from this URL:

Quoted 7 lines
Quoted 5 lines

http://downloadcenter.intel.com/filter_results.aspx?strTypes=all&ProductID=2643&OSFullName=Linux*&lang=eng&strOSs=39&submit=Go!

which gets me a microcode-20080401.dat that does the same thing. Is there some *other* Intel-supplied microcode data I should be getting instead?

Oh, sorry, I assumed that Intel distribute the data in the format that driver expects.

There are two format of Intel CPU microcode and two methods to load it.
- old: the microcodes are in a big file, which include multiple microcodes (for multiple CPU). The driver require a char device and a user space loader ("microcode_ctl")
- new: one microcode per file, using the 'request_firmware' infrastructure. No user space support needed.

Actually Intel provides only the old methods.
There was talks with Arjan and Intel about the distribution format for the new methods. But I don't have any new.
I think that when the new format is fully specified (directory structure, tar, gzip,...) Intel will distribute the microcodes in the new form.

ciao
cate

#21 lore raw
Quoted 8 lines · On Thu, 03 Jul 2008 11:24:34 +0200 "Giacomo A. Catenazzi" <[email protected]>…

There are two format of Intel CPU microcode and two methods to load it.
- old: the microcodes are in a big file, which include multiple microcodes (for multiple CPU). The driver require a char device and a user space loader ("microcode_ctl")
- new: one microcode per file, using the 'request_firmware' infrastructure. No user space support needed.

Quoted 6 lines

Actually Intel provides only the old methods.
There was talks with Arjan and Intel about the distribution format for the new methods. But I don't have any new.
I think that when the new format is fully specified (directory structure, tar, gzip,...) Intel will distribute the microcodes in the new form.

we hope to switch to the new form but there's the small case of "installed base" using ancient kernels, and it's kind of not nice to have to release 2 sets. At some point we will switch over though.

#22 lore raw
Quoted 10 lines · Arjan van de Ven
Quoted 7 lines · On Thu, 03 Jul 2008 11:24:34 +0200 "Giacomo A. Catenazzi" <[email protected]>…

There are two format of Intel CPU microcode and two methods to load it.
- old: the microcodes are in a big file, which include multiple microcodes (for multiple CPU). The driver require a char device and a user space loader ("microcode_ctl")
- new: one microcode per file, using the 'request_firmware' infrastructure. No user space support needed.

Actually there are three:

1. The text format 'microcode.dat', including updates for all CPUs.
2. The binary format output by microcode_ctl, still including all CPUs.
3. The smaller files with just the relevant subset of #2.

The kernel (since 2006) can actually take either #2 or #3. The udev scripts I just posted will use microcode_ctl to feed it #2, when they find #1 on the file system.

A small amount of extra work in the userspace tool would let those udev scripts feed #3 to the kernel, and then the code in the kernel to select the appropriate update could be removed.

Quoted 6 lines
Quoted 6 lines

Actually Intel provides only the old methods.
There was talks with Arjan and Intel about the distribution format for the new methods. But I don't have any new.
I think that when the new format is fully specified (directory structure, tar, gzip,...) Intel will distribute the microcodes in the new form.

Doesn't the "new format" (#3) involve hard links too, since there are some cases where the same microcode update applies to more than one CPU revision?

Quoted 3 lines

we hope to switch to the new form but there's the small case of "installed base" using ancient kernels, and it's kind of not nice to have to release 2 sets. At some point we will switch over though.

Do we really need to _ship_ it in a different form? It's not exactly hard to convert from the text form (#1) to either of the other two -- either on the fly in udev scripts, or at installation time.

Signature · 1 line
dwmw2
#23 lore raw

[recipient clean-up and adding Simon]

Quoted 9 lines · David Woodhouse
Quoted 2 lines · Arjan van de Ven

On Thu, 03 Jul 2008 11:24:34 +0200
"Giacomo A. Catenazzi" <[email protected]> wrote:

Actually there are three:

1. The text format 'microcode.dat', including updates for all CPUs.
2. The binary format output by microcode_ctl, still including all CPUs.
3. The smaller files with just the relevant subset of #2.

we are diverging :-(

The #2 is not on upstream microcode_ctl. BTW Debian/Ubuntu already diverge a lot in respect of upstream.
Further, IIRC, Simon slowed/stopped developing microcode_ctl, because he think (IIRC) that distributions used other solutions or going to use #3.

So if we need #2, I really want Simon (or someone) that take care of continuing microcode_ctl upstream and merging distribution patches.

Quoted 3 lines

The kernel (since 2006) can actually take either #2 or #3. The udev scripts I just posted will use microcode_ctl to feed it #2, when they find #1 on the file system.

#1 and #2 have some problem in Debian/Ubuntu:
- microcode_ctl is in /usr
- it require a module, so we load the module and we wait that the device is created (it is asynchronous)
- initram would require an additional program, to an early load
- split on architectures [32/64bit CPU] (but this is an internal autobuild / non-free Debian problem)

Quoted 4 lines

A small amount of extra work in the userspace tool would let those udev scripts feed #3 to the kernel, and then the code in the kernel to select the appropriate update could be removed.

I don't think Intel want this solution.
Earlier Intel preferred to give us two version of microcode (new and old kernel) instead of let us to filter out one short microcode, which trigged a kernel bug.

I'm ready to do such script (really I've already
a similar tool)

Quoted 11 lines
Quoted 7 lines
Quoted 7 lines

Actually Intel provides only the old methods.
There was talks with Arjan and Intel about the distribution format for the new methods. But I don't have any new.
I think that when the new format is fully specified (directory structure, tar, gzip,...) Intel will distribute the microcodes in the new form.

Doesn't the "new format" (#3) involve hard links too, since there are some cases where the same microcode update applies to more than one CPU revision?

no, I don't think. Intel documentation on BIOS writing, document that microcode should be discarded if the header information don't match with current CPU. I didn't checked if some microcode contains same data with different header (which anyway is not feeded to the CPU)

Quoted 8 lines
Quoted 4 lines

we hope to switch to the new form but there's the small case of "installed base" using ancient kernels, and it's kind of not nice to have to release 2 sets. At some point we will switch over though.

Do we really need to _ship_ it in a different form? It's not exactly hard to convert from the text form (#1) to either of the other two -- either on the fly in udev scripts, or at installation time.

It was a personal interpretation of Intel wishes.

ciao
cate