Skip to contentSkip to conversation

e1000: "eeprom checksum is not valid" after kexec

DISCUSSION Jiri Slaby 3 participants 13 messages Apr 23, 2009 – May 11, 2009

lore
Share & tools
lore

Jiri Slaby

author
#0 lore raw

Hi,

4a865905f685eaefaedf6ade362323dc52aa703b
(PCI PM: Make pci_set_power_state() handle devices with no PM support) breaks e1000 after being kexec'ed. These reverts fix the problem:
Revert "PCI PM: Make pci_set_power_state() handle devices with no PM
support"
Revert "PCI PM: Introduce __pci_[start|complete]_power_transition()
(rev. 2)"

I reverted the second one just for an easy revert of the former one, which is actually the culprit.

The symptoms:
e1000 0000:02:01.0: enabling device (0000 -> 0003)
e1000 0000:02:01.0: PCI INT A -> Link[LNKA] -> GSI 11 (level, low) -> IRQ 11 e1000 0000:02:01.0: setting latency timer to 64
e1000: 0000:02:01.0: e1000_probe: The EEPROM Checksum Is Not Valid Switched to high resolution mode on CPU 0 /*********************/
Current EEPROM Checksum : 0xffff
Calculated : 0xbaf9
Offset Values
======== ======
00000000: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff 00000010: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff 00000020: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff 00000030: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff 00000040: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff 00000050: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff 00000060: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff 00000070: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff Include this output when contacting your support provider.
This is not a software error! Something bad happened to your hardware or EEPROM image. Ignoring this problem could result in further problems, possibly loss of data, corruption or system hangs!
The MAC Address will be reset to 00:00:00:00:00:00, which is invalid and requires you to set the proper MAC address manually before continuing to enable this network device.
Please inspect the EEPROM dump and report the issue to your hardware vendor or Intel Customer Support.
/*********************/
e1000: 0000:02:01.0: e1000_probe: Invalid MAC Address
e1000: 0000:02:01.0: e1000_probe: (PCI-X:33MHz:64-bit) 00:00:00:00:00:00

#1 lore raw
Quoted 13 lines · Jiri Slaby

Hi,

4a865905f685eaefaedf6ade362323dc52aa703b
(PCI PM: Make pci_set_power_state() handle devices with no PM support) breaks e1000 after being kexec'ed. These reverts fix the problem:
Revert "PCI PM: Make pci_set_power_state() handle devices with no PM
support"
Revert "PCI PM: Introduce __pci_[start|complete]_power_transition()
(rev. 2)"

I reverted the second one just for an easy revert of the former one, which is actually the culprit.

Quoted 6 lines

The symptoms:
e1000 0000:02:01.0: enabling device (0000 -> 0003)
e1000 0000:02:01.0: PCI INT A -> Link[LNKA] -> GSI 11 (level, low) -> IRQ 11 e1000 0000:02:01.0: setting latency timer to 64
e1000: 0000:02:01.0: e1000_probe: The EEPROM Checksum Is Not Valid Switched to high resolution mode on CPU 0

Have you tried b43fcd7dc7b, found in v2.6.30-rc3?

author
#2 lore raw
Quoted 14 lines · Thadeu Lima de Souza Cascardo
Quoted 13 lines · Jiri Slaby

Hi,

4a865905f685eaefaedf6ade362323dc52aa703b
(PCI PM: Make pci_set_power_state() handle devices with no PM support) breaks e1000 after being kexec'ed. These reverts fix the problem:
Revert "PCI PM: Make pci_set_power_state() handle devices with no PM
support"
Revert "PCI PM: Introduce __pci_[start|complete]_power_transition()
(rev. 2)"

I reverted the second one just for an easy revert of the former one, which is actually the culprit.

Quoted 8 lines
Quoted 6 lines

The symptoms:
e1000 0000:02:01.0: enabling device (0000 -> 0003)
e1000 0000:02:01.0: PCI INT A -> Link[LNKA] -> GSI 11 (level, low) -> IRQ 11 e1000 0000:02:01.0: setting latency timer to 64
e1000: 0000:02:01.0: e1000_probe: The EEPROM Checksum Is Not Valid Switched to high resolution mode on CPU 0

Have you tried b43fcd7dc7b, found in v2.6.30-rc3?

I've tried 2.6.30-rc3-next-20090423 without success.

#10 lore raw
In reply to Jiri Slaby
Quoted 1 line · Jiri Slaby

Hi,

Hi,

Quoted 9 lines

4a865905f685eaefaedf6ade362323dc52aa703b
(PCI PM: Make pci_set_power_state() handle devices with no PM support) breaks e1000 after being kexec'ed. These reverts fix the problem:
Revert "PCI PM: Make pci_set_power_state() handle devices with no PM
support"
Revert "PCI PM: Introduce __pci_[start|complete]_power_transition()
(rev. 2)"

I reverted the second one

I don't think it can be reverted.

Quoted 1 line

just for an easy revert of the former one, which is actually the culprit.

Can you just try to revert the changes in pci_raw_set_power_state() and check if that has any effect (it shouldn't)?

Quoted 23 lines

The symptoms:
e1000 0000:02:01.0: enabling device (0000 -> 0003)
e1000 0000:02:01.0: PCI INT A -> Link[LNKA] -> GSI 11 (level, low) -> IRQ 11 e1000 0000:02:01.0: setting latency timer to 64
e1000: 0000:02:01.0: e1000_probe: The EEPROM Checksum Is Not Valid Switched to high resolution mode on CPU 0 /*********************/
Current EEPROM Checksum : 0xffff
Calculated : 0xbaf9
Offset Values
======== ======
00000000: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff 00000010: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff 00000020: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff 00000030: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff 00000040: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff 00000050: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff 00000060: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff 00000070: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff Include this output when contacting your support provider.
This is not a software error! Something bad happened to your hardware or EEPROM image. Ignoring this problem could result in further problems, possibly loss of data, corruption or system hangs!

Quoted 8 lines

The MAC Address will be reset to 00:00:00:00:00:00, which is invalid and requires you to set the proper MAC address manually before continuing to enable this network device.
Please inspect the EEPROM dump and report the issue to your hardware vendor or Intel Customer Support.
/*********************/
e1000: 0000:02:01.0: e1000_probe: Invalid MAC Address
e1000: 0000:02:01.0: e1000_probe: (PCI-X:33MHz:64-bit) 00:00:00:00:00:00

So this is after kexec?

What happens if you remove just the

	/* Check if we're already there */
	if (dev->current_state == state)
		return 0;

part from pci_set_power_state()?

Rafael

#3 lore raw
In reply to Jiri Slaby
Quoted 22 lines · Jiri Slaby
Quoted 21 lines · Thadeu Lima de Souza Cascardo
Quoted 19 lines · Jiri Slaby

Hi,

4a865905f685eaefaedf6ade362323dc52aa703b
(PCI PM: Make pci_set_power_state() handle devices with no PM support) breaks e1000 after being kexec'ed. These reverts fix the problem:
Revert "PCI PM: Make pci_set_power_state() handle devices with no PM
support"
Revert "PCI PM: Introduce __pci_[start|complete]_power_transition()
(rev. 2)"

I reverted the second one just for an easy revert of the former one, which is actually the culprit.

The symptoms:
e1000 0000:02:01.0: enabling device (0000 -> 0003)
e1000 0000:02:01.0: PCI INT A -> Link[LNKA] -> GSI 11 (level, low) -> IRQ 11 e1000 0000:02:01.0: setting latency timer to 64
e1000: 0000:02:01.0: e1000_probe: The EEPROM Checksum Is Not Valid Switched to high resolution mode on CPU 0

Quoted 3 lines
Quoted 1 line

Have you tried b43fcd7dc7b, found in v2.6.30-rc3?

I've tried 2.6.30-rc3-next-20090423 without success.

You mean next-20090423. The patch is really found there.

But, then, I realize you mean reverting these patches for the kernel that is running or the kernel that is being kexec'd?

If b43fcd7dc7b is applied to the running kernel, it fixes the shutdown issue, and the next loaded kernel probes e1000 fine.

If you are reverting 4a865905f in the kexec'd kernel and the running kernel does not have b43fcd7dc7b, then I'd like to test the revert for my case here, which is e100.

Which is it?

Regards,
Cascardo.

author
#11 lore raw
In reply to Rafael J. Wysocki
Quoted 9 lines · Rafael J. Wysocki
Quoted 8 lines · Jiri Slaby

4a865905f685eaefaedf6ade362323dc52aa703b
(PCI PM: Make pci_set_power_state() handle devices with no PM support) breaks e1000 after being kexec'ed. These reverts fix the problem:
Revert "PCI PM: Make pci_set_power_state() handle devices with no PM
support"
Revert "PCI PM: Introduce __pci_[start|complete]_power_transition()
(rev. 2)"

Quoted 3 lines
Quoted 1 line

I reverted the second one

I don't think it can be reverted.

But it works :). I'm not saying, it's correct to revert them upstream. It was just confirmation, that it causes the problem.

Quoted 4 lines
Quoted 1 line

just for an easy revert of the former one, which is actually the culprit.

Can you just try to revert the changes in pci_raw_set_power_state() and check if that has any effect (it shouldn't)?

Please send a patch. I'm lost in the changes done there. Let's say against the top of 20090423 next tree.

Quoted 4 lines
Quoted 2 lines

e1000: 0000:02:01.0: e1000_probe: Invalid MAC Address
e1000: 0000:02:01.0: e1000_probe: (PCI-X:33MHz:64-bit) 00:00:00:00:00:00

So this is after kexec?

yes

Quoted 7 lines

What happens if you remove just the

	/* Check if we're already there */
	if (dev->current_state == state)
		return 0;

part from pci_set_power_state()?

Will try later.

Thanks.

author
#4 lore raw
In reply to Thadeu Lima de Souza Cascardo
Quoted 9 lines · Thadeu Lima de Souza Cascardo
Quoted 3 lines · Jiri Slaby
Quoted 1 line · Thadeu Lima de Souza Cascardo

Have you tried b43fcd7dc7b, found in v2.6.30-rc3?

I've tried 2.6.30-rc3-next-20090423 without success.

You mean next-20090423. The patch is really found there.

But, then, I realize you mean reverting these patches for the kernel that is running or the kernel that is being kexec'd?

The latter.

Quoted 2 lines

If b43fcd7dc7b is applied to the running kernel, it fixes the shutdown issue, and the next loaded kernel probes e1000 fine.

Makes sense.

Quoted 3 lines

If you are reverting 4a865905f in the kexec'd kernel and the running kernel does not have b43fcd7dc7b, then I'd like to test the revert for my case here, which is e100.

To make things clear: on that machine, there was stock opensuse 11.1 distro kernel which is 2.6.27-based (no b43fcd7dc7b). I needed to debug a wireless bug, so I kexec'ed wireless-testing (contains 4a865905f already).

So in fact, 4a865905f from the testing kernel triggered a bug fixed in near past by b43fcd7dc7b.

Did the other two e100* drivers suffer from the same and were fixed recently? It would render kexec pretty unusable from the older kernels if this is not going to be fixed anyhow :(.

#5 lore raw
Quoted 26 lines · Jiri Slaby
Quoted 9 lines · Thadeu Lima de Souza Cascardo
Quoted 3 lines · Jiri Slaby
Quoted 1 line · Thadeu Lima de Souza Cascardo

Have you tried b43fcd7dc7b, found in v2.6.30-rc3?

I've tried 2.6.30-rc3-next-20090423 without success.

You mean next-20090423. The patch is really found there.

But, then, I realize you mean reverting these patches for the kernel that is running or the kernel that is being kexec'd?

The latter.

Quoted 2 lines

If b43fcd7dc7b is applied to the running kernel, it fixes the shutdown issue, and the next loaded kernel probes e1000 fine.

Makes sense.

Quoted 3 lines

If you are reverting 4a865905f in the kexec'd kernel and the running kernel does not have b43fcd7dc7b, then I'd like to test the revert for my case here, which is e100.

To make things clear: on that machine, there was stock opensuse 11.1 distro kernel which is 2.6.27-based (no b43fcd7dc7b). I needed to debug a wireless bug, so I kexec'ed wireless-testing (contains 4a865905f already).

Quoted 6 lines

So in fact, 4a865905f from the testing kernel triggered a bug fixed in near past by b43fcd7dc7b.

Did the other two e100* drivers suffer from the same and were fixed recently? It would render kexec pretty unusable from the older kernels if this is not going to be fixed anyhow :(.

Yes, as well as some other network drivers, it seems. My fix for e100 should be in Jeffrey Kirsher's tree by now and go into netdev and rc4 soon, I expect.

But, since I also thought that it would be good to fix that and allow people to kexec from earlier kernels, I did a followup to e100-devel, linux-pci, netdev and Rafael Wysocki. I didn't include linux-kernel, which I have just fixed, bouncing the message (oops!). I may bounce it to you too, if you want that.

Your findings shed a light into that problem. But I could find it in very early kernels too for some configurations, and these commits you are reverting may only fix the issue for the most common configurations out there. That is, it was very easy to trigger the shutdown bug with these patches. But I think there are some other bugs out there that will trigger it, and they are not that easy bisecting, it seems, since only some very particular configurations trigger it.

I will do some tests with the commits you mention and reproduce the problem using as earlier kernels as I can and send the config.

Regards,
Cascardo.

#12 lore raw
In reply to Jiri Slaby
Quoted 17 lines · Jiri Slaby
Quoted 9 lines · Rafael J. Wysocki
Quoted 8 lines · Jiri Slaby

4a865905f685eaefaedf6ade362323dc52aa703b
(PCI PM: Make pci_set_power_state() handle devices with no PM support) breaks e1000 after being kexec'ed. These reverts fix the problem:
Revert "PCI PM: Make pci_set_power_state() handle devices with no PM
support"
Revert "PCI PM: Introduce __pci_[start|complete]_power_transition()
(rev. 2)"

Quoted 3 lines
Quoted 1 line

I reverted the second one

I don't think it can be reverted.

But it works :). I'm not saying, it's correct to revert them upstream. It was just confirmation, that it causes the problem.

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

just for an easy revert of the former one, which is actually the culprit.

Can you just try to revert the changes in pci_raw_set_power_state() and check if that has any effect (it shouldn't)?

Please send a patch. I'm lost in the changes done there. Let's say against the top of 20090423 next tree.

I can easily send you a patch against -rc3 if that's sufficient.

Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

e1000: 0000:02:01.0: e1000_probe: Invalid MAC Address
e1000: 0000:02:01.0: e1000_probe: (PCI-X:33MHz:64-bit) 00:00:00:00:00:00

So this is after kexec?

yes

Do you kexec the same kernel or any other kernel?

Quoted 9 lines
Quoted 7 lines

What happens if you remove just the

	/* Check if we're already there */
	if (dev->current_state == state)
		return 0;

part from pci_set_power_state()?

Will try later.

OK, thanks.

Best,
Rafael

#6 lore raw
In reply to Thadeu Lima de Souza Cascardo
Quoted 52 lines · Thadeu Lima de Souza Cascardo
Quoted 26 lines · Jiri Slaby
Quoted 9 lines · Thadeu Lima de Souza Cascardo
Quoted 3 lines · Jiri Slaby
Quoted 1 line · Thadeu Lima de Souza Cascardo

Have you tried b43fcd7dc7b, found in v2.6.30-rc3?

I've tried 2.6.30-rc3-next-20090423 without success.

You mean next-20090423. The patch is really found there.

But, then, I realize you mean reverting these patches for the kernel that is running or the kernel that is being kexec'd?

The latter.

Quoted 2 lines

If b43fcd7dc7b is applied to the running kernel, it fixes the shutdown issue, and the next loaded kernel probes e1000 fine.

Makes sense.

Quoted 3 lines

If you are reverting 4a865905f in the kexec'd kernel and the running kernel does not have b43fcd7dc7b, then I'd like to test the revert for my case here, which is e100.

To make things clear: on that machine, there was stock opensuse 11.1 distro kernel which is 2.6.27-based (no b43fcd7dc7b). I needed to debug a wireless bug, so I kexec'ed wireless-testing (contains 4a865905f already).

Quoted 6 lines

So in fact, 4a865905f from the testing kernel triggered a bug fixed in near past by b43fcd7dc7b.

Did the other two e100* drivers suffer from the same and were fixed recently? It would render kexec pretty unusable from the older kernels if this is not going to be fixed anyhow :(.

Yes, as well as some other network drivers, it seems. My fix for e100 should be in Jeffrey Kirsher's tree by now and go into netdev and rc4 soon, I expect.

But, since I also thought that it would be good to fix that and allow people to kexec from earlier kernels, I did a followup to e100-devel, linux-pci, netdev and Rafael Wysocki. I didn't include linux-kernel, which I have just fixed, bouncing the message (oops!). I may bounce it to you too, if you want that.

Your findings shed a light into that problem. But I could find it in very early kernels too for some configurations, and these commits you are reverting may only fix the issue for the most common configurations out there. That is, it was very easy to trigger the shutdown bug with these patches. But I think there are some other bugs out there that will trigger it, and they are not that easy bisecting, it seems, since only some very particular configurations trigger it.

Quoted 2 lines

I will do some tests with the commits you mention and reproduce the problem using as earlier kernels as I can and send the config.

Cascardo, Jiri, can you tell me please what the status here is?

My understanding is that the commit pointed to by Jiri caused a problem if the current mainline kernel was kexeced from an older kernel (2.6.27.x from openSUSE-11.1 in this particular case), because the older kernel didn't have the recent network driver fixes applied. Is this correct?

Also, I'm still interested in whether or not removig the following three lines:

        /* Check if we're already there */
        if (dev->current_state == state)
                return 0;

from pci_set_power_state() in the current mainline kernel fixes the problem in the configuration where it is readily reproducible.

Thanks,
Rafael

author
#7 lore raw
Quoted 4 lines · Rafael J. Wysocki

My understanding is that the commit pointed to by Jiri caused a problem if the current mainline kernel was kexeced from an older kernel (2.6.27.x from openSUSE-11.1 in this particular case), because the older kernel didn't have the recent network driver fixes applied. Is this correct?

Exactly!

Quoted 8 lines

Also, I'm still interested in whether or not removig the following three lines:

        /* Check if we're already there */
        if (dev->current_state == state)
                return 0;

from pci_set_power_state() in the current mainline kernel fixes the problem in the configuration where it is readily reproducible.

After removing those lines, the problem still persists:
e1000: 0000:02:01.0: e1000_probe: The EEPROM Checksum Is Not Valid

#8 lore raw
Quoted 14 lines · Jiri Slaby
Quoted 4 lines · Rafael J. Wysocki

My understanding is that the commit pointed to by Jiri caused a problem if the current mainline kernel was kexeced from an older kernel (2.6.27.x from openSUSE-11.1 in this particular case), because the older kernel didn't have the recent network driver fixes applied. Is this correct?

Exactly!

Quoted 6 lines

Also, I'm still interested in whether or not removig the following three lines:

        /* Check if we're already there */
        if (dev->current_state == state)
                return 0;
Quoted 5 lines
Quoted 2 lines

from pci_set_power_state() in the current mainline kernel fixes the problem in the configuration where it is readily reproducible.

After removing those lines, the problem still persists:
e1000: 0000:02:01.0: e1000_probe: The EEPROM Checksum Is Not Valid

So it's more complicated than I thought. Well ...

What if the driver in question is rmmoded before kexec?

#9 lore raw
Quoted 18 lines · Rafael J. Wysocki
Quoted 17 lines · Jiri Slaby
Quoted 4 lines · Rafael J. Wysocki

My understanding is that the commit pointed to by Jiri caused a problem if the current mainline kernel was kexeced from an older kernel (2.6.27.x from openSUSE-11.1 in this particular case), because the older kernel didn't have the recent network driver fixes applied. Is this correct?

Exactly!

Quoted 8 lines

Also, I'm still interested in whether or not removig the following three lines:

        /* Check if we're already there */
        if (dev->current_state == state)
                return 0;

from pci_set_power_state() in the current mainline kernel fixes the problem in the configuration where it is readily reproducible.

Quoted 6 lines
Quoted 2 lines

After removing those lines, the problem still persists:
e1000: 0000:02:01.0: e1000_probe: The EEPROM Checksum Is Not Valid

So it's more complicated than I thought. Well ...

What if the driver in question is rmmoded before kexec?

Well, it should be the same, never mind.