igb: fix kexec with igb

PATCH In review Yinghai Lu 8 participants 43 messages Mar 7, 2009 – Mar 31, 2009

Subsystems (3)

Networking Network drivers PCI

lore

Share & tools

lore

Yinghai Lu

author

2009-03-07 04:35 UTC #0 lore raw

PATCH 1/1 igb: fix kexec with igb

Impact: could probe igb

Found one system with 82575EB, in the kernel that is kexeced, probe igb failed with -2.

it looks like the same behavior happened on forcedeth.

try to check system_state to make sure if put it on D3

Signed-off-by Yinghai Lu <[email protected]>

---
 drivers/net/igb/igb_main.c |   19 ++++++++++++++-----
 1 file changed, 14 insertions(+), 5 deletions(-)

drivers/net/igb/igb_main.c +14 -5

Index: linux-2.6/drivers/net/igb/igb_main.c
===================================================================
@@ -4358,14 +4358,22 @@ static int igb_suspend(struct pci_dev *p
 		wr32(E1000_WUFC, 0);
 	}
 
+	/*
+	 * Apparently it is not possible to reinitialise from D3 hot,
+	 * only put the device into D3 if we really go for poweroff.
+	 */
 	/* make sure adapter isn't asleep if manageability/wol is enabled */
 	if (wufc || adapter->en_mng_pt) {
-		pci_enable_wake(pdev, PCI_D3hot, 1);
-		pci_enable_wake(pdev, PCI_D3cold, 1);
+		if (system_state == SYSTEM_POWER_OFF) {
+			pci_enable_wake(pdev, PCI_D3hot, 1);
+			pci_enable_wake(pdev, PCI_D3cold, 1);
+		}
 	} else {
 		igb_shutdown_fiber_serdes_link_82575(hw);
-		pci_enable_wake(pdev, PCI_D3hot, 0);
-		pci_enable_wake(pdev, PCI_D3cold, 0);
+		if (system_state == SYSTEM_POWER_OFF) {
+			pci_enable_wake(pdev, PCI_D3hot, 0);
+			pci_enable_wake(pdev, PCI_D3cold, 0);
+		}
 	}
 
 	/* Release control of h/w to f/w.  If f/w is AMT enabled, this

@@ -4374,7 +4382,8 @@ static int igb_suspend(struct pci_dev *p
 
 	pci_disable_device(pdev);
 
-	pci_set_power_state(pdev, pci_choose_state(pdev, state));
+	if (system_state == SYSTEM_POWER_OFF)
+		pci_set_power_state(pdev, pci_choose_state(pdev, state));
 
 	return 0;
 }

Jesse Brandeburg

2009-03-07 07:18 UTC #1 lore raw

Quoted 8 lines · Yinghai Lu

Impact: could probe igb
Found one system with 82575EB, in the kernel that is kexeced, probe igb failed with -2.
it looks like the same behavior happened on forcedeth.

Quoted 7 lines

try to check system_state to make sure if put it on D3
Signed-off-by Yinghai Lu <[email protected]>
---
drivers/net/igb/igb_main.c | 19 ++++++++++++++-----
 1 file changed, 14 insertions(+), 5 deletions(-)

I see the point of the patch, but I know for a fact that ixgbe when enabled for MSI-X also doesn't work with kexec.

so my questions are:
are you going to change every driver?
why can't this be fixed in core kernel code instead? Shouldn't pci_enable_device take it out of D3?
Or maybe it should be taken out of D3 immediately if someone tries to ioremap any of the BARx registers?

Jesse

Yinghai Lu

author

2009-03-07 07:31 UTC #2 lore raw

Quoted 17 lines · Jesse Brandeburg

Quoted 8 lines · Yinghai Lu
Impact: could probe igb
Found one system with 82575EB, in the kernel that is kexeced, probe igb failed with -2.
it looks like the same behavior happened on forcedeth.
Quoted 7 lines
try to check system_state to make sure if put it on D3
Signed-off-by Yinghai Lu <[email protected]>
---
drivers/net/igb/igb_main.c | 19 ++++++++++++++-----
 1 file changed, 14 insertions(+), 5 deletions(-)

Quoted 5 lines

I see the point of the patch, but I know for a fact that ixgbe when enabled for MSI-X also doesn't work with kexec.
so my questions are:
are you going to change every driver?

i tend to only change driver that i have related HW.

Quoted 1 line

why can't this be fixed in core kernel code instead?

will check it.

Quoted 3 lines

Shouldn't pci_enable_device take it out of D3?
Or maybe it should be taken out of D3 immediately if someone tries to ioremap any of the BARx registers?

looks like second kernel can not detect the state any more.

Eric W. Biederman

2009-03-07 18:20 UTC #3 lore raw

Quoted 28 lines · Yinghai Lu

Quoted 17 lines · Jesse Brandeburg
Quoted 8 lines · Yinghai Lu
Impact: could probe igb
Found one system with 82575EB, in the kernel that is kexeced, probe igb failed with -2.
it looks like the same behavior happened on forcedeth.
Quoted 7 lines
try to check system_state to make sure if put it on D3
Signed-off-by Yinghai Lu <[email protected]>
---
drivers/net/igb/igb_main.c | 19 ++++++++++++++-----
 1 file changed, 14 insertions(+), 5 deletions(-)
Quoted 5 lines
I see the point of the patch, but I know for a fact that ixgbe when enabled for MSI-X also doesn't work with kexec.
so my questions are:
are you going to change every driver?
i tend to only change driver that i have related HW.
Quoted 1 line
why can't this be fixed in core kernel code instead?

Quoted 8 lines

will check it.
Quoted 3 lines
Shouldn't pci_enable_device take it out of D3?
Or maybe it should be taken out of D3 immediately if someone tries to ioremap any of the BARx registers?
looks like second kernel can not detect the state any more.

I know this has historically been a problem with the e1000 NICs. Placing the hardware in a state they can not get them out of on the reboot path.

Last I heard (a couple of weeks ago?) we had code to bring devices out of a low power state that was working for the e1000 driver.

YH can you look and see if you can find that code and if it works?

<rant>
Frankly I don't understand why anyone would want to power down a device when they are rebooting or shutting down a computer. That is a system state change. But it seems to be bleed over from the confusion that has been the power management code.
</rant>

If we can teach the kernel to handle this case with the proper enables and disables that would be great. Otherwise let's look at getting the responsibilities of the various methods sorted out so we can at least say with certainty what the various methods are supposed to do and not do.

Eric

Yinghai Lu

author

2009-03-07 18:50 UTC #4 lore raw

Quoted 37 lines · Eric W. Biederman

Quoted 35 lines · Yinghai Lu
Quoted 17 lines · Jesse Brandeburg
Quoted 8 lines · Yinghai Lu
Impact: could probe igb
Found one system with 82575EB, in the kernel that is kexeced, probe igb failed with -2.
it looks like the same behavior happened on forcedeth.
Quoted 7 lines
try to check system_state to make sure if put it on D3
Signed-off-by Yinghai Lu <[email protected]>
---
drivers/net/igb/igb_main.c | 19 ++++++++++++++-----
 1 file changed, 14 insertions(+), 5 deletions(-)
Quoted 5 lines
I see the point of the patch, but I know for a fact that ixgbe when enabled for MSI-X also doesn't work with kexec.
so my questions are:
are you going to change every driver?
i tend to only change driver that i have related HW.
Quoted 1 line
why can't this be fixed in core kernel code instead?
will check it.
Quoted 3 lines
Shouldn't pci_enable_device take it out of D3?
Or maybe it should be taken out of D3 immediately if someone tries to ioremap any of the BARx registers?

Quoted 8 lines

Quoted 1 line
looks like second kernel can not detect the state any more.
I know this has historically been a problem with the e1000 NICs. Placing the hardware in a state they can not get them out of on the reboot path.
Last I heard (a couple of weeks ago?) we had code to bring devices out of a low power state that was working for the e1000 driver.

in net-next tree?

Quoted 2 lines

YH can you look and see if you can find that code and if it works?

it seems e1000 and e1000e works well for a long time.

Quoted 7 lines

<rant>
Frankly I don't understand why anyone would want to power down a device when they are rebooting or shutting down a computer. That is a system state change. But it seems to be bleed over from the confusion that has been the power management code.
</rant>

agreed.

Quoted 6 lines

If we can teach the kernel to handle this case with the proper enables and disables that would be great. Otherwise let's look at getting the responsibilities of the various methods sorted out so we can at least say with certainty what the various methods are supposed to do and not do.

root cause could be some BIOS/acpi lying to Kernel about device state querying.

for igb, it only has problem on one platform. but other one platform is ok. for forcedeth, it seems all platform has the problem, aka you can not put it in D3 in kexec path.

Rafael J. Wysocki

2009-03-08 10:46 UTC #5 lore raw

Quoted 56 lines · Yinghai Lu

Quoted 37 lines · Eric W. Biederman
Quoted 35 lines · Yinghai Lu
Quoted 17 lines · Jesse Brandeburg
Quoted 8 lines · Yinghai Lu
Impact: could probe igb
Found one system with 82575EB, in the kernel that is kexeced, probe igb failed with -2.
it looks like the same behavior happened on forcedeth.
Quoted 7 lines
try to check system_state to make sure if put it on D3
Signed-off-by Yinghai Lu <[email protected]>
---
 drivers/net/igb/igb_main.c |   19 ++++++++++++++-----
 1 file changed, 14 insertions(+), 5 deletions(-)
Quoted 5 lines
I see the point of the patch, but I know for a fact that ixgbe when enabled for MSI-X also doesn't work with kexec.
so my questions are:
are you going to change every driver?
i tend to only change driver that i have related HW.
Quoted 1 line
why can't this be fixed in core kernel code instead?
will check it.
Quoted 3 lines
Shouldn't pci_enable_device take it out of D3?
Or maybe it should be taken out of D3 immediately if someone tries to ioremap any of the BARx registers?
Quoted 8 lines
Quoted 1 line
looks like second kernel can not detect the state any more.
I know this has historically been a problem with the e1000 NICs. Placing the hardware in a state they can not get them out of on the reboot path.
Last I heard (a couple of weeks ago?) we had code to bring devices out of a low power state that was working for the e1000 driver.
in net-next tree?
Quoted 2 lines
YH can you look and see if you can find that code and if it works?
it seems e1000 and e1000e works well for a long time.
Quoted 1 line

Quoted 8 lines

Quoted 6 lines
<rant>
Frankly I don't understand why anyone would want to power down a device when they are rebooting or shutting down a computer. That is a system state change. But it seems to be bleed over from the confusion that has been the power management code.
</rant>
agreed.

Well, well.

IMHO, the confusion is that ->shutdown() is used for too many _different_ things, because kexec is sufficiently different from power off, for example, to be handled by a separate driver callback. I'm totally unsure what power management has to do with it, though.

Thanks,
Rafael

Rafael J. Wysocki

2009-03-08 10:57 UTC #6 lore raw

In reply to Yinghai Lu

Quoted 28 lines · Yinghai Lu

Quoted 17 lines · Jesse Brandeburg
Quoted 8 lines · Yinghai Lu
Impact: could probe igb
Found one system with 82575EB, in the kernel that is kexeced, probe igb failed with -2.
it looks like the same behavior happened on forcedeth.
Quoted 7 lines
try to check system_state to make sure if put it on D3
Signed-off-by Yinghai Lu <[email protected]>
---
 drivers/net/igb/igb_main.c |   19 ++++++++++++++-----
 1 file changed, 14 insertions(+), 5 deletions(-)
Quoted 5 lines
I see the point of the patch, but I know for a fact that ixgbe when enabled for MSI-X also doesn't work with kexec.
so my questions are:
are you going to change every driver?
i tend to only change driver that i have related HW.
Quoted 1 line
why can't this be fixed in core kernel code instead?

Quoted 8 lines

will check it.
Quoted 3 lines
Shouldn't pci_enable_device take it out of D3?
Or maybe it should be taken out of D3 immediately if someone tries to ioremap any of the BARx registers?
looks like second kernel can not detect the state any more.

In fact pci_enable_device() calls pci_set_power_state(dev, PCI_D0) as the first thing. The question is why it doesn't work as expected.

What kernel(s) have you tested?

Thanks,
Rafael

Yinghai Lu

author

2009-03-08 18:04 UTC #7 lore raw

Quoted 7 lines · Rafael J. Wysocki

Quoted 2 lines
looks like second kernel can not detect the state any more.
In fact pci_enable_device() calls pci_set_power_state(dev, PCI_D0) as the first thing. The question is why it doesn't work as expected.
What kernel(s) have you tested?

tip/master

also RHEL 5.3 with igb + kexec have the same problem.

Yinghai Lu

author Tested-by Acked-by Jeff Garzik

2009-03-08 18:11 UTC #8 lore raw

In reply to Rafael J. Wysocki

Quoted 28 lines · Rafael J. Wysocki

Quoted 27 lines · Yinghai Lu
Quoted 12 lines · Jesse Brandeburg
Quoted 11 lines · Yinghai Lu
Impact: could probe igb
Found one system with 82575EB, in the kernel that is kexeced, probe igb failed with -2.
it looks like the same behavior happened on forcedeth.
try to check system_state to make sure if put it on D3
Signed-off-by Yinghai Lu <[email protected]>
Quoted 8 lines
Quoted 3 lines
---
 drivers/net/igb/igb_main.c |   19 ++++++++++++++-----
 1 file changed, 14 insertions(+), 5 deletions(-)
I see the point of the patch, but I know for a fact that ixgbe when enabled for MSI-X also doesn't work with kexec.
so my questions are:
are you going to change every driver?
i tend to only change driver that i have related HW.
Quoted 1 line
why can't this be fixed in core kernel code instead?
will check it.

Quoted 8 lines

Quoted 5 lines
Quoted 3 lines
Shouldn't pci_enable_device take it out of D3?
Or maybe it should be taken out of D3 immediately if someone tries to ioremap any of the BARx registers?
looks like second kernel can not detect the state any more.
In fact pci_enable_device() calls pci_set_power_state(dev, PCI_D0) as the first thing. The question is why it doesn't work as expected.

not sure... please check the version for forcedeth that you made.

commit 3cb5599a84c557c0dd9a19feb63a3788268cf249

Author: Rafael J. Wysocki <[email protected]>
Date:   Fri Sep 5 14:00:19 2008 -0700

forcedeth: fix kexec regression

Fix regression tracked as http://bugzilla.kernel.org/show_bug.cgi?id=11361
and caused by commit f735a2a1a4f2a0f5cd823ce323e82675990469e2 ("[netdrvr]
forcedeth: setup wake-on-lan before shutting down") that makes network adapters integrated into the NVidia MCP55 chipsets fail to work in kexeced kernels. The problem appears to be that if the adapter is put into D3_hot during ->shutdown(), it cannot be brought back into D0 after kexec (ref.
http://marc.info/?l=linux-kernel&m=121900062814967&w=4). Therefore, only put forcedeth into D3 during ->shutdown() if the system is to be powered off.

Signed-off-by Rafael J. Wysocki <[email protected]>

Tested-by Yinghai Lu <[email protected]>

Cc Ayaz Abdulla <[email protected]>

Acked-by Jeff Garzik <[email protected]>

Signed-off-by Andrew Morton <[email protected]>

Signed-off-by Linus Torvalds <[email protected]>

drivers/net/forcedeth.c +5 -3

index 331b86b..0b6ecef 100644
@@ -5975,10 +5975,12 @@ static void nv_shutdown(struct pci_dev *pdev)
        if (netif_running(dev))
                nv_close(dev);
 
-       pci_enable_wake(pdev, PCI_D3hot, np->wolenabled);
-       pci_enable_wake(pdev, PCI_D3cold, np->wolenabled);
        pci_disable_device(pdev);
-       pci_set_power_state(pdev, PCI_D3hot);
+       if (system_state == SYSTEM_POWER_OFF) {
+               if (pci_enable_wake(pdev, PCI_D3cold, np->wolenabled))
+                       pci_enable_wake(pdev, PCI_D3hot, np->wolenabled);
+               pci_set_power_state(pdev, PCI_D3hot);
+       }
 }
 #else
 #define nv_suspend NULL

Rafael J. Wysocki

2009-03-08 21:08 UTC #9 lore raw

Quoted 46 lines · Yinghai Lu

Quoted 28 lines · Rafael J. Wysocki
Quoted 27 lines · Yinghai Lu
Quoted 12 lines · Jesse Brandeburg
Quoted 11 lines · Yinghai Lu
Impact: could probe igb
Found one system with 82575EB, in the kernel that is kexeced, probe igb failed with -2.
it looks like the same behavior happened on forcedeth.
try to check system_state to make sure if put it on D3
Signed-off-by Yinghai Lu <[email protected]>
Quoted 8 lines
Quoted 3 lines
---
 drivers/net/igb/igb_main.c |   19 ++++++++++++++-----
 1 file changed, 14 insertions(+), 5 deletions(-)
I see the point of the patch, but I know for a fact that ixgbe when enabled for MSI-X also doesn't work with kexec.
so my questions are:
are you going to change every driver?
i tend to only change driver that i have related HW.
Quoted 1 line
why can't this be fixed in core kernel code instead?
will check it.
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Shouldn't pci_enable_device take it out of D3?
Or maybe it should be taken out of D3 immediately if someone tries to ioremap any of the BARx registers?
looks like second kernel can not detect the state any more.
In fact pci_enable_device() calls pci_set_power_state(dev, PCI_D0) as the first thing. The question is why it doesn't work as expected.
not sure... please check the version for forcedeth that you made.
commit 3cb5599a84c557c0dd9a19feb63a3788268cf249
Author: Rafael J. Wysocki <[email protected]>
Date:   Fri Sep 5 14:00:19 2008 -0700
forcedeth: fix kexec regression

Quoted 9 lines

Fix regression tracked as http://bugzilla.kernel.org/show_bug.cgi?id=11361
and caused by commit f735a2a1a4f2a0f5cd823ce323e82675990469e2 ("[netdrvr]
forcedeth: setup wake-on-lan before shutting down") that makes network adapters integrated into the NVidia MCP55 chipsets fail to work in kexeced kernels. The problem appears to be that if the adapter is put into D3_hot during ->shutdown(), it cannot be brought back into D0 after kexec (ref.
http://marc.info/?l=linux-kernel&m=121900062814967&w=4). Therefore, only put forcedeth into D3 during ->shutdown() if the system is to be powered off.

Thanks, I remember now.

This appears to be quirky hardware, doesn't it?

Rafael

Yinghai Lu

author

2009-03-08 21:20 UTC #10 lore raw

Quoted 54 lines · Rafael J. Wysocki

Quoted 43 lines · Yinghai Lu
Quoted 26 lines · Rafael J. Wysocki
Quoted 25 lines · Yinghai Lu
Quoted 12 lines · Jesse Brandeburg
Quoted 11 lines · Yinghai Lu
Impact: could probe igb
Found one system with 82575EB, in the kernel that is kexeced, probe igb failed with -2.
it looks like the same behavior happened on forcedeth.
try to check system_state to make sure if put it on D3
Signed-off-by Yinghai Lu <[email protected]>
Quoted 8 lines
Quoted 3 lines
---
 drivers/net/igb/igb_main.c |   19 ++++++++++++++-----
 1 file changed, 14 insertions(+), 5 deletions(-)
I see the point of the patch, but I know for a fact that ixgbe when enabled for MSI-X also doesn't work with kexec.
so my questions are:
are you going to change every driver?
i tend to only change driver that i have related HW.
Quoted 1 line
why can't this be fixed in core kernel code instead?
Quoted 8 lines
Quoted 6 lines
will check it.
Quoted 3 lines
Shouldn't pci_enable_device take it out of D3?
Or maybe it should be taken out of D3 immediately if someone tries to ioremap any of the BARx registers?
looks like second kernel can not detect the state any more.
In fact pci_enable_device() calls pci_set_power_state(dev, PCI_D0) as the first thing. The question is why it doesn't work as expected.
not sure... please check the version for forcedeth that you made.
commit 3cb5599a84c557c0dd9a19feb63a3788268cf249
Author: Rafael J. Wysocki <[email protected]>
Date:   Fri Sep 5 14:00:19 2008 -0700
forcedeth: fix kexec regression
Quoted 9 lines
Fix regression tracked as http://bugzilla.kernel.org/show_bug.cgi?id=11361
and caused by commit f735a2a1a4f2a0f5cd823ce323e82675990469e2 ("[netdrvr]
forcedeth: setup wake-on-lan before shutting down") that makes network adapters integrated into the NVidia MCP55 chipsets fail to work in kexeced kernels. The problem appears to be that if the adapter is put into D3_hot during ->shutdown(), it cannot be brought back into D0 after kexec (ref.
http://marc.info/?l=linux-kernel&m=121900062814967&w=4). Therefore, only put forcedeth into D3 during ->shutdown() if the system is to be powered off.

Quoted 4 lines

Thanks, I remember now.
This appears to be quirky hardware, doesn't it?

all my systems with ck804 and mcp55 need this trick.

for igb: one system need this trick, and other doesn't need this trick. could BIOS/ACPI difference or the nic option rom difference.

but for kexec path, we really don't need put pci devices to D3.

Rafael J. Wysocki

2009-03-08 21:40 UTC #11 lore raw

Quoted 57 lines · Yinghai Lu

Quoted 56 lines · Rafael J. Wysocki
Quoted 43 lines · Yinghai Lu
Quoted 26 lines · Rafael J. Wysocki
Quoted 25 lines · Yinghai Lu
Quoted 12 lines · Jesse Brandeburg
Quoted 11 lines · Yinghai Lu
Impact: could probe igb
Found one system with 82575EB, in the kernel that is kexeced, probe igb failed with -2.
it looks like the same behavior happened on forcedeth.
try to check system_state to make sure if put it on D3
Signed-off-by Yinghai Lu <[email protected]>
Quoted 8 lines
Quoted 3 lines
---
 drivers/net/igb/igb_main.c |   19 ++++++++++++++-----
 1 file changed, 14 insertions(+), 5 deletions(-)
I see the point of the patch, but I know for a fact that ixgbe when enabled for MSI-X also doesn't work with kexec.
so my questions are:
are you going to change every driver?
i tend to only change driver that i have related HW.
Quoted 1 line
why can't this be fixed in core kernel code instead?
Quoted 8 lines
Quoted 6 lines
will check it.
Quoted 3 lines
Shouldn't pci_enable_device take it out of D3?
Or maybe it should be taken out of D3 immediately if someone tries to ioremap any of the BARx registers?
looks like second kernel can not detect the state any more.
In fact pci_enable_device() calls pci_set_power_state(dev, PCI_D0) as the first thing. The question is why it doesn't work as expected.
not sure... please check the version for forcedeth that you made.
commit 3cb5599a84c557c0dd9a19feb63a3788268cf249
Author: Rafael J. Wysocki <[email protected]>
Date:   Fri Sep 5 14:00:19 2008 -0700
forcedeth: fix kexec regression
Quoted 9 lines
Fix regression tracked as http://bugzilla.kernel.org/show_bug.cgi?id=11361
and caused by commit f735a2a1a4f2a0f5cd823ce323e82675990469e2 ("[netdrvr]
forcedeth: setup wake-on-lan before shutting down") that makes network adapters integrated into the NVidia MCP55 chipsets fail to work in kexeced kernels. The problem appears to be that if the adapter is put into D3_hot during ->shutdown(), it cannot be brought back into D0 after kexec (ref.
http://marc.info/?l=linux-kernel&m=121900062814967&w=4). Therefore, only put forcedeth into D3 during ->shutdown() if the system is to be powered off.
Thanks, I remember now.

Quoted 8 lines

Quoted 2 lines
This appears to be quirky hardware, doesn't it?
all my systems with ck804 and mcp55 need this trick.
for igb: one system need this trick, and other doesn't need this trick. could BIOS/ACPI difference or the nic option rom difference.
but for kexec path, we really don't need put pci devices to D3.

Agreed, but pci_set_power_state() may be called for too many different reasons to be a suitable place for a fix IMO.

Thanks,
Rafael

Rafael J. Wysocki

2009-03-08 21:32 UTC #12 lore raw

In reply to Rafael J. Wysocki

Quoted 57 lines · Rafael J. Wysocki

Quoted 46 lines · Yinghai Lu
Quoted 28 lines · Rafael J. Wysocki
Quoted 27 lines · Yinghai Lu
Quoted 12 lines · Jesse Brandeburg
Quoted 11 lines · Yinghai Lu
Impact: could probe igb
Found one system with 82575EB, in the kernel that is kexeced, probe igb failed with -2.
it looks like the same behavior happened on forcedeth.
try to check system_state to make sure if put it on D3
Signed-off-by Yinghai Lu <[email protected]>
Quoted 8 lines
Quoted 3 lines
---
 drivers/net/igb/igb_main.c |   19 ++++++++++++++-----
 1 file changed, 14 insertions(+), 5 deletions(-)
I see the point of the patch, but I know for a fact that ixgbe when enabled for MSI-X also doesn't work with kexec.
so my questions are:
are you going to change every driver?
i tend to only change driver that i have related HW.
Quoted 1 line
why can't this be fixed in core kernel code instead?
will check it.
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Shouldn't pci_enable_device take it out of D3?
Or maybe it should be taken out of D3 immediately if someone tries to ioremap any of the BARx registers?
looks like second kernel can not detect the state any more.
In fact pci_enable_device() calls pci_set_power_state(dev, PCI_D0) as the first thing. The question is why it doesn't work as expected.
not sure... please check the version for forcedeth that you made.
commit 3cb5599a84c557c0dd9a19feb63a3788268cf249
Author: Rafael J. Wysocki <[email protected]>
Date:   Fri Sep 5 14:00:19 2008 -0700
forcedeth: fix kexec regression
Quoted 9 lines
Fix regression tracked as http://bugzilla.kernel.org/show_bug.cgi?id=11361
and caused by commit f735a2a1a4f2a0f5cd823ce323e82675990469e2 ("[netdrvr]
forcedeth: setup wake-on-lan before shutting down") that makes network adapters integrated into the NVidia MCP55 chipsets fail to work in kexeced kernels. The problem appears to be that if the adapter is put into D3_hot during ->shutdown(), it cannot be brought back into D0 after kexec (ref.
http://marc.info/?l=linux-kernel&m=121900062814967&w=4). Therefore, only put forcedeth into D3 during ->shutdown() if the system is to be powered off.

Quoted 1 line

Thanks, I remember now.

In which case you need to rework igb_shutdown() rather than igb_suspend().

Something like the patch below, perhaps (totally untested).

Thanks,
Rafael

---
 drivers/net/igb/igb_main.c |   24 +++++++++++++++++++-----
 1 file changed, 19 insertions(+), 5 deletions(-)

drivers/net/igb/igb_main.c +19 -5

Index: linux-2.6/drivers/net/igb/igb_main.c
===================================================================
@@ -4299,7 +4299,7 @@ int igb_set_spd_dplx(struct igb_adapter
 }
 
 
-static int igb_suspend(struct pci_dev *pdev, pm_message_t state)
+static int __igb_shutdown(struct pci_dev *pdev, bool enable_wake)
 {
 	struct net_device *netdev = pci_get_drvdata(pdev);
 	struct igb_adapter *adapter = netdev_priv(netdev);

@@ -4359,7 +4359,7 @@ static int igb_suspend(struct pci_dev *p
 	}
 
 	/* make sure adapter isn't asleep if manageability/wol is enabled */
-	if (wufc || adapter->en_mng_pt) {
+	if ((wufc || adapter->en_mng_pt) && enable_wake) {
 		pci_enable_wake(pdev, PCI_D3hot, 1);
 		pci_enable_wake(pdev, PCI_D3cold, 1);
 	} else {

@@ -4374,12 +4374,21 @@ static int igb_suspend(struct pci_dev *p
 
 	pci_disable_device(pdev);
 
-	pci_set_power_state(pdev, pci_choose_state(pdev, state));
-
 	return 0;
 }
 
 #ifdef CONFIG_PM
+static int igb_suspend(struct pci_dev *pdev, pm_message_t state)
+{
+	int retval;
+
+	retval = __igb_shutdown(pdev, true);
+	if (!retval)
+		pci_set_power_state(pdev, PCI_D3hot);
+
+	return retval;
+}
+
 static int igb_resume(struct pci_dev *pdev)
 {
 	struct net_device *netdev = pci_get_drvdata(pdev);

@@ -4434,7 +4443,12 @@ static int igb_resume(struct pci_dev *pd
 
 static void igb_shutdown(struct pci_dev *pdev)
 {
-	igb_suspend(pdev, PMSG_SUSPEND);
+	if (system_state == SYSTEM_POWER_OFF) {
+		__igb_shutdown(pdev, true);
+		pci_set_power_state(pdev, PCI_D3hot);
+	} else {
+		__igb_shutdown(pdev, false);
+	}
 }
 
 #ifdef CONFIG_NET_POLL_CONTROLLER

Yinghai Lu

author

2009-03-08 22:35 UTC #13 lore raw

Quoted 118 lines · Rafael J. Wysocki

Quoted 57 lines · Rafael J. Wysocki
Quoted 46 lines · Yinghai Lu
Quoted 28 lines · Rafael J. Wysocki
Quoted 27 lines · Yinghai Lu
Quoted 12 lines · Jesse Brandeburg
Quoted 11 lines · Yinghai Lu
Impact: could probe igb
Found one system with 82575EB, in the kernel that is kexeced, probe igb failed with -2.
it looks like the same behavior happened on forcedeth.
try to check system_state to make sure if put it on D3
Signed-off-by Yinghai Lu <[email protected]>
Quoted 8 lines
Quoted 3 lines
---
drivers/net/igb/igb_main.c | 19 ++++++++++++++-----
 1 file changed, 14 insertions(+), 5 deletions(-)
I see the point of the patch, but I know for a fact that ixgbe when enabled for MSI-X also doesn't work with kexec.
so my questions are:
are you going to change every driver?
i tend to only change driver that i have related HW.
Quoted 1 line
why can't this be fixed in core kernel code instead?
will check it.
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Shouldn't pci_enable_device take it out of D3?
Or maybe it should be taken out of D3 immediately if someone tries to ioremap any of the BARx registers?
looks like second kernel can not detect the state any more.
In fact pci_enable_device() calls pci_set_power_state(dev, PCI_D0) as the first thing. The question is why it doesn't work as expected.
not sure... please check the version for forcedeth that you made.
commit 3cb5599a84c557c0dd9a19feb63a3788268cf249
Author: Rafael J. Wysocki <[email protected]>
Date: Fri Sep 5 14:00:19 2008 -0700
forcedeth: fix kexec regression
Quoted 9 lines
Fix regression tracked as http://bugzilla.kernel.org/show_bug.cgi?id=11361
and caused by commit f735a2a1a4f2a0f5cd823ce323e82675990469e2 ("[netdrvr]
forcedeth: setup wake-on-lan before shutting down") that makes network adapters integrated into the NVidia MCP55 chipsets fail to work in kexeced kernels. The problem appears to be that if the adapter is put into D3_hot during ->shutdown(), it cannot be brought back into D0 after kexec (ref.
http://marc.info/?l=linux-kernel&m=121900062814967&w=4). Therefore, only put forcedeth into D3 during ->shutdown() if the system is to be powered off.
Quoted 1 line
Thanks, I remember now.
In which case you need to rework igb_shutdown() rather than igb_suspend().
Something like the patch below, perhaps (totally untested).
Thanks,
Rafael
---
drivers/net/igb/igb_main.c | 24 +++++++++++++++++++-----
 1 file changed, 19 insertions(+), 5 deletions(-)
Index: linux-2.6/drivers/net/igb/igb_main.c
===================================================================
drivers/net/igb/igb_main.c +13 -4
@@ -4299,7 +4299,7 @@ int igb_set_spd_dplx(struct igb_adapter
  }
 
 
-static int igb_suspend(struct pci_dev *pdev, pm_message_t state)
+static int __igb_shutdown(struct pci_dev *pdev, bool enable_wake)
  {
        struct net_device *netdev = pci_get_drvdata(pdev);
        struct igb_adapter *adapter = netdev_priv(netdev);
@@ -4359,7 +4359,7 @@ static int igb_suspend(struct pci_dev *p
        }
 
        /* make sure adapter isn't asleep if manageability/wol is enabled */
-       if (wufc || adapter->en_mng_pt) {
+       if ((wufc || adapter->en_mng_pt) && enable_wake) {
                pci_enable_wake(pdev, PCI_D3hot, 1);
                pci_enable_wake(pdev, PCI_D3cold, 1);
        } else {
@@ -4374,12 +4374,21 @@ static int igb_suspend(struct pci_dev *p
 
        pci_disable_device(pdev);
 
-       pci_set_power_state(pdev, pci_choose_state(pdev, state));
-
        return 0;
  }
 
  #ifdef CONFIG_PM
+static int igb_suspend(struct pci_dev *pdev, pm_message_t state)
+{
+       int retval;
+
+       retval = __igb_shutdown(pdev, true);
+       if (!retval)
+               pci_set_power_state(pdev, PCI_D3hot);
+
+       return retval;
+}
+
  static int igb_resume(struct pci_dev *pdev)
  {
        struct net_device *netdev = pci_get_drvdata(pdev);

drivers/net/igb/igb_main.c static int igb_resume(struct pci_dev *pd · line 4448

Quoted 7 lines

drivers/net/igb/igb_main.c +3 -1

@@ -4434,7 +4443,12 @@ static int igb_resume(struct pci_dev *pd
 
  static void igb_shutdown(struct pci_dev *pdev)
  {
-       igb_suspend(pdev, PMSG_SUSPEND);
+       if (system_state == SYSTEM_POWER_OFF) {
+               __igb_shutdown(pdev, true);
+               pci_set_power_state(pdev, PCI_D3hot);

you don't need to use pci_choose_state(pdev, state), but use PCI-D3hot directly?

Rafael J. Wysocki

2009-03-08 22:57 UTC #14 lore raw

Quoted 120 lines · Yinghai Lu

Quoted 119 lines · Rafael J. Wysocki
Quoted 57 lines · Rafael J. Wysocki
Quoted 46 lines · Yinghai Lu
Quoted 28 lines · Rafael J. Wysocki
Quoted 27 lines · Yinghai Lu
Quoted 12 lines · Jesse Brandeburg
Quoted 11 lines · Yinghai Lu
Impact: could probe igb
Found one system with 82575EB, in the kernel that is kexeced, probe igb failed with -2.
it looks like the same behavior happened on forcedeth.
try to check system_state to make sure if put it on D3
Signed-off-by Yinghai Lu <[email protected]>
Quoted 8 lines
Quoted 3 lines
---
 drivers/net/igb/igb_main.c |   19 ++++++++++++++-----
 1 file changed, 14 insertions(+), 5 deletions(-)
I see the point of the patch, but I know for a fact that ixgbe when enabled for MSI-X also doesn't work with kexec.
so my questions are:
are you going to change every driver?
i tend to only change driver that i have related HW.
Quoted 1 line
why can't this be fixed in core kernel code instead?
will check it.
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Shouldn't pci_enable_device take it out of D3?
Or maybe it should be taken out of D3 immediately if someone tries to ioremap any of the BARx registers?
looks like second kernel can not detect the state any more.
In fact pci_enable_device() calls pci_set_power_state(dev, PCI_D0) as the first thing. The question is why it doesn't work as expected.
not sure... please check the version for forcedeth that you made.
commit 3cb5599a84c557c0dd9a19feb63a3788268cf249
Author: Rafael J. Wysocki <[email protected]>
Date:   Fri Sep 5 14:00:19 2008 -0700
forcedeth: fix kexec regression
Quoted 9 lines
Fix regression tracked as http://bugzilla.kernel.org/show_bug.cgi?id=11361
and caused by commit f735a2a1a4f2a0f5cd823ce323e82675990469e2 ("[netdrvr]
forcedeth: setup wake-on-lan before shutting down") that makes network adapters integrated into the NVidia MCP55 chipsets fail to work in kexeced kernels. The problem appears to be that if the adapter is put into D3_hot during ->shutdown(), it cannot be brought back into D0 after kexec (ref.
http://marc.info/?l=linux-kernel&m=121900062814967&w=4). Therefore, only put forcedeth into D3 during ->shutdown() if the system is to be powered off.
Quoted 1 line
Thanks, I remember now.
In which case you need to rework igb_shutdown() rather than igb_suspend().
Something like the patch below, perhaps (totally untested).
Thanks,
Rafael
---
 drivers/net/igb/igb_main.c |   24 +++++++++++++++++++-----
 1 file changed, 19 insertions(+), 5 deletions(-)
Index: linux-2.6/drivers/net/igb/igb_main.c
===================================================================
drivers/net/igb/igb_main.c +13 -4
@@ -4299,7 +4299,7 @@ int igb_set_spd_dplx(struct igb_adapter
 }
 
 
-static int igb_suspend(struct pci_dev *pdev, pm_message_t state)
+static int __igb_shutdown(struct pci_dev *pdev, bool enable_wake)
 {
       struct net_device *netdev = pci_get_drvdata(pdev);
       struct igb_adapter *adapter = netdev_priv(netdev);
@@ -4359,7 +4359,7 @@ static int igb_suspend(struct pci_dev *p
       }
 
       /* make sure adapter isn't asleep if manageability/wol is enabled */
-       if (wufc || adapter->en_mng_pt) {
+       if ((wufc || adapter->en_mng_pt) && enable_wake) {
               pci_enable_wake(pdev, PCI_D3hot, 1);
               pci_enable_wake(pdev, PCI_D3cold, 1);
       } else {
@@ -4374,12 +4374,21 @@ static int igb_suspend(struct pci_dev *p
 
       pci_disable_device(pdev);
 
-       pci_set_power_state(pdev, pci_choose_state(pdev, state));
-
       return 0;
 }
 
 #ifdef CONFIG_PM
+static int igb_suspend(struct pci_dev *pdev, pm_message_t state)
+{
+       int retval;
+
+       retval = __igb_shutdown(pdev, true);
+       if (!retval)
+               pci_set_power_state(pdev, PCI_D3hot);
+
+       return retval;
+}
+
 static int igb_resume(struct pci_dev *pdev)
 {
       struct net_device *netdev = pci_get_drvdata(pdev);
@@ -4434,7 +4443,12 @@ static int igb_resume(struct pci_dev *pd
 

Quoted 8 lines

drivers/net/igb/igb_main.c static int igb_resume(struct pci_dev *pd · line 4448
Quoted 6 lines
drivers/net/igb/igb_main.c +3 -1
 static void igb_shutdown(struct pci_dev *pdev)
 {
-       igb_suspend(pdev, PMSG_SUSPEND);
+       if (system_state == SYSTEM_POWER_OFF) {
+               __igb_shutdown(pdev, true);
+               pci_set_power_state(pdev, PCI_D3hot);
you don't need to use pci_choose_state(pdev, state), but use PCI-D3hot directly?

Yes, because we've already decided we'd put the device into D3_hot by calling pci_enable_wake(pdev, PCI_D3hot, 1) in __igb_shutdown().

In fact we should first determine the target state and _then_ call pci_enable_wake() and pci_set_power_state() with that as the argument, but this is a separate issue. For now, IMO, it's better to use D3_hot in both places directly.

On a slightly related note, the sequence

                pci_enable_wake(pdev, PCI_D3hot, 1);
                pci_enable_wake(pdev, PCI_D3cold, 1);

is (slightly) incorrect by itself, because ACPI doesn't distinguish bewteen D3_hot and D3_cold, so this just causes the same platform code to run twice, which may or may not work (it theory it should always work, but still).

This is yet another problem, though.

Thanks,
Rafael

Yinghai Lu

author

2009-03-08 23:04 UTC #15 lore raw

Quoted 140 lines · Rafael J. Wysocki

Quoted 120 lines · Yinghai Lu
Quoted 119 lines · Rafael J. Wysocki
Quoted 57 lines · Rafael J. Wysocki
Quoted 46 lines · Yinghai Lu
Quoted 28 lines · Rafael J. Wysocki
Quoted 27 lines · Yinghai Lu
Quoted 12 lines · Jesse Brandeburg
Quoted 11 lines · Yinghai Lu
Impact: could probe igb
Found one system with 82575EB, in the kernel that is kexeced, probe igb failed with -2.
it looks like the same behavior happened on forcedeth.
try to check system_state to make sure if put it on D3
Signed-off-by Yinghai Lu <[email protected]>
Quoted 8 lines
Quoted 3 lines
---
drivers/net/igb/igb_main.c | 19 ++++++++++++++-----
 1 file changed, 14 insertions(+), 5 deletions(-)
I see the point of the patch, but I know for a fact that ixgbe when enabled for MSI-X also doesn't work with kexec.
so my questions are:
are you going to change every driver?
i tend to only change driver that i have related HW.
Quoted 1 line
why can't this be fixed in core kernel code instead?
will check it.
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Shouldn't pci_enable_device take it out of D3?
Or maybe it should be taken out of D3 immediately if someone tries to ioremap any of the BARx registers?
looks like second kernel can not detect the state any more.
In fact pci_enable_device() calls pci_set_power_state(dev, PCI_D0) as the first thing. The question is why it doesn't work as expected.
not sure... please check the version for forcedeth that you made.
commit 3cb5599a84c557c0dd9a19feb63a3788268cf249
Author: Rafael J. Wysocki <[email protected]>
Date: Fri Sep 5 14:00:19 2008 -0700
forcedeth: fix kexec regression
Quoted 9 lines
Fix regression tracked as http://bugzilla.kernel.org/show_bug.cgi?id=11361
and caused by commit f735a2a1a4f2a0f5cd823ce323e82675990469e2 ("[netdrvr]
forcedeth: setup wake-on-lan before shutting down") that makes network adapters integrated into the NVidia MCP55 chipsets fail to work in kexeced kernels. The problem appears to be that if the adapter is put into D3_hot during ->shutdown(), it cannot be brought back into D0 after kexec (ref.
http://marc.info/?l=linux-kernel&m=121900062814967&w=4). Therefore, only put forcedeth into D3 during ->shutdown() if the system is to be powered off.
Quoted 1 line
Thanks, I remember now.
In which case you need to rework igb_shutdown() rather than igb_suspend().
Something like the patch below, perhaps (totally untested).
Thanks,
Rafael
---
drivers/net/igb/igb_main.c | 24 +++++++++++++++++++-----
 1 file changed, 19 insertions(+), 5 deletions(-)
Index: linux-2.6/drivers/net/igb/igb_main.c
===================================================================
drivers/net/igb/igb_main.c +13 -4
@@ -4299,7 +4299,7 @@ int igb_set_spd_dplx(struct igb_adapter
  }
 
 
-static int igb_suspend(struct pci_dev *pdev, pm_message_t state)
+static int __igb_shutdown(struct pci_dev *pdev, bool enable_wake)
  {
        struct net_device *netdev = pci_get_drvdata(pdev);
        struct igb_adapter *adapter = netdev_priv(netdev);
@@ -4359,7 +4359,7 @@ static int igb_suspend(struct pci_dev *p
        }
 
        /* make sure adapter isn't asleep if manageability/wol is enabled */
-       if (wufc || adapter->en_mng_pt) {
+       if ((wufc || adapter->en_mng_pt) && enable_wake) {
                pci_enable_wake(pdev, PCI_D3hot, 1);
                pci_enable_wake(pdev, PCI_D3cold, 1);
        } else {
@@ -4374,12 +4374,21 @@ static int igb_suspend(struct pci_dev *p
 
        pci_disable_device(pdev);
 
-       pci_set_power_state(pdev, pci_choose_state(pdev, state));
-
        return 0;
  }
 
  #ifdef CONFIG_PM
+static int igb_suspend(struct pci_dev *pdev, pm_message_t state)
+{
+       int retval;
+
+       retval = __igb_shutdown(pdev, true);
+       if (!retval)
+               pci_set_power_state(pdev, PCI_D3hot);
+
+       return retval;
+}
+
  static int igb_resume(struct pci_dev *pdev)
  {
        struct net_device *netdev = pci_get_drvdata(pdev);
@@ -4434,7 +4443,12 @@ static int igb_resume(struct pci_dev *pd
 
Quoted 8 lines
drivers/net/igb/igb_main.c static int igb_resume(struct pci_dev *pd · line 4448
Quoted 6 lines
drivers/net/igb/igb_main.c +3 -1
  static void igb_shutdown(struct pci_dev *pdev)
  {
-       igb_suspend(pdev, PMSG_SUSPEND);
+       if (system_state == SYSTEM_POWER_OFF) {
+               __igb_shutdown(pdev, true);
+               pci_set_power_state(pdev, PCI_D3hot);
you don't need to use pci_choose_state(pdev, state), but use PCI-D3hot directly?
Yes, because we've already decided we'd put the device into D3_hot by calling pci_enable_wake(pdev, PCI_D3hot, 1) in __igb_shutdown().
In fact we should first determine the target state and _then_ call pci_enable_wake() and pci_set_power_state() with that as the argument, but this is a separate issue. For now, IMO, it's better to use D3_hot in both places directly.
On a slightly related note, the sequence

Quoted 8 lines

pci_enable_wake(pdev, PCI_D3hot, 1); pci_enable_wake(pdev, PCI_D3cold, 1);
is (slightly) incorrect by itself, because ACPI doesn't distinguish bewteen D3_hot and D3_cold, so this just causes the same platform code to run twice, which may or may not work (it theory it should always work, but still).
This is yet another problem, though.

can you go through e1000, e1000e, ixgb, ixgbe too?

Rafael J. Wysocki

2009-03-08 23:58 UTC #16 lore raw

Quoted 144 lines · Yinghai Lu

Quoted 143 lines · Rafael J. Wysocki
Quoted 120 lines · Yinghai Lu
Quoted 119 lines · Rafael J. Wysocki
Quoted 57 lines · Rafael J. Wysocki
Quoted 46 lines · Yinghai Lu
Quoted 28 lines · Rafael J. Wysocki
Quoted 27 lines · Yinghai Lu
Quoted 12 lines · Jesse Brandeburg
Quoted 11 lines · Yinghai Lu
Impact: could probe igb
Found one system with 82575EB, in the kernel that is kexeced, probe igb failed with -2.
it looks like the same behavior happened on forcedeth.
try to check system_state to make sure if put it on D3
Signed-off-by Yinghai Lu <[email protected]>
Quoted 8 lines
Quoted 3 lines
---
 drivers/net/igb/igb_main.c |   19 ++++++++++++++-----
 1 file changed, 14 insertions(+), 5 deletions(-)
I see the point of the patch, but I know for a fact that ixgbe when enabled for MSI-X also doesn't work with kexec.
so my questions are:
are you going to change every driver?
i tend to only change driver that i have related HW.
Quoted 1 line
why can't this be fixed in core kernel code instead?
will check it.
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Shouldn't pci_enable_device take it out of D3?
Or maybe it should be taken out of D3 immediately if someone tries to ioremap any of the BARx registers?
looks like second kernel can not detect the state any more.
In fact pci_enable_device() calls pci_set_power_state(dev, PCI_D0) as the first thing. The question is why it doesn't work as expected.
not sure... please check the version for forcedeth that you made.
commit 3cb5599a84c557c0dd9a19feb63a3788268cf249
Author: Rafael J. Wysocki <[email protected]>
Date:   Fri Sep 5 14:00:19 2008 -0700
forcedeth: fix kexec regression
Quoted 9 lines
Fix regression tracked as http://bugzilla.kernel.org/show_bug.cgi?id=11361
and caused by commit f735a2a1a4f2a0f5cd823ce323e82675990469e2 ("[netdrvr]
forcedeth: setup wake-on-lan before shutting down") that makes network adapters integrated into the NVidia MCP55 chipsets fail to work in kexeced kernels. The problem appears to be that if the adapter is put into D3_hot during ->shutdown(), it cannot be brought back into D0 after kexec (ref.
http://marc.info/?l=linux-kernel&m=121900062814967&w=4). Therefore, only put forcedeth into D3 during ->shutdown() if the system is to be powered off.
Quoted 1 line
Thanks, I remember now.
In which case you need to rework igb_shutdown() rather than igb_suspend().
Something like the patch below, perhaps (totally untested).
Thanks,
Rafael
---
 drivers/net/igb/igb_main.c |   24 +++++++++++++++++++-----
 1 file changed, 19 insertions(+), 5 deletions(-)
Index: linux-2.6/drivers/net/igb/igb_main.c
===================================================================
drivers/net/igb/igb_main.c +13 -4
@@ -4299,7 +4299,7 @@ int igb_set_spd_dplx(struct igb_adapter
 }
 
 
-static int igb_suspend(struct pci_dev *pdev, pm_message_t state)
+static int __igb_shutdown(struct pci_dev *pdev, bool enable_wake)
 {
       struct net_device *netdev = pci_get_drvdata(pdev);
       struct igb_adapter *adapter = netdev_priv(netdev);
@@ -4359,7 +4359,7 @@ static int igb_suspend(struct pci_dev *p
       }
 
       /* make sure adapter isn't asleep if manageability/wol is enabled */
-       if (wufc || adapter->en_mng_pt) {
+       if ((wufc || adapter->en_mng_pt) && enable_wake) {
               pci_enable_wake(pdev, PCI_D3hot, 1);
               pci_enable_wake(pdev, PCI_D3cold, 1);
       } else {
@@ -4374,12 +4374,21 @@ static int igb_suspend(struct pci_dev *p
 
       pci_disable_device(pdev);
 
-       pci_set_power_state(pdev, pci_choose_state(pdev, state));
-
       return 0;
 }
 
 #ifdef CONFIG_PM
+static int igb_suspend(struct pci_dev *pdev, pm_message_t state)
+{
+       int retval;
+
+       retval = __igb_shutdown(pdev, true);
+       if (!retval)
+               pci_set_power_state(pdev, PCI_D3hot);
+
+       return retval;
+}
+
 static int igb_resume(struct pci_dev *pdev)
 {
       struct net_device *netdev = pci_get_drvdata(pdev);
@@ -4434,7 +4443,12 @@ static int igb_resume(struct pci_dev *pd
 
Quoted 8 lines
drivers/net/igb/igb_main.c static int igb_resume(struct pci_dev *pd · line 4448
Quoted 6 lines
drivers/net/igb/igb_main.c +3 -1
 static void igb_shutdown(struct pci_dev *pdev)
 {
-       igb_suspend(pdev, PMSG_SUSPEND);
+       if (system_state == SYSTEM_POWER_OFF) {
+               __igb_shutdown(pdev, true);
+               pci_set_power_state(pdev, PCI_D3hot);
you don't need to use pci_choose_state(pdev, state), but use PCI-D3hot directly?
Yes, because we've already decided we'd put the device into D3_hot by calling pci_enable_wake(pdev, PCI_D3hot, 1) in __igb_shutdown().
In fact we should first determine the target state and _then_ call pci_enable_wake() and pci_set_power_state() with that as the argument, but this is a separate issue. For now, IMO, it's better to use D3_hot in both places directly.
On a slightly related note, the sequence
               pci_enable_wake(pdev, PCI_D3hot, 1);
               pci_enable_wake(pdev, PCI_D3cold, 1);

Quoted 7 lines

Quoted 5 lines
is (slightly) incorrect by itself, because ACPI doesn't distinguish bewteen D3_hot and D3_cold, so this just causes the same platform code to run twice, which may or may not work (it theory it should always work, but still).
This is yet another problem, though.
can you go through e1000, e1000e, ixgb, ixgbe too?

OK, I will, but it's going to take some time.

Thanks,
Rafael

Yinghai Lu

author

2009-03-11 23:37 UTC #17 lore raw

In reply to Rafael J. Wysocki

Quoted 54 lines · Rafael J. Wysocki

Quoted 53 lines · Rafael J. Wysocki
Quoted 52 lines · Yinghai Lu
Quoted 26 lines · Rafael J. Wysocki
Quoted 25 lines · Yinghai Lu
Quoted 12 lines · Jesse Brandeburg
Quoted 11 lines · Yinghai Lu
Impact: could probe igb
Found one system with 82575EB, in the kernel that is kexeced, probe igb failed with -2.
it looks like the same behavior happened on forcedeth.
try to check system_state to make sure if put it on D3
Signed-off-by Yinghai Lu <[email protected]>
Quoted 8 lines
Quoted 3 lines
---
 drivers/net/igb/igb_main.c |   19 ++++++++++++++-----
 1 file changed, 14 insertions(+), 5 deletions(-)
I see the point of the patch, but I know for a fact that ixgbe when enabled for MSI-X also doesn't work with kexec.
so my questions are:
are you going to change every driver?
i tend to only change driver that i have related HW.
Quoted 1 line
why can't this be fixed in core kernel code instead?
Quoted 8 lines
Quoted 6 lines
will check it.
Quoted 3 lines
Shouldn't pci_enable_device take it out of D3?
Or maybe it should be taken out of D3 immediately if someone tries to ioremap any of the BARx registers?
looks like second kernel can not detect the state any more.
In fact pci_enable_device() calls pci_set_power_state(dev, PCI_D0) as the first thing. The question is why it doesn't work as expected.
not sure... please check the version for forcedeth that you made.
commit 3cb5599a84c557c0dd9a19feb63a3788268cf249
Author: Rafael J. Wysocki <[email protected]>
Date:   Fri Sep 5 14:00:19 2008 -0700
forcedeth: fix kexec regression
Fix regression tracked as http://bugzilla.kernel.org/show_bug.cgi?id=11361
and caused by commit f735a2a1a4f2a0f5cd823ce323e82675990469e2 ("[netdrvr]
forcedeth: setup wake-on-lan before shutting down") that makes network adapters integrated into the NVidia MCP55 chipsets fail to work in kexeced kernels. The problem appears to be that if the adapter is put into D3_hot during ->shutdown(), it cannot be brought back into D0 after kexec (ref.
http://marc.info/?l=linux-kernel&m=121900062814967&w=4). Therefore, only put forcedeth into D3 during ->shutdown() if the system is to be powered off.

Quoted 5 lines

Quoted 1 line
Thanks, I remember now.
In which case you need to rework igb_shutdown() rather than igb_suspend().
Something like the patch below, perhaps (totally untested).

it works, David, can you picked it up

---

From: Rafael J. Wysocki <[email protected]>

[PATCH] igb: fix kexec with igb -v2

Impact: could probe igb

Found one system with 82575EB, in the kernel that is kexeced, probe igb failed with -2.

it looks like the same behavior happened on forcedeth.

try to check system_state to make sure if put it on D3

v2: Rafael J. Wysocki seperate igb_shutdown and igb_suspend code

Signed-off-by Yinghai Lu <[email protected]>

---
drivers/net/igb/igb_main.c | 24 +++++++++++++++++++-----

 1 file changed, 19 insertions(+), 5 deletions(-)

drivers/net/igb/igb_main.c +19 -5

Index: linux-2.6/drivers/net/igb/igb_main.c
===================================================================
@@ -4299,7 +4299,7 @@ int igb_set_spd_dplx(struct igb_adapter
 }
 
 
-static int igb_suspend(struct pci_dev *pdev, pm_message_t state)
+static int __igb_shutdown(struct pci_dev *pdev, bool enable_wake)
 {
 	struct net_device *netdev = pci_get_drvdata(pdev);
 	struct igb_adapter *adapter = netdev_priv(netdev);

@@ -4359,7 +4359,7 @@ static int igb_suspend(struct pci_dev *p
 	}
 
 	/* make sure adapter isn't asleep if manageability/wol is enabled */
-	if (wufc || adapter->en_mng_pt) {
+	if ((wufc || adapter->en_mng_pt) && enable_wake) {
 		pci_enable_wake(pdev, PCI_D3hot, 1);
 		pci_enable_wake(pdev, PCI_D3cold, 1);
 	} else {

@@ -4374,12 +4374,21 @@ static int igb_suspend(struct pci_dev *p
 
 	pci_disable_device(pdev);
 
-	pci_set_power_state(pdev, pci_choose_state(pdev, state));
-
 	return 0;
 }
 
 #ifdef CONFIG_PM
+static int igb_suspend(struct pci_dev *pdev, pm_message_t state)
+{
+	int retval;
+
+	retval = __igb_shutdown(pdev, true);
+	if (!retval)
+		pci_set_power_state(pdev, PCI_D3hot);
+
+	return retval;
+}
+
 static int igb_resume(struct pci_dev *pdev)
 {
 	struct net_device *netdev = pci_get_drvdata(pdev);

@@ -4434,7 +4443,12 @@ static int igb_resume(struct pci_dev *pd
 
 static void igb_shutdown(struct pci_dev *pdev)
 {
-	igb_suspend(pdev, PMSG_SUSPEND);
+	if (system_state == SYSTEM_POWER_OFF) {
+		__igb_shutdown(pdev, true);
+		pci_set_power_state(pdev, PCI_D3hot);
+	} else {
+		__igb_shutdown(pdev, false);
+	}
 }
 
 #ifdef CONFIG_NET_POLL_CONTROLLER

Rafael J. Wysocki

2009-03-21 22:04 UTC #18 lore raw

Quoted 55 lines · Yinghai Lu

Quoted 54 lines · Rafael J. Wysocki
Quoted 53 lines · Rafael J. Wysocki
Quoted 52 lines · Yinghai Lu
Quoted 26 lines · Rafael J. Wysocki
Quoted 25 lines · Yinghai Lu
Quoted 12 lines · Jesse Brandeburg
Quoted 11 lines · Yinghai Lu
Impact: could probe igb
Found one system with 82575EB, in the kernel that is kexeced, probe igb failed with -2.
it looks like the same behavior happened on forcedeth.
try to check system_state to make sure if put it on D3
Signed-off-by Yinghai Lu <[email protected]>
Quoted 8 lines
Quoted 3 lines
---
 drivers/net/igb/igb_main.c |   19 ++++++++++++++-----
 1 file changed, 14 insertions(+), 5 deletions(-)
I see the point of the patch, but I know for a fact that ixgbe when enabled for MSI-X also doesn't work with kexec.
so my questions are:
are you going to change every driver?
i tend to only change driver that i have related HW.
Quoted 1 line
why can't this be fixed in core kernel code instead?
Quoted 8 lines
Quoted 6 lines
will check it.
Quoted 3 lines
Shouldn't pci_enable_device take it out of D3?
Or maybe it should be taken out of D3 immediately if someone tries to ioremap any of the BARx registers?
looks like second kernel can not detect the state any more.
In fact pci_enable_device() calls pci_set_power_state(dev, PCI_D0) as the first thing. The question is why it doesn't work as expected.
not sure... please check the version for forcedeth that you made.
commit 3cb5599a84c557c0dd9a19feb63a3788268cf249
Author: Rafael J. Wysocki <[email protected]>
Date:   Fri Sep 5 14:00:19 2008 -0700
forcedeth: fix kexec regression
Fix regression tracked as http://bugzilla.kernel.org/show_bug.cgi?id=11361
and caused by commit f735a2a1a4f2a0f5cd823ce323e82675990469e2 ("[netdrvr]
forcedeth: setup wake-on-lan before shutting down") that makes network adapters integrated into the NVidia MCP55 chipsets fail to work in kexeced kernels. The problem appears to be that if the adapter is put into D3_hot during ->shutdown(), it cannot be brought back into D0 after kexec (ref.
http://marc.info/?l=linux-kernel&m=121900062814967&w=4). Therefore, only put forcedeth into D3 during ->shutdown() if the system is to be powered off.

Quoted 7 lines

Quoted 5 lines
Quoted 1 line
Thanks, I remember now.
In which case you need to rework igb_shutdown() rather than igb_suspend().
Something like the patch below, perhaps (totally untested).
it works, David, can you picked it up

Still, Yinghai, can you please also test the patch below? It fixes all shortcomings in the driver's suspend and shutdown methods I was talking about in one of the previous messages. If it works, IMO it will be a preferable fix (in particular, it would be good to check if WoL still works with it, but I don't have the hardware).

Thanks,
Rafael

---
From: Rafael J. Wysocki <[email protected]>
Subject: net/igb: Fix kexec with igb (rev. 3)
Impact: Fix

Yinghai Lu found one system with 82575EB where, in the kernel that is kexeced, probe igb failed with -2, the reason being that the adapter could not be brought back from D3 by the kexec kernel, most probably due to quirky hardware (it looks like the same behavior happened on forcedeth).

Prevent igb from putting the adapter into D3 during shutdown except when we going to power off the system. For this purpose, seperate igb_shutdown() from igb_suspend() and use the appropriate PCI PM callbacks in both of them.

Signed-off-by Rafael J. Wysocki <[email protected]>

Reported-by Yinghai Lu <[email protected]>

---
 drivers/net/igb/igb_main.c |   42 ++++++++++++++++++++++++++++++------------
 1 file changed, 30 insertions(+), 12 deletions(-)

drivers/net/igb/igb_main.c +30 ~~-12~~

Index: linux-2.6/drivers/net/igb/igb_main.c
===================================================================
@@ -4277,7 +4277,7 @@ int igb_set_spd_dplx(struct igb_adapter 
 }
 
 
-static int igb_suspend(struct pci_dev *pdev, pm_message_t state)
+static int __igb_shutdown(struct pci_dev *pdev, bool *enable_wake)
 {
 	struct net_device *netdev = pci_get_drvdata(pdev);
 	struct igb_adapter *adapter = netdev_priv(netdev);

@@ -4336,15 +4336,9 @@ static int igb_suspend(struct pci_dev *p
 		wr32(E1000_WUFC, 0);
 	}
 
-	/* make sure adapter isn't asleep if manageability/wol is enabled */
-	if (wufc || adapter->en_mng_pt) {
-		pci_enable_wake(pdev, PCI_D3hot, 1);
-		pci_enable_wake(pdev, PCI_D3cold, 1);
-	} else {
+	*enable_wake = wufc || adapter->en_mng_pt;
+	if (!*enable_wake)
 		igb_shutdown_fiber_serdes_link_82575(hw);
-		pci_enable_wake(pdev, PCI_D3hot, 0);
-		pci_enable_wake(pdev, PCI_D3cold, 0);
-	}
 
 	/* Release control of h/w to f/w.  If f/w is AMT enabled, this
 	 * would have already happened in close and is redundant. */

@@ -4352,12 +4346,29 @@ static int igb_suspend(struct pci_dev *p
 
 	pci_disable_device(pdev);
 
-	pci_set_power_state(pdev, pci_choose_state(pdev, state));
-
 	return 0;
 }
 
 #ifdef CONFIG_PM
+static int igb_suspend(struct pci_dev *pdev, pm_message_t state)
+{
+	int retval;
+	bool wake;
+
+	retval = __igb_shutdown(pdev, &wake);
+	if (retval)
+		return retval;
+
+	if (wake) {
+		pci_prepare_to_sleep(pdev);
+	} else {
+		pci_wake_from_d3(pdev, false);
+		pci_set_power_state(pdev, PCI_D3hot);
+	}
+
+	return 0;
+}
+
 static int igb_resume(struct pci_dev *pdev)
 {
 	struct net_device *netdev = pci_get_drvdata(pdev);

@@ -4412,7 +4423,14 @@ static int igb_resume(struct pci_dev *pd
 
 static void igb_shutdown(struct pci_dev *pdev)
 {
-	igb_suspend(pdev, PMSG_SUSPEND);
+	bool wake;
+
+	__igb_shutdown(pdev, &wake);
+
+	if (system_state == SYSTEM_POWER_OFF) {
+		pci_wake_from_d3(pdev, wake);
+		pci_set_power_state(pdev, PCI_D3hot);
+	}
 }
 
 #ifdef CONFIG_NET_POLL_CONTROLLER

Yinghai Lu

author

2009-03-24 22:37 UTC #19 lore raw

Quoted 162 lines · Rafael J. Wysocki

Quoted 55 lines · Yinghai Lu
Quoted 54 lines · Rafael J. Wysocki
Quoted 53 lines · Rafael J. Wysocki
Quoted 52 lines · Yinghai Lu
Quoted 26 lines · Rafael J. Wysocki
Quoted 25 lines · Yinghai Lu
Quoted 12 lines · Jesse Brandeburg
Quoted 11 lines · Yinghai Lu
Impact: could probe igb
Found one system with 82575EB, in the kernel that is kexeced, probe igb failed with -2.
it looks like the same behavior happened on forcedeth.
try to check system_state to make sure if put it on D3
Signed-off-by Yinghai Lu <[email protected]>
Quoted 8 lines
Quoted 3 lines
---
 drivers/net/igb/igb_main.c |   19 ++++++++++++++-----
 1 file changed, 14 insertions(+), 5 deletions(-)
I see the point of the patch, but I know for a fact that ixgbe when enabled for MSI-X also doesn't work with kexec.
so my questions are:
are you going to change every driver?
i tend to only change driver that i have related HW.
Quoted 1 line
why can't this be fixed in core kernel code instead?
Quoted 8 lines
Quoted 6 lines
will check it.
Quoted 3 lines
Shouldn't pci_enable_device take it out of D3?
Or maybe it should be taken out of D3 immediately if someone tries to ioremap any of the BARx registers?
looks like second kernel can not detect the state any more.
In fact pci_enable_device() calls pci_set_power_state(dev, PCI_D0) as the first thing. The question is why it doesn't work as expected.
not sure... please check the version for forcedeth that you made.
commit 3cb5599a84c557c0dd9a19feb63a3788268cf249
Author: Rafael J. Wysocki <[email protected]>
Date:   Fri Sep 5 14:00:19 2008 -0700
forcedeth: fix kexec regression
Fix regression tracked as http://bugzilla.kernel.org/show_bug.cgi?id=11361
and caused by commit f735a2a1a4f2a0f5cd823ce323e82675990469e2 ("[netdrvr]
forcedeth: setup wake-on-lan before shutting down") that makes network adapters integrated into the NVidia MCP55 chipsets fail to work in kexeced kernels. The problem appears to be that if the adapter is put into D3_hot during ->shutdown(), it cannot be brought back into D0 after kexec (ref.
http://marc.info/?l=linux-kernel&m=121900062814967&w=4). Therefore, only put forcedeth into D3 during ->shutdown() if the system is to be powered off.
Quoted 5 lines
Quoted 4 lines
Quoted 1 line
Thanks, I remember now.
In which case you need to rework igb_shutdown() rather than igb_suspend().
Something like the patch below, perhaps (totally untested).
it works, David, can you picked it up
Still, Yinghai, can you please also test the patch below? It fixes all shortcomings in the driver's suspend and shutdown methods I was talking about in one of the previous messages. If it works, IMO it will be a preferable fix (in particular, it would be good to check if WoL still works with it, but I don't have the hardware).
Thanks,
Rafael
---
From: Rafael J. Wysocki <[email protected]>
Subject: net/igb: Fix kexec with igb (rev. 3)
Impact: Fix
Yinghai Lu found one system with 82575EB where, in the kernel that is kexeced, probe igb failed with -2, the reason being that the adapter could not be brought back from D3 by the kexec kernel, most probably due to quirky hardware (it looks like the same behavior happened on forcedeth).
Prevent igb from putting the adapter into D3 during shutdown except when we going to power off the system. For this purpose, seperate igb_shutdown() from igb_suspend() and use the appropriate PCI PM callbacks in both of them.
Signed-off-by Rafael J. Wysocki <[email protected]>
Reported-by Yinghai Lu <[email protected]>
---
 drivers/net/igb/igb_main.c |   42 ++++++++++++++++++++++++++++++------------
 1 file changed, 30 insertions(+), 12 deletions(-)
Index: linux-2.6/drivers/net/igb/igb_main.c
===================================================================
drivers/net/igb/igb_main.c +23 ~~-12~~
@@ -4277,7 +4277,7 @@ int igb_set_spd_dplx(struct igb_adapter 
 }
 
 
-static int igb_suspend(struct pci_dev *pdev, pm_message_t state)
+static int __igb_shutdown(struct pci_dev *pdev, bool *enable_wake)
 {
 	struct net_device *netdev = pci_get_drvdata(pdev);
 	struct igb_adapter *adapter = netdev_priv(netdev);
@@ -4336,15 +4336,9 @@ static int igb_suspend(struct pci_dev *p
 		wr32(E1000_WUFC, 0);
 	}
 
-	/* make sure adapter isn't asleep if manageability/wol is enabled */
-	if (wufc || adapter->en_mng_pt) {
-		pci_enable_wake(pdev, PCI_D3hot, 1);
-		pci_enable_wake(pdev, PCI_D3cold, 1);
-	} else {
+	*enable_wake = wufc || adapter->en_mng_pt;
+	if (!*enable_wake)
 		igb_shutdown_fiber_serdes_link_82575(hw);
-		pci_enable_wake(pdev, PCI_D3hot, 0);
-		pci_enable_wake(pdev, PCI_D3cold, 0);
-	}
 
 	/* Release control of h/w to f/w.  If f/w is AMT enabled, this
 	 * would have already happened in close and is redundant. */
@@ -4352,12 +4346,29 @@ static int igb_suspend(struct pci_dev *p
 
 	pci_disable_device(pdev);
 
-	pci_set_power_state(pdev, pci_choose_state(pdev, state));
-
 	return 0;
 }
 
 #ifdef CONFIG_PM
+static int igb_suspend(struct pci_dev *pdev, pm_message_t state)
+{
+	int retval;
+	bool wake;
+
+	retval = __igb_shutdown(pdev, &wake);
+	if (retval)
+		return retval;
+
+	if (wake) {
+		pci_prepare_to_sleep(pdev);
+	} else {
+		pci_wake_from_d3(pdev, false);
+		pci_set_power_state(pdev, PCI_D3hot);
+	}
+
+	return 0;
+}
+
 static int igb_resume(struct pci_dev *pdev)
 {
 	struct net_device *netdev = pci_get_drvdata(pdev);
@@ -4412,7 +4423,14 @@ static int igb_resume(struct pci_dev *pd
 
 static void igb_shutdown(struct pci_dev *pdev)
 {
-	igb_suspend(pdev, PMSG_SUSPEND);
+	bool wake;

drivers/net/igb/igb_main.c static int igb_resume(struct pci_dev *pd · line 4436

Quoted 10 lines

drivers/net/igb/igb_main.c +7 -0

+
+	__igb_shutdown(pdev, &wake);
+
+	if (system_state == SYSTEM_POWER_OFF) {
+		pci_wake_from_d3(pdev, wake);
+		pci_set_power_state(pdev, PCI_D3hot);
+	}
 }
 
 #ifdef CONFIG_NET_POLL_CONTROLLER

it works too.

Rafael J. Wysocki

2009-03-28 21:27 UTC #20 lore raw

Quoted 172 lines · Yinghai Lu

Quoted 171 lines · Rafael J. Wysocki
Quoted 55 lines · Yinghai Lu
Quoted 54 lines · Rafael J. Wysocki
Quoted 53 lines · Rafael J. Wysocki
Quoted 52 lines · Yinghai Lu
Quoted 26 lines · Rafael J. Wysocki
Quoted 25 lines · Yinghai Lu
Quoted 12 lines · Jesse Brandeburg
Quoted 11 lines · Yinghai Lu
Impact: could probe igb
Found one system with 82575EB, in the kernel that is kexeced, probe igb failed with -2.
it looks like the same behavior happened on forcedeth.
try to check system_state to make sure if put it on D3
Signed-off-by Yinghai Lu <[email protected]>
Quoted 8 lines
Quoted 3 lines
---
 drivers/net/igb/igb_main.c |   19 ++++++++++++++-----
 1 file changed, 14 insertions(+), 5 deletions(-)
I see the point of the patch, but I know for a fact that ixgbe when enabled for MSI-X also doesn't work with kexec.
so my questions are:
are you going to change every driver?
i tend to only change driver that i have related HW.
Quoted 1 line
why can't this be fixed in core kernel code instead?
Quoted 8 lines
Quoted 6 lines
will check it.
Quoted 3 lines
Shouldn't pci_enable_device take it out of D3?
Or maybe it should be taken out of D3 immediately if someone tries to ioremap any of the BARx registers?
looks like second kernel can not detect the state any more.
In fact pci_enable_device() calls pci_set_power_state(dev, PCI_D0) as the first thing. The question is why it doesn't work as expected.
not sure... please check the version for forcedeth that you made.
commit 3cb5599a84c557c0dd9a19feb63a3788268cf249
Author: Rafael J. Wysocki <[email protected]>
Date:   Fri Sep 5 14:00:19 2008 -0700
forcedeth: fix kexec regression
Fix regression tracked as http://bugzilla.kernel.org/show_bug.cgi?id=11361
and caused by commit f735a2a1a4f2a0f5cd823ce323e82675990469e2 ("[netdrvr]
forcedeth: setup wake-on-lan before shutting down") that makes network adapters integrated into the NVidia MCP55 chipsets fail to work in kexeced kernels. The problem appears to be that if the adapter is put into D3_hot during ->shutdown(), it cannot be brought back into D0 after kexec (ref.
http://marc.info/?l=linux-kernel&m=121900062814967&w=4). Therefore, only put forcedeth into D3 during ->shutdown() if the system is to be powered off.
Quoted 5 lines
Quoted 4 lines
Quoted 1 line
Thanks, I remember now.
In which case you need to rework igb_shutdown() rather than igb_suspend().
Something like the patch below, perhaps (totally untested).
it works, David, can you picked it up
Still, Yinghai, can you please also test the patch below? It fixes all shortcomings in the driver's suspend and shutdown methods I was talking about in one of the previous messages. If it works, IMO it will be a preferable fix (in particular, it would be good to check if WoL still works with it, but I don't have the hardware).
Thanks,
Rafael
---
From: Rafael J. Wysocki <[email protected]>
Subject: net/igb: Fix kexec with igb (rev. 3)
Impact: Fix
Yinghai Lu found one system with 82575EB where, in the kernel that is kexeced, probe igb failed with -2, the reason being that the adapter could not be brought back from D3 by the kexec kernel, most probably due to quirky hardware (it looks like the same behavior happened on forcedeth).
Prevent igb from putting the adapter into D3 during shutdown except when we going to power off the system. For this purpose, seperate igb_shutdown() from igb_suspend() and use the appropriate PCI PM callbacks in both of them.
Signed-off-by Rafael J. Wysocki <[email protected]>
Reported-by Yinghai Lu <[email protected]>
---
 drivers/net/igb/igb_main.c |   42 ++++++++++++++++++++++++++++++------------
 1 file changed, 30 insertions(+), 12 deletions(-)
Index: linux-2.6/drivers/net/igb/igb_main.c
===================================================================
drivers/net/igb/igb_main.c +30 ~~-12~~
@@ -4277,7 +4277,7 @@ int igb_set_spd_dplx(struct igb_adapter 
 }
 
 
-static int igb_suspend(struct pci_dev *pdev, pm_message_t state)
+static int __igb_shutdown(struct pci_dev *pdev, bool *enable_wake)
 {
 	struct net_device *netdev = pci_get_drvdata(pdev);
 	struct igb_adapter *adapter = netdev_priv(netdev);
@@ -4336,15 +4336,9 @@ static int igb_suspend(struct pci_dev *p
 		wr32(E1000_WUFC, 0);
 	}
 
-	/* make sure adapter isn't asleep if manageability/wol is enabled */
-	if (wufc || adapter->en_mng_pt) {
-		pci_enable_wake(pdev, PCI_D3hot, 1);
-		pci_enable_wake(pdev, PCI_D3cold, 1);
-	} else {
+	*enable_wake = wufc || adapter->en_mng_pt;
+	if (!*enable_wake)
 		igb_shutdown_fiber_serdes_link_82575(hw);
-		pci_enable_wake(pdev, PCI_D3hot, 0);
-		pci_enable_wake(pdev, PCI_D3cold, 0);
-	}
 
 	/* Release control of h/w to f/w.  If f/w is AMT enabled, this
 	 * would have already happened in close and is redundant. */
@@ -4352,12 +4346,29 @@ static int igb_suspend(struct pci_dev *p
 
 	pci_disable_device(pdev);
 
-	pci_set_power_state(pdev, pci_choose_state(pdev, state));
-
 	return 0;
 }
 
 #ifdef CONFIG_PM
+static int igb_suspend(struct pci_dev *pdev, pm_message_t state)
+{
+	int retval;
+	bool wake;
+
+	retval = __igb_shutdown(pdev, &wake);
+	if (retval)
+		return retval;
+
+	if (wake) {
+		pci_prepare_to_sleep(pdev);
+	} else {
+		pci_wake_from_d3(pdev, false);
+		pci_set_power_state(pdev, PCI_D3hot);
+	}
+
+	return 0;
+}
+
 static int igb_resume(struct pci_dev *pdev)
 {
 	struct net_device *netdev = pci_get_drvdata(pdev);
@@ -4412,7 +4423,14 @@ static int igb_resume(struct pci_dev *pd
 
 static void igb_shutdown(struct pci_dev *pdev)
 {
-	igb_suspend(pdev, PMSG_SUSPEND);
+	bool wake;
+
+	__igb_shutdown(pdev, &wake);
+
+	if (system_state == SYSTEM_POWER_OFF) {
+		pci_wake_from_d3(pdev, wake);
+		pci_set_power_state(pdev, PCI_D3hot);
+	}
 }
 

Quoted 3 lines

drivers/net/igb/igb_main.c static int igb_resume(struct pci_dev *pd · line 4436
Quoted 1 line
drivers/net/igb/igb_main.c +0 -0
 #ifdef CONFIG_NET_POLL_CONTROLLER
it works too.

Great, thanks for testing.

Jeff, Jesse, is the patch fine with you?

Rafael

Jeff Kirsher

2009-03-29 02:30 UTC #21 lore raw

Quoted 177 lines · Rafael J. Wysocki

Quoted 172 lines · Yinghai Lu
Quoted 171 lines · Rafael J. Wysocki
Quoted 55 lines · Yinghai Lu
Quoted 54 lines · Rafael J. Wysocki
Quoted 53 lines · Rafael J. Wysocki
Quoted 52 lines · Yinghai Lu
Quoted 26 lines · Rafael J. Wysocki
Quoted 25 lines · Yinghai Lu
Quoted 12 lines · Jesse Brandeburg
Quoted 11 lines · Yinghai Lu
Impact: could probe igb
Found one system with 82575EB, in the kernel that is kexeced, probe igb failed with -2.
it looks like the same behavior happened on forcedeth.
try to check system_state to make sure if put it on D3
Signed-off-by Yinghai Lu <[email protected]>
Quoted 8 lines
Quoted 3 lines
---
drivers/net/igb/igb_main.c | 19 ++++++++++++++-----
 1 file changed, 14 insertions(+), 5 deletions(-)
I see the point of the patch, but I know for a fact that ixgbe when enabled for MSI-X also doesn't work with kexec.
so my questions are:
are you going to change every driver?
i tend to only change driver that i have related HW.
Quoted 1 line
why can't this be fixed in core kernel code instead?
Quoted 8 lines
Quoted 6 lines
will check it.
Quoted 3 lines
Shouldn't pci_enable_device take it out of D3?
Or maybe it should be taken out of D3 immediately if someone tries to ioremap any of the BARx registers?
looks like second kernel can not detect the state any more.
In fact pci_enable_device() calls pci_set_power_state(dev, PCI_D0) as the first thing. The question is why it doesn't work as expected.
not sure... please check the version for forcedeth that you made.
commit 3cb5599a84c557c0dd9a19feb63a3788268cf249
Author: Rafael J. Wysocki <[email protected]>
Date: Fri Sep 5 14:00:19 2008 -0700
forcedeth: fix kexec regression
Fix regression tracked as http://bugzilla.kernel.org/show_bug.cgi?id=11361
and caused by commit f735a2a1a4f2a0f5cd823ce323e82675990469e2 ("[netdrvr]
forcedeth: setup wake-on-lan before shutting down") that makes network adapters integrated into the NVidia MCP55 chipsets fail to work in kexeced kernels. The problem appears to be that if the adapter is put into D3_hot during ->shutdown(), it cannot be brought back into D0 after kexec (ref.
http://marc.info/?l=linux-kernel&m=121900062814967&w=4). Therefore, only put forcedeth into D3 during ->shutdown() if the system is to be powered off.
Quoted 5 lines
Quoted 4 lines
Quoted 1 line
Thanks, I remember now.
In which case you need to rework igb_shutdown() rather than igb_suspend().
Something like the patch below, perhaps (totally untested).
it works, David, can you picked it up
Still, Yinghai, can you please also test the patch below? It fixes all shortcomings in the driver's suspend and shutdown methods I was talking about in one of the previous messages. If it works, IMO it will be a preferable fix (in particular, it would be good to check if WoL still works with it, but I don't have the hardware).
Thanks,
Rafael
---
From: Rafael J. Wysocki <[email protected]>
Subject: net/igb: Fix kexec with igb (rev. 3)
Impact: Fix
Yinghai Lu found one system with 82575EB where, in the kernel that is kexeced, probe igb failed with -2, the reason being that the adapter could not be brought back from D3 by the kexec kernel, most probably due to quirky hardware (it looks like the same behavior happened on forcedeth).
Prevent igb from putting the adapter into D3 during shutdown except when we going to power off the system. For this purpose, seperate igb_shutdown() from igb_suspend() and use the appropriate PCI PM callbacks in both of them.
Signed-off-by Rafael J. Wysocki <[email protected]>
Reported-by Yinghai Lu <[email protected]>
---
drivers/net/igb/igb_main.c | 42 ++++++++++++++++++++++++++++++------------
 1 file changed, 30 insertions(+), 12 deletions(-)
Index: linux-2.6/drivers/net/igb/igb_main.c
===================================================================
drivers/net/igb/igb_main.c +30 ~~-12~~
@@ -4277,7 +4277,7 @@ int igb_set_spd_dplx(struct igb_adapter
  }
 
 
-static int igb_suspend(struct pci_dev *pdev, pm_message_t state)
+static int __igb_shutdown(struct pci_dev *pdev, bool *enable_wake)
  {
     struct net_device *netdev = pci_get_drvdata(pdev);
     struct igb_adapter *adapter = netdev_priv(netdev);
@@ -4336,15 +4336,9 @@ static int igb_suspend(struct pci_dev *p
             wr32(E1000_WUFC, 0);
     }
 
-   /* make sure adapter isn't asleep if manageability/wol is enabled */
-   if (wufc || adapter->en_mng_pt) {
-           pci_enable_wake(pdev, PCI_D3hot, 1);
-           pci_enable_wake(pdev, PCI_D3cold, 1);
-   } else {
+   *enable_wake = wufc || adapter->en_mng_pt;
+   if (!*enable_wake)
             igb_shutdown_fiber_serdes_link_82575(hw);
-           pci_enable_wake(pdev, PCI_D3hot, 0);
-           pci_enable_wake(pdev, PCI_D3cold, 0);
-   }
 
     /* Release control of h/w to f/w.  If f/w is AMT enabled, this
      * would have already happened in close and is redundant. */
@@ -4352,12 +4346,29 @@ static int igb_suspend(struct pci_dev *p
 
     pci_disable_device(pdev);
 
-   pci_set_power_state(pdev, pci_choose_state(pdev, state));
-
     return 0;
  }
 
  #ifdef CONFIG_PM
+static int igb_suspend(struct pci_dev *pdev, pm_message_t state)
+{
+   int retval;
+   bool wake;
+
+   retval = __igb_shutdown(pdev, &wake);
+   if (retval)
+           return retval;
+
+   if (wake) {
+           pci_prepare_to_sleep(pdev);
+   } else {
+           pci_wake_from_d3(pdev, false);
+           pci_set_power_state(pdev, PCI_D3hot);
+   }
+
+   return 0;
+}
+
  static int igb_resume(struct pci_dev *pdev)
  {
     struct net_device *netdev = pci_get_drvdata(pdev);
@@ -4412,7 +4423,14 @@ static int igb_resume(struct pci_dev *pd
 
  static void igb_shutdown(struct pci_dev *pdev)
  {
-   igb_suspend(pdev, PMSG_SUSPEND);
+   bool wake;
+
+   __igb_shutdown(pdev, &wake);
+
+   if (system_state == SYSTEM_POWER_OFF) {
+           pci_wake_from_d3(pdev, wake);
+           pci_set_power_state(pdev, PCI_D3hot);
+   }
  }
 
Quoted 3 lines
drivers/net/igb/igb_main.c static int igb_resume(struct pci_dev *pd · line 4436
Quoted 1 line
drivers/net/igb/igb_main.c +0 -0
  #ifdef CONFIG_NET_POLL_CONTROLLER
it works too.

Quoted 5 lines

Great, thanks for testing.
Jeff, Jesse, is the patch fine with you?
Rafael

Let me talk with Jesse on Monday about it. I know that Jesse's concern's were that there were more than one driver afflicted with the same or similar issue and that it made more sense to fix this in the core for all drivers. For instance ixgbe, IIRC, but because Yinghai did not have any other hardware, it was unclear if this was an issue on other drivers.

After discussing this with Jesse, one of us will respond. I will apply this patch to my tree anyway and if we are alright with the patch, I will push it out with my other patches to Dave.

Signature · 2 lines

Cheers,
Jeff

Rafael J. Wysocki

2009-03-29 11:19 UTC #22 lore raw

Quoted 182 lines · Jeff Kirsher

Quoted 181 lines · Rafael J. Wysocki
Quoted 172 lines · Yinghai Lu
Quoted 171 lines · Rafael J. Wysocki
Quoted 55 lines · Yinghai Lu
Quoted 54 lines · Rafael J. Wysocki
Quoted 53 lines · Rafael J. Wysocki
Quoted 52 lines · Yinghai Lu
Quoted 26 lines · Rafael J. Wysocki
Quoted 25 lines · Yinghai Lu
Quoted 12 lines · Jesse Brandeburg
Quoted 11 lines · Yinghai Lu
Impact: could probe igb
Found one system with 82575EB, in the kernel that is kexeced, probe igb failed with -2.
it looks like the same behavior happened on forcedeth.
try to check system_state to make sure if put it on D3
Signed-off-by Yinghai Lu <[email protected]>
Quoted 8 lines
Quoted 3 lines
---
 drivers/net/igb/igb_main.c |   19 ++++++++++++++-----
 1 file changed, 14 insertions(+), 5 deletions(-)
I see the point of the patch, but I know for a fact that ixgbe when enabled for MSI-X also doesn't work with kexec.
so my questions are:
are you going to change every driver?
i tend to only change driver that i have related HW.
Quoted 1 line
why can't this be fixed in core kernel code instead?
Quoted 8 lines
Quoted 6 lines
will check it.
Quoted 3 lines
Shouldn't pci_enable_device take it out of D3?
Or maybe it should be taken out of D3 immediately if someone tries to ioremap any of the BARx registers?
looks like second kernel can not detect the state any more.
In fact pci_enable_device() calls pci_set_power_state(dev, PCI_D0) as the first thing. The question is why it doesn't work as expected.
not sure... please check the version for forcedeth that you made.
commit 3cb5599a84c557c0dd9a19feb63a3788268cf249
Author: Rafael J. Wysocki <[email protected]>
Date:   Fri Sep 5 14:00:19 2008 -0700
forcedeth: fix kexec regression
Fix regression tracked as http://bugzilla.kernel.org/show_bug.cgi?id=11361
and caused by commit f735a2a1a4f2a0f5cd823ce323e82675990469e2 ("[netdrvr]
forcedeth: setup wake-on-lan before shutting down") that makes network adapters integrated into the NVidia MCP55 chipsets fail to work in kexeced kernels. The problem appears to be that if the adapter is put into D3_hot during ->shutdown(), it cannot be brought back into D0 after kexec (ref.
http://marc.info/?l=linux-kernel&m=121900062814967&w=4). Therefore, only put forcedeth into D3 during ->shutdown() if the system is to be powered off.
Quoted 5 lines
Quoted 4 lines
Quoted 1 line
Thanks, I remember now.
In which case you need to rework igb_shutdown() rather than igb_suspend().
Something like the patch below, perhaps (totally untested).
it works, David, can you picked it up
Still, Yinghai, can you please also test the patch below? It fixes all shortcomings in the driver's suspend and shutdown methods I was talking about in one of the previous messages. If it works, IMO it will be a preferable fix (in particular, it would be good to check if WoL still works with it, but I don't have the hardware).
Thanks,
Rafael
---
From: Rafael J. Wysocki <[email protected]>
Subject: net/igb: Fix kexec with igb (rev. 3)
Impact: Fix
Yinghai Lu found one system with 82575EB where, in the kernel that is kexeced, probe igb failed with -2, the reason being that the adapter could not be brought back from D3 by the kexec kernel, most probably due to quirky hardware (it looks like the same behavior happened on forcedeth).
Prevent igb from putting the adapter into D3 during shutdown except when we going to power off the system. For this purpose, seperate igb_shutdown() from igb_suspend() and use the appropriate PCI PM callbacks in both of them.
Signed-off-by Rafael J. Wysocki <[email protected]>
Reported-by Yinghai Lu <[email protected]>
---
 drivers/net/igb/igb_main.c |   42 ++++++++++++++++++++++++++++++------------
 1 file changed, 30 insertions(+), 12 deletions(-)
Index: linux-2.6/drivers/net/igb/igb_main.c
===================================================================
drivers/net/igb/igb_main.c +30 ~~-12~~
@@ -4277,7 +4277,7 @@ int igb_set_spd_dplx(struct igb_adapter
 }
 
 
-static int igb_suspend(struct pci_dev *pdev, pm_message_t state)
+static int __igb_shutdown(struct pci_dev *pdev, bool *enable_wake)
 {
    struct net_device *netdev = pci_get_drvdata(pdev);
    struct igb_adapter *adapter = netdev_priv(netdev);
@@ -4336,15 +4336,9 @@ static int igb_suspend(struct pci_dev *p
            wr32(E1000_WUFC, 0);
    }
 
-   /* make sure adapter isn't asleep if manageability/wol is enabled */
-   if (wufc || adapter->en_mng_pt) {
-           pci_enable_wake(pdev, PCI_D3hot, 1);
-           pci_enable_wake(pdev, PCI_D3cold, 1);
-   } else {
+   *enable_wake = wufc || adapter->en_mng_pt;
+   if (!*enable_wake)
            igb_shutdown_fiber_serdes_link_82575(hw);
-           pci_enable_wake(pdev, PCI_D3hot, 0);
-           pci_enable_wake(pdev, PCI_D3cold, 0);
-   }
 
    /* Release control of h/w to f/w.  If f/w is AMT enabled, this
     * would have already happened in close and is redundant. */
@@ -4352,12 +4346,29 @@ static int igb_suspend(struct pci_dev *p
 
    pci_disable_device(pdev);
 
-   pci_set_power_state(pdev, pci_choose_state(pdev, state));
-
    return 0;
 }
 
 #ifdef CONFIG_PM
+static int igb_suspend(struct pci_dev *pdev, pm_message_t state)
+{
+   int retval;
+   bool wake;
+
+   retval = __igb_shutdown(pdev, &wake);
+   if (retval)
+           return retval;
+
+   if (wake) {
+           pci_prepare_to_sleep(pdev);
+   } else {
+           pci_wake_from_d3(pdev, false);
+           pci_set_power_state(pdev, PCI_D3hot);
+   }
+
+   return 0;
+}
+
 static int igb_resume(struct pci_dev *pdev)
 {
    struct net_device *netdev = pci_get_drvdata(pdev);
@@ -4412,7 +4423,14 @@ static int igb_resume(struct pci_dev *pd
 
 static void igb_shutdown(struct pci_dev *pdev)
 {
-   igb_suspend(pdev, PMSG_SUSPEND);
+   bool wake;
+
+   __igb_shutdown(pdev, &wake);
+
+   if (system_state == SYSTEM_POWER_OFF) {
+           pci_wake_from_d3(pdev, wake);
+           pci_set_power_state(pdev, PCI_D3hot);
+   }
 }
 
Quoted 3 lines
drivers/net/igb/igb_main.c static int igb_resume(struct pci_dev *pd · line 4436
Quoted 1 line
drivers/net/igb/igb_main.c +0 -0
 #ifdef CONFIG_NET_POLL_CONTROLLER
it works too.
Great, thanks for testing.
Jeff, Jesse, is the patch fine with you?

Quoted 8 lines

Quoted 1 line
Rafael
Let me talk with Jesse on Monday about it. I know that Jesse's concern's were that there were more than one driver afflicted with the same or similar issue and that it made more sense to fix this in the core for all drivers. For instance ixgbe, IIRC, but because Yinghai did not have any other hardware, it was unclear if this was an issue on other drivers.

I think it was, at least theoretically. We should generally avoid powering off things on shutdown when it's not really necessary. Also, the use of pci_enable_wake() in these drivers is not really correct.

And in fact this is why I'm asking, because I have analogouns patches for some other drivers in the works too. If you are fine with the approach, I'll post the whole series.

Quoted 3 lines

After discussing this with Jesse, one of us will respond. I will apply this patch to my tree anyway and if we are alright with the patch, I will push it out with my other patches to Dave.

Thanks!

Best,
Rafael

Jeff Kirsher

2009-03-30 21:36 UTC #23 lore raw

Quoted 204 lines · Rafael J. Wysocki

Quoted 182 lines · Jeff Kirsher
Quoted 181 lines · Rafael J. Wysocki
Quoted 172 lines · Yinghai Lu
Quoted 171 lines · Rafael J. Wysocki
Quoted 55 lines · Yinghai Lu
Quoted 54 lines · Rafael J. Wysocki
Quoted 53 lines · Rafael J. Wysocki
Quoted 52 lines · Yinghai Lu
Quoted 26 lines · Rafael J. Wysocki
Quoted 25 lines · Yinghai Lu
Quoted 12 lines · Jesse Brandeburg
Quoted 11 lines · Yinghai Lu
Impact: could probe igb
Found one system with 82575EB, in the kernel that is kexeced, probe igb
failed with -2.
it looks like the same behavior happened on forcedeth.
try to check system_state to make sure if put it on D3
Signed-off-by: Yinghai Lu <[email protected]>
Quoted 8 lines
Quoted 3 lines
---
drivers/net/igb/igb_main.c | 19 ++++++++++++++-----
1 file changed, 14 insertions(+), 5 deletions(-)
I see the point of the patch, but I know for a fact that ixgbe when enabled for MSI-X also doesn't work with kexec.
so my questions are:
are you going to change every driver?
i tend to only change driver that i have related HW.
Quoted 1 line
why can't this be fixed in core kernel code instead?
Quoted 8 lines
Quoted 6 lines
will check it.
Quoted 3 lines
Shouldn't pci_enable_device take it out of D3?
Or maybe it should be taken out of D3 immediately if someone tries to ioremap any of the BARx registers?
looks like second kernel can not detect the state any more.
In fact pci_enable_device() calls pci_set_power_state(dev, PCI_D0) as the first thing. The question is why it doesn't work as expected.
not sure... please check the version for forcedeth that you made.
commit 3cb5599a84c557c0dd9a19feb63a3788268cf249
Author: Rafael J. Wysocki <[email protected]>
Date: Fri Sep 5 14:00:19 2008 -0700
forcedeth: fix kexec regression
Fix regression tracked as http://bugzilla.kernel.org/show_bug.cgi?id=11361
and caused by commit f735a2a1a4f2a0f5cd823ce323e82675990469e2 ("[netdrvr]
forcedeth: setup wake-on-lan before shutting down") that makes network adapters integrated into the NVidia MCP55 chipsets fail to work in kexeced kernels. The problem appears to be that if the adapter is put into D3_hot during ->shutdown(), it cannot be brought back into D0 after kexec (ref.
http://marc.info/?l=linux-kernel&m=121900062814967&w=4). Therefore, only put forcedeth into D3 during ->shutdown() if the system is to be powered off.
Quoted 5 lines
Quoted 4 lines
Quoted 1 line
Thanks, I remember now.
In which case you need to rework igb_shutdown() rather than igb_suspend().
Something like the patch below, perhaps (totally untested).
it works, David, can you picked it up
Still, Yinghai, can you please also test the patch below? It fixes all shortcomings in the driver's suspend and shutdown methods I was talking about in one of the previous messages. If it works, IMO it will be a preferable fix (in particular, it would be good to check if WoL still works with it, but I don't have the hardware).
Thanks,
Rafael
---
From: Rafael J. Wysocki <[email protected]>
Subject: net/igb: Fix kexec with igb (rev. 3)
Impact: Fix
Yinghai Lu found one system with 82575EB where, in the kernel that is kexeced, probe igb failed with -2, the reason being that the adapter could not be brought back from D3 by the kexec kernel, most probably due to quirky hardware (it looks like the same behavior happened on forcedeth).
Prevent igb from putting the adapter into D3 during shutdown except when we going to power off the system. For this purpose, seperate igb_shutdown() from igb_suspend() and use the appropriate PCI PM callbacks in both of them.
Signed-off-by Rafael J. Wysocki <[email protected]>
Reported-by Yinghai Lu <[email protected]>
---
drivers/net/igb/igb_main.c | 42 ++++++++++++++++++++++++++++++------------
 1 file changed, 30 insertions(+), 12 deletions(-)
Index: linux-2.6/drivers/net/igb/igb_main.c
===================================================================
drivers/net/igb/igb_main.c +30 ~~-12~~
@@ -4277,7 +4277,7 @@ int igb_set_spd_dplx(struct igb_adapter
  }
 
 
-static int igb_suspend(struct pci_dev *pdev, pm_message_t state)
+static int __igb_shutdown(struct pci_dev *pdev, bool *enable_wake)
  {
     struct net_device *netdev = pci_get_drvdata(pdev);
     struct igb_adapter *adapter = netdev_priv(netdev);
@@ -4336,15 +4336,9 @@ static int igb_suspend(struct pci_dev *p
             wr32(E1000_WUFC, 0);
     }
 
-   /* make sure adapter isn't asleep if manageability/wol is enabled */
-   if (wufc || adapter->en_mng_pt) {
-           pci_enable_wake(pdev, PCI_D3hot, 1);
-           pci_enable_wake(pdev, PCI_D3cold, 1);
-   } else {
+   *enable_wake = wufc || adapter->en_mng_pt;
+   if (!*enable_wake)
             igb_shutdown_fiber_serdes_link_82575(hw);
-           pci_enable_wake(pdev, PCI_D3hot, 0);
-           pci_enable_wake(pdev, PCI_D3cold, 0);
-   }
 
     /* Release control of h/w to f/w.  If f/w is AMT enabled, this
      * would have already happened in close and is redundant. */
@@ -4352,12 +4346,29 @@ static int igb_suspend(struct pci_dev *p
 
     pci_disable_device(pdev);
 
-   pci_set_power_state(pdev, pci_choose_state(pdev, state));
-
     return 0;
  }
 
  #ifdef CONFIG_PM
+static int igb_suspend(struct pci_dev *pdev, pm_message_t state)
+{
+   int retval;
+   bool wake;
+
+   retval = __igb_shutdown(pdev, &wake);
+   if (retval)
+           return retval;
+
+   if (wake) {
+           pci_prepare_to_sleep(pdev);
+   } else {
+           pci_wake_from_d3(pdev, false);
+           pci_set_power_state(pdev, PCI_D3hot);
+   }
+
+   return 0;
+}
+
  static int igb_resume(struct pci_dev *pdev)
  {
     struct net_device *netdev = pci_get_drvdata(pdev);
@@ -4412,7 +4423,14 @@ static int igb_resume(struct pci_dev *pd
 
  static void igb_shutdown(struct pci_dev *pdev)
  {
-   igb_suspend(pdev, PMSG_SUSPEND);
+   bool wake;
+
+   __igb_shutdown(pdev, &wake);
+
+   if (system_state == SYSTEM_POWER_OFF) {
+           pci_wake_from_d3(pdev, wake);
+           pci_set_power_state(pdev, PCI_D3hot);
+   }
  }
 
Quoted 3 lines
drivers/net/igb/igb_main.c static int igb_resume(struct pci_dev *pd · line 4436
Quoted 1 line
drivers/net/igb/igb_main.c +0 -0
  #ifdef CONFIG_NET_POLL_CONTROLLER
it works too.
Great, thanks for testing.
Jeff, Jesse, is the patch fine with you?
Quoted 8 lines
Quoted 1 line
Rafael
Let me talk with Jesse on Monday about it. I know that Jesse's concern's were that there were more than one driver afflicted with the same or similar issue and that it made more sense to fix this in the core for all drivers. For instance ixgbe, IIRC, but because Yinghai did not have any other hardware, it was unclear if this was an issue on other drivers.
I think it was, at least theoretically. We should generally avoid powering off things on shutdown when it's not really necessary. Also, the use of pci_enable_wake() in these drivers is not really correct.
And in fact this is why I'm asking, because I have analogouns patches for some other drivers in the works too. If you are fine with the approach, I'll post the whole series.
Quoted 3 lines
After discussing this with Jesse, one of us will respond. I will apply this patch to my tree anyway and if we are alright with the patch, I will push it out with my other patches to Dave.

Quoted 5 lines

Thanks!
Best,
Rafael
--

After talking with Jesse, we are fine with it if our testers buy off on it. I have the patch in my tree and so Emil will run a barrage of regression/stress tests on the patch overnight. If everything looks good, I will send it out with the other igb patches I have.

Thanks.

Signature · 2 lines

Cheers,
Jeff

Rafael J. Wysocki

2009-03-30 21:39 UTC #24 lore raw

Quoted 207 lines · Jeff Kirsher

Quoted 206 lines · Rafael J. Wysocki
Quoted 182 lines · Jeff Kirsher
Quoted 181 lines · Rafael J. Wysocki
Quoted 172 lines · Yinghai Lu
Quoted 171 lines · Rafael J. Wysocki
Quoted 55 lines · Yinghai Lu
Quoted 54 lines · Rafael J. Wysocki
Quoted 53 lines · Rafael J. Wysocki
Quoted 52 lines · Yinghai Lu
Quoted 26 lines · Rafael J. Wysocki
Quoted 25 lines · Yinghai Lu
Quoted 20 lines · Jesse Brandeburg
On Fri, Mar 6, 2009 at 8:33 PM, Yinghai Lu <[email protected]> wrote:
Impact: could probe igb
Found one system with 82575EB, in the kernel that is kexeced, probe igb
failed with -2.
it looks like the same behavior happened on forcedeth.
try to check system_state to make sure if put it on D3
Signed-off-by: Yinghai Lu <[email protected]>
---
drivers/net/igb/igb_main.c | 19 ++++++++++++++-----
1 file changed, 14 insertions(+), 5 deletions(-)
I see the point of the patch, but I know for a fact that ixgbe when
enabled for MSI-X also doesn't work with kexec.
so my questions are:
are you going to change every driver?
i tend to only change driver that i have related HW.
Quoted 1 line
why can't this be fixed in core kernel code instead?
Quoted 8 lines
Quoted 6 lines
will check it.
Quoted 3 lines
Shouldn't pci_enable_device take it out of D3?
Or maybe it should be taken out of D3 immediately if someone tries to
ioremap any of the BARx registers?
looks like second kernel can not detect the state any more.
In fact pci_enable_device() calls pci_set_power_state(dev, PCI_D0) as the first thing. The question is why it doesn't work as expected.
not sure... please check the version for forcedeth that you made.
commit 3cb5599a84c557c0dd9a19feb63a3788268cf249
Author: Rafael J. Wysocki <[email protected]>
Date:   Fri Sep 5 14:00:19 2008 -0700
forcedeth: fix kexec regression
Fix regression tracked as http://bugzilla.kernel.org/show_bug.cgi?id=11361
and caused by commit f735a2a1a4f2a0f5cd823ce323e82675990469e2 ("[netdrvr]
forcedeth: setup wake-on-lan before shutting down") that makes network adapters integrated into the NVidia MCP55 chipsets fail to work in kexeced kernels. The problem appears to be that if the adapter is put into D3_hot during ->shutdown(), it cannot be brought back into D0 after kexec (ref.
http://marc.info/?l=linux-kernel&m=121900062814967&w=4). Therefore, only put forcedeth into D3 during ->shutdown() if the system is to be powered off.
Quoted 5 lines
Quoted 4 lines
Quoted 1 line
Thanks, I remember now.
In which case you need to rework igb_shutdown() rather than igb_suspend().
Something like the patch below, perhaps (totally untested).
it works, David, can you picked it up
Still, Yinghai, can you please also test the patch below? It fixes all shortcomings in the driver's suspend and shutdown methods I was talking about in one of the previous messages. If it works, IMO it will be a preferable fix (in particular, it would be good to check if WoL still works with it, but I don't have the hardware).
Thanks,
Rafael
---
From: Rafael J. Wysocki <[email protected]>
Subject: net/igb: Fix kexec with igb (rev. 3)
Impact: Fix
Yinghai Lu found one system with 82575EB where, in the kernel that is kexeced, probe igb failed with -2, the reason being that the adapter could not be brought back from D3 by the kexec kernel, most probably due to quirky hardware (it looks like the same behavior happened on forcedeth).
Prevent igb from putting the adapter into D3 during shutdown except when we going to power off the system. For this purpose, seperate igb_shutdown() from igb_suspend() and use the appropriate PCI PM callbacks in both of them.
Signed-off-by Rafael J. Wysocki <[email protected]>
Reported-by Yinghai Lu <[email protected]>
---
 drivers/net/igb/igb_main.c |   42 ++++++++++++++++++++++++++++++------------
 1 file changed, 30 insertions(+), 12 deletions(-)
Index: linux-2.6/drivers/net/igb/igb_main.c
===================================================================
drivers/net/igb/igb_main.c +30 ~~-12~~
@@ -4277,7 +4277,7 @@ int igb_set_spd_dplx(struct igb_adapter
 }
 
 
-static int igb_suspend(struct pci_dev *pdev, pm_message_t state)
+static int __igb_shutdown(struct pci_dev *pdev, bool *enable_wake)
 {
    struct net_device *netdev = pci_get_drvdata(pdev);
    struct igb_adapter *adapter = netdev_priv(netdev);
@@ -4336,15 +4336,9 @@ static int igb_suspend(struct pci_dev *p
            wr32(E1000_WUFC, 0);
    }
 
-   /* make sure adapter isn't asleep if manageability/wol is enabled */
-   if (wufc || adapter->en_mng_pt) {
-           pci_enable_wake(pdev, PCI_D3hot, 1);
-           pci_enable_wake(pdev, PCI_D3cold, 1);
-   } else {
+   *enable_wake = wufc || adapter->en_mng_pt;
+   if (!*enable_wake)
            igb_shutdown_fiber_serdes_link_82575(hw);
-           pci_enable_wake(pdev, PCI_D3hot, 0);
-           pci_enable_wake(pdev, PCI_D3cold, 0);
-   }
 
    /* Release control of h/w to f/w.  If f/w is AMT enabled, this
     * would have already happened in close and is redundant. */
@@ -4352,12 +4346,29 @@ static int igb_suspend(struct pci_dev *p
 
    pci_disable_device(pdev);
 
-   pci_set_power_state(pdev, pci_choose_state(pdev, state));
-
    return 0;
 }
 
 #ifdef CONFIG_PM
+static int igb_suspend(struct pci_dev *pdev, pm_message_t state)
+{
+   int retval;
+   bool wake;
+
+   retval = __igb_shutdown(pdev, &wake);
+   if (retval)
+           return retval;
+
+   if (wake) {
+           pci_prepare_to_sleep(pdev);
+   } else {
+           pci_wake_from_d3(pdev, false);
+           pci_set_power_state(pdev, PCI_D3hot);
+   }
+
+   return 0;
+}
+
 static int igb_resume(struct pci_dev *pdev)
 {
    struct net_device *netdev = pci_get_drvdata(pdev);
@@ -4412,7 +4423,14 @@ static int igb_resume(struct pci_dev *pd
 
 static void igb_shutdown(struct pci_dev *pdev)
 {
-   igb_suspend(pdev, PMSG_SUSPEND);
+   bool wake;
+
+   __igb_shutdown(pdev, &wake);
+
+   if (system_state == SYSTEM_POWER_OFF) {
+           pci_wake_from_d3(pdev, wake);
+           pci_set_power_state(pdev, PCI_D3hot);
+   }
 }
 
Quoted 3 lines
drivers/net/igb/igb_main.c static int igb_resume(struct pci_dev *pd · line 4436
Quoted 1 line
drivers/net/igb/igb_main.c +0 -0
 #ifdef CONFIG_NET_POLL_CONTROLLER
it works too.
Great, thanks for testing.
Jeff, Jesse, is the patch fine with you?
Quoted 8 lines
Quoted 1 line
Rafael
Let me talk with Jesse on Monday about it. I know that Jesse's concern's were that there were more than one driver afflicted with the same or similar issue and that it made more sense to fix this in the core for all drivers. For instance ixgbe, IIRC, but because Yinghai did not have any other hardware, it was unclear if this was an issue on other drivers.
I think it was, at least theoretically. We should generally avoid powering off things on shutdown when it's not really necessary. Also, the use of pci_enable_wake() in these drivers is not really correct.
And in fact this is why I'm asking, because I have analogouns patches for some other drivers in the works too. If you are fine with the approach, I'll post the whole series.
Quoted 3 lines
After discussing this with Jesse, one of us will respond. I will apply this patch to my tree anyway and if we are alright with the patch, I will push it out with my other patches to Dave.
Thanks!

Quoted 8 lines

Quoted 3 lines
Best,
Rafael
--
After talking with Jesse, we are fine with it if our testers buy off on it. I have the patch in my tree and so Emil will run a barrage of regression/stress tests on the patch overnight. If everything looks good, I will send it out with the other igb patches I have.

Great, thanks!

Rafael

Tantilov, Emil S

2009-03-31 19:15 UTC #25 lore raw

Quoted 225 lines · Rafael J. Wysocki

Quoted 224 lines · Jeff Kirsher
Quoted 222 lines · Rafael J. Wysocki
Quoted 197 lines · Jeff Kirsher
Quoted 195 lines · Rafael J. Wysocki
Quoted 186 lines · Yinghai Lu
Quoted 185 lines · Rafael J. Wysocki
Quoted 64 lines · Yinghai Lu
Quoted 63 lines · Rafael J. Wysocki
Quoted 62 lines · Rafael J. Wysocki
Quoted 61 lines · Yinghai Lu
Quoted 30 lines · Rafael J. Wysocki
Quoted 29 lines · Yinghai Lu
On Fri, Mar 6, 2009 at 11:18 PM, Jesse Brandeburg
<[email protected]> wrote:
On Fri, Mar 6, 2009 at 8:33 PM, Yinghai Lu
<[email protected]> wrote:
Impact: could probe igb
Found one system with 82575EB, in the kernel that is
kexeced, probe igb failed with -2.
it looks like the same behavior happened on forcedeth.
try to check system_state to make sure if put it on D3
Signed-off-by: Yinghai Lu <[email protected]>
---
drivers/net/igb/igb_main.c | 19 ++++++++++++++-----
1 file changed, 14 insertions(+), 5 deletions(-)
I see the point of the patch, but I know for a fact that
ixgbe when enabled for MSI-X also doesn't work with
kexec.
so my questions are:
are you going to change every driver?
i tend to only change driver that i have related HW.
why can't this be fixed in core kernel code instead?
will check it.
Quoted 8 lines
Quoted 5 lines
Shouldn't pci_enable_device take it out of D3?
Or maybe it should be taken out of D3 immediately if
someone tries to ioremap any of the BARx registers?
looks like second kernel can not detect the state any
more.
In fact pci_enable_device() calls pci_set_power_state(dev, PCI_D0) as the first thing. The question is why it doesn't work as expected.
not sure... please check the version for forcedeth that you made.
commit 3cb5599a84c557c0dd9a19feb63a3788268cf249
Author: Rafael J. Wysocki <[email protected]>
Date:   Fri Sep 5 14:00:19 2008 -0700
forcedeth: fix kexec regression
Fix regression tracked as
http://bugzilla.kernel.org/show_bug.cgi?id=11361 and caused by commit f735a2a1a4f2a0f5cd823ce323e82675990469e2 ("[netdrvr]
forcedeth: setup wake-on-lan before shutting down") that makes network adapters integrated into the NVidia MCP55 chipsets fail to work in kexeced kernels. The problem appears to be that if the adapter is put into D3_hot during ->shutdown(), it cannot be brought back
into D0 after kexec (ref.
http://marc.info/?l=linux-kernel&m=121900062814967&w=4).
Therefore, only put forcedeth into D3 during ->shutdown() if the system is to be powered off.
Quoted 6 lines
Quoted 5 lines
Quoted 1 line
Thanks, I remember now.
In which case you need to rework igb_shutdown() rather than igb_suspend().
Something like the patch below, perhaps (totally untested).
it works, David, can you picked it up
Still, Yinghai, can you please also test the patch below? It fixes all shortcomings in the driver's suspend and shutdown methods I was talking about in one of the previous messages. If it works, IMO it will be a preferable fix (in particular, it would be good to check if WoL still works with it, but I don't have the hardware).
Thanks,
Rafael
---
From: Rafael J. Wysocki <[email protected]>
Subject: net/igb: Fix kexec with igb (rev. 3)
Impact: Fix
Yinghai Lu found one system with 82575EB where, in the kernel that is kexeced, probe igb failed with -2, the reason being that the adapter
could not be brought back from D3 by the kexec kernel, most probably
due to quirky hardware (it looks like the same behavior happened on forcedeth).
Prevent igb from putting the adapter into D3 during shutdown except
when we going to power off the system. For this purpose, seperate igb_shutdown() from igb_suspend() and use the appropriate PCI PM
callbacks in both of them.
Signed-off-by Rafael J. Wysocki <[email protected]>
Reported-by Yinghai Lu <[email protected]>
---
 drivers/net/igb/igb_main.c |   42
++++++++++++++++++++++++++++++------------ 1 file changed, 30
insertions(+), 12 deletions(-)
Index: linux-2.6/drivers/net/igb/igb_main.c
===================================================================
drivers/net/igb/igb_main.c +0 -1
@@ -4277,7 +4277,7 @@ int igb_set_spd_dplx(struct igb_adapter  }
 
 
-static int igb_suspend(struct pci_dev *pdev, pm_message_t
state) +static int __igb_shutdown(struct pci_dev *pdev, bool
*enable_wake) { struct net_device *netdev = pci_get_drvdata(pdev); struct igb_adapter *adapter =
netdev_priv(netdev); @@ -4336,15 +4336,9 @@ static int
            igb_suspend(struct pci_dev *p wr32(E1000_WUFC, 0);
    }
- /* make sure adapter isn't asleep if manageability/wol is
enabled */
drivers/net/igb/igb_main.c +2 -7
-   if (wufc || adapter->en_mng_pt) {
-           pci_enable_wake(pdev, PCI_D3hot, 1);
-           pci_enable_wake(pdev, PCI_D3cold, 1);
-   } else {
+   *enable_wake = wufc || adapter->en_mng_pt;
+   if (!*enable_wake)
            igb_shutdown_fiber_serdes_link_82575(hw);
-           pci_enable_wake(pdev, PCI_D3hot, 0);
-           pci_enable_wake(pdev, PCI_D3cold, 0);
-   }
 
    /* Release control of h/w to f/w.  If f/w is AMT enabled,
     this * would have already happened in close and is
redundant. */ @@ -4352,12 +4346,29 @@ static int
igb_suspend(struct pci_dev *p
    pci_disable_device(pdev);
drivers/net/igb/igb_main.c +1 -1
-   pci_set_power_state(pdev, pci_choose_state(pdev, state)); -
    return 0;
 }
 
 #ifdef CONFIG_PM
+static int igb_suspend(struct pci_dev *pdev, pm_message_t
state) +{ + int retval;
drivers/net/igb/igb_main.c +24 -1
+   bool wake;
+
+   retval = __igb_shutdown(pdev, &wake);
+   if (retval)
+           return retval;
+
+   if (wake) {
+           pci_prepare_to_sleep(pdev);
+   } else {
+           pci_wake_from_d3(pdev, false);
+           pci_set_power_state(pdev, PCI_D3hot);
+   }
+
+   return 0;
+}
+
 static int igb_resume(struct pci_dev *pdev)
 {
    struct net_device *netdev = pci_get_drvdata(pdev);
@@ -4412,7 +4423,14 @@ static int igb_resume(struct pci_dev *pd
 
 static void igb_shutdown(struct pci_dev *pdev)
 {
-   igb_suspend(pdev, PMSG_SUSPEND);
+   bool wake;
+
+   __igb_shutdown(pdev, &wake);
+
+   if (system_state == SYSTEM_POWER_OFF) {
+           pci_wake_from_d3(pdev, wake);
+           pci_set_power_state(pdev, PCI_D3hot);
+   }
 }
 
Quoted 3 lines
drivers/net/igb/igb_main.c static int igb_resume(struct pci_dev *pd · line 4436
Quoted 1 line
drivers/net/igb/igb_main.c +0 -0
 #ifdef CONFIG_NET_POLL_CONTROLLER
it works too.
Great, thanks for testing.
Jeff, Jesse, is the patch fine with you?
Quoted 8 lines
Quoted 1 line
Rafael
Let me talk with Jesse on Monday about it. I know that Jesse's concern's were that there were more than one driver afflicted with the same or similar issue and that it made more sense to fix this in the core for all drivers. For instance ixgbe, IIRC, but because Yinghai did not have any other hardware, it was unclear if this was an issue on other drivers.
I think it was, at least theoretically. We should generally avoid powering off things on shutdown when it's not really necessary. Also, the use of pci_enable_wake() in these drivers is not really correct.
And in fact this is why I'm asking, because I have analogouns patches for some other drivers in the works too. If you are fine with the approach, I'll post the whole series.
Quoted 3 lines
After discussing this with Jesse, one of us will respond. I will apply this patch to my tree anyway and if we are alright with the patch, I will push it out with my other patches to Dave.
Thanks!

Quoted 8 lines

Quoted 8 lines
Quoted 3 lines
Best,
Rafael
--
After talking with Jesse, we are fine with it if our testers buy off on it. I have the patch in my tree and so Emil will run a barrage of regression/stress tests on the patch overnight. If everything looks good, I will send it out with the other igb patches I have.

The patch checks out. I tested suspend/resume (including WOL) and kexec. There is only a small issue with warning on compile when CONFIG_PM is disabled:

drivers/net/igb/igb_main.c +1 -1

index 13fe162..0a4f8f4 100644
@@ -135,8 +135,8 @@ static inline int igb_set_vf_rlpml(struct igb_adapter *, int, int);
 static int igb_set_vf_mac(struct igb_adapter *adapter, int, unsigned char *);
 static void igb_restore_vf_multicasts(struct igb_adapter *adapter);
 
-static int igb_suspend(struct pci_dev *, pm_message_t);
 #ifdef CONFIG_PM
+static int igb_suspend(struct pci_dev *, pm_message_t);
 static int igb_resume(struct pci_dev *);
 #endif
 static void igb_shutdown(struct pci_dev *);

Thanks,
Emil

Jeff Kirsher

2009-03-31 19:51 UTC #26 lore raw

Quoted 18 lines · Emil S

The patch checks out. I tested suspend/resume (including WOL) and kexec. There is only a small issue with warning on compile when CONFIG_PM is disabled:

drivers/net/igb/igb_main.c +1 -1

index 13fe162..0a4f8f4 100644
@@ -135,8 +135,8 @@ static inline int igb_set_vf_rlpml(struct igb_adapter *, int, int);
  static int igb_set_vf_mac(struct igb_adapter *adapter, int, unsigned char *);
  static void igb_restore_vf_multicasts(struct igb_adapter *adapter);
 
-static int igb_suspend(struct pci_dev *, pm_message_t);
  #ifdef CONFIG_PM
+static int igb_suspend(struct pci_dev *, pm_message_t);
  static int igb_resume(struct pci_dev *);
  #endif
  static void igb_shutdown(struct pci_dev *);

Quoted 3 lines

Thanks,
Emil

Rafael - Based on Emil's testing, I will modify the patch with Emil's suggestion and push to Dave, ok?

Signature · 2 lines

Cheers,
Jeff

Rafael J. Wysocki

2009-03-31 20:27 UTC #27 lore raw

Quoted 20 lines · Jeff Kirsher

Quoted 18 lines · Emil S

The patch checks out. I tested suspend/resume (including WOL) and kexec. There is only a small issue with warning on compile when CONFIG_PM is disabled:

drivers/net/igb/igb_main.c +1 -1

index 13fe162..0a4f8f4 100644
@@ -135,8 +135,8 @@ static inline int igb_set_vf_rlpml(struct igb_adapter *, int, int);
 static int igb_set_vf_mac(struct igb_adapter *adapter, int, unsigned char *);
 static void igb_restore_vf_multicasts(struct igb_adapter *adapter);
 
-static int igb_suspend(struct pci_dev *, pm_message_t);
 #ifdef CONFIG_PM
+static int igb_suspend(struct pci_dev *, pm_message_t);
 static int igb_resume(struct pci_dev *);
 #endif
 static void igb_shutdown(struct pci_dev *);

Quoted 6 lines

Quoted 3 lines
Thanks,
Emil
Rafael - Based on Emil's testing, I will modify the patch with Emil's suggestion and push to Dave, ok?

Fine with me.

Thanks,
Rafael

Yinghai Lu

author

2009-03-08 08:09 UTC #28 lore raw

PATCH 1/1 pci: fix kexec with power state D3

In reply to Jesse Brandeburg

Impact: second kernel by kexec will have some pci devices working

Found one system with 82575EB, in the kernel that is kexeced, probe igb failed with -2.

it looks like the same behavior happened on forcedeth. try to check system_state to make sure if put it on D3

Jesse Brandeburg said that we should do that check in core code instead of every device driver.

Signed-off-by Yinghai Lu <[email protected]>

---
 drivers/pci/pci.c |   17 +++++++++++++++++
 1 file changed, 17 insertions(+)

drivers/pci/pci.c +17 -0

Index: linux-2.6/drivers/pci/pci.c
===================================================================
@@ -593,6 +593,14 @@ int pci_set_power_state(struct pci_dev *
 	if (state == PCI_D3hot && (dev->dev_flags & PCI_DEV_FLAGS_NO_D3))
 		return 0;
 
+	/*
+	 * Apparently it is not possible to reinitialise from D3 hot,
+	 * only put the device into D3 if we really go for poweroff.
+	 */
+	if (system_state != SYSTEM_POWER_OFF &&
+	    (state == PCI_D3hot || state == PCI_D3cold))
+		return 0;
+
 	error = pci_raw_set_power_state(dev, state, true);
 
 	if (state > PCI_D0 && platform_pci_power_manageable(dev)) {

@@ -1124,6 +1132,15 @@ int pci_enable_wake(struct pci_dev *dev,
 	int error = 0;
 	bool pme_done = false;
 
+	/*
+	 * Apparently it is not possible to reinitialise from D3 hot,
+	 * only put the device into D3 if we really go for poweroff.
+	 * we only need to enable wake when we are going to power off
+	 */
+	if (enable && system_state != SYSTEM_POWER_OFF &&
+	    (state == PCI_D3hot || state == PCI_D3cold))
+		return 0;
+
 	if (enable && !device_may_wakeup(&dev->dev))
 		return -EINVAL;

Rafael J. Wysocki

2009-03-08 10:08 UTC #29 lore raw

Re: [PATCH] pci: fix kexec with power state D3

Quoted 8 lines · Yinghai Lu

Impact: second kernel by kexec will have some pci devices working
Found one system with 82575EB, in the kernel that is kexeced, probe igb failed with -2.
it looks like the same behavior happened on forcedeth. try to check system_state to make sure if put it on D3

This is not enough, because the PM code doesn't change system_state and it uses pci_set_power_state too.

Quoted 2 lines

Jesse Brandeburg said that we should do that check in core code instead of every device driver.

Well, I'm not really sure. The drivers are where the bug is.

Quoted 12 lines

Signed-off-by Yinghai Lu <[email protected]>
---
 drivers/pci/pci.c |   17 +++++++++++++++++
 1 file changed, 17 insertions(+)
Index: linux-2.6/drivers/pci/pci.c
===================================================================
drivers/pci/pci.c +0 -0
@@ -593,6 +593,14 @@ int pci_set_power_state(struct pci_dev *
 	if (state == PCI_D3hot && (dev->dev_flags & PCI_DEV_FLAGS_NO_D3))

drivers/pci/pci.c int pci_set_power_state(struct pci_dev * · line 603

Quoted 10 lines

drivers/pci/pci.c +8 -0

 		return 0;
 
+	/*
+	 * Apparently it is not possible to reinitialise from D3 hot,
+	 * only put the device into D3 if we really go for poweroff.
+	 */
+	if (system_state != SYSTEM_POWER_OFF &&
+	    (state == PCI_D3hot || state == PCI_D3cold))
+		return 0;
+

This breaks suspend/hibernation, doesn't it? Surely, we want to put devices into D3 when going for suspend, for example.

That's apart from the fact that the 'state == PCI_D3cold' is redundant.

Quoted 8 lines

drivers/pci/pci.c +1 -0

 	error = pci_raw_set_power_state(dev, state, true);
 
 	if (state > PCI_D0 && platform_pci_power_manageable(dev)) {

@@ -1124,6 +1132,15 @@ int pci_enable_wake(struct pci_dev *dev,
 	int error = 0;
 	bool pme_done = false;
 
+	/*

drivers/pci/pci.c int pci_enable_wake(struct pci_dev *dev, · line 1145

Quoted 10 lines

drivers/pci/pci.c +8 -0

+	 * Apparently it is not possible to reinitialise from D3 hot,
+	 * only put the device into D3 if we really go for poweroff.
+	 * we only need to enable wake when we are going to power off
+	 */
+	if (enable && system_state != SYSTEM_POWER_OFF &&
+	    (state == PCI_D3hot || state == PCI_D3cold))
+		return 0;
+
 	if (enable && !device_may_wakeup(&dev->dev))
 		return -EINVAL;

I don't like this at all, sorry.

I thought we were supposed to avoid using system_state in such a way, apart from the other things.

Thanks,
Rafael

Ingo Molnar

2009-03-08 10:15 UTC #30 lore raw

Re: [PATCH] pci: fix kexec with power state D3

Quoted 10 lines · Rafael J. Wysocki

Quoted 8 lines · Yinghai Lu
Impact: second kernel by kexec will have some pci devices working
Found one system with 82575EB, in the kernel that is kexeced, probe igb failed with -2.
it looks like the same behavior happened on forcedeth. try to check system_state to make sure if put it on D3

Quoted 7 lines

This is not enough, because the PM code doesn't change system_state and it uses pci_set_power_state too.
Quoted 2 lines
Jesse Brandeburg said that we should do that check in core code instead of every device driver.
Well, I'm not really sure. The drivers are where the bug is.

So Yinghai has now written a driver fix and a generic code fix as well and both are being rejected pointing to the other side?

This really sucks for users (who'd be happy with any of the patches) and the disagreement needs to be resolved ASAP.

Ingo

Rafael J. Wysocki

2009-03-08 10:28 UTC #31 lore raw

Re: [PATCH] pci: fix kexec with power state D3

Quoted 19 lines · Ingo Molnar

Quoted 16 lines · Rafael J. Wysocki
Quoted 8 lines · Yinghai Lu
Impact: second kernel by kexec will have some pci devices working
Found one system with 82575EB, in the kernel that is kexeced, probe igb failed with -2.
it looks like the same behavior happened on forcedeth. try to check system_state to make sure if put it on D3
This is not enough, because the PM code doesn't change system_state and it uses pci_set_power_state too.
Quoted 2 lines
Jesse Brandeburg said that we should do that check in core code instead of every device driver.

Quoted 7 lines

Quoted 1 line
Well, I'm not really sure. The drivers are where the bug is.
So Yinghai has now written a driver fix and a generic code fix as well and both are being rejected pointing to the other side?
This really sucks for users (who'd be happy with any of the patches) and the disagreement needs to be resolved ASAP.

I hope you read my other comment (that you deleted from the reply)?

If we're going to fix this at the core level along the Yinghai's patch lines, we'll have to introduce a few additional values for system_state and make suspend and hibernation code use them.

Do you think it's worth the effort, given that only a couple of drivers have been found to be affected so far?

Rafael

Rafael J. Wysocki

2009-03-08 10:33 UTC #32 lore raw

Re: [PATCH] pci: fix kexec with power state D3

Quoted 28 lines · Rafael J. Wysocki

Quoted 19 lines · Ingo Molnar
Quoted 16 lines · Rafael J. Wysocki
Quoted 8 lines · Yinghai Lu
Impact: second kernel by kexec will have some pci devices working
Found one system with 82575EB, in the kernel that is kexeced, probe igb failed with -2.
it looks like the same behavior happened on forcedeth. try to check system_state to make sure if put it on D3
This is not enough, because the PM code doesn't change system_state and it uses pci_set_power_state too.
Quoted 2 lines
Jesse Brandeburg said that we should do that check in core code instead of every device driver.
Quoted 7 lines
Quoted 1 line
Well, I'm not really sure. The drivers are where the bug is.
So Yinghai has now written a driver fix and a generic code fix as well and both are being rejected pointing to the other side?
This really sucks for users (who'd be happy with any of the patches) and the disagreement needs to be resolved ASAP.

Quoted 8 lines

I hope you read my other comment (that you deleted from the reply)?
If we're going to fix this at the core level along the Yinghai's patch lines, we'll have to introduce a few additional values for system_state and make suspend and hibernation code use them.
Do you think it's worth the effort, given that only a couple of drivers have been found to be affected so far?

And, quite frankly, I'm not sure if users will be happy with the $subject patch, because it _really_ breaks things (well, the kexec users who don't use suspend might be, but surely suspend users who don't use kexec won't).

Thanks,
Rafael

Ingo Molnar

2009-03-08 11:08 UTC #33 lore raw

Re: [PATCH] pci: fix kexec with power state D3

Quoted 35 lines · Rafael J. Wysocki

Quoted 34 lines · Rafael J. Wysocki
Quoted 19 lines · Ingo Molnar
Quoted 16 lines · Rafael J. Wysocki
Quoted 8 lines · Yinghai Lu
Impact: second kernel by kexec will have some pci devices working
Found one system with 82575EB, in the kernel that is kexeced, probe igb failed with -2.
it looks like the same behavior happened on forcedeth. try to check system_state to make sure if put it on D3
This is not enough, because the PM code doesn't change system_state and it uses pci_set_power_state too.
Quoted 2 lines
Jesse Brandeburg said that we should do that check in core code instead of every device driver.
Quoted 7 lines
Quoted 1 line
Well, I'm not really sure. The drivers are where the bug is.
So Yinghai has now written a driver fix and a generic code fix as well and both are being rejected pointing to the other side?
This really sucks for users (who'd be happy with any of the patches) and the disagreement needs to be resolved ASAP.
I hope you read my other comment (that you deleted from the reply)?
If we're going to fix this at the core level along the Yinghai's patch lines, we'll have to introduce a few additional values for system_state and make suspend and hibernation code use them.

Quoted 7 lines

Quoted 2 lines
Do you think it's worth the effort, given that only a couple of drivers have been found to be affected so far?
And, quite frankly, I'm not sure if users will be happy with the $subject patch, because it _really_ breaks things (well, the kexec users who don't use suspend might be, but surely suspend users who don't use kexec won't).

Please note that i havent reviewed the patches and i did not take any sides in the discussion - i just flagged the maintainer ping-pong. As long as we pick one of the patches (or a third one) within a bound amount of time we should be fine :)

Ingo

Ingo Molnar

2009-03-08 11:09 UTC #34 lore raw

Re: [PATCH] pci: fix kexec with power state D3

In reply to Rafael J. Wysocki

Quoted 35 lines · Rafael J. Wysocki

Quoted 34 lines · Rafael J. Wysocki
Quoted 19 lines · Ingo Molnar
Quoted 16 lines · Rafael J. Wysocki
Quoted 8 lines · Yinghai Lu
Impact: second kernel by kexec will have some pci devices working
Found one system with 82575EB, in the kernel that is kexeced, probe igb failed with -2.
it looks like the same behavior happened on forcedeth. try to check system_state to make sure if put it on D3
This is not enough, because the PM code doesn't change system_state and it uses pci_set_power_state too.
Quoted 2 lines
Jesse Brandeburg said that we should do that check in core code instead of every device driver.
Quoted 7 lines
Quoted 1 line
Well, I'm not really sure. The drivers are where the bug is.
So Yinghai has now written a driver fix and a generic code fix as well and both are being rejected pointing to the other side?
This really sucks for users (who'd be happy with any of the patches) and the disagreement needs to be resolved ASAP.
I hope you read my other comment (that you deleted from the reply)?
If we're going to fix this at the core level along the Yinghai's patch lines, we'll have to introduce a few additional values for system_state and make suspend and hibernation code use them.

Quoted 7 lines

Quoted 2 lines
Do you think it's worth the effort, given that only a couple of drivers have been found to be affected so far?
And, quite frankly, I'm not sure if users will be happy with the $subject patch, because it _really_ breaks things (well, the kexec users who don't use suspend might be, but surely suspend users who don't use kexec won't).

Ingo

Jesse Barnes

2009-03-20 01:49 UTC #35 lore raw

Re: [PATCH] pci: fix kexec with power state D3

Quoted 9 lines

Quoted 4 lines
And, quite frankly, I'm not sure if users will be happy with the $subject patch, because it _really_ breaks things (well, the kexec users who don't use suspend might be, but surely suspend users who don't use kexec won't).
Please note that i havent reviewed the patches and i did not take any sides in the discussion - i just flagged the maintainer ping-pong. As long as we pick one of the patches (or a third one) within a bound amount of time we should be fine :)

I'll defer to Rafael here; he's been working the most in this area. The changelog wasn't very complete for the original patch, but it sounds like in the kexec case the newly booted kernel will get an igb device in D3 which it can't handle? That really does sound like a driver bug, not something we should mess with in the core.

Signature · 1 line

Jesse Barnes, Intel Open Source Technology Center

Rafael J. Wysocki

2009-03-20 11:29 UTC #36 lore raw

Re: [PATCH] pci: fix kexec with power state D3

Quoted 10 lines · Jesse Barnes

Quoted 9 lines
Quoted 4 lines
And, quite frankly, I'm not sure if users will be happy with the $subject patch, because it _really_ breaks things (well, the kexec users who don't use suspend might be, but surely suspend users who don't use kexec won't).
Please note that i havent reviewed the patches and i did not take any sides in the discussion - i just flagged the maintainer ping-pong. As long as we pick one of the patches (or a third one) within a bound amount of time we should be fine :)

Quoted 5 lines

I'll defer to Rafael here; he's been working the most in this area. The changelog wasn't very complete for the original patch, but it sounds like in the kexec case the newly booted kernel will get an igb device in D3 which it can't handle? That really does sound like a driver bug, not something we should mess with in the core.

I have already posted an alternative patch for igb that's been reported to fix the problem.

Thanks,
Rafael

Rafael J. Wysocki

2009-03-20 11:29 UTC #37 lore raw

Re: [PATCH] pci: fix kexec with power state D3

In reply to Jesse Barnes

Quoted 10 lines · Jesse Barnes

Quoted 9 lines
Quoted 4 lines
And, quite frankly, I'm not sure if users will be happy with the $subject patch, because it _really_ breaks things (well, the kexec users who don't use suspend might be, but surely suspend users who don't use kexec won't).
Please note that i havent reviewed the patches and i did not take any sides in the discussion - i just flagged the maintainer ping-pong. As long as we pick one of the patches (or a third one) within a bound amount of time we should be fine :)

Quoted 5 lines

I'll defer to Rafael here; he's been working the most in this area. The changelog wasn't very complete for the original patch, but it sounds like in the kexec case the newly booted kernel will get an igb device in D3 which it can't handle? That really does sound like a driver bug, not something we should mess with in the core.

I have already posted an alternative patch for igb that's been reported to fix the problem.

Thanks,
Rafael

Jesse Barnes

2009-03-20 01:49 UTC #38 lore raw

Re: [PATCH] pci: fix kexec with power state D3

In reply to Ingo Molnar

Quoted 9 lines

Quoted 4 lines
And, quite frankly, I'm not sure if users will be happy with the $subject patch, because it _really_ breaks things (well, the kexec users who don't use suspend might be, but surely suspend users who don't use kexec won't).
Please note that i havent reviewed the patches and i did not take any sides in the discussion - i just flagged the maintainer ping-pong. As long as we pick one of the patches (or a third one) within a bound amount of time we should be fine :)

Signature · 1 line

Jesse Barnes, Intel Open Source Technology Center

Rafael J. Wysocki

2009-03-08 10:33 UTC #39 lore raw

Re: [PATCH] pci: fix kexec with power state D3

In reply to Rafael J. Wysocki

Quoted 28 lines · Rafael J. Wysocki

Quoted 19 lines · Ingo Molnar
Quoted 16 lines · Rafael J. Wysocki
Quoted 8 lines · Yinghai Lu
Impact: second kernel by kexec will have some pci devices working
Found one system with 82575EB, in the kernel that is kexeced, probe igb failed with -2.
it looks like the same behavior happened on forcedeth. try to check system_state to make sure if put it on D3
This is not enough, because the PM code doesn't change system_state and it uses pci_set_power_state too.
Quoted 2 lines
Jesse Brandeburg said that we should do that check in core code instead of every device driver.
Quoted 7 lines
Quoted 1 line
Well, I'm not really sure. The drivers are where the bug is.
So Yinghai has now written a driver fix and a generic code fix as well and both are being rejected pointing to the other side?
This really sucks for users (who'd be happy with any of the patches) and the disagreement needs to be resolved ASAP.

Quoted 8 lines

I hope you read my other comment (that you deleted from the reply)?
If we're going to fix this at the core level along the Yinghai's patch lines, we'll have to introduce a few additional values for system_state and make suspend and hibernation code use them.
Do you think it's worth the effort, given that only a couple of drivers have been found to be affected so far?

Thanks,
Rafael

Rafael J. Wysocki

2009-03-08 10:28 UTC #40 lore raw

Re: [PATCH] pci: fix kexec with power state D3

In reply to Ingo Molnar

Quoted 19 lines · Ingo Molnar

Quoted 16 lines · Rafael J. Wysocki
Quoted 8 lines · Yinghai Lu
Impact: second kernel by kexec will have some pci devices working
Found one system with 82575EB, in the kernel that is kexeced, probe igb failed with -2.
it looks like the same behavior happened on forcedeth. try to check system_state to make sure if put it on D3
This is not enough, because the PM code doesn't change system_state and it uses pci_set_power_state too.
Quoted 2 lines
Jesse Brandeburg said that we should do that check in core code instead of every device driver.

Quoted 7 lines

Quoted 1 line
Well, I'm not really sure. The drivers are where the bug is.
So Yinghai has now written a driver fix and a generic code fix as well and both are being rejected pointing to the other side?
This really sucks for users (who'd be happy with any of the patches) and the disagreement needs to be resolved ASAP.

I hope you read my other comment (that you deleted from the reply)?

If we're going to fix this at the core level along the Yinghai's patch lines, we'll have to introduce a few additional values for system_state and make suspend and hibernation code use them.

Do you think it's worth the effort, given that only a couple of drivers have been found to be affected so far?

Rafael

Ingo Molnar

2009-03-08 10:15 UTC #41 lore raw

Re: [PATCH] pci: fix kexec with power state D3

In reply to Rafael J. Wysocki

Quoted 10 lines · Rafael J. Wysocki

Quoted 8 lines · Yinghai Lu
Impact: second kernel by kexec will have some pci devices working
Found one system with 82575EB, in the kernel that is kexeced, probe igb failed with -2.
it looks like the same behavior happened on forcedeth. try to check system_state to make sure if put it on D3

Quoted 7 lines

This is not enough, because the PM code doesn't change system_state and it uses pci_set_power_state too.
Quoted 2 lines
Jesse Brandeburg said that we should do that check in core code instead of every device driver.
Well, I'm not really sure. The drivers are where the bug is.

So Yinghai has now written a driver fix and a generic code fix as well and both are being rejected pointing to the other side?

This really sucks for users (who'd be happy with any of the patches) and the disagreement needs to be resolved ASAP.

Ingo

Rafael J. Wysocki

2009-03-08 10:08 UTC #42 lore raw

Re: [PATCH] pci: fix kexec with power state D3

In reply to Yinghai Lu

Quoted 8 lines · Yinghai Lu

Impact: second kernel by kexec will have some pci devices working
Found one system with 82575EB, in the kernel that is kexeced, probe igb failed with -2.
it looks like the same behavior happened on forcedeth. try to check system_state to make sure if put it on D3

This is not enough, because the PM code doesn't change system_state and it uses pci_set_power_state too.

Quoted 2 lines

Jesse Brandeburg said that we should do that check in core code instead of every device driver.

Well, I'm not really sure. The drivers are where the bug is.

Quoted 12 lines

Signed-off-by Yinghai Lu <[email protected]>
---
 drivers/pci/pci.c |   17 +++++++++++++++++
 1 file changed, 17 insertions(+)
Index: linux-2.6/drivers/pci/pci.c
===================================================================
drivers/pci/pci.c +0 -0
@@ -593,6 +593,14 @@ int pci_set_power_state(struct pci_dev *
 	if (state == PCI_D3hot && (dev->dev_flags & PCI_DEV_FLAGS_NO_D3))

drivers/pci/pci.c int pci_set_power_state(struct pci_dev * · line 603

Quoted 10 lines

drivers/pci/pci.c +8 -0

 		return 0;
 
+	/*
+	 * Apparently it is not possible to reinitialise from D3 hot,
+	 * only put the device into D3 if we really go for poweroff.
+	 */
+	if (system_state != SYSTEM_POWER_OFF &&
+	    (state == PCI_D3hot || state == PCI_D3cold))
+		return 0;
+

This breaks suspend/hibernation, doesn't it? Surely, we want to put devices into D3 when going for suspend, for example.

That's apart from the fact that the 'state == PCI_D3cold' is redundant.

Quoted 8 lines

drivers/pci/pci.c +1 -0

 	error = pci_raw_set_power_state(dev, state, true);
 
 	if (state > PCI_D0 && platform_pci_power_manageable(dev)) {

@@ -1124,6 +1132,15 @@ int pci_enable_wake(struct pci_dev *dev,
 	int error = 0;
 	bool pme_done = false;
 
+	/*

drivers/pci/pci.c int pci_enable_wake(struct pci_dev *dev, · line 1145

Quoted 10 lines

drivers/pci/pci.c +8 -0

+	 * Apparently it is not possible to reinitialise from D3 hot,
+	 * only put the device into D3 if we really go for poweroff.
+	 * we only need to enable wake when we are going to power off
+	 */
+	if (enable && system_state != SYSTEM_POWER_OFF &&
+	    (state == PCI_D3hot || state == PCI_D3cold))
+		return 0;
+
 	if (enable && !device_may_wakeup(&dev->dev))
 		return -EINVAL;

I don't like this at all, sorry.

I thought we were supposed to avoid using system_state in such a way, apart from the other things.

Thanks,
Rafael