A question about PCI suspend-resume functionallity
Hi,
We are testing Intel Gigabit adapter driver (igb) under Fedora 20, kernel 3.14.4 for the following use-case:
(*) Adapter is connected to the PCIE slot
(*) We put the system under suspend by running pm-suspend from user-space (*) Remove the adapter from the PCIE slot
(*) Wake up the system
Currenlty, we got kernel panics and the system got stuck.
My question is - does the PCI subsystem logic calls the driver remove function when driver resume function returns with error code?
Or should I implement the call to igb_remove from igb_resume in the Intel driver?
Thanks,
Igor Bezukh
Bjorn Helgaas
maintainer[+cc linux-pm]
Quoted 10 lines · Igor Bezukh
Hi,
We are testing Intel Gigabit adapter driver (igb) under Fedora 20, kernel 3.14.4 for the following use-case:
(*) Adapter is connected to the PCIE slot
(*) We put the system under suspend by running pm-suspend from user-space (*) Remove the adapter from the PCIE slot
(*) Wake up the system
Quoted 5 lines
Currenlty, we got kernel panics and the system got stuck.
My question is - does the PCI subsystem logic calls the driver remove function when driver resume function returns with error code?
Or should I implement the call to
igb_removefromigb_resumein the Intel driver?
I don't know what the best design is here. I suspect that when we resume, we should re-enumerate the PCI topology to see if anything changed, but I don't think we do that. I think there *is* something like that for ACPI -- the BIOS can send a Bus Check notification on resume if it knows something has changed. But my guess is that if you remove an adapter below a switch that is powered down because the system is suspended, the interrupt we would normally get is lost forever.
It doesn't seem like something that should be solved in the driver, because you could just as easily have *added* a device while suspended, and the core has to re-enumerate to find that.
Bjorn
Quoted 17 lines · Bjorn Helgaas
[+cc linux-pm]
Quoted 14 lines · Igor Bezukh
Hi,
We are testing Intel Gigabit adapter driver (igb) under Fedora 20, kernel 3.14.4 for the following use-case:
(*) Adapter is connected to the PCIE slot
(*) We put the system under suspend by running pm-suspend from user-space (*) Remove the adapter from the PCIE slot
(*) Wake up the systemCurrenlty, we got kernel panics and the system got stuck.
My question is - does the PCI subsystem logic calls the driver remove function when driver resume function returns with error code?
Quoted 7 lines
Quoted 1 line
Or should I implement the call to
igb_removefromigb_resumein the Intel driver?I don't know what the best design is here. I suspect that when we resume, we should re-enumerate the PCI topology to see if anything changed, but I don't think we do that. I think there *is* something like that for ACPI -- the BIOS can send a Bus Check notification on resume if it knows something has changed.
In ACPI we wait for system resume to complete and handle the notifications then. Calling .remove() callbacks from drivers during system resume doesn't really work, especially if they happen too early (e.g. during the "noirq resume" stage).
Quoted 8 lines
But my guess is that if you
remove an adapter below a switch that is powered down because the system is suspended, the interrupt we would normally get is lost forever.It doesn't seem like something that should be solved in the driver, because you could just as easily have *added* a device while suspended, and the core has to re-enumerate to find that.
The driver's system resume callbacks need to be able to cope with missing devices.
Rafael
Quoted 26 lines · Rafael J. Wysocki
Quoted 17 lines · Bjorn Helgaas
[+cc linux-pm]
Quoted 14 lines · Igor Bezukh
Hi,
We are testing Intel Gigabit adapter driver (igb) under Fedora 20, kernel 3.14.4 for the following use-case:
(*) Adapter is connected to the PCIE slot
(*) We put the system under suspend by running pm-suspend from user-space (*) Remove the adapter from the PCIE slot
(*) Wake up the systemCurrenlty, we got kernel panics and the system got stuck.
My question is - does the PCI subsystem logic calls the driver remove function when driver resume function returns with error code?
Quoted 7 lines
Quoted 1 line
Or should I implement the call to
igb_removefromigb_resumein the Intel driver?I don't know what the best design is here. I suspect that when we resume, we should re-enumerate the PCI topology to see if anything changed, but I don't think we do that. I think there *is* something like that for ACPI -- the BIOS can send a Bus Check notification on resume if it knows something has changed.
Quoted 4 lines
In ACPI we wait for system resume to complete and handle the notifications then. Calling .remove() callbacks from drivers during system resume doesn't really work, especially if they happen too early (e.g. during the "noirq resume" stage).
USB uses a similar approach. Hot-unplugs are handled by the khubd thread, which is freezable and therefore doesn't run until after the system is resumed.
Quoted 11 lines
Quoted 8 lines
But my guess is that if you
remove an adapter below a switch that is powered down because the system is suspended, the interrupt we would normally get is lost forever.It doesn't seem like something that should be solved in the driver, because you could just as easily have *added* a device while suspended, and the core has to re-enumerate to find that.
The driver's system resume callbacks need to be able to cope with missing devices.
In the USB stack, the subsystem core resume code checks to see if a device has been unplugged before the driver's ->resume callback is invoked. If a device is gone, the driver callback is skipped. Thus drivers don't have to worry about trying to resume a device that has been unplugged.
Alan Stern
Bjorn Helgaas
maintainerQuoted 18 lines · Alan Stern
Quoted 16 lines · Rafael J. Wysocki
Quoted 15 lines · Bjorn Helgaas
[+cc linux-pm]
Quoted 12 lines · Igor Bezukh
Hi,
We are testing Intel Gigabit adapter driver (igb) under Fedora 20, kernel 3.14.4 for the following use-case:
(*) Adapter is connected to the PCIE slot
(*) We put the system under suspend by running pm-suspend from user-space (*) Remove the adapter from the PCIE slot
(*) Wake up the systemCurrenlty, we got kernel panics and the system got stuck.
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
My question is - does the PCI subsystem logic calls the driver remove function when driver resume function returns with error code?
Or should I implement the call to
igb_removefromigb_resumein the Intel driver?...
The driver's system resume callbacks need to be able to cope with missing devices.
Based on this, it sounds like igb_resume() should call igb_remove() when it figures out the device is missing.
That might be the best we can do right now, but it doesn't sound like a general-purpose solution. Detecting device removal sounds like a core function, not a driver function. It doesn't seem like drivers should have to implement ->resume just to deal with this case. Calling ->remove from ->resume is a little strange and may expose locking issues or races. I suspect that most ->remove methods are not prepared to deal with missing devices (they may want to stop DMA, disable interrupts, etc.)
Quoted 5 lines
In the USB stack, the subsystem core resume code checks to see if a device has been unplugged before the driver's ->resume callback is invoked. If a device is gone, the driver callback is skipped. Thus drivers don't have to worry about trying to resume a device that has been unplugged.
I assume that you do want to call the driver's ->remove method to free any per-device state. Do USB drivers just have to be smart enough to do that without touching the device?
Bjorn
Quoted 21 lines · Bjorn Helgaas
Quoted 20 lines · Alan Stern
Quoted 18 lines · Rafael J. Wysocki
Quoted 17 lines · Bjorn Helgaas
[+cc linux-pm]
Quoted 14 lines · Igor Bezukh
Hi,
We are testing Intel Gigabit adapter driver (igb) under Fedora 20, kernel 3.14.4 for the following use-case:
(*) Adapter is connected to the PCIE slot
(*) We put the system under suspend by running pm-suspend from user-space (*) Remove the adapter from the PCIE slot
(*) Wake up the systemCurrenlty, we got kernel panics and the system got stuck.
My question is - does the PCI subsystem logic calls the driver remove function when driver resume function returns with error code?
Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
Quoted 1 line
Quoted 1 line
Or should I implement the call to
igb_removefromigb_resumein the Intel driver?...
The driver's system resume callbacks need to be able to cope with missing devices.Based on this, it sounds like
igb_resume()should calligb_remove()when it figures out the device is missing.
I wouldn't say so. igb_resume() should not crash when the device is missing and should just handle that situation cleanly. Obviously it is not its role to remove the device from the hierarchy.
Quoted 4 lines
That might be the best we can do right now, but it doesn't sound like a general-purpose solution. Detecting device removal sounds like a core function, not a driver function. It doesn't seem like drivers should have to implement ->resume just to deal with this case.
No, they shouldn't. They just need to be able to cope with missing devices cleanly.
Devices (and PCI devices in particular) can go away at any time, including during system resume, without notice anyway and drivers need to be able to cope with that regardless.
The notification can actually come in *after* the device has gone in any case and then whoever gets the notification should handle the device removal. That is not the driver in particular, but in the meantime the driver should still work without crashing.
Rafael
Quoted 9 lines · Bjorn Helgaas
Quoted 5 lines
In the USB stack, the subsystem core resume code checks to see if a device has been unplugged before the driver's ->resume callback is invoked. If a device is gone, the driver callback is skipped. Thus drivers don't have to worry about trying to resume a device that has been unplugged.
I assume that you do want to call the driver's ->remove method to free any per-device state. Do USB drivers just have to be smart enough to do that without touching the device?
Not quite how I'd describe it. They have to be smart enough for the remove routine to run correctly even if the device is already gone -- _any_ driver for a hot-unpluggable device has to be like this.
Obviously no driver can touch a device that is gone. But it can try. If a USB driver tries to touch a device that has been unplugged, the corresponding function call just returns an error. It doesn't crash.
Alan Stern
Bjorn Helgaas
maintainerQuoted 28 lines · Rafael J. Wysocki
Quoted 27 lines · Bjorn Helgaas
Quoted 25 lines · Alan Stern
Quoted 23 lines · Rafael J. Wysocki
Quoted 18 lines · Bjorn Helgaas
[+cc linux-pm]
Quoted 15 lines · Igor Bezukh
Hi,
We are testing Intel Gigabit adapter driver (igb) under Fedora 20, kernel 3.14.4 for the following use-case:
(*) Adapter is connected to the PCIE slot
(*) We put the system under suspend by running pm-suspend from user-space (*) Remove the adapter from the PCIE slot
(*) Wake up the systemCurrenlty, we got kernel panics and the system got stuck.
My question is - does the PCI subsystem logic calls the driver remove function when driver resume function returns with error code?
Or should I implement the call to
igb_removefromigb_resumein the Intel driver?...
The driver's system resume callbacks need to be able to cope with missing devices.
Quoted 6 lines
Quoted 2 lines
Based on this, it sounds like
igb_resume()should calligb_remove()when it figures out the device is missing.I wouldn't say so.
igb_resume()should not crash when the device is missing and should just handle that situation cleanly. Obviously it is not its role to remove the device from the hierarchy.
OK, that makes sense.
However, I don't know of anything in the PCI core that will notice that the device has disappeared, so I doubt it will be removed from the hierarchy. I think that means the slot will become unusable until a reboot, because the original device is gone and we can't add a new one because the original one is still in the hierarchy. That's not very good, but it is better than a panic.
Quoted 8 lines
Quoted 4 lines
That might be the best we can do right now, but it doesn't sound like a general-purpose solution. Detecting device removal sounds like a core function, not a driver function. It doesn't seem like drivers should have to implement ->resume just to deal with this case.
No, they shouldn't. They just need to be able to cope with missing devices cleanly.
Quoted 8 lines
Devices (and PCI devices in particular) can go away at any time, including during system resume, without notice anyway and drivers need to be able to cope with that regardless.
The notification can actually come in *after* the device has gone in any case and then whoever gets the notification should handle the device removal. That is not the driver in particular, but in the meantime the driver should still work without crashing.
Yep. So the panic Igor is seeing is probably an igb_resume() problem, but after that's fixed, we'll probably trip over the PCI bug about not handling the remove.
Bjorn
Quoted 36 lines · Bjorn Helgaas
Quoted 28 lines · Rafael J. Wysocki
Quoted 27 lines · Bjorn Helgaas
Quoted 25 lines · Alan Stern
Quoted 23 lines · Rafael J. Wysocki
Quoted 18 lines · Bjorn Helgaas
[+cc linux-pm]
Quoted 15 lines · Igor Bezukh
Hi,
We are testing Intel Gigabit adapter driver (igb) under Fedora 20, kernel 3.14.4 for the following use-case:
(*) Adapter is connected to the PCIE slot
(*) We put the system under suspend by running pm-suspend from user-space (*) Remove the adapter from the PCIE slot
(*) Wake up the systemCurrenlty, we got kernel panics and the system got stuck.
My question is - does the PCI subsystem logic calls the driver remove function when driver resume function returns with error code?
Or should I implement the call to
igb_removefromigb_resumein the Intel driver?...
The driver's system resume callbacks need to be able to cope with missing devices.Quoted 6 lines
Quoted 2 lines
Based on this, it sounds like
igb_resume()should calligb_remove()when it figures out the device is missing.I wouldn't say so.
igb_resume()should not crash when the device is missing and should just handle that situation cleanly. Obviously it is not its role to remove the device from the hierarchy.
Quoted 5 lines
OK, that makes sense.
However, I don't know of anything in the PCI core that will notice that the device has disappeared, so I doubt it will be removed from the hierarchy.
If we don't get a notification via ACPI or PCIe hotplug or anything, then no, it won't be removed automatically.
However, it still can be removed manually via sysfs, can't it?
Quoted 17 lines
I think that means the slot will become unusable until
a reboot, because the original device is gone and we can't add a new one because the original one is still in the hierarchy. That's not very good, but it is better than a panic.Quoted 12 lines
Quoted 4 lines
That might be the best we can do right now, but it doesn't sound like a general-purpose solution. Detecting device removal sounds like a core function, not a driver function. It doesn't seem like drivers should have to implement ->resume just to deal with this case.
No, they shouldn't. They just need to be able to cope with missing devices cleanly.
Devices (and PCI devices in particular) can go away at any time, including during system resume, without notice anyway and drivers need to be able to cope with that regardless.
Quoted 8 lines
Quoted 4 lines
The notification can actually come in *after* the device has gone in any case and then whoever gets the notification should handle the device removal. That is not the driver in particular, but in the meantime the driver should still work without crashing.
Yep. So the panic Igor is seeing is probably an
igb_resume()problem, but after that's fixed, we'll probably trip over the PCI bug about not handling the remove.
If there's any kind of notification coming in either before of after the event, we should act on it and remove the device. It's a bug if we don't I'd say.
Rafael
Bjorn Helgaas
maintainerQuoted 39 lines · Rafael J. Wysocki
Quoted 38 lines · Bjorn Helgaas
Quoted 28 lines · Rafael J. Wysocki
Quoted 27 lines · Bjorn Helgaas
Quoted 25 lines · Alan Stern
Quoted 23 lines · Rafael J. Wysocki
Quoted 18 lines · Bjorn Helgaas
[+cc linux-pm]
Quoted 15 lines · Igor Bezukh
Hi,
We are testing Intel Gigabit adapter driver (igb) under Fedora 20, kernel 3.14.4 for the following use-case:
(*) Adapter is connected to the PCIE slot
(*) We put the system under suspend by running pm-suspend from user-space (*) Remove the adapter from the PCIE slot
(*) Wake up the systemCurrenlty, we got kernel panics and the system got stuck.
My question is - does the PCI subsystem logic calls the driver remove function when driver resume function returns with error code?
Or should I implement the call to
igb_removefromigb_resumein the Intel driver?...
The driver's system resume callbacks need to be able to cope with missing devices.Quoted 6 lines
Quoted 2 lines
Based on this, it sounds like
igb_resume()should calligb_remove()when it figures out the device is missing.I wouldn't say so.
igb_resume()should not crash when the device is missing and should just handle that situation cleanly. Obviously it is not its role to remove the device from the hierarchy.OK, that makes sense.
Quoted 8 lines
Quoted 3 lines
However, I don't know of anything in the PCI core that will notice that the device has disappeared, so I doubt it will be removed from the hierarchy.
If we don't get a notification via ACPI or PCIe hotplug or anything, then no, it won't be removed automatically.
However, it still can be removed manually via sysfs, can't it?
Yes, I would think so. So I guess there's a workaround at least.
Igor, can you test this scenario (after fixing igb_resume() so it doesn't crash when the device is missing)? I.e., suspend the system, remove the adapter, resume the system, then do an "lspci" to see if the kernel thinks the adapter is still there, then put an adapter in the slot again (either hot-add if the the slot supports it, or suspend/add/resume)?
Bjorn
Quoted 50 lines · Rafael J. Wysocki
Quoted 39 lines · Bjorn Helgaas
Quoted 38 lines · Rafael J. Wysocki
Quoted 28 lines · Bjorn Helgaas
Quoted 27 lines · Alan Stern
Quoted 25 lines · Rafael J. Wysocki
Quoted 24 lines · Bjorn Helgaas
Quoted 18 lines
[+cc linux-pm]
Quoted 15 lines · Igor Bezukh
Hi,
We are testing Intel Gigabit adapter driver (igb) under Fedora 20, kernel 3.14.4 for the following use-case:
(*) Adapter is connected to the PCIE slot
(*) We put the system under suspend by running pm-suspend from user-space (*) Remove the adapter from the PCIE slot
(*) Wake up the systemCurrenlty, we got kernel panics and the system got stuck.
My question is - does the PCI subsystem logic calls the driver remove function when driver resume function returns with error code?
Or should I implement the call to
igb_removefromigb_resumein the Intel driver?...
The driver's system resume callbacks need to be able to cope with missing devices.Quoted 6 lines
Quoted 2 lines
Based on this, it sounds like
igb_resume()should calligb_remove()when it figures out the device is missing.I wouldn't say so.
igb_resume()should not crash when the device is missing and should just handle that situation cleanly. Obviously it is not its role to remove the device from the hierarchy.OK, that makes sense.
Quoted 8 lines
Quoted 3 lines
However, I don't know of anything in the PCI core that will notice that the device has disappeared, so I doubt it will be removed from the hierarchy.
If we don't get a notification via ACPI or PCIe hotplug or anything, then no, it won't be removed automatically.
However, it still can be removed manually via sysfs, can't it?
Quoted 8 lines
Yes, I would think so. So I guess there's a workaround at least.
Igor, can you test this scenario (after fixing
igb_resume()so it doesn't crash when the device is missing)? I.e., suspend the system, remove the adapter, resume the system, then do an "lspci" to see if the kernel thinks the adapter is still there, then put an adapter in the slot again (either hot-add if the the slot supports it, or suspend/add/resume)?
Sure. I think I already found the root cause of the kernel panic. I will test it ( and submit a patch if it is correct) .
I will also test the PCI enumeration with the fixed driver and I wll update you soon.
Alan, Rafael, Bjorn, thank you for the information!
Igor