PCI/ERR: Fix reset logic in pcie_do_recovery() call
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 21 lines · Sathyanarayanan
Quoted 18 lines · Bjorn Helgaas
Quoted 4 lines · On Fri, Jul 24, 2020 at 12:07:55PM -0700…
From: Kuppuswamy Sathyanarayanan <[email protected]>
Current
pcie_do_recovery()implementation has following two issues:I'm having trouble parsing this out, probably just lack of my understanding...
Quoted 8 lines
1. Fatal (DPC) error recovery is currently broken for non-hotplug capable devices. Current fatal error recovery implementation relies on PCIe hotplug (pciehp) handler for detaching and re-enumerating the affected devices/drivers. pciehp handler listens for DLLSC state changes and handles device/driver detachment on
DLLSC_LINK_DOWNevent and re-enumeration onDLLSC_LINK_UPevent. So when dealing with non-hotplug capable devices, recovery code does not restore the state of the affected devices correctly.
Quoted 5 lines
Quoted 2 lines
Apparently in the hotplug case, something *does* restore the state of affected devices?
Yes, in hotplug case, DLLSC state change handler takes over detachment /cleanup and re-attachment of affected devices/drivers.
Where does the restore happen here? I.e., what function does this?
Quoted 22 lines · Bjorn Helgaas
Quoted 21 lines · Sathyanarayanan
Quoted 18 lines · Bjorn Helgaas
Quoted 4 lines · On Fri, Jul 24, 2020 at 12:07:55PM -0700…
From: Kuppuswamy Sathyanarayanan <[email protected]>
Current
pcie_do_recovery()implementation has following two issues:I'm having trouble parsing this out, probably just lack of my understanding...
Quoted 8 lines
1. Fatal (DPC) error recovery is currently broken for non-hotplug capable devices. Current fatal error recovery implementation relies on PCIe hotplug (pciehp) handler for detaching and re-enumerating the affected devices/drivers. pciehp handler listens for DLLSC state changes and handles device/driver detachment on
DLLSC_LINK_DOWNevent and re-enumeration onDLLSC_LINK_UPevent. So when dealing with non-hotplug capable devices, recovery code does not restore the state of the affected devices correctly.
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
Apparently in the hotplug case, something *does* restore the state of affected devices?
Yes, in hotplug case, DLLSC state change handler takes over detachment /cleanup and re-attachment of affected devices/drivers.
Where does the restore happen here? I.e., what function does this?
DLLSC link down event will remove affected devices/drivers. And link up event will re-create all devices.
on DLLSC link down event
->pciehp_ist()
->pciehp_handle_presence_or_link_change()
->pciehp_disable_slot()
->__pciehp_disable_slot()
->remove_board()
->pciehp_unconfigure_device()on DLLSC link up event
->pciehp_ist()
->pciehp_handle_presence_or_link_change()
->pciehp_enable_slot()
->__pciehp_enable_slot()
->board_added()
->pciehp_configure_device()Quoted 1 line
Signature · 2 lines
Sathyanarayanan Kuppuswamy Linux Kernel Developer
Quoted 14 lines · Sathyanarayanan
Quoted 1 line
here does the restore happen here? I.e., what function does this?
DLLSC link down event will remove affected devices/drivers. And link up event
will re-create all devices.on DLLSC link down event
->pciehp_ist()
->pciehp_handle_presence_or_link_change()
->pciehp_disable_slot()
->__pciehp_disable_slot()
->remove_board()
->pciehp_unconfigure_device()
Quoted 7 lines
on DLLSC link up event
->pciehp_ist()
->pciehp_handle_presence_or_link_change()
->pciehp_enable_slot()
->__pciehp_enable_slot()
->board_added()
->pciehp_configure_device()
AFAIK, DLLSC is a requirement not optional. Why is this not supported by non-hotplug ports?
Quoted 23 lines · Sinan Kaya
Quoted 14 lines · Sathyanarayanan
Quoted 1 line
here does the restore happen here? I.e., what function does this?
DLLSC link down event will remove affected devices/drivers. And link up event
will re-create all devices.on DLLSC link down event
->pciehp_ist()
->pciehp_handle_presence_or_link_change()
->pciehp_disable_slot()
->__pciehp_disable_slot()
->remove_board()
->pciehp_unconfigure_device()Quoted 7 lines
on DLLSC link up event
->pciehp_ist()
->pciehp_handle_presence_or_link_change()
->pciehp_enable_slot()
->__pciehp_enable_slot()
->board_added()
->pciehp_configure_device()
Quoted 2 lines
AFAIK, DLLSC is a requirement not optional. Why is this not supported by non-hotplug ports?
Its required for hotplug capable ports. Please check PCIe spec v5.0 sec 6.7.3.3.
The Data Link Layer State Changed event provides an indication that the state of the Data Link Layer Link Active bit in the Link Status Register has changed. Support for Data Link Layer State Changed events and software notification of these events are required for hot-plug capable Downstream Ports.
Quoted 1 line
Signature · 2 lines
Sathyanarayanan Kuppuswamy Linux Kernel Developer
Quoted 5 lines · Sathyanarayanan
Quoted 2 lines
AFAIK, DLLSC is a requirement not optional. Why is this not supported by non-hotplug ports?
Its required for hotplug capable ports. Please check PCIe spec v5.0 sec 6.7.3.3.
Quoted 7 lines
The Data Link Layer State Changed event provides an indication that the state of
the Data Link Layer Link Active bit in the Link Status Register has changed.
Support for Data Link Layer State Changed events and software notification of these
events are required for hot-plug capable Downstream Ports.
I see. Can I assume that your system supports DPC? DPC is supposed to recover the link via dpc_reset_link().
Quoted 14 lines · Sinan Kaya
Quoted 5 lines · Sathyanarayanan
Quoted 2 lines
AFAIK, DLLSC is a requirement not optional. Why is this not supported by non-hotplug ports?
Its required for hotplug capable ports. Please check PCIe spec v5.0 sec 6.7.3.3.
Quoted 7 lines
The Data Link Layer State Changed event provides an indication that the state of
the Data Link Layer Link Active bit in the Link Status Register has changed.
Support for Data Link Layer State Changed events and software notification of these
events are required for hot-plug capable Downstream Ports.
Quoted 2 lines
I see. Can I assume that your system supports DPC? DPC is supposed to recover the link via
dpc_reset_link().
Yes. But the affected device/drivers cleanup during error recovery is handled by hotplug handler. So we are facing issue when dealing with non hotplug capable ports.
Quoted 1 line
Signature · 2 lines
Sathyanarayanan Kuppuswamy Linux Kernel Developer
Quoted 6 lines · Sathyanarayanan
Quoted 3 lines
I see. Can I assume that your system supports DPC? DPC is supposed to recover the link via
dpc_reset_link().Yes. But the affected device/drivers cleanup during error recovery is handled by hotplug handler. So we are facing issue when dealing with non hotplug capable ports.
This is confusing.
Why would hotplug driver be involved unless port supports hotplug and the link goes down? You said that DLLSC is only supported on hotplug capable ports.
Need a better description of symptoms and what triggers hotplug driver to activate.
Can you expand this a little bit?
Quoted 8 lines · Sinan Kaya
Quoted 6 lines · Sathyanarayanan
Quoted 3 lines
I see. Can I assume that your system supports DPC? DPC is supposed to recover the link via
dpc_reset_link().Yes. But the affected device/drivers cleanup during error recovery is handled by hotplug handler. So we are facing issue when dealing with non hotplug capable ports.
Quoted 5 lines
This is confusing.
Why would hotplug driver be involved unless port supports hotplug and the link goes down? You said that DLLSC is only supported on hotplug capable ports.
hotplug driver is *only* involved when dealing with recovery of hotplug capable ports. For hotplug capable ports, when DPC is triggered and link goes down, DLLSC handler in pciehp driver will remove the affected devices/drivers. Once the link comes back it will re-attach them.
Quoted 3 lines
Need a better description of symptoms and what triggers hotplug driver to activate.
For problem description, please check the following details
Current pcie_do_recovery() implementation has following two issues:
1. Fatal (DPC) error recovery is currently broken for non-hotplug capable devices. Current fatal error recovery implementation relies on PCIe hotplug (pciehp) handler for detaching and re-enumerating the affected devices/drivers. pciehp handler listens for DLLSC state changes and handles device/driver detachment on DLLSC_LINK_DOWN event and re-enumeration on DLLSC_LINK_UP event. So when dealing with non-hotplug capable devices, recovery code does not restore the state of the affected devices correctly. Correct implementation should restore the device state and call report_slot_reset() function after resetting the link to restore the state of the device/driver.
You can find fatal non-hotplug related issues reported in following links:
https://lore.kernel.org/linux-pci/[email protected]/
https://lore.kernel.org/linux-pci/12115.1588207324@famine/
https://lore.kernel.org/linux-pci/0e6f89cd6b9e4a72293cc90fafe93487d7c2d295.1585000084.git.sathyanarayanan.kuppuswamy@linux.intel.com/
2. For non-fatal errors if report_error_detected() or report_mmio_enabled() functions requests PCI_ERS_RESULT_NEED_RESET then current pcie_do_recovery() implementation does not do the requested explicit device reset, instead just calls the report_slot_reset() on all affected devices. Notifying about the reset via report_slot_reset() without doing the actual device reset is incorrect.
To fix above issues, use PCI_ERS_RESULT_NEED_RESET as error state after successful reset_link() operation. This will ensure ->slot_reset() be called after reset_link() operation for fatal errors. Also call pci_bus_reset() to do slot/bus reset() before triggering device specific ->slot_reset() callback. Also, using pci_bus_reset() will restore the state of the devices after performing the reset operation.
Even though using pci_bus_reset() will do redundant reset operation after ->reset_link() for fatal errors, it should should affect the functional behavior.
Quoted 3 lines
Can you expand this a little bit?
Signature · 2 lines
Sathyanarayanan Kuppuswamy Linux Kernel Developer
Quoted 4 lines · Sathyanarayanan
For problem description, please check the following details
Current
pcie_do_recovery()implementation has following two issues:
Quoted 11 lines
1. Fatal (DPC) error recovery is currently broken for non-hotplug capable devices. Current fatal error recovery implementation relies on PCIe hotplug (pciehp) handler for detaching and re-enumerating the affected devices/drivers. pciehp handler listens for DLLSC state changes and handles device/driver detachment on
DLLSC_LINK_DOWNevent and re-enumeration onDLLSC_LINK_UPevent. So when dealing with non-hotplug capable devices, recovery code does not restore the state of the affected devices correctly. Correct implementation should restore the device state and callreport_slot_reset()function after resetting the link to restore the state of the device/driver.
So, this is a matter of moving the save/restore logic from the hotplug driver into common code so that DPC slot reset takes advantage of it?
If that's direction we are going, this is fine change IMO.
Quoted 8 lines
You can find fatal non-hotplug related issues reported in following links:
https://lore.kernel.org/linux-pci/[email protected]/
https://lore.kernel.org/linux-pci/12115.1588207324@famine/
https://lore.kernel.org/linux-pci/0e6f89cd6b9e4a72293cc90fafe93487d7c2d295.1585000084.git.sathyanarayanan.kuppuswamy@linux.intel.com/
Quoted 7 lines
2. For non-fatal errors if
report_error_detected()orreport_mmio_enabled()functions requestsPCI_ERS_RESULT_NEED_RESETthen currentpcie_do_recovery()implementation does not do the requested explicit device reset, instead just calls thereport_slot_reset()on all affected devices. Notifying about the reset viareport_slot_reset()without doing the actual device reset is incorrect.
This makes sens too. There seems to be an ordering issue per your description.
Quoted 3 lines
To fix above issues, use
PCI_ERS_RESULT_NEED_RESETas error state after successfulreset_link()operation. This will ensure ->slot_reset() be called afterreset_link()operation for fatal errors.
You lost me here. Why do we want to do secondary bus reset on top of DPC reset?
Quoted 8 lines
Also call
pci_bus_reset()to do slot/busreset()before triggering device specific ->slot_reset() callback. Also, usingpci_bus_reset()will restore the state of the devices after performing the reset operation.Even though using
pci_bus_reset()will do redundant reset operation after ->reset_link() for fatal errors, it should should affect the functional behavior.
Quoted 17 lines · Sinan Kaya
Quoted 4 lines · Sathyanarayanan
For problem description, please check the following details
Current
pcie_do_recovery()implementation has following two issues:Quoted 11 lines
1. Fatal (DPC) error recovery is currently broken for non-hotplug capable devices. Current fatal error recovery implementation relies on PCIe hotplug (pciehp) handler for detaching and re-enumerating the affected devices/drivers. pciehp handler listens for DLLSC state changes and handles device/driver detachment on
DLLSC_LINK_DOWNevent and re-enumeration onDLLSC_LINK_UPevent. So when dealing with non-hotplug capable devices, recovery code does not restore the state of the affected devices correctly. Correct implementation should restore the device state and callreport_slot_reset()function after resetting the link to restore the state of the device/driver.
Quoted 2 lines
So, this is a matter of moving the save/restore logic from the hotplug driver into common code so that DPC slot reset takes advantage of it?
We are not moving it out of hotplug path. But fixing it in this code path. With this fix, we will not depend on hotplug driver to restore the state.
Quoted 19 lines
If that's direction we are going, this is fine change IMO.
Quoted 8 lines
You can find fatal non-hotplug related issues reported in following links:
https://lore.kernel.org/linux-pci/[email protected]/
https://lore.kernel.org/linux-pci/12115.1588207324@famine/
https://lore.kernel.org/linux-pci/0e6f89cd6b9e4a72293cc90fafe93487d7c2d295.1585000084.git.sathyanarayanan.kuppuswamy@linux.intel.com/Quoted 7 lines
2. For non-fatal errors if
report_error_detected()orreport_mmio_enabled()functions requestsPCI_ERS_RESULT_NEED_RESETthen currentpcie_do_recovery()implementation does not do the requested explicit device reset, instead just calls thereport_slot_reset()on all affected devices. Notifying about the reset viareport_slot_reset()without doing the actual device reset is incorrect.
Quoted 2 lines
This makes sens too. There seems to be an ordering issue per your description.
Yes
Quoted 7 lines
Quoted 4 lines
To fix above issues, use
PCI_ERS_RESULT_NEED_RESETas error state after successfulreset_link()operation. This will ensure ->slot_reset() be called afterreset_link()operation for fatal errors.You lost me here. Why do we want to do secondary bus reset on top of DPC reset?
For non-hotplug capable slots, when reset (PCI_ERS_RESULT_NEED_RESET) is requested, we want to reset it before calling ->slot_reset() callback.
Quoted 10 lines
Quoted 9 lines
Also call
pci_bus_reset()to do slot/busreset()before triggering device specific ->slot_reset() callback. Also, usingpci_bus_reset()will restore the state of the devices after performing the reset operation.Even though using
pci_bus_reset()will do redundant reset operation after ->reset_link() for fatal errors, it should should affect the functional behavior.
Signature · 2 lines
Sathyanarayanan Kuppuswamy Linux Kernel Developer
Quoted 10 lines · Sathyanarayanan
Quoted 5 lines · Sinan Kaya
On 9/24/2020 12:06 AM, Kuppuswamy, Sathyanarayanan wrote:
So, this is a matter of moving the save/restore logic from the hotplug driver into common code so that DPC slot reset takes advantage of it?
We are not moving it out of hotplug path. But fixing it in this code path. With this fix, we will not depend on hotplug driver to restore the state.
Any possibility of unification?
[snip]
Quoted 9 lines
Quoted 7 lines
Quoted 4 lines
To fix above issues, use
PCI_ERS_RESULT_NEED_RESETas error state after successfulreset_link()operation. This will ensure ->slot_reset() be called afterreset_link()operation for fatal errors.You lost me here. Why do we want to do secondary bus reset on top of DPC reset?
For non-hotplug capable slots, when reset (
PCI_ERS_RESULT_NEED_RESET) is requested, we want to reset it before calling ->slot_reset() callback.
Why? Isn't DPC slot reset enough?
What will bus reset do that DPC slot reset won't do?
I can understand calling bus reset if DPC is not supported. I don't understand the requirement to do double reset.
Quoted 7 lines · Sinan Kaya
Quoted 6 lines · Sathyanarayanan
Quoted 3 lines · Sinan Kaya
On 9/24/2020 12:06 AM, Kuppuswamy, Sathyanarayanan wrote:
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
So, this is a matter of moving the save/restore logic from the hotplug driver into common code so that DPC slot reset takes advantage of it?
We are not moving it out of hotplug path. But fixing it in this code path. With this fix, we will not depend on hotplug driver to restore the state.
Any possibility of unification?
If we do that, it might need rework of hotplug driver. It will be a big change. IMO, its better not to touch that bee hive.
Quoted 8 lines
[snip]
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
To fix above issues, use
PCI_ERS_RESULT_NEED_RESETas error state after successfulreset_link()operation. This will ensure ->slot_reset() be called afterreset_link()operation for fatal errors.
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
You lost me here. Why do we want to do secondary bus reset on top of DPC reset?
For non-hotplug capable slots, when reset (
PCI_ERS_RESULT_NEED_RESET) is requested, we want to reset it before calling ->slot_reset() callback.Why? Isn't DPC slot reset enough?
It will do the reset at hardware level. But driver state is not cleaned up. So doing bus reset will restore both driver and hardware states.
Also for non-fatal errors, if reset is requested then we still need some kind of bus reset call here.
Quoted 5 lines
What will bus reset do that DPC slot reset won't do?
I can understand calling bus reset if DPC is not supported. I don't understand the requirement to do double reset.
Signature · 2 lines
Sathyanarayanan Kuppuswamy Linux Kernel Developer
Quoted 4 lines · Sathyanarayanan
Quoted 1 line
Why? Isn't DPC slot reset enough?
It will do the reset at hardware level. But driver state is not cleaned up. So doing bus reset will restore both driver and hardware states.
I really don't like this. If hotplug driver is restoring the state and DPC driver is not; let's fix the DPC driver rather than causing two resets and hope for the best.
One approach is to share the restore code between hotplug driver and DPC driver.
If this is a too involved change, DPC driver should restore state when hotplug is not supported.
DPC driver should be self-sufficient by itself.
Quoted 2 lines
Also for non-fatal errors, if reset is requested then we still need some kind of bus reset call here.
DPC should handle both fatal and non-fatal cases and cause a bus reset in hardware already before triggering an interrupt.
I disagree that you need an additional reset on top of DPC reset. Isn't one reset enough?
What will the second reset provide that first reset won't provide?
I see that you are trying to do the second reset only because second reset restores state.
That looks like a short-term fix only to explode on the next iteration.
Quoted 10 lines · Sinan Kaya
Quoted 4 lines · Sathyanarayanan
Quoted 1 line
Why? Isn't DPC slot reset enough?
It will do the reset at hardware level. But driver state is not cleaned up. So doing bus reset will restore both driver and hardware states.
I really don't like this. If hotplug driver is restoring the state and DPC driver is not; let's fix the DPC driver rather than causing two resets and hope for the best.
Quoted 5 lines
One approach is to share the restore code between hotplug driver and DPC driver.
If this is a too involved change, DPC driver should restore state when hotplug is not supported.
Yes. we can add a condition for hotplug capability check.
Quoted 7 lines
DPC driver should be self-sufficient by itself.
Quoted 2 lines
Also for non-fatal errors, if reset is requested then we still need some kind of bus reset call here
DPC should handle both fatal and non-fatal cases
Currently DPC is only triggered for FATAL errors.
and cause a bus reset
Quoted 1 line
in hardware already before triggering an interrupt.
Error recovery is not triggered only DPC driver. AER also uses the same error recovery code. If DPC is not supported, then we still need reset logic.
Quoted 11 lines
I disagree that you need an additional reset on top of DPC reset. Isn't one reset enough?
What will the second reset provide that first reset won't provide?
I see that you are trying to do the second reset only because second reset restores state.
That looks like a short-term fix only to explode on the next iteration.
Signature · 2 lines
Sathyanarayanan Kuppuswamy Linux Kernel Developer
Quoted 7 lines · Sathyanarayanan
Quoted 3 lines
If this is a too involved change, DPC driver should restore state when hotplug is not supported.
Yes. we can add a condition for hotplug capability check.
Quoted 3 lines
DPC driver should be self-sufficient by itself.
Sounds good.
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
Also for non-fatal errors, if reset is requested then we still need some kind of bus reset call here
DPC should handle both fatal and non-fatal cases
Currently DPC is only triggered for FATAL errors.
and cause a bus reset
Thanks for the heads up.
This seems to have changed since I looked at the DPC code.
Quoted 4 lines
Quoted 1 line
in hardware already before triggering an interrupt.
Error recovery is not triggered only DPC driver. AER also uses the same error recovery code. If DPC is not supported, then we still need reset logic.
It sounds like we are cross-talking two issues.
1. no state restore on DPC after FATAL error.
Let's fix this.
2. no bus reset on NON_FATAL error through AER driver path. This already tells me that you need to split your change into multiple patches.
Let's talk about this too. bus reset should be triggered via AER driver before informing the recovery.
if (status == PCI_ERS_RESULT_NEED_RESET) {
/*
* TODO: Should call platform-specific
* functions to reset slot before calling
* drivers' slot_reset callbacks?
*/
status = PCI_ERS_RESULT_RECOVERED;
pci_dbg(dev, "broadcast slot_reset message\n");
pci_walk_bus(bus, report_slot_reset, &status);
}Hi,
Quoted 22 lines · Sinan Kaya
Quoted 7 lines · Sathyanarayanan
Quoted 3 lines
If this is a too involved change, DPC driver should restore state when hotplug is not supported.
Yes. we can add a condition for hotplug capability check.
Quoted 3 lines
DPC driver should be self-sufficient by itself.
Sounds good.
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
Also for non-fatal errors, if reset is requested then we still need some kind of bus reset call here
DPC should handle both fatal and non-fatal cases
Currently DPC is only triggered for FATAL errors.
and cause a bus resetThanks for the heads up.
This seems to have changed since I looked at the DPC code.Quoted 1 line
Quoted 1 line
in hardware already before triggering an interrupt.
Quoted 8 lines
Quoted 3 lines
Error recovery is not triggered only DPC driver. AER also uses the same error recovery code. If DPC is not supported, then we still need reset logic.
It sounds like we are cross-talking two issues.
1. no state restore on DPC after FATAL error.
Let's fix this.
Agree. Few more detail about the above issue is,
There are two cases under FATAL error.
FATAL + hotplug - In this case, link will be reseted. And hotplug handler will remove the driver state. This case works well with current code.
FATAL + no-hotplug - In this case, link will still be reseted. But currently driver state is not properly restored. So I attempted to restore it using pci_reset_bus().
status = reset_link(dev);
- if (status != PCI_ERS_RESULT_RECOVERED) {
+ if (status == PCI_ERS_RESULT_RECOVERED) {
+ status = PCI_ERS_RESULT_NEED_RESET;
...
if (status == PCI_ERS_RESULT_NEED_RESET) {
/*
- * TODO: Should call platform-specific
- * functions to reset slot before calling
- * drivers' slot_reset callbacks?
+ * TODO: Optimize the call to pci_reset_bus()
+ *
+ * There are two components to pci_reset_bus().
+ *
+ * 1. Do platform specific slot/bus reset.
+ * 2. Save/Restore all devices in the bus.
+ *
+ * For hotplug capable devices and fatal errors,
+ * device is already in reset state due to link
+ * reset. So repeating platform specific slot/bus
+ * reset via pci_reset_bus() call is redundant. So
+ * can optimize this logic and conditionally call
+ * pci_reset_bus().
*/
+ pci_reset_bus(dev);
Quoted 7 lines
2. no bus reset on
NON_FATALerror through AER driver path. This already tells me that you need to split your change into multiple patches.Let's talk about this too. bus reset should be triggered via AER driver before informing the recovery.
But as per error recovery documentation, any call to ->error_detected() or ->mmio_enabled() can request PCI_ERS_RESULT_NEED_RESET. So we need to add code to do the actual reset before calling ->slot_reset() callback. So call to pci_reset_bus() fixes this issue.
if (status == PCI_ERS_RESULT_NEED_RESET) {
+ pci_reset_bus(dev);
Quoted 12 lines
if (status == PCI_ERS_RESULT_NEED_RESET) {
/*
* TODO: Should call platform-specific
* functions to reset slot before calling
* drivers' slot_reset callbacks?
*/
status = PCI_ERS_RESULT_RECOVERED;
pci_dbg(dev, "broadcast slot_reset message\n");
pci_walk_bus(bus, report_slot_reset, &status);
}Signature · 2 lines
Sathyanarayanan Kuppuswamy Linux Kernel Developer
Sathyanarayanan,
Quoted 36 lines · Sathyanarayanan
Hi,
Quoted 22 lines · Sinan Kaya
Quoted 7 lines · Sathyanarayanan
Quoted 3 lines
If this is a too involved change, DPC driver should restore state when hotplug is not supported.
Yes. we can add a condition for hotplug capability check.
Quoted 3 lines
DPC driver should be self-sufficient by itself.
Sounds good.
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
Also for non-fatal errors, if reset is requested then we still need some kind of bus reset call here
DPC should handle both fatal and non-fatal cases
Currently DPC is only triggered for FATAL errors.
and cause a bus resetThanks for the heads up.
This seems to have changed since I looked at the DPC code.Quoted 1 line
Quoted 1 line
in hardware already before triggering an interrupt.
Quoted 8 lines
Quoted 3 lines
Error recovery is not triggered only DPC driver. AER also uses the same error recovery code. If DPC is not supported, then we still need reset logic.
It sounds like we are cross-talking two issues.
1. no state restore on DPC after FATAL error.
Let's fix this.Agree. Few more detail about the above issue is,
Quoted 8 lines
There are two cases under FATAL error.
FATAL + hotplug - In this case, link will be reseted. And hotplug handler will remove the driver state. This case works well with current code.
FATAL + no-hotplug - In this case, link will still be reseted. But currently driver state is not properly restored. So I attempted to restore it using
pci_reset_bus().
Seems you should fix something at device driver side, not do double-reset in DPC driver, one reset is done by hardware, and you want to do another by DPC driver ?
Why hardware initiated reset is not enough for you ?
Thanks,
Ethan
Quoted 62 lines
status = reset_link(dev);(quoted diff) +16
-4- if (status != PCI_ERS_RESULT_RECOVERED) { + if (status == PCI_ERS_RESULT_RECOVERED) { + status = PCI_ERS_RESULT_NEED_RESET; if (status == PCI_ERS_RESULT_NEED_RESET) { /* - * TODO: Should call platform-specific - * functions to reset slot before calling - * drivers' slot_reset callbacks? + * TODO: Optimize the call to pci_reset_bus() + * + * There are two components to pci_reset_bus(). + * + * 1. Do platform specific slot/bus reset. + * 2. Save/Restore all devices in the bus. + * + * For hotplug capable devices and fatal errors, + * device is already in reset state due to link + * reset. So repeating platform specific slot/bus + * reset via pci_reset_bus() call is redundant. So + * can optimize this logic and conditionally call + * pci_reset_bus(). */ + pci_reset_bus(dev);Quoted 8 lines
2. no bus reset on
NON_FATALerror through AER driver path. This already tells me that you need to split your change into multiple patches.Let's talk about this too. bus reset should be triggered via AER driver before informing the recovery.
But as per error recovery documentation, any call to ->error_detected() or ->mmio_enabled() can request
PCI_ERS_RESULT_NEED_RESET. So we need to add code to do the actual reset before calling ->slot_reset() callback. So call topci_reset_bus()fixes this issue.if (status == PCI_ERS_RESULT_NEED_RESET) {+
pci_reset_bus(dev);Quoted 12 lines
if (status == PCI_ERS_RESULT_NEED_RESET) { /* * TODO: Should call platform-specific * functions to reset slot before calling * drivers' slot_reset callbacks? */ status = PCI_ERS_RESULT_RECOVERED; pci_dbg(dev, "broadcast slot_reset message\n"); pci_walk_bus(bus, report_slot_reset, &status); }--
Sathyanarayanan Kuppuswamy
Linux Kernel Developer
Quoted 23 lines · Sathyanarayanan
FATAL + no-hotplug - In this case, link will still be reseted. But currently driver state is not properly restored. So I attempted to restore it using
pci_reset_bus().status =
reset_link(dev);
- if (status !=PCI_ERS_RESULT_RECOVERED) { + if (status ==PCI_ERS_RESULT_RECOVERED) { + status =PCI_ERS_RESULT_NEED_RESET;...
if (status ==
PCI_ERS_RESULT_NEED_RESET) { /*
- * TODO: Should call platform-specific - * functions to reset slot before calling - * drivers'slot_resetcallbacks?
+ * TODO: Optimize the call topci_reset_bus()+ *
+ * There are two components topci_reset_bus(). + *
+ * 1. Do platform specific slot/bus reset. + * 2. Save/Restore all devices in the bus. + *
Quoted 8 lines
+ * For hotplug capable devices and fatal errors, + * device is already in reset state due to link + * reset. So repeating platform specific slot/bus + * reset via
pci_reset_bus()call is redundant. So + * can optimize this logic and conditionally call + *pci_reset_bus().
*/
+pci_reset_bus(dev);
I think we have to go to remove/rescan for this case as you also mentioned above. There is no state to save. All BAR assignments are gone. Entire device programming is also lost.
I don't think pci_reset_bus() can recover from this situation safely. It will make things worse by saving/restoring the hardware default state.
This should remove/rescan logic should be inside DPC's slot_reset() function BTW. Not here.
Quoted 2 lines · Sinan Kaya
This should remove/rescan logic should be inside DPC's
slot_reset()function BTW. Not here.
Correct function name is dpc_handler().
I hope I did not create confusion with slot_reset() that gets called for each driver post recovery.
Quoted 37 lines · Sinan Kaya
Quoted 23 lines · Sathyanarayanan
FATAL + no-hotplug - In this case, link will still be reseted. But currently driver state is not properly restored. So I attempted to restore it using
pci_reset_bus().status =
reset_link(dev);
- if (status !=PCI_ERS_RESULT_RECOVERED) { + if (status ==PCI_ERS_RESULT_RECOVERED) { + status =PCI_ERS_RESULT_NEED_RESET;...
if (status ==
PCI_ERS_RESULT_NEED_RESET) { /*
- * TODO: Should call platform-specific - * functions to reset slot before calling - * drivers'slot_resetcallbacks?
+ * TODO: Optimize the call topci_reset_bus()+ *
+ * There are two components topci_reset_bus(). + *
+ * 1. Do platform specific slot/bus reset. + * 2. Save/Restore all devices in the bus. + *Quoted 8 lines
+ * For hotplug capable devices and fatal errors, + * device is already in reset state due to link + * reset. So repeating platform specific slot/bus + * reset via
pci_reset_bus()call is redundant. So + * can optimize this logic and conditionally call + *pci_reset_bus().
*/
+pci_reset_bus(dev);I think we have to go to remove/rescan for this case as you also mentioned above. There is no state to save. All BAR assignments are gone. Entire device programming is also lost.
Quoted 6 lines
I don't think
pci_reset_bus()can recover from this situation safely. It will make things worse by saving/restoring the hardware default state.This should remove/rescan logic should be inside DPC's
slot_reset()function BTW. Not here.
Since there is no state restoration for FATAL errors, I am wondering whether calls to ->error_detected(), ->mmio_enabled() and ->slot_reset() are required?
Let me know your comments about following pseudo code.
if (fatal error & hotplug_supported)
do nothing // if fatal triggered by DPC, clear DPC state.
if (fatal error & no-hotplug)
perform slot_reset and renumerate affected devices.
Quoted 1 line
Signature · 2 lines
Sathyanarayanan Kuppuswamy Linux Kernel Developer
Quoted 4 lines · Sathyanarayanan
Since there is no state restoration for FATAL errors, I am wondering whether
calls to ->error_detected(), ->mmio_enabled() and ->slot_reset() are required?
Good question,
Initially when we started, we were trying to handle both NON_FATAL and FATAL errors in DPC.
We have seen value in unifying AER's callback mechanism with DPC. It looks like this no longer applies for DPC.
Some drivers want these indication to stop outgoing DMA/timers so that system can recover quickly.
There is value in calling them with existing AER based design.
I agree it doesn't apply here anymore if we are going to remove the device driver. Maybe, you should stop calling pcie_do_recovery() in DPC as well.
Quoted 8 lines
Let me know your comments about following pseudo code.
if (fatal error &
hotplug_supported)
do nothing // if fatal triggered by DPC, clear DPC state.if (fatal error & no-hotplug)
performslot_resetand renumerate affected devices.
LGTM,
I apologize for calling this slot_reset but slot_reset in err.c code is for post recovery callback to endpoint drivers. Let's not use this term here anymore to not confuse ourselves.
remove device + rescan similar to how hotplug remove + hotplug insertion notifications does eventually.
All of this to be done in DPC driver without any err.c involvement.
Bjorn,
What do you think? Is this a good direction?
Sinan
Quoted 4 lines · Sinan Kaya
Since there is no state restoration for FATAL errors, I am wondering whether
calls to ->error_detected(), ->mmio_enabled() and ->slot_reset() are required?
I also would like to ask someone closer to the spec language double check this.
When we recover the link at the end of the DPC handler, what is the expected state of the endpoint?
Is it a some kind of a reset like secondary bus reset? (I assumed this
one)
Undefined?
or just plain link recovery with everything else as intact as it used to be?
Quoted 6 lines · Sinan Kaya
Quoted 4 lines · Sinan Kaya
Since there is no state restoration for FATAL errors, I am wondering whether
calls to ->error_detected(), ->mmio_enabled() and ->slot_reset() are required?
Quoted 8 lines
I also would like to ask someone closer to the spec language double check this.
When we recover the link at the end of the DPC handler, what is the expected state of the endpoint?
Is it a some kind of a reset like secondary bus reset? (I assumed this
one)
I think it will be in reset state.
Quoted 6 lines
Undefined?
or just plain link recovery with everything else as intact as it used to be?
Signature · 2 lines
Sathyanarayanan Kuppuswamy Linux Kernel Developer
Hi Sinan,
Quoted 12 lines · Sathyanarayanan
Quoted 9 lines · Sinan Kaya
Quoted 4 lines · Sinan Kaya
Since there is no state restoration for FATAL errors, I am wondering whether
calls to ->error_detected(), ->mmio_enabled() and ->slot_reset() are required?I also would like to ask someone closer to the spec language double check this.
Quoted 13 lines
Quoted 5 lines
When we recover the link at the end of the DPC handler, what is the expected state of the endpoint?
Is it a some kind of a reset like secondary bus reset? (I assumed this one)
I think it will be in reset state.
Quoted 6 lines
Undefined?
or just plain link recovery with everything else as intact as it used to be?
Please check the following version. It should fix most of the reset issues properly.
Signature · 2 lines
Sathyanarayanan Kuppuswamy Linux Kernel Developer
Quoted 29 lines · Sathyanarayanan
Hi Sinan,
Quoted 12 lines · Sathyanarayanan
Quoted 9 lines · Sinan Kaya
Quoted 4 lines · Sinan Kaya
Since there is no state restoration for FATAL errors, I am wondering whether
calls to ->error_detected(), ->mmio_enabled() and ->slot_reset() are required?I also would like to ask someone closer to the spec language double check this.
Quoted 13 lines
Quoted 5 lines
When we recover the link at the end of the DPC handler, what is the expected state of the endpoint?
Is it a some kind of a reset like secondary bus reset? (I assumed this one)
I think it will be in reset state.
Quoted 6 lines
Undefined?
or just plain link recovery with everything else as intact as it used to be?
Quoted 6 lines
Please check the following version. It should fix most of the reset issues properly.
Thanks, good stuff.
Quoted 13 lines · Sathyanarayanan
Quoted 6 lines
2. no bus reset on
NON_FATALerror through AER driver path. This already tells me that you need to split your change into multiple patches.Let's talk about this too. bus reset should be triggered via AER driver before informing the recovery.
But as per error recovery documentation, any call to ->error_detected() or ->mmio_enabled() can request
PCI_ERS_RESULT_NEED_RESET. So we need to add code to do the actual reset before calling ->slot_reset() callback. So call topci_reset_bus()fixes this issue.
Quoted 2 lines
if (status ==
PCI_ERS_RESULT_NEED_RESET) {
+pci_reset_bus(dev);
This part seems to make sense as you already highlighted there is a TO-DO in the code.
This is an independent change that deserves its own patch.
Quoted 6 lines · Sathyanarayanan
Quoted 5 lines
One approach is to share the restore code between hotplug driver and DPC driver.
If this is a too involved change, DPC driver should restore state when hotplug is not supported.
Yes. we can add a condition for hotplug capability check.
Now that I think about this more...
This won't work. Link is brought down automatically by the DPC hardware. Therefore, all software state is lost. Restore won't help here.
The only solution I can see is to force driver disconnect and rescan.