PCI/ERR: Resolve regression in pcie_do_recovery
Subsystems (2)
PATCH 1/1 PCI/ERR: Resolve regression in pcie_do_recovery
Commit 6d2c89441571 ("PCI/ERR: Update error status afterreset_link()"), introduced a regression, as pcie_do_recovery will discard the status result from report_frozen_detected. This can cause a failure to recover if _NEED_RESET is returned by report_frozen_detected and report_slot_reset is not invoked.
Such an event can be induced for testing purposes by reducing
the Max_Payload_Size of a PCIe bridge to less than that of a device downstream from the bridge, and then initating I/O through the device, resulting in oversize transactions. In the presence of DPC, this results in a containment event and attempted reset and recovery via pcie_do_recovery. After 6d2c89441571 report_slot_reset is not invoked, and the device does not recover.
Inspection shows a similar path is plausible for a return of_CAN_RECOVER and the invocation of report_mmio_enabled.
Resolve this by preserving the result of report_frozen_detected ifreset_link does not return _DISCONNECT.
--- drivers/pci/pcie/err.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)
drivers/pci/pcie/err.c +9 -2
@@ -164,10 +164,17 @@ pci_ers_result_t pcie_do_recovery(struct pci_dev *dev,pci_dbg (dev,"broadcast error_detected message\n" );if (state == pci_channel_io_frozen) { +pci_ers_result_t status2; +pci_walk_bus (bus, report_frozen_detected, &status); - status =reset_link (dev); -if (status !=PCI_ERS_RESULT_RECOVERED ) { +/* preserve status from report_frozen_detected to +* insure report_mmio_enabled or report_slot_reset are +* invoked even if reset_link returns _RECOVERED. +*/ + status2 =reset_link (dev); +if (status2 !=PCI_ERS_RESULT_RECOVERED ) {pci_warn (dev,"link reset failed\n" ); + status = status2; goto failed; } }else {
Signature · 1 line
2.7.4
Quoted 6 lines · Jay Vosburgh
Commit
6d2c89441571("PCI/ERR: Update error status afterreset_link()"), introduced a regression, aspcie_do_recoverywill discard the status result fromreport_frozen_detected. This can cause a failure to recover if_NEED_RESETis returned byreport_frozen_detectedandreport_slot_resetis not invoked.
Quoted 7 lines
Such an event can be induced for testing purposes by reducing
theMax_Payload_Sizeof a PCIe bridge to less than that of a device downstream from the bridge, and then initating I/O through the device, resulting in oversize transactions. In the presence of DPC, this results in a containment event and attempted reset and recovery viapcie_do_recovery. After6d2c89441571report_slot_resetis not invoked, and the device does not recover.
I think this issue is related to the issue discussed in following thread (DPC non-hotplug support).
https://lkml.org/lkml/2020/3/28/328
If my assumption is correct, you are dealing with devices which are not hotplug capable. If the devices are hotplug capable then you don't need to proceed to report_slot_reset(), since hotplug handler will remove/re-enumerate the devices correctly.
Quoted 39 lines
Inspection shows a similar path is plausible for a return of
_CAN_RECOVERand the invocation ofreport_mmio_enabled.Resolve this by preserving the result of
report_frozen_detectedifreset_linkdoes not return_DISCONNECT.Fixes 6d2c89441571 ("PCI/ERR: Update error status after reset_link()")Signed-off-by Jay Vosburgh <[email protected]>--- drivers/pci/pcie/err.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)drivers/pci/pcie/err.c +9
-2@@ -164,10 +164,17 @@ pci_ers_result_t pcie_do_recovery(struct pci_dev *dev,pci_dbg (dev,"broadcast error_detected message\n" );if (state == pci_channel_io_frozen) { +pci_ers_result_t status2; +pci_walk_bus (bus, report_frozen_detected, &status); - status =reset_link (dev); -if (status !=PCI_ERS_RESULT_RECOVERED ) { +/* preserve status from report_frozen_detected to +* insure report_mmio_enabled or report_slot_reset are +* invoked even if reset_link returns _RECOVERED. +*/ + status2 =reset_link (dev); +if (status2 !=PCI_ERS_RESULT_RECOVERED ) {pci_warn (dev,"link reset failed\n" ); + status = status2; goto failed; } }else {
Hi Jay,
Quoted 20 lines · Sathyanarayanan
Quoted 6 lines · Jay Vosburgh
Commit
6d2c89441571("PCI/ERR: Update error status afterreset_link()"), introduced a regression, aspcie_do_recoverywill discard the status result from report_frozen_detected. This can cause a failure to recover if_NEED_RESETis returned byreport_frozen_detectedandreport_slot_resetis not invoked.Quoted 7 lines
Such an event can be induced for testing purposes by reducing the
Max_Payload_Sizeof a PCIe bridge to less than that of a device downstream from the bridge, and then initating I/O through the device, resulting in oversize transactions. In the presence of DPC, this results in a containment event and attempted reset and recovery via pcie_do_recovery. After6d2c89441571report_slot_resetis not invoked, and the device does not recover.I think this issue is related to the issue discussed in following thread (DPC non-hotplug support).
Quoted 6 lines
https://lkml.org/lkml/2020/3/28/328
If my assumption is correct, you are dealing with devices which are not hotplug capable. If the devices are hotplug capable then you don't need to proceed to
report_slot_reset(), since hotplug handler will remove/re-enumerate the devices correctly.
Can you check whether following fix works for you?
This includes support for bus_reset in recovery function itself.
index 14bb8f54723e..c9eaab68ab7a 100644
drivers/pci/pcie/err.c +14 -4
@@ -165,13 +165,23 @@ pci_ers_result_t pcie_do_recovery(struct pci_dev *dev,pci_dbg (dev,"broadcast error_detected message\n" );if (state == pci_channel_io_frozen) {if (state == pci_channel_io_frozen) {pci_walk_bus (bus, report_frozen_detected, &status); - status =reset_link (dev); -if (status !=PCI_ERS_RESULT_RECOVERED ) { + status =PCI_ERS_RESULT_NEED_RESET ; + }else { +pci_walk_bus (bus, report_normal_detected, &status); + } + +if (status ==PCI_ERS_RESULT_NEED_RESET ) { +if (reset_link) +if (reset_link (dev) !=PCI_ERS_RESULT_RECOVERED ) + status =PCI_ERS_RESULT_DISCONNECT ; +else +if (pci_bus_error_reset (dev)) + status =PCI_ERS_RESULT_DISCONNECT ; + +if (status ==PCI_ERS_RESULT_DISCONNECT ) {pci_warn (dev,"link reset failed\n" ); goto failed; } - }else { -pci_walk_bus (bus, report_normal_detected, &status); }
if (status == PCI_ERS_RESULT_CAN_RECOVER) {Quoted 40 lines
Quoted 40 lines
Inspection shows a similar path is plausible for a return of
_CAN_RECOVERand the invocation ofreport_mmio_enabled.Resolve this by preserving the result of
report_frozen_detectedifreset_linkdoes not return_DISCONNECT.Fixes 6d2c89441571 ("PCI/ERR: Update error status after reset_link()")Signed-off-by Jay Vosburgh <[email protected]>---
drivers/pci/pcie/err.c| 11 +++++++++--1 file changed, 9 insertions(+), 2 deletions(-)drivers/pci/pcie/err.c +9
-2@@ -164,10 +164,17 @@ pci_ers_result_t pcie_do_recovery(struct pci_dev *dev, pci_dbg(dev, "broadcast error_detected message\n"); if (state == pci_channel_io_frozen) { + pci_ers_result_t status2; + pci_walk_bus(bus, report_frozen_detected, &status); - status = reset_link(dev); - if (status != PCI_ERS_RESULT_RECOVERED) { + /* preserve status from report_frozen_detected to + * insure report_mmio_enabled or report_slot_reset are + * invoked even if reset_link returns _RECOVERED. + */ + status2 = reset_link(dev); + if (status2 != PCI_ERS_RESULT_RECOVERED) { pci_warn(dev, "link reset failed\n"); + status = status2; goto failed; } } else {
Quoted 29 lines · Sathyanarayanan
Hi Jay,
Quoted 26 lines · Sathyanarayanan
Quoted 6 lines · Jay Vosburgh
Commit
6d2c89441571("PCI/ERR: Update error status afterreset_link()"), introduced a regression, aspcie_do_recoverywill discard the status result from report_frozen_detected. This can cause a failure to recover if_NEED_RESETis returned byreport_frozen_detectedandreport_slot_resetis not invoked.Quoted 7 lines
Such an event can be induced for testing purposes by reducing the
Max_Payload_Sizeof a PCIe bridge to less than that of a device downstream from the bridge, and then initating I/O through the device, resulting in oversize transactions. In the presence of DPC, this results in a containment event and attempted reset and recovery via pcie_do_recovery. After6d2c89441571report_slot_resetis not invoked, and the device does not recover.I think this issue is related to the issue discussed in following thread (DPC non-hotplug support).
https://lkml.org/lkml/2020/3/28/328
If my assumption is correct, you are dealing with devices which are not hotplug capable. If the devices are hotplug capable then you don't need to proceed to
report_slot_reset(), since hotplug handler will remove/re-enumerate the devices correctly.
Correct, this particular device (a network card) is in a
non-hotplug slot.
Quoted 1 line
Can you check whether following fix works for you?
Yes, it does.
I fixed up the whitespace and made a minor change to add braces
in what look like the correct places around the "if (reset_link)" block; the patch I tested with is below. I'll also install this on another machine with hotplug capable slots to test there as well.
drivers/pci/pcie/err.c +15 -4
@@ -165,13 +165,24 @@ pci_ers_result_t pcie_do_recovery(struct pci_dev *dev,pci_dbg (dev,"broadcast error_detected message\n" );if (state == pci_channel_io_frozen) {pci_walk_bus (bus, report_frozen_detected, &status); - status =reset_link (dev); -if (status !=PCI_ERS_RESULT_RECOVERED ) { + status =PCI_ERS_RESULT_NEED_RESET ; + }else { +pci_walk_bus (bus, report_normal_detected, &status); + } + +if (status ==PCI_ERS_RESULT_NEED_RESET ) { +if (reset_link) { +if (reset_link (dev) !=PCI_ERS_RESULT_RECOVERED ) + status =PCI_ERS_RESULT_DISCONNECT ; + }else { +if (pci_bus_error_reset (dev)) + status =PCI_ERS_RESULT_DISCONNECT ; + } + +if (status ==PCI_ERS_RESULT_DISCONNECT ) {pci_warn (dev,"link reset failed\n" ); goto failed; } - }else { -pci_walk_bus (bus, report_normal_detected, &status); }if (status ==PCI_ERS_RESULT_CAN_RECOVER ) {
-J
Quoted 68 lines
This includes support for
bus_resetin recovery function itself.index
14bb8f54723e..c9eaab68ab7a100644drivers/pci/pcie/err.c +14
-4@@ -165,13 +165,23 @@ pci_ers_result_t pcie_do_recovery(struct pci_dev *dev,pci_dbg (dev,"broadcast error_detected message\n" );if (state == pci_channel_io_frozen) {if (state == pci_channel_io_frozen) {pci_walk_bus (bus, report_frozen_detected, &status); - status =reset_link (dev); -if (status !=PCI_ERS_RESULT_RECOVERED ) { + status =PCI_ERS_RESULT_NEED_RESET ; + }else { +pci_walk_bus (bus, report_normal_detected, &status); + } + +if (status ==PCI_ERS_RESULT_NEED_RESET ) { +if (reset_link) +if (reset_link (dev) !=PCI_ERS_RESULT_RECOVERED ) + status =PCI_ERS_RESULT_DISCONNECT ; +else +if (pci_bus_error_reset (dev)) + status =PCI_ERS_RESULT_DISCONNECT ; + +if (status ==PCI_ERS_RESULT_DISCONNECT ) {pci_warn (dev,"link reset failed\n" ); goto failed; } - }else { -pci_walk_bus (bus, report_normal_detected, &status); }if (status == PCI_ERS_RESULT_CAN_RECOVER) {Quoted 32 lines
Quoted 32 lines
Inspection shows a similar path is plausible for a return of
_CAN_RECOVERand the invocation ofreport_mmio_enabled.Resolve this by preserving the result of
report_frozen_detectedifreset_linkdoes not return_DISCONNECT.Fixes 6d2c89441571 ("PCI/ERR: Update error status after reset_link()")Signed-off-by Jay Vosburgh <[email protected]>---
drivers/pci/pcie/err.c| 11 +++++++++--1 file changed, 9 insertions(+), 2 deletions(-)drivers/pci/pcie/err.c +6
-2@@ -164,10 +164,17 @@ pci_ers_result_t pcie_do_recovery(struct pci_dev *dev, pci_dbg(dev, "broadcast error_detected message\n"); if (state == pci_channel_io_frozen) { + pci_ers_result_t status2; + pci_walk_bus(bus, report_frozen_detected, &status); - status = reset_link(dev); - if (status != PCI_ERS_RESULT_RECOVERED) { + /* preserve status from report_frozen_detected to +* insure report_mmio_enabled or report_slot_reset are +* invoked even if reset_link returns _RECOVERED. +*/
Quoted 8 lines
Quoted 8 lines
drivers/pci/pcie/err.c pci_ers_result_t pcie_do_recovery(struct pci_dev · line 180
Quoted 8 lines
drivers/pci/pcie/err.c +3
-0+ status2 = reset_link(dev); + if (status2 != PCI_ERS_RESULT_RECOVERED) { pci_warn(dev, "link reset failed\n"); + status = status2; goto failed; } } else {
---
-Jay Vosburgh, [email protected]
Quoted 35 lines · Jay Vosburgh
Quoted 29 lines · Sathyanarayanan
Hi Jay,
Quoted 26 lines · Sathyanarayanan
Quoted 6 lines · Jay Vosburgh
Commit
6d2c89441571("PCI/ERR: Update error status afterreset_link()"), introduced a regression, aspcie_do_recoverywill discard the status result from report_frozen_detected. This can cause a failure to recover if_NEED_RESETis returned byreport_frozen_detectedandreport_slot_resetis not invoked.Quoted 7 lines
Such an event can be induced for testing purposes by reducing the
Max_Payload_Sizeof a PCIe bridge to less than that of a device downstream from the bridge, and then initating I/O through the device, resulting in oversize transactions. In the presence of DPC, this results in a containment event and attempted reset and recovery via pcie_do_recovery. After6d2c89441571report_slot_resetis not invoked, and the device does not recover.I think this issue is related to the issue discussed in following thread (DPC non-hotplug support).
https://lkml.org/lkml/2020/3/28/328
If my assumption is correct, you are dealing with devices which are not hotplug capable. If the devices are hotplug capable then you don't need to proceed to
report_slot_reset(), since hotplug handler will remove/re-enumerate the devices correctly.Correct, this particular device (a network card) is in a
non-hotplug slot.
Quoted 8 lines
Quoted 1 line
Can you check whether following fix works for you?
Yes, it does.
I fixed up the whitespace and made a minor change to add braces
in what look like the correct places around the "if (reset_link)" block; the patch I tested with is below. I'll also install this on another machine with hotplug capable slots to test there as well.
We've tested the below patch on a couple of different machines
and devices (network card, NVMe device) and it appears to solve the recovery issue in our testing.
Is there anything further we need to do, or can this be
considered for inclusion upstream at this time?
-J
Quoted 116 lines
drivers/pci/pcie/err.c +15
-4@@ -165,13 +165,24 @@ pci_ers_result_t pcie_do_recovery(struct pci_dev *dev,pci_dbg (dev,"broadcast error_detected message\n" );if (state == pci_channel_io_frozen) {pci_walk_bus (bus, report_frozen_detected, &status); - status =reset_link (dev); -if (status !=PCI_ERS_RESULT_RECOVERED ) { + status =PCI_ERS_RESULT_NEED_RESET ; + }else { +pci_walk_bus (bus, report_normal_detected, &status); + } + +if (status ==PCI_ERS_RESULT_NEED_RESET ) { +if (reset_link) { +if (reset_link (dev) !=PCI_ERS_RESULT_RECOVERED ) + status =PCI_ERS_RESULT_DISCONNECT ; + }else { +if (pci_bus_error_reset (dev)) + status =PCI_ERS_RESULT_DISCONNECT ; + } + +if (status ==PCI_ERS_RESULT_DISCONNECT ) {pci_warn (dev,"link reset failed\n" ); goto failed; } - }else { -pci_walk_bus (bus, report_normal_detected, &status); }if (status ==PCI_ERS_RESULT_CAN_RECOVER ) {-JQuoted 68 lines
This includes support for
bus_resetin recovery function itself.index
14bb8f54723e..c9eaab68ab7a100644drivers/pci/pcie/err.c +14
-4@@ -165,13 +165,23 @@ pci_ers_result_t pcie_do_recovery(struct pci_dev *dev,pci_dbg (dev,"broadcast error_detected message\n" );if (state == pci_channel_io_frozen) {if (state == pci_channel_io_frozen) {pci_walk_bus (bus, report_frozen_detected, &status); - status =reset_link (dev); -if (status !=PCI_ERS_RESULT_RECOVERED ) { + status =PCI_ERS_RESULT_NEED_RESET ; + }else { +pci_walk_bus (bus, report_normal_detected, &status); + } + +if (status ==PCI_ERS_RESULT_NEED_RESET ) { +if (reset_link) +if (reset_link (dev) !=PCI_ERS_RESULT_RECOVERED ) + status =PCI_ERS_RESULT_DISCONNECT ; +else +if (pci_bus_error_reset (dev)) + status =PCI_ERS_RESULT_DISCONNECT ; + +if (status ==PCI_ERS_RESULT_DISCONNECT ) {pci_warn (dev,"link reset failed\n" ); goto failed; } - }else { -pci_walk_bus (bus, report_normal_detected, &status); }if (status == PCI_ERS_RESULT_CAN_RECOVER) {Quoted 32 lines
Quoted 32 lines
Inspection shows a similar path is plausible for a return of
_CAN_RECOVERand the invocation ofreport_mmio_enabled.Resolve this by preserving the result of
report_frozen_detectedifreset_linkdoes not return_DISCONNECT.Fixes 6d2c89441571 ("PCI/ERR: Update error status after reset_link()")Signed-off-by Jay Vosburgh <[email protected]>---
drivers/pci/pcie/err.c| 11 +++++++++--1 file changed, 9 insertions(+), 2 deletions(-)drivers/pci/pcie/err.c +6
-2@@ -164,10 +164,17 @@ pci_ers_result_t pcie_do_recovery(struct pci_dev *dev, pci_dbg(dev, "broadcast error_detected message\n"); if (state == pci_channel_io_frozen) { + pci_ers_result_t status2; + pci_walk_bus(bus, report_frozen_detected, &status); - status = reset_link(dev); - if (status != PCI_ERS_RESULT_RECOVERED) { + /* preserve status from report_frozen_detected to +* insure report_mmio_enabled or report_slot_reset are +* invoked even if reset_link returns _RECOVERED. +*/ Quoted 8 lines
Quoted 8 lines
drivers/pci/pcie/err.c pci_ers_result_t pcie_do_recovery(struct pci_dev · line 180
Quoted 8 lines
drivers/pci/pcie/err.c +3
-0+ status2 = reset_link(dev); + if (status2 != PCI_ERS_RESULT_RECOVERED) { pci_warn(dev, "link reset failed\n"); + status = status2; goto failed; } } else {---
-Jay Vosburgh, [email protected]
Hi Jay, Kuppuswamy
Quoted 54 lines · Jay Vosburgh
Quoted 27 lines · Sathyanarayanan
Hi Jay,
Quoted 24 lines · Sathyanarayanan
Quoted 6 lines · Jay Vosburgh
Commit
6d2c89441571("PCI/ERR: Update error status afterreset_link()"), introduced a regression, aspcie_do_recoverywill discard the status result fromreport_frozen_detected. This can cause a failure to recover if_NEED_RESETis returned byreport_frozen_detectedandreport_slot_resetis not invoked.Quoted 7 lines
Such an event can be induced for testing purposes by reducing
theMax_Payload_Sizeof a PCIe bridge to less than that of a device downstream from the bridge, and then initating I/O through the device, resulting in oversize transactions. In the presence of DPC, this results in a containment event and attempted reset and recovery viapcie_do_recovery. After6d2c89441571report_slot_resetis not invoked, and the device does not recover.I think this issue is related to the issue discussed in following thread (DPC non-hotplug support).
https://lkml.org/lkml/2020/3/28/328
If my assumption is correct, you are dealing with devices which are not hotplug capable. If the devices are hotplug capable then you don't need to proceed to
report_slot_reset(), since hotplug handler will remove/re-enumerate the devices correctly.Correct, this particular device (a network card) is in a
non-hotplug slot.Quoted 1 line
Can you check whether following fix works for you?
Yes, it does.
I fixed up the whitespace and made a minor change to add braces
in what look like the correct places around the "if (reset_link)" block; the patch I tested with is below. I'll also install this on another machine with hotplug capable slots to test there as well.drivers/pci/pcie/err.c +4
-2@@ -165,13 +165,24 @@ pci_ers_result_t pcie_do_recovery(struct pci_dev *dev,pci_dbg (dev,"broadcast error_detected message\n" );if (state == pci_channel_io_frozen) {pci_walk_bus (bus, report_frozen_detected, &status); - status =reset_link (dev); -if (status !=PCI_ERS_RESULT_RECOVERED ) { + status =PCI_ERS_RESULT_NEED_RESET ; + }else { +pci_walk_bus (bus, report_normal_detected, &status); + }
drivers/pci/pcie/err.c pci_ers_result_t pcie_do_recovery(struct pci_dev *dev, · line 181
Quoted 10 lines
drivers/pci/pcie/err.c +10
-0+ +if (status ==PCI_ERS_RESULT_NEED_RESET ) { +if (reset_link) { +if (reset_link (dev) !=PCI_ERS_RESULT_RECOVERED ) + status =PCI_ERS_RESULT_DISCONNECT ; + }else { +if (pci_bus_error_reset (dev)) + status =PCI_ERS_RESULT_DISCONNECT ; + } +
The PCI_ERS_RESULT_NEED_RESET may indicate that the driver requires a *slot* reset. With this patch, seems later slot reset broadcast may not be performed.
if (status == PCI_ERS_RESULT_NEED_RESET) {
status = PCI_ERS_RESULT_RECOVERED;
pci_dbg(dev, "broadcast slot_reset message\n");
pci_walk_bus(bus, report_slot_reset, &status);
}One minor question, currently the callers of pcie_do_recovery() will always pass a reset_link pointer, so is the condition necessary?
Yicong
Quoted 91 lines
drivers/pci/pcie/err.c +1
-2+if (status ==PCI_ERS_RESULT_DISCONNECT ) {pci_warn (dev,"link reset failed\n" ); goto failed; } - }else { -pci_walk_bus (bus, report_normal_detected, &status); }if (status ==PCI_ERS_RESULT_CAN_RECOVER ) {-JQuoted 66 lines
This includes support for
bus_resetin recovery function itself.index
14bb8f54723e..c9eaab68ab7a100644drivers/pci/pcie/err.c +14
-4@@ -165,13 +165,23 @@ pci_ers_result_t pcie_do_recovery(struct pci_dev *dev,pci_dbg (dev,"broadcast error_detected message\n" );if (state == pci_channel_io_frozen) {if (state == pci_channel_io_frozen) {pci_walk_bus (bus, report_frozen_detected, &status); - status =reset_link (dev); -if (status !=PCI_ERS_RESULT_RECOVERED ) { + status =PCI_ERS_RESULT_NEED_RESET ; + }else { +pci_walk_bus (bus, report_normal_detected, &status); + } + +if (status ==PCI_ERS_RESULT_NEED_RESET ) { +if (reset_link) +if (reset_link (dev) !=PCI_ERS_RESULT_RECOVERED ) + status =PCI_ERS_RESULT_DISCONNECT ; +else +if (pci_bus_error_reset (dev)) + status =PCI_ERS_RESULT_DISCONNECT ; + +if (status ==PCI_ERS_RESULT_DISCONNECT ) {pci_warn (dev,"link reset failed\n" ); goto failed; } - }else { -pci_walk_bus (bus, report_normal_detected, &status); }if (status == PCI_ERS_RESULT_CAN_RECOVER) {Quoted 30 lines
Quoted 30 lines
Inspection shows a similar path is plausible for a return of
_CAN_RECOVERand the invocation ofreport_mmio_enabled.Resolve this by preserving the result of
report_frozen_detectedifreset_linkdoes not return_DISCONNECT.Fixes 6d2c89441571 ("PCI/ERR: Update error status after reset_link()")Signed-off-by Jay Vosburgh <[email protected]>--- drivers/pci/pcie/err.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)drivers/pci/pcie/err.c +6
-2@@ -164,10 +164,17 @@ pci_ers_result_t pcie_do_recovery(struct pci_dev *dev,pci_dbg (dev,"broadcast error_detected message\n" );if (state == pci_channel_io_frozen) { +pci_ers_result_t status2; +pci_walk_bus (bus, report_frozen_detected, &status); - status =reset_link (dev); -if (status !=PCI_ERS_RESULT_RECOVERED ) { +/* preserve status from report_frozen_detected to +* insure report_mmio_enabled or report_slot_reset are +* invoked even if reset_link returns _RECOVERED. +*/ Quoted 8 lines
Quoted 8 lines
drivers/pci/pcie/err.c pci_ers_result_t pcie_do_recovery(struct pci_dev · line 180
Quoted 8 lines
drivers/pci/pcie/err.c +3
-0+ status2 =reset_link (dev); +if (status2 !=PCI_ERS_RESULT_RECOVERED ) {pci_warn (dev,"link reset failed\n" ); + status = status2; goto failed; } }else {---
-Jay Vosburgh, [email protected]
.
[ +cc dongdong as we met the issue ]
Hi,
The regression happened with our intel 82599 network adaptor. A DPC event happened on the device, recovery returns successful but the device is not enabled. As we only reset link, but don't call driver specific handler(mmio enable/slot reset).
If we met an error with IO blocked, the logic should be:
1. try to reset link
2. broadcast report_frozen_detected on the bus
3. do what drivers suggest(mmio enable/slot reset)
Currently only step 1 is performed, and returns directly. I tried the patch below and it solves the issues. The differences from Jay's one is that it merge the return value to the status and it match the logic I mentioned above.
Regards,
Yicong
drivers/pci/pcie/err.c +1 -1
@@ -164,12 +164,12 @@ pci_ers_result_t pcie_do_recovery(struct pci_dev *dev,pci_dbg (dev,"broadcast error_detected message\n" );if (state == pci_channel_io_frozen) { -pci_walk_bus (bus, report_frozen_detected, &status); status =reset_link (dev);if (status !=PCI_ERS_RESULT_RECOVERED ) {pci_warn (dev,"link reset failed\n" ); goto failed; } +pci_walk_bus (bus, report_frozen_detected, &status); }else {pci_walk_bus (bus, report_normal_detected, &status); }
Quoted 51 lines · Jay Vosburgh
Commit
6d2c89441571("PCI/ERR: Update error status afterreset_link()"), introduced a regression, aspcie_do_recoverywill discard the status result fromreport_frozen_detected. This can cause a failure to recover if_NEED_RESETis returned byreport_frozen_detectedandreport_slot_resetis not invoked.Such an event can be induced for testing purposes by reducing
theMax_Payload_Sizeof a PCIe bridge to less than that of a device downstream from the bridge, and then initating I/O through the device, resulting in oversize transactions. In the presence of DPC, this results in a containment event and attempted reset and recovery viapcie_do_recovery. After6d2c89441571report_slot_resetis not invoked, and the device does not recover.Inspection shows a similar path is plausible for a return of
_CAN_RECOVERand the invocation ofreport_mmio_enabled.Resolve this by preserving the result of
report_frozen_detectedifreset_linkdoes not return_DISCONNECT.Fixes 6d2c89441571 ("PCI/ERR: Update error status after reset_link()")Signed-off-by Jay Vosburgh <[email protected]>--- drivers/pci/pcie/err.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)drivers/pci/pcie/err.c +9
-2@@ -164,10 +164,17 @@ pci_ers_result_t pcie_do_recovery(struct pci_dev *dev,pci_dbg (dev,"broadcast error_detected message\n" );if (state == pci_channel_io_frozen) { +pci_ers_result_t status2; +pci_walk_bus (bus, report_frozen_detected, &status); - status =reset_link (dev); -if (status !=PCI_ERS_RESULT_RECOVERED ) { +/* preserve status from report_frozen_detected to +* insure report_mmio_enabled or report_slot_reset are +* invoked even if reset_link returns _RECOVERED. +*/ + status2 =reset_link (dev); +if (status2 !=PCI_ERS_RESULT_RECOVERED ) {pci_warn (dev,"link reset failed\n" ); + status = status2; goto failed; } }else {
Quoted 62 lines · Yicong Yang
Hi Jay, Kuppuswamy
Quoted 59 lines · Jay Vosburgh
Quoted 27 lines · Sathyanarayanan
Hi Jay,
Quoted 24 lines · Sathyanarayanan
Quoted 6 lines · Jay Vosburgh
Commit
6d2c89441571("PCI/ERR: Update error status afterreset_link()"), introduced a regression, aspcie_do_recoverywill discard the status result fromreport_frozen_detected. This can cause a failure to recover if_NEED_RESETis returned byreport_frozen_detectedandreport_slot_resetis not invoked.Quoted 7 lines
Such an event can be induced for testing purposes by reducing
theMax_Payload_Sizeof a PCIe bridge to less than that of a device downstream from the bridge, and then initating I/O through the device, resulting in oversize transactions. In the presence of DPC, this results in a containment event and attempted reset and recovery viapcie_do_recovery. After6d2c89441571report_slot_resetis not invoked, and the device does not recover.I think this issue is related to the issue discussed in following thread (DPC non-hotplug support).
https://lkml.org/lkml/2020/3/28/328
If my assumption is correct, you are dealing with devices which are not hotplug capable. If the devices are hotplug capable then you don't need to proceed to
report_slot_reset(), since hotplug handler will remove/re-enumerate the devices correctly.Correct, this particular device (a network card) is in a
non-hotplug slot.Quoted 1 line
Can you check whether following fix works for you?
Yes, it does.
I fixed up the whitespace and made a minor change to add braces
in what look like the correct places around the "if (reset_link)" block; the patch I tested with is below. I'll also install this on another machine with hotplug capable slots to test there as well.drivers/pci/pcie/err.c +9
-2@@ -165,13 +165,24 @@ pci_ers_result_t pcie_do_recovery(struct pci_dev *dev,pci_dbg (dev,"broadcast error_detected message\n" );if (state == pci_channel_io_frozen) {pci_walk_bus (bus, report_frozen_detected, &status); - status =reset_link (dev); -if (status !=PCI_ERS_RESULT_RECOVERED ) { + status =PCI_ERS_RESULT_NEED_RESET ; + }else { +pci_walk_bus (bus, report_normal_detected, &status); + } + +if (status ==PCI_ERS_RESULT_NEED_RESET ) { +if (reset_link) { +if (reset_link (dev) !=PCI_ERS_RESULT_RECOVERED ) + status =PCI_ERS_RESULT_DISCONNECT ;
Quoted 8 lines
drivers/pci/pcie/err.c pci_ers_result_t pcie_do_recovery(struct pci_dev *dev, · line 181
Quoted 5 lines
drivers/pci/pcie/err.c +5
-0+ }else { +if (pci_bus_error_reset (dev)) + status =PCI_ERS_RESULT_DISCONNECT ; + } +The
PCI_ERS_RESULT_NEED_RESETmay indicate that the driver requires a *slot* reset. With this patch, seems later slot reset broadcast may not be performed.
Slot reset wont be performed only if reset_link or pci_bus_error_reset returns error. Otherwise, we will still call pci_slot_reset later.
Quoted 9 lines
if (status == PCI_ERS_RESULT_NEED_RESET) { status = PCI_ERS_RESULT_RECOVERED; pci_dbg(dev, "broadcast slot_reset message\n"); pci_walk_bus(bus, report_slot_reset, &status); }One minor question, currently the callers of
pcie_do_recovery()will always pass areset_linkpointer, so is the condition necessary?
Yes, currently we don't need it. I added it to cover future use cases. But we can remove it if not needed.
Quoted 95 lines
Yicong
Quoted 91 lines
drivers/pci/pcie/err.c +1
-2+if (status ==PCI_ERS_RESULT_DISCONNECT ) {pci_warn (dev,"link reset failed\n" ); goto failed; } - }else { -pci_walk_bus (bus, report_normal_detected, &status); }if (status ==PCI_ERS_RESULT_CAN_RECOVER ) {
pci_bus_error_reset-JQuoted 66 lines
This includes support for
bus_resetin recovery function itself.index
14bb8f54723e..c9eaab68ab7a100644drivers/pci/pcie/err.c +14
-4@@ -165,13 +165,23 @@ pci_ers_result_t pcie_do_recovery(struct pci_dev *dev,pci_dbg (dev,"broadcast error_detected message\n" );if (state == pci_channel_io_frozen) {if (state == pci_channel_io_frozen) {pci_walk_bus (bus, report_frozen_detected, &status); - status =reset_link (dev); -if (status !=PCI_ERS_RESULT_RECOVERED ) { + status =PCI_ERS_RESULT_NEED_RESET ; + }else { +pci_walk_bus (bus, report_normal_detected, &status); + } + +if (status ==PCI_ERS_RESULT_NEED_RESET ) { +if (reset_link) +if (reset_link (dev) !=PCI_ERS_RESULT_RECOVERED ) + status =PCI_ERS_RESULT_DISCONNECT ; +else +if (pci_bus_error_reset (dev)) + status =PCI_ERS_RESULT_DISCONNECT ; + +if (status ==PCI_ERS_RESULT_DISCONNECT ) {pci_warn (dev,"link reset failed\n" ); goto failed; } - }else { -pci_walk_bus (bus, report_normal_detected, &status); }if (status == PCI_ERS_RESULT_CAN_RECOVER) {Quoted 30 lines
Quoted 30 lines
Inspection shows a similar path is plausible for a return of
_CAN_RECOVERand the invocation ofreport_mmio_enabled.Resolve this by preserving the result of
report_frozen_detectedifreset_linkdoes not return_DISCONNECT.Fixes 6d2c89441571 ("PCI/ERR: Update error status after reset_link()")Signed-off-by Jay Vosburgh <[email protected]>--- drivers/pci/pcie/err.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)drivers/pci/pcie/err.c +6
-2@@ -164,10 +164,17 @@ pci_ers_result_t pcie_do_recovery(struct pci_dev *dev,pci_dbg (dev,"broadcast error_detected message\n" );if (state == pci_channel_io_frozen) { +pci_ers_result_t status2; +pci_walk_bus (bus, report_frozen_detected, &status); - status =reset_link (dev); -if (status !=PCI_ERS_RESULT_RECOVERED ) { +/* preserve status from report_frozen_detected to +* insure report_mmio_enabled or report_slot_reset are +* invoked even if reset_link returns _RECOVERED. +*/ Quoted 8 lines
Quoted 8 lines
drivers/pci/pcie/err.c pci_ers_result_t pcie_do_recovery(struct pci_dev · line 180
Quoted 8 lines
drivers/pci/pcie/err.c +3
-0+ status2 =reset_link (dev); +if (status2 !=PCI_ERS_RESULT_RECOVERED ) {pci_warn (dev,"link reset failed\n" ); + status = status2; goto failed; } }else {---
-Jay Vosburgh, [email protected]
.