PCI/ERR: Resolve regression in pcie_do_recovery
Subsystems (2)
A message that is not in the archive (sent off-list or not yet received): 20200506203249.GA453633@bjorn-Precision-5520
Quoted 11 lines · Bjorn Helgaas
Quoted 10 lines · Jay Vosburgh
Quoted 8 lines · Jay Vosburgh
Quoted 6 lines · Sathyanarayanan
Hi Jay,
Quoted 3 lines · Sathyanarayanan
On 4/29/20 5:42 PM, Jay Vosburgh wrote:
[...]
Quoted 26 lines
Quoted 26 lines
Quoted 13 lines
Quoted 9 lines
Quoted 9 lines
I think this issue is related to the issue discussed in following thread (DPC non-hotplug support).
https://lkml.org/lkml/2020/3/28/328
If my assumption is correct, you are dealing with devices which are not hotplug capable. If the devices are hotplug capable then you don't need to proceed to
report_slot_reset(), since hotplug handler will remove/re-enumerate the devices correctly.Correct, this particular device (a network card) is in a
non-hotplug slot.Quoted 8 lines
Quoted 1 line
Can you check whether following fix works for you?
Yes, it does.
I fixed up the whitespace and made a minor change to add braces
in what look like the correct places around the "if (reset_link)" block; the patch I tested with is below. I'll also install this on another machine with hotplug capable slots to test there as well.We've tested the below patch on a couple of different machines
and devices (network card, NVMe device) and it appears to solve the recovery issue in our testing.
Quoted 8 lines
Quoted 2 lines
Is there anything further we need to do, or can this be
considered for inclusion upstream at this time?Can somebody please post a clean version of what we should merge? There was the initial patch plus a follow-up fix, so it's not clear where we ended up.
Bjorn
Below is the patch we tested, from Sathyanarayanan's test patch
(slightly edited to clarify ambiguous "if else" nesting), along with an edited version of the commit log from my original patch. I have not seen a Signed-off-by from Sathyanarayanan, so I didn't include one here.
One question I have is that, after the patch is applied, the
"status" filled in by pci_walk_bus(... report_frozen_detected ...) is discarded regardless of its value. Is that correct behavior in all cases? The original issue I was trying to solve was that the status set by report_frozen_detected was thrown away starting with 6d2c89441571 ("PCI/ERR: Update error status after reset_link()"), causing a _NEED_RESET to be lost. With the below patch, all cases of pci_channel_io_frozen will call reset_link unconditionally.
-J
Subject: [PATCH] PCI/ERR: Resolve regression in pcie_do_recovery
From: Kuppuswamy Sathyanarayanan <[email protected]>
Commit 6d2c89441571 ("PCI/ERR: Update error status afterreset_link()"), introduced a regression, as pcie_do_recovery will discard the status result from report_frozen_detected. This can cause a failure to recover if _NEED_RESET is returned by report_frozen_detected and report_slot_reset is not invoked.
Such an event can be induced for testing purposes by reducing
the Max_Payload_Size of a PCIe bridge to less than that of a device downstream from the bridge, and then initating I/O through the device, resulting in oversize transactions. In the presence of DPC, this results in a containment event and attempted reset and recovery via pcie_do_recovery. After 6d2c89441571 report_slot_reset is not invoked, and the device does not recover.
drivers/pci/pcie/err.c +15 -4
@@ -165,13 +165,24 @@ pci_ers_result_t pcie_do_recovery(struct pci_dev *dev,pci_dbg (dev,"broadcast error_detected message\n" );if (state == pci_channel_io_frozen) {pci_walk_bus (bus, report_frozen_detected, &status); - status =reset_link (dev); -if (status !=PCI_ERS_RESULT_RECOVERED ) { + status =PCI_ERS_RESULT_NEED_RESET ; + }else { +pci_walk_bus (bus, report_normal_detected, &status); + } + +if (status ==PCI_ERS_RESULT_NEED_RESET ) { +if (reset_link) { +if (reset_link (dev) !=PCI_ERS_RESULT_RECOVERED ) + status =PCI_ERS_RESULT_DISCONNECT ; + }else { +if (pci_bus_error_reset (dev)) + status =PCI_ERS_RESULT_DISCONNECT ; + } + +if (status ==PCI_ERS_RESULT_DISCONNECT ) {pci_warn (dev,"link reset failed\n" ); goto failed; } - }else { -pci_walk_bus (bus, report_normal_detected, &status); }if (status ==PCI_ERS_RESULT_CAN_RECOVER ) {
---
-Jay Vosburgh, [email protected]
PATCH 1/1 PCI/ERR: Handle fatal error recovery for non-hotplug capable devices
From: Kuppuswamy Sathyanarayanan <[email protected]>
If there are non-hotplug capable devices connected to a given port, then during the fatal error recovery(triggered by DPC or AER), after calling reset_link() function, we cannot rely on hotplug handler to detach and re-enumerate the device drivers in the affected bus. Instead, we will have to let the error recovery handler call report_slot_reset() for all devices in the bus to notify about the reset operation. Although this is only required for non hot-plug capable devices, doing it for hotplug capable devices should not affect the functionality.
Along with above issue, this fix also applicable to following issue.
Commit 6d2c89441571 ("PCI/ERR: Update error status after reset_link()") added support to store status of reset_link() call. Although this fixed the error recovery issue observed if the initial value of error status is PCI_ERS_RESULT_DISCONNECT or PCI_ERS_RESULT_NO_AER_DRIVER, it also discarded the status result from report_frozen_detected. This can cause a failure to recover if _NEED_RESET is returned by report_frozen_detected and report_slot_reset is not invoked.
Such an event can be induced for testing purposes by reducing the Max_Payload_Size of a PCIe bridge to less than that of a device downstream from the bridge, and then initiating I/O through the device, resulting in oversize transactions. In the presence of DPC, this results in a containment event and attempted reset and recovery via pcie_do_recovery. After 6d2c89441571 report_slot_reset is not invoked, and the device does not recover.
--- drivers/pci/pcie/err.c | 19 +++++++++++++++---- 1 file changed, 15 insertions(+), 4 deletions(-)
drivers/pci/pcie/err.c +15 -4
@@ -165,13 +165,24 @@ pci_ers_result_t pcie_do_recovery(struct pci_dev *dev,pci_dbg (dev,"broadcast error_detected message\n" );if (state == pci_channel_io_frozen) {pci_walk_bus (bus, report_frozen_detected, &status); - status =reset_link (dev); -if (status !=PCI_ERS_RESULT_RECOVERED ) { + status =PCI_ERS_RESULT_NEED_RESET ; + }else { +pci_walk_bus (bus, report_normal_detected, &status); + } + +if (status ==PCI_ERS_RESULT_NEED_RESET ) { +if (reset_link) { +if (reset_link (dev) !=PCI_ERS_RESULT_RECOVERED ) + status =PCI_ERS_RESULT_DISCONNECT ; + }else { +if (pci_bus_error_reset (dev)) + status =PCI_ERS_RESULT_DISCONNECT ; + } + +if (status ==PCI_ERS_RESULT_DISCONNECT ) {pci_warn (dev,"link reset failed\n" ); goto failed; } - }else { -pci_walk_bus (bus, report_normal_detected, &status); }if (status ==PCI_ERS_RESULT_CAN_RECOVER ) {
Signature · 1 line
2.17.1
Re: [PATCH v1 1/1] PCI/ERR: Handle fatal error recovery for non-hotplug capable devices
Quoted 11 lines · [email protected] wrote:
From: Kuppuswamy Sathyanarayanan <[email protected]>
If there are non-hotplug capable devices connected to a given port, then during the fatal error recovery(triggered by DPC or AER), after calling
reset_link()function, we cannot rely on hotplug handler to detach and re-enumerate the device drivers in the affected bus. Instead, we will have to let the error recovery handler callreport_slot_reset()for all devices in the bus to notify about the reset operation. Although this is only required for non hot-plug capable devices, doing it for hotplug capable devices should not affect the functionality.
Yicong,
Does the patch below also resolve the issue for you, as with
your changed version of my original patch?
-J
Quoted 65 lines
Along with above issue, this fix also applicable to following issue.
Commit
6d2c89441571("PCI/ERR: Update error status afterreset_link()") added support to store status ofreset_link()call. Although this fixed the error recovery issue observed if the initial value of error status isPCI_ERS_RESULT_DISCONNECTorPCI_ERS_RESULT_NO_AER_DRIVER, it also discarded the status result fromreport_frozen_detected. This can cause a failure to recover if_NEED_RESETis returned byreport_frozen_detectedandreport_slot_resetis not invoked.Such an event can be induced for testing purposes by reducing the
Max_Payload_Sizeof a PCIe bridge to less than that of a device downstream from the bridge, and then initiating I/O through the device, resulting in oversize transactions. In the presence of DPC, this results in a containment event and attempted reset and recovery viapcie_do_recovery. After6d2c89441571report_slot_resetis not invoked, and the device does not recover.[original patch is from [email protected]][original patch link https://lore.kernel.org/linux-pci/18609.1588812972@famine/]Fixes 6d2c89441571 ("PCI/ERR: Update error status after reset_link()")Signed-off-by Jay Vosburgh <[email protected]>Signed-off-by Kuppuswamy Sathyanarayanan <[email protected]>---
drivers/pci/pcie/err.c| 19 +++++++++++++++----1 file changed, 15 insertions(+), 4 deletions(-)drivers/pci/pcie/err.c +15
-4@@ -165,13 +165,24 @@ pci_ers_result_t pcie_do_recovery(struct pci_dev *dev,pci_dbg (dev,"broadcast error_detected message\n" );if (state == pci_channel_io_frozen) {pci_walk_bus (bus, report_frozen_detected, &status); - status =reset_link (dev); -if (status !=PCI_ERS_RESULT_RECOVERED ) { + status =PCI_ERS_RESULT_NEED_RESET ; + }else { +pci_walk_bus (bus, report_normal_detected, &status); + } + +if (status ==PCI_ERS_RESULT_NEED_RESET ) { +if (reset_link) { +if (reset_link (dev) !=PCI_ERS_RESULT_RECOVERED ) + status =PCI_ERS_RESULT_DISCONNECT ; + }else { +if (pci_bus_error_reset (dev)) + status =PCI_ERS_RESULT_DISCONNECT ; + } + +if (status ==PCI_ERS_RESULT_DISCONNECT ) {pci_warn (dev,"link reset failed\n" ); goto failed; } - }else { -pci_walk_bus (bus, report_normal_detected, &status); }if (status ==PCI_ERS_RESULT_CAN_RECOVER ) {Signature · 1 line
2.17.1
---
-Jay Vosburgh, [email protected]
Re: [PATCH v1 1/1] PCI/ERR: Handle fatal error recovery for non-hotplug capable devices
Quoted 13 lines · Jay Vosburgh
Quoted 11 lines · [email protected] wrote:
From: Kuppuswamy Sathyanarayanan <[email protected]>
If there are non-hotplug capable devices connected to a given port, then during the fatal error recovery(triggered by DPC or AER), after calling
reset_link()function, we cannot rely on hotplug handler to detach and re-enumerate the device drivers in the affected bus. Instead, we will have to let the error recovery handler callreport_slot_reset()for all devices in the bus to notify about the reset operation. Although this is only required for non hot-plug capable devices, doing it for hotplug capable devices should not affect the functionality.
Quoted 4 lines
Yicong,
Does the patch below also resolve the issue for you, as with
your changed version of my original patch?
Yes. It works.
Quoted 72 lines
-JQuoted 65 lines
Along with above issue, this fix also applicable to following issue.
Commit
6d2c89441571("PCI/ERR: Update error status afterreset_link()") added support to store status ofreset_link()call. Although this fixed the error recovery issue observed if the initial value of error status isPCI_ERS_RESULT_DISCONNECTorPCI_ERS_RESULT_NO_AER_DRIVER, it also discarded the status result fromreport_frozen_detected. This can cause a failure to recover if_NEED_RESETis returned byreport_frozen_detectedandreport_slot_resetis not invoked.Such an event can be induced for testing purposes by reducing the
Max_Payload_Sizeof a PCIe bridge to less than that of a device downstream from the bridge, and then initiating I/O through the device, resulting in oversize transactions. In the presence of DPC, this results in a containment event and attempted reset and recovery viapcie_do_recovery. After6d2c89441571report_slot_resetis not invoked, and the device does not recover.[original patch is from [email protected]][original patch link https://lore.kernel.org/linux-pci/18609.1588812972@famine/]Fixes 6d2c89441571 ("PCI/ERR: Update error status after reset_link()")Signed-off-by Jay Vosburgh <[email protected]>Signed-off-by Kuppuswamy Sathyanarayanan <[email protected]>---
drivers/pci/pcie/err.c| 19 +++++++++++++++----1 file changed, 15 insertions(+), 4 deletions(-)drivers/pci/pcie/err.c +15
-4@@ -165,13 +165,24 @@ pci_ers_result_t pcie_do_recovery(struct pci_dev *dev,pci_dbg (dev,"broadcast error_detected message\n" );if (state == pci_channel_io_frozen) {pci_walk_bus (bus, report_frozen_detected, &status); - status =reset_link (dev); -if (status !=PCI_ERS_RESULT_RECOVERED ) { + status =PCI_ERS_RESULT_NEED_RESET ; + }else { +pci_walk_bus (bus, report_normal_detected, &status); + } + +if (status ==PCI_ERS_RESULT_NEED_RESET ) { +if (reset_link) { +if (reset_link (dev) !=PCI_ERS_RESULT_RECOVERED ) + status =PCI_ERS_RESULT_DISCONNECT ; + }else { +if (pci_bus_error_reset (dev)) + status =PCI_ERS_RESULT_DISCONNECT ; + } + +if (status ==PCI_ERS_RESULT_DISCONNECT ) {pci_warn (dev,"link reset failed\n" ); goto failed; } - }else { -pci_walk_bus (bus, report_normal_detected, &status); }if (status ==PCI_ERS_RESULT_CAN_RECOVER ) {Signature · 1 line
2.17.1---
-Jay Vosburgh, [email protected]
.
Re: [PATCH v1 1/1] PCI/ERR: Handle fatal error recovery for non-hotplug capable devices
In reply to [email protected]Quoted 11 lines · On Wed, May 06, 2020 at 08:32:59PM -0700…
From: Kuppuswamy Sathyanarayanan <[email protected]>
If there are non-hotplug capable devices connected to a given port, then during the fatal error recovery(triggered by DPC or AER), after calling
reset_link()function, we cannot rely on hotplug handler to detach and re-enumerate the device drivers in the affected bus. Instead, we will have to let the error recovery handler callreport_slot_reset()for all devices in the bus to notify about the reset operation. Although this is only required for non hot-plug capable devices, doing it for hotplug capable devices should not affect the functionality.
Apparently this fixes a bug. Can you include a link to a problem report? The above is a description of a *solution*, but it's hard to understand without starting with the problem itself.
The above talks about reset_link(), which is only used in the io_frozen case. In that case, I think the only thing this patch changes is that when reset_link() fails, we'll return with PCI_ERS_RESULT_DISCONNECT instead of whatever reset_link() returned.
What this patch *does* change is that in the normal_detected case, we previously never called reset_link() but now we will if an .error_detected() callback requested it.
Quoted 11 lines
Along with above issue, this fix also applicable to following issue.
Commit
6d2c89441571("PCI/ERR: Update error status afterreset_link()") added support to store status ofreset_link()call. Although this fixed the error recovery issue observed if the initial value of error status isPCI_ERS_RESULT_DISCONNECTorPCI_ERS_RESULT_NO_AER_DRIVER, it also discarded the status result fromreport_frozen_detected. This can cause a failure to recover if_NEED_RESETis returned byreport_frozen_detectedandreport_slot_resetis not invoked.
Sorry, I'm not following this explanation. IIUC this fixes a bug when the channel is frozen and .error_detected() returns NEED_RESET.
In that case, the current code does:
* state == io_frozen
* .error_detected() returns status == NEED_RESET
* status = reset_link()
* if status != RECOVERED, we return status
* otherwise status == RECOVERED
* we do not call report_slot_reset()
* we return RECOVERED
It does seem like we *should* call the .slot_reset() callbacks, but the only time we return recovery failure is if reset_link() failed.
I can certainly understand if drivers don't recover when we reset their device but don't call their .slot_reset() callback. Is that what this fixes?
After the patch,
* state == io_frozen
* .error_detected() returns status == NEED_RESET
* we set status = NEED_RESET (doesn't change anything in this case)
* we call reset_link()
* if reset_link() failed, we return DISCONNECT
* otherwise continue with status == NEED_RESET
* we *do* call report_slot_reset()
* we return whatever .slot_reset() returned
I think the change to call .slot_reset() makes sense, but that's not at all clear from the commit log. Am I understanding the behavior correctly?
Quoted 7 lines
Such an event can be induced for testing purposes by reducing the
Max_Payload_Sizeof a PCIe bridge to less than that of a device downstream from the bridge, and then initiating I/O through the device, resulting in oversize transactions. In the presence of DPC, this results in a containment event and attempted reset and recovery viapcie_do_recovery. After6d2c89441571report_slot_resetis not invoked, and the device does not recover.
Use "pcie_do_recovery()" and "report_slot_reset()" (including the parentheses) to follow convention. Also fix other occurrences above.
Quoted 21 lines
[original patch is from [email protected]][original patch link https://lore.kernel.org/linux-pci/18609.1588812972@famine/]Fixes 6d2c89441571 ("PCI/ERR: Update error status after reset_link()")Signed-off-by Jay Vosburgh <[email protected]>Signed-off-by Kuppuswamy Sathyanarayanan <[email protected]>--- drivers/pci/pcie/err.c | 19 +++++++++++++++---- 1 file changed, 15 insertions(+), 4 deletions(-)drivers/pci/pcie/err.c +2
-2@@ -165,13 +165,24 @@ pci_ers_result_t pcie_do_recovery(struct pci_dev *dev,pci_dbg (dev,"broadcast error_detected message\n" );if (state == pci_channel_io_frozen) {pci_walk_bus (bus, report_frozen_detected, &status); - status =reset_link (dev); -if (status !=PCI_ERS_RESULT_RECOVERED ) { + status =PCI_ERS_RESULT_NEED_RESET ; + }else {
drivers/pci/pcie/err.c pci_ers_result_t pcie_do_recovery(struct pci_dev *dev, · line 179
Quoted 10 lines
drivers/pci/pcie/err.c +10
-0+pci_walk_bus (bus, report_normal_detected, &status); + } + +if (status ==PCI_ERS_RESULT_NEED_RESET ) { +if (reset_link) { +if (reset_link (dev) !=PCI_ERS_RESULT_RECOVERED ) + status =PCI_ERS_RESULT_DISCONNECT ; + }else { +if (pci_bus_error_reset (dev)) + status =PCI_ERS_RESULT_DISCONNECT ;
As far as I can tell, there is no caller of pcie_do_recovery() where reset_link is NULL, so this call of pci_bus_error_reset() looks like dead code.
We did not previously check whether reset_link was NULL, and this patch changes nothing that could result in it being NULL.
Quoted 14 lines
drivers/pci/pcie/err.c +3
-2+ } + +if (status ==PCI_ERS_RESULT_DISCONNECT ) {pci_warn (dev,"link reset failed\n" ); goto failed; } - }else { -pci_walk_bus (bus, report_normal_detected, &status); }if (status ==PCI_ERS_RESULT_CAN_RECOVER ) {Signature · 1 line
2.17.1
Re: [PATCH v1 1/1] PCI/ERR: Handle fatal error recovery for non-hotplug capable devices
Hi Bjorn,
Quoted 13 lines · Bjorn Helgaas
Quoted 11 lines · On Wed, May 06, 2020 at 08:32:59PM -0700…
From: Kuppuswamy Sathyanarayanan <[email protected]>
If there are non-hotplug capable devices connected to a given port, then during the fatal error recovery(triggered by DPC or AER), after calling
reset_link()function, we cannot rely on hotplug handler to detach and re-enumerate the device drivers in the affected bus. Instead, we will have to let the error recovery handler callreport_slot_reset()for all devices in the bus to notify about the reset operation. Although this is only required for non hot-plug capable devices, doing it for hotplug capable devices should not affect the functionality.
Quoted 3 lines
Apparently this fixes a bug. Can you include a link to a problem report? The above is a description of a *solution*, but it's hard to understand without starting with the problem itself.
Let me add the problem description to the commit log in next version.
Adding it here for discussion.
Current pcie_do_recovery() implementation has following three issues:
1. Fatal (DPC) error recovery is currently broken for non-hotplug capable devices. Current DPC recovery implementation relies on hotplug handler for detaching/re-enumerating the affected devices/drivers on DLLSC state changes. So when dealing with non-hotplug capable devices, recovery code does not restore the state of the affected devices correctly. Correct implementation should call report_mmio_enabled() and report_slot_reset() functions after reseting the link/device which is currently missing.
2. For non-fatal errors if report_error_detected() or report_mmio_enabled() functions requests PCI_ERS_RESULT_NEED_RESET then current pcie_do_recovery() implementation does not do the requested explicit device reset, instead just calls the report_slot_reset() on all affected devices. Notifying about the reset via report_slot_reset() without doing the actual device reset is incorrect.
3. For fatal errors, pcie_do_recovery() function currently completely ignores the status result of report_error_detected() call. After calling reset_link() function then status value gets updated to either PCI_ERS_RESULT_DISCONNECT or PCI_ERS_RESULT_RECOVERED, and the initial value returned from report_error_detected() is totally ignored. For solving the issue # 1, we need to update the status only if reset_link() call failed and returns PCI_ERS_RESULT_DISCONNECT. Otherwise recovery handler should proceed to call report_mmio_enabled() and report_slot_reset().
Quoted 6 lines
The above talks about
reset_link(), which is only used in theio_frozencase. In that case, I think the only thing this patch changes is that whenreset_link()fails, we'll return withPCI_ERS_RESULT_DISCONNECTinstead of whateverreset_link()returned.
Yes, if reset_link() is successful then we let pcie_do_recovery() proceed to call report_slot_reset() with initial status value returned from report_error_detected() function call.
Quoted 4 lines
What this patch *does* change is that in the
normal_detectedcase, we previously never calledreset_link()but now we will if an .error_detected() callback requested it.
yes, as highlighted in issue # 1 this patch adds support to call reset_link() if report_error_detected() function requests it.
Quoted 44 lines
Quoted 4 lines
Along with above issue, this fix also applicable to following issue.
Quoted 8 lines
Commit
6d2c89441571("PCI/ERR: Update error status afterreset_link()") added support to store status ofreset_link()call. Although this fixed the error recovery issue observed if the initial value of error status isPCI_ERS_RESULT_DISCONNECTorPCI_ERS_RESULT_NO_AER_DRIVER, it also discarded the status result fromreport_frozen_detected. This can cause a failure to recover if_NEED_RESETis returned byreport_frozen_detectedandreport_slot_resetis not invoked.Sorry, I'm not following this explanation. IIUC this fixes a bug when the channel is frozen and .error_detected() returns
NEED_RESET.In that case, the current code does:
* state ==
io_frozen
* .error_detected() returns status ==NEED_RESET
* status =reset_link()
* if status != RECOVERED, we return status
* otherwise status == RECOVERED
* we do not callreport_slot_reset()
* we return RECOVEREDIt does seem like we *should* call the .slot_reset() callbacks, but the only time we return recovery failure is if
reset_link()failed.I can certainly understand if drivers don't recover when we reset their device but don't call their .slot_reset() callback. Is that what this fixes?
After the patch,
* state ==
io_frozen
* .error_detected() returns status ==NEED_RESET
* we set status =NEED_RESET(doesn't change anything in this case)
* we callreset_link()
* ifreset_link()failed, we return DISCONNECT
* otherwise continue with status ==NEED_RESET
* we *do* callreport_slot_reset()
* we return whatever .slot_reset() returned
Quoted 3 lines
I think the change to call .slot_reset() makes sense, but that's not at all clear from the commit log. Am I understanding the behavior correctly?
Yes, your understanding is correct. I hope above mentioned error description clarifies it.
Quoted 11 lines
Quoted 8 lines
Such an event can be induced for testing purposes by reducing the
Max_Payload_Sizeof a PCIe bridge to less than that of a device downstream from the bridge, and then initiating I/O through the device, resulting in oversize transactions. In the presence of DPC, this results in a containment event and attempted reset and recovery viapcie_do_recovery. After6d2c89441571report_slot_resetis not invoked, and the device does not recover.Use "
pcie_do_recovery()" and "report_slot_reset()" (including the parentheses) to follow convention. Also fix other occurrences above.
will fix it in next version.
Quoted 28 lines
Quoted 28 lines
[original patch is from [email protected]][original patch link https://lore.kernel.org/linux-pci/18609.1588812972@famine/]Fixes 6d2c89441571 ("PCI/ERR: Update error status after reset_link()")Signed-off-by Jay Vosburgh <[email protected]>Signed-off-by Kuppuswamy Sathyanarayanan <[email protected]>--- drivers/pci/pcie/err.c | 19 +++++++++++++++---- 1 file changed, 15 insertions(+), 4 deletions(-)drivers/pci/pcie/err.c +8
-2@@ -165,13 +165,24 @@ pci_ers_result_t pcie_do_recovery(struct pci_dev *dev,pci_dbg (dev,"broadcast error_detected message\n" );if (state == pci_channel_io_frozen) {pci_walk_bus (bus, report_frozen_detected, &status); - status =reset_link (dev); -if (status !=PCI_ERS_RESULT_RECOVERED ) { + status =PCI_ERS_RESULT_NEED_RESET ; + }else { +pci_walk_bus (bus, report_normal_detected, &status); + } + +if (status ==PCI_ERS_RESULT_NEED_RESET ) { +if (reset_link) { +if (reset_link (dev) !=PCI_ERS_RESULT_RECOVERED )
Quoted 8 lines
drivers/pci/pcie/err.c pci_ers_result_t pcie_do_recovery(struct pci_dev *dev, · line 179
Quoted 4 lines
drivers/pci/pcie/err.c +4
-0+ status =PCI_ERS_RESULT_DISCONNECT ; + }else { +if (pci_bus_error_reset (dev)) + status =PCI_ERS_RESULT_DISCONNECT ;As far as I can tell, there is no caller of
pcie_do_recovery()wherereset_linkis NULL, so this call ofpci_bus_error_reset()looks like dead code.
Yes, I will remove the if (reset_link) check.
Quoted 18 lines
We did not previously check whether
reset_linkwas NULL, and this patch changes nothing that could result in it being NULL.Quoted 14 lines
drivers/pci/pcie/err.c +3
-2+ } + +if (status ==PCI_ERS_RESULT_DISCONNECT ) {pci_warn (dev,"link reset failed\n" ); goto failed; } - }else { -pci_walk_bus (bus, report_normal_detected, &status); }if (status ==PCI_ERS_RESULT_CAN_RECOVER ) {Signature · 1 line
2.17.1
Re: [PATCH v1 1/1] PCI/ERR: Handle fatal error recovery for non-hotplug capable devices
In reply to [email protected]Quoted 49 lines · On 2020/5/7 11:32, [email protected] wrote:
From: Kuppuswamy Sathyanarayanan <[email protected]>
If there are non-hotplug capable devices connected to a given port, then during the fatal error recovery(triggered by DPC or AER), after calling
reset_link()function, we cannot rely on hotplug handler to detach and re-enumerate the device drivers in the affected bus. Instead, we will have to let the error recovery handler callreport_slot_reset()for all devices in the bus to notify about the reset operation. Although this is only required for non hot-plug capable devices, doing it for hotplug capable devices should not affect the functionality.Along with above issue, this fix also applicable to following issue.
Commit
6d2c89441571("PCI/ERR: Update error status afterreset_link()") added support to store status ofreset_link()call. Although this fixed the error recovery issue observed if the initial value of error status isPCI_ERS_RESULT_DISCONNECTorPCI_ERS_RESULT_NO_AER_DRIVER, it also discarded the status result fromreport_frozen_detected. This can cause a failure to recover if_NEED_RESETis returned byreport_frozen_detectedandreport_slot_resetis not invoked.Such an event can be induced for testing purposes by reducing the
Max_Payload_Sizeof a PCIe bridge to less than that of a device downstream from the bridge, and then initiating I/O through the device, resulting in oversize transactions. In the presence of DPC, this results in a containment event and attempted reset and recovery viapcie_do_recovery. After6d2c89441571report_slot_resetis not invoked, and the device does not recover.[original patch is from [email protected]][original patch link https://lore.kernel.org/linux-pci/18609.1588812972@famine/]Fixes 6d2c89441571 ("PCI/ERR: Update error status after reset_link()")Signed-off-by Jay Vosburgh <[email protected]>Signed-off-by Kuppuswamy Sathyanarayanan <[email protected]>--- drivers/pci/pcie/err.c | 19 +++++++++++++++---- 1 file changed, 15 insertions(+), 4 deletions(-)drivers/pci/pcie/err.c +0
-0@@ -165,13 +165,24 @@ pci_ers_result_t pcie_do_recovery(struct pci_dev *dev,pci_dbg (dev,"broadcast error_detected message\n" );if (state == pci_channel_io_frozen) {pci_walk_bus (bus, report_frozen_detected, &status);
drivers/pci/pcie/err.c pci_ers_result_t pcie_do_recovery(struct pci_dev *dev, · line 175
Quoted 10 lines
drivers/pci/pcie/err.c +8
-2- status =reset_link (dev); -if (status !=PCI_ERS_RESULT_RECOVERED ) { + status =PCI_ERS_RESULT_NEED_RESET ; + }else { +pci_walk_bus (bus, report_normal_detected, &status); + } + +if (status ==PCI_ERS_RESULT_NEED_RESET ) { +if (reset_link) { +if (reset_link (dev) !=PCI_ERS_RESULT_RECOVERED )
we'll call reset_link() only if link is frozen. so it may have problem here.
Thanks,
Yicong
Quoted 15 lines
drivers/pci/pcie/err.c +7
-2+ status =PCI_ERS_RESULT_DISCONNECT ; + }else { +if (pci_bus_error_reset (dev)) + status =PCI_ERS_RESULT_DISCONNECT ; + } + +if (status ==PCI_ERS_RESULT_DISCONNECT ) {pci_warn (dev,"link reset failed\n" ); goto failed; } - }else { -pci_walk_bus (bus, report_normal_detected, &status); }if (status ==PCI_ERS_RESULT_CAN_RECOVER ) {
Re: [PATCH v1 1/1] PCI/ERR: Handle fatal error recovery for non-hotplug capable devices
Quoted 54 lines · Yicong Yang
Quoted 53 lines · On 2020/5/7 11:32, [email protected] wrote:
From: Kuppuswamy Sathyanarayanan <[email protected]>
If there are non-hotplug capable devices connected to a given port, then during the fatal error recovery(triggered by DPC or AER), after calling
reset_link()function, we cannot rely on hotplug handler to detach and re-enumerate the device drivers in the affected bus. Instead, we will have to let the error recovery handler callreport_slot_reset()for all devices in the bus to notify about the reset operation. Although this is only required for non hot-plug capable devices, doing it for hotplug capable devices should not affect the functionality.Along with above issue, this fix also applicable to following issue.
Commit
6d2c89441571("PCI/ERR: Update error status afterreset_link()") added support to store status ofreset_link()call. Although this fixed the error recovery issue observed if the initial value of error status isPCI_ERS_RESULT_DISCONNECTorPCI_ERS_RESULT_NO_AER_DRIVER, it also discarded the status result fromreport_frozen_detected. This can cause a failure to recover if_NEED_RESETis returned byreport_frozen_detectedandreport_slot_resetis not invoked.Such an event can be induced for testing purposes by reducing the
Max_Payload_Sizeof a PCIe bridge to less than that of a device downstream from the bridge, and then initiating I/O through the device, resulting in oversize transactions. In the presence of DPC, this results in a containment event and attempted reset and recovery viapcie_do_recovery. After6d2c89441571report_slot_resetis not invoked, and the device does not recover.[original patch is from [email protected]][original patch link https://lore.kernel.org/linux-pci/18609.1588812972@famine/]Fixes 6d2c89441571 ("PCI/ERR: Update error status after reset_link()")Signed-off-by Jay Vosburgh <[email protected]>Signed-off-by Kuppuswamy Sathyanarayanan <[email protected]>--- drivers/pci/pcie/err.c | 19 +++++++++++++++---- 1 file changed, 15 insertions(+), 4 deletions(-)drivers/pci/pcie/err.c +2
-2@@ -165,13 +165,24 @@ pci_ers_result_t pcie_do_recovery(struct pci_dev *dev,pci_dbg (dev,"broadcast error_detected message\n" );if (state == pci_channel_io_frozen) {pci_walk_bus (bus, report_frozen_detected, &status); - status =reset_link (dev); -if (status !=PCI_ERS_RESULT_RECOVERED ) { + status =PCI_ERS_RESULT_NEED_RESET ; + }else {
Quoted 8 lines
drivers/pci/pcie/err.c pci_ers_result_t pcie_do_recovery(struct pci_dev *dev, · line 175
Quoted 6 lines
drivers/pci/pcie/err.c +6
-0+pci_walk_bus (bus, report_normal_detected, &status); + } + +if (status ==PCI_ERS_RESULT_NEED_RESET ) { +if (reset_link) { +if (reset_link (dev) !=PCI_ERS_RESULT_RECOVERED )we'll call
reset_link()only if link is frozen. so it may have problem here.
you mean before this change right?
After this change, reset_link() will be called as long as status is PCI_ERS_RESULT_NEED_RESET.
Quoted 21 lines
Thanks,
YicongQuoted 15 lines
drivers/pci/pcie/err.c +7
-2+ status =PCI_ERS_RESULT_DISCONNECT ; + }else { +if (pci_bus_error_reset (dev)) + status =PCI_ERS_RESULT_DISCONNECT ; + } + +if (status ==PCI_ERS_RESULT_DISCONNECT ) {pci_warn (dev,"link reset failed\n" ); goto failed; } - }else { -pci_walk_bus (bus, report_normal_detected, &status); }if (status ==PCI_ERS_RESULT_CAN_RECOVER ) {
Re: [PATCH v1 1/1] PCI/ERR: Handle fatal error recovery for non-hotplug capable devices
Quoted 65 lines · Sathyanarayanan
Quoted 54 lines · Yicong Yang
Quoted 53 lines · On 2020/5/7 11:32, [email protected] wrote:
From: Kuppuswamy Sathyanarayanan <[email protected]>
If there are non-hotplug capable devices connected to a given port, then during the fatal error recovery(triggered by DPC or AER), after calling
reset_link()function, we cannot rely on hotplug handler to detach and re-enumerate the device drivers in the affected bus. Instead, we will have to let the error recovery handler callreport_slot_reset()for all devices in the bus to notify about the reset operation. Although this is only required for non hot-plug capable devices, doing it for hotplug capable devices should not affect the functionality.Along with above issue, this fix also applicable to following issue.
Commit
6d2c89441571("PCI/ERR: Update error status afterreset_link()") added support to store status ofreset_link()call. Although this fixed the error recovery issue observed if the initial value of error status isPCI_ERS_RESULT_DISCONNECTorPCI_ERS_RESULT_NO_AER_DRIVER, it also discarded the status result fromreport_frozen_detected. This can cause a failure to recover if_NEED_RESETis returned byreport_frozen_detectedandreport_slot_resetis not invoked.Such an event can be induced for testing purposes by reducing the
Max_Payload_Sizeof a PCIe bridge to less than that of a device downstream from the bridge, and then initiating I/O through the device, resulting in oversize transactions. In the presence of DPC, this results in a containment event and attempted reset and recovery viapcie_do_recovery. After6d2c89441571report_slot_resetis not invoked, and the device does not recover.[original patch is from [email protected]][original patch link https://lore.kernel.org/linux-pci/18609.1588812972@famine/]Fixes 6d2c89441571 ("PCI/ERR: Update error status after reset_link()")Signed-off-by Jay Vosburgh <[email protected]>Signed-off-by Kuppuswamy Sathyanarayanan <[email protected]>--- drivers/pci/pcie/err.c | 19 +++++++++++++++---- 1 file changed, 15 insertions(+), 4 deletions(-)drivers/pci/pcie/err.c +2
-2@@ -165,13 +165,24 @@ pci_ers_result_t pcie_do_recovery(struct pci_dev *dev,pci_dbg (dev,"broadcast error_detected message\n" );if (state == pci_channel_io_frozen) {pci_walk_bus (bus, report_frozen_detected, &status); - status =reset_link (dev); -if (status !=PCI_ERS_RESULT_RECOVERED ) { + status =PCI_ERS_RESULT_NEED_RESET ; + }else {Quoted 8 lines
drivers/pci/pcie/err.c pci_ers_result_t pcie_do_recovery(struct pci_dev *dev, · line 175
Quoted 6 lines
drivers/pci/pcie/err.c +6
-0+pci_walk_bus (bus, report_normal_detected, &status); + } + +if (status ==PCI_ERS_RESULT_NEED_RESET ) { +if (reset_link) { +if (reset_link (dev) !=PCI_ERS_RESULT_RECOVERED )we'll call
reset_link()only if link is frozen. so it may have problem here.
Quoted 3 lines
you mean before this change right?
After this change,reset_link()will be called as long as status isPCI_ERS_RESULT_NEED_RESET.
Yes. I think we should reset the link only if the io is blocked as before. There's no reason to reset a normal link.
Furthermore, PCI_ERS_RESULT_NEED_RESET means device driver requires a slot reset rather than a link reset, so it maybe improper to use it to judge whether a link reset is needed. We decide whether to do a link reset only by the io state.
Thanks,
Yicong
Quoted 22 lines
Quoted 20 lines
Thanks,
YicongQuoted 14 lines
drivers/pci/pcie/err.c +7
-2+ status =PCI_ERS_RESULT_DISCONNECT ; + }else { +if (pci_bus_error_reset (dev)) + status =PCI_ERS_RESULT_DISCONNECT ; + } + +if (status ==PCI_ERS_RESULT_DISCONNECT ) {pci_warn (dev,"link reset failed\n" ); goto failed; } - }else { -pci_walk_bus (bus, report_normal_detected, &status); }if (status ==PCI_ERS_RESULT_CAN_RECOVER ) {.
Re: [PATCH v1 1/1] PCI/ERR: Handle fatal error recovery for non-hotplug capable devices
Quoted 66 lines · Yicong Yang
Quoted 65 lines · Sathyanarayanan
Quoted 62 lines · Yicong Yang
Quoted 49 lines · On 2020/5/7 11:32, [email protected] wrote:
From: Kuppuswamy Sathyanarayanan <[email protected]>
If there are non-hotplug capable devices connected to a given port, then during the fatal error recovery(triggered by DPC or AER), after calling
reset_link()function, we cannot rely on hotplug handler to detach and re-enumerate the device drivers in the affected bus. Instead, we will have to let the error recovery handler callreport_slot_reset()for all devices in the bus to notify about the reset operation. Although this is only required for non hot-plug capable devices, doing it for hotplug capable devices should not affect the functionality.Along with above issue, this fix also applicable to following issue.
Commit
6d2c89441571("PCI/ERR: Update error status afterreset_link()") added support to store status ofreset_link()call. Although this fixed the error recovery issue observed if the initial value of error status isPCI_ERS_RESULT_DISCONNECTorPCI_ERS_RESULT_NO_AER_DRIVER, it also discarded the status result fromreport_frozen_detected. This can cause a failure to recover if_NEED_RESETis returned byreport_frozen_detectedandreport_slot_resetis not invoked.Such an event can be induced for testing purposes by reducing the
Max_Payload_Sizeof a PCIe bridge to less than that of a device downstream from the bridge, and then initiating I/O through the device, resulting in oversize transactions. In the presence of DPC, this results in a containment event and attempted reset and recovery viapcie_do_recovery. After6d2c89441571report_slot_resetis not invoked, and the device does not recover.[original patch is from [email protected]][original patch link https://lore.kernel.org/linux-pci/18609.1588812972@famine/]Fixes 6d2c89441571 ("PCI/ERR: Update error status after reset_link()")Signed-off-by Jay Vosburgh <[email protected]>Signed-off-by Kuppuswamy Sathyanarayanan <[email protected]>--- drivers/pci/pcie/err.c | 19 +++++++++++++++---- 1 file changed, 15 insertions(+), 4 deletions(-)drivers/pci/pcie/err.c +0
-0@@ -165,13 +165,24 @@ pci_ers_result_t pcie_do_recovery(struct pci_dev *dev,pci_dbg (dev,"broadcast error_detected message\n" );if (state == pci_channel_io_frozen) {pci_walk_bus (bus, report_frozen_detected, &status);drivers/pci/pcie/err.c pci_ers_result_t pcie_do_recovery(struct pci_dev *dev, · line 175
Quoted 10 lines
drivers/pci/pcie/err.c +8
-2- status =reset_link (dev); -if (status !=PCI_ERS_RESULT_RECOVERED ) { + status =PCI_ERS_RESULT_NEED_RESET ; + }else { +pci_walk_bus (bus, report_normal_detected, &status); + } + +if (status ==PCI_ERS_RESULT_NEED_RESET ) { +if (reset_link) { +if (reset_link (dev) !=PCI_ERS_RESULT_RECOVERED )we'll call
reset_link()only if link is frozen. so it may have problem here.
Quoted 6 lines
Quoted 3 lines
you mean before this change right?
After this change,reset_link()will be called as long as status isPCI_ERS_RESULT_NEED_RESET.Yes. I think we should reset the link only if the io is blocked as before. There's no reason to reset a normal link.
Currently, only AER and DPC driver uses pcie_do_recovery() call. So the possible reset_link options are dpc_reset_link() and aer_root_reset().
In dpc_reset_link() case, the link is already disabled and hence we don't need to do another reset. In case of aer_root_reset() it uses pci_bus_error_reset() to reset the slot.
Quoted 32 lines
Furthermore,
PCI_ERS_RESULT_NEED_RESETmeans device driver requires a slot reset rather than a link reset, so it maybe improper to use it to judge whether a link reset is needed. We decide whether to do a link reset only by the io state.Thanks,
YicongQuoted 22 lines
Quoted 20 lines
Thanks,
YicongQuoted 14 lines
drivers/pci/pcie/err.c +7
-2+ status =PCI_ERS_RESULT_DISCONNECT ; + }else { +if (pci_bus_error_reset (dev)) + status =PCI_ERS_RESULT_DISCONNECT ; + } + +if (status ==PCI_ERS_RESULT_DISCONNECT ) {pci_warn (dev,"link reset failed\n" ); goto failed; } - }else { -pci_walk_bus (bus, report_normal_detected, &status); }if (status ==PCI_ERS_RESULT_CAN_RECOVER ) {.
Re: [PATCH v1 1/1] PCI/ERR: Handle fatal error recovery for non-hotplug capable devices
Quoted 73 lines · Sathyanarayanan
Quoted 70 lines · Yicong Yang
Quoted 68 lines · Sathyanarayanan
Quoted 54 lines · Yicong Yang
Quoted 53 lines · On 2020/5/7 11:32, [email protected] wrote:
From: Kuppuswamy Sathyanarayanan <[email protected]>
If there are non-hotplug capable devices connected to a given port, then during the fatal error recovery(triggered by DPC or AER), after calling
reset_link()function, we cannot rely on hotplug handler to detach and re-enumerate the device drivers in the affected bus. Instead, we will have to let the error recovery handler callreport_slot_reset()for all devices in the bus to notify about the reset operation. Although this is only required for non hot-plug capable devices, doing it for hotplug capable devices should not affect the functionality.Along with above issue, this fix also applicable to following issue.
Commit
6d2c89441571("PCI/ERR: Update error status afterreset_link()") added support to store status ofreset_link()call. Although this fixed the error recovery issue observed if the initial value of error status isPCI_ERS_RESULT_DISCONNECTorPCI_ERS_RESULT_NO_AER_DRIVER, it also discarded the status result fromreport_frozen_detected. This can cause a failure to recover if_NEED_RESETis returned byreport_frozen_detectedandreport_slot_resetis not invoked.Such an event can be induced for testing purposes by reducing the
Max_Payload_Sizeof a PCIe bridge to less than that of a device downstream from the bridge, and then initiating I/O through the device, resulting in oversize transactions. In the presence of DPC, this results in a containment event and attempted reset and recovery viapcie_do_recovery. After6d2c89441571report_slot_resetis not invoked, and the device does not recover.[original patch is from [email protected]][original patch link https://lore.kernel.org/linux-pci/18609.1588812972@famine/]Fixes 6d2c89441571 ("PCI/ERR: Update error status after reset_link()")Signed-off-by Jay Vosburgh <[email protected]>Signed-off-by Kuppuswamy Sathyanarayanan <[email protected]>--- drivers/pci/pcie/err.c | 19 +++++++++++++++---- 1 file changed, 15 insertions(+), 4 deletions(-)drivers/pci/pcie/err.c +2
-2@@ -165,13 +165,24 @@ pci_ers_result_t pcie_do_recovery(struct pci_dev *dev,pci_dbg (dev,"broadcast error_detected message\n" );if (state == pci_channel_io_frozen) {pci_walk_bus (bus, report_frozen_detected, &status); - status =reset_link (dev); -if (status !=PCI_ERS_RESULT_RECOVERED ) { + status =PCI_ERS_RESULT_NEED_RESET ; + }else {Quoted 8 lines
drivers/pci/pcie/err.c pci_ers_result_t pcie_do_recovery(struct pci_dev *dev, · line 175
Quoted 6 lines
drivers/pci/pcie/err.c +6
-0+pci_walk_bus (bus, report_normal_detected, &status); + } + +if (status ==PCI_ERS_RESULT_NEED_RESET ) { +if (reset_link) { +if (reset_link (dev) !=PCI_ERS_RESULT_RECOVERED )we'll call
reset_link()only if link is frozen. so it may have problem here.you mean before this change right?
After this change,reset_link()will be called as long as status isPCI_ERS_RESULT_NEED_RESET.
Quoted 8 lines
Quoted 2 lines
Yes. I think we should reset the link only if the io is blocked as before. There's no reason to reset a normal link.
Currently, only AER and DPC driver uses
pcie_do_recovery()call. So the possiblereset_linkoptions aredpc_reset_link()andaer_root_reset().In
dpc_reset_link()case, the link is already disabled and hence we don't need to do another reset. In case ofaer_root_reset()it usespci_bus_error_reset()to reset the slot.
Not exactly. In pci_bus_error_reset(), we call pci_slot_reset() only if it's hotpluggable. But we always call pci_bus_reset() to perform a secondary bus reset for the bridge. That's what I think is unnecessary for a normal link,
and that's what reset link indicates us to do. The slot reset is introduced
in the process only to solve side effects. (c4eed62a2143, PCI/ERR: Use slot reset if available)
PCI_ERS_RESULT_NEED_RESET indicates that the driver
wants a platform-dependent slot reset and its ->slot_reset() method to be called then. I don't think it's same as slot reset mentioned above, which is only for hotpluggable ones.
Previously, if link is normal and the driver reports PCI_ERS_RESULT_NEED_RESET, we'll only call ->slot_reset() without slot reset in reset_link(). Maybe it's better to perform just like before.
Thanks.
Quoted 34 lines
Quoted 32 lines
Furthermore,
PCI_ERS_RESULT_NEED_RESETmeans device driver requires a slot reset rather than a link reset, so it maybe improper to use it to judge whether a link reset is needed. We decide whether to do a link reset only by the io state.Thanks,
YicongQuoted 22 lines
Quoted 20 lines
Thanks,
YicongQuoted 14 lines
drivers/pci/pcie/err.c +7
-2+ status =PCI_ERS_RESULT_DISCONNECT ; + }else { +if (pci_bus_error_reset (dev)) + status =PCI_ERS_RESULT_DISCONNECT ; + } + +if (status ==PCI_ERS_RESULT_DISCONNECT ) {pci_warn (dev,"link reset failed\n" ); goto failed; } - }else { -pci_walk_bus (bus, report_normal_detected, &status); }if (status ==PCI_ERS_RESULT_CAN_RECOVER ) {.
.
Re: [PATCH v1 1/1] PCI/ERR: Handle fatal error recovery for non-hotplug capable devices
Hi,
Quoted 85 lines · Yicong Yang
Quoted 73 lines · Sathyanarayanan
Quoted 70 lines · Yicong Yang
Quoted 68 lines · Sathyanarayanan
Quoted 54 lines · Yicong Yang
Quoted 53 lines · On 2020/5/7 11:32, [email protected] wrote:
From: Kuppuswamy Sathyanarayanan <[email protected]>
If there are non-hotplug capable devices connected to a given port, then during the fatal error recovery(triggered by DPC or AER), after calling
reset_link()function, we cannot rely on hotplug handler to detach and re-enumerate the device drivers in the affected bus. Instead, we will have to let the error recovery handler callreport_slot_reset()for all devices in the bus to notify about the reset operation. Although this is only required for non hot-plug capable devices, doing it for hotplug capable devices should not affect the functionality.Along with above issue, this fix also applicable to following issue.
Commit
6d2c89441571("PCI/ERR: Update error status afterreset_link()") added support to store status ofreset_link()call. Although this fixed the error recovery issue observed if the initial value of error status isPCI_ERS_RESULT_DISCONNECTorPCI_ERS_RESULT_NO_AER_DRIVER, it also discarded the status result fromreport_frozen_detected. This can cause a failure to recover if_NEED_RESETis returned byreport_frozen_detectedandreport_slot_resetis not invoked.Such an event can be induced for testing purposes by reducing the
Max_Payload_Sizeof a PCIe bridge to less than that of a device downstream from the bridge, and then initiating I/O through the device, resulting in oversize transactions. In the presence of DPC, this results in a containment event and attempted reset and recovery viapcie_do_recovery. After6d2c89441571report_slot_resetis not invoked, and the device does not recover.[original patch is from [email protected]][original patch link https://lore.kernel.org/linux-pci/18609.1588812972@famine/]Fixes 6d2c89441571 ("PCI/ERR: Update error status after reset_link()")Signed-off-by Jay Vosburgh <[email protected]>Signed-off-by Kuppuswamy Sathyanarayanan <[email protected]>--- drivers/pci/pcie/err.c | 19 +++++++++++++++---- 1 file changed, 15 insertions(+), 4 deletions(-)drivers/pci/pcie/err.c +2
-2@@ -165,13 +165,24 @@ pci_ers_result_t pcie_do_recovery(struct pci_dev *dev,pci_dbg (dev,"broadcast error_detected message\n" );if (state == pci_channel_io_frozen) {pci_walk_bus (bus, report_frozen_detected, &status); - status =reset_link (dev); -if (status !=PCI_ERS_RESULT_RECOVERED ) { + status =PCI_ERS_RESULT_NEED_RESET ; + }else {Quoted 8 lines
drivers/pci/pcie/err.c pci_ers_result_t pcie_do_recovery(struct pci_dev *dev, · line 175
Quoted 6 lines
drivers/pci/pcie/err.c +6
-0+pci_walk_bus (bus, report_normal_detected, &status); + } + +if (status ==PCI_ERS_RESULT_NEED_RESET ) { +if (reset_link) { +if (reset_link (dev) !=PCI_ERS_RESULT_RECOVERED )we'll call
reset_link()only if link is frozen. so it may have problem here.you mean before this change right?
After this change,reset_link()will be called as long as status isPCI_ERS_RESULT_NEED_RESET.Quoted 8 lines
Quoted 2 lines
Yes. I think we should reset the link only if the io is blocked as before. There's no reason to reset a normal link.
Currently, only AER and DPC driver uses
pcie_do_recovery()call. So the possiblereset_linkoptions aredpc_reset_link()andaer_root_reset().In
dpc_reset_link()case, the link is already disabled and hence we don't need to do another reset. In case ofaer_root_reset()it usespci_bus_error_reset()to reset the slot.
Quoted 5 lines
Not exactly. In
pci_bus_error_reset(), we callpci_slot_reset()only if it's hotpluggable. But we always callpci_bus_reset()to perform a secondary bus reset for the bridge. That's what I think is unnecessary for a normal link,
and that's what reset link indicates us to do. The slot reset is introduced
in the process only to solve side effects. (c4eed62a2143, PCI/ERR: Use slot reset if available)
IIUC, pci_bus_reset() will do slot reset if its supported (hot-plug capable slots). If its not supported then it will attempt secondary bus reset. So secondary bus reset will be attempted only if slot reset is not supported.
Since reported_error_detected() requests us to do reset, we will have to attempt some kind of reset before we call ->slot_reset() right? What is the side effect in calling secondary bus reset?
Quoted 5 lines
PCI_ERS_RESULT_NEED_RESETindicates that the driver
wants a platform-dependent slot reset and its ->slot_reset() method to be called then. I don't think it's same as slot reset mentioned above, which is only for hotpluggable ones.
What you think is the correct reset implementation ? Is it something like this?
if (hotplug capable)
try_slot_reset()
else
do_nothing()
Quoted 43 lines
Previously, if link is normal and the driver reports
PCI_ERS_RESULT_NEED_RESET, we'll only call ->slot_reset() without slot reset inreset_link(). Maybe it's better to perform just like before.Thanks.
Quoted 34 lines
Quoted 32 lines
Furthermore,
PCI_ERS_RESULT_NEED_RESETmeans device driver requires a slot reset rather than a link reset, so it maybe improper to use it to judge whether a link reset is needed. We decide whether to do a link reset only by the io state.Thanks,
YicongQuoted 22 lines
Quoted 20 lines
Thanks,
YicongQuoted 14 lines
drivers/pci/pcie/err.c +7
-2+ status =PCI_ERS_RESULT_DISCONNECT ; + }else { +if (pci_bus_error_reset (dev)) + status =PCI_ERS_RESULT_DISCONNECT ; + } + +if (status ==PCI_ERS_RESULT_DISCONNECT ) {pci_warn (dev,"link reset failed\n" ); goto failed; } - }else { -pci_walk_bus (bus, report_normal_detected, &status); }if (status ==PCI_ERS_RESULT_CAN_RECOVER ) {.
.
Oliver O'Halloran
reviewerRe: [PATCH v1 1/1] PCI/ERR: Handle fatal error recovery for non-hotplug capable devices
Quoted 14 lines · Sathyanarayanan
Hi,
Quoted 9 lines · Yicong Yang
Quoted 1 line · Sathyanarayanan
Not exactly. In
pci_bus_error_reset(), we callpci_slot_reset()only if it's hotpluggable. But we always callpci_bus_reset()to perform a secondary bus reset for the bridge. That's what I think is unnecessary for a normal link,
and that's what reset link indicates us to do. The slot reset is introduced
in the process only to solve side effects. (c4eed62a2143, PCI/ERR: Use slot reset if available)
Quoted 7 lines
IIUC,
pci_bus_reset()will do slot reset if its supported (hot-plug capable slots). If its not supported then it will attempt secondary bus reset. So secondary bus reset will be attempted only if slot reset is not supported.Since
reported_error_detected()requests us to do reset, we will have to attempt some kind of reset before we call ->slot_reset() right?
Yes, the driver returns PCI_ERS_RESULT_NEED_RESET from ->error_detected() to indicate that it doesn't know how to recover from the error. How that reset is performed doesn't really matter, but it does need to happen.
Quoted 11 lines
Quoted 4 lines
PCI_ERS_RESULT_NEED_RESETindicates that the driver
wants a platform-dependent slot reset and its ->slot_reset() method to be called then. I don't think it's same as slot reset mentioned above, which is only for hotpluggable ones.What you think is the correct reset implementation ? Is it something like this?
if (hotplug capable)
try_slot_reset()
else
do_nothing()
Looks broken to me, but all the reset handling is a rat's nest so maybe I'm missing something. In the case of a DPC trip the link is disabled which has the side-effect of hot-resetting the downstream device. Maybe it's fine?
As an aside, why do we have both ->slot_reset() and ->reset_done() in the error handling callbacks? Seems like their roles are almost identical.
Oliver
Re: [PATCH v1 1/1] PCI/ERR: Handle fatal error recovery for non-hotplug capable devices
Hi,
Quoted 42 lines · Oliver O'Halloran
Quoted 14 lines · Sathyanarayanan
Hi,
Quoted 9 lines · Yicong Yang
Quoted 1 line · Sathyanarayanan
Not exactly. In
pci_bus_error_reset(), we callpci_slot_reset()only if it's hotpluggable. But we always callpci_bus_reset()to perform a secondary bus reset for the bridge. That's what I think is unnecessary for a normal link,
and that's what reset link indicates us to do. The slot reset is introduced
in the process only to solve side effects. (c4eed62a2143, PCI/ERR: Use slot reset if available)Quoted 7 lines
IIUC,
pci_bus_reset()will do slot reset if its supported (hot-plug capable slots). If its not supported then it will attempt secondary bus reset. So secondary bus reset will be attempted only if slot reset is not supported.Since
reported_error_detected()requests us to do reset, we will have to attempt some kind of reset before we call ->slot_reset() right?Yes, the driver returns
PCI_ERS_RESULT_NEED_RESETfrom ->error_detected() to indicate that it doesn't know how to recover from the error. How that reset is performed doesn't really matter, but it does need to happen.Quoted 11 lines
Quoted 4 lines
PCI_ERS_RESULT_NEED_RESETindicates that the driver
wants a platform-dependent slot reset and its ->slot_reset() method to be called then. I don't think it's same as slot reset mentioned above, which is only for hotpluggable ones.What you think is the correct reset implementation ? Is it something like this?
if (hotplug capable)
try_slot_reset()
else
do_nothing()
Quoted 4 lines
Looks broken to me, but all the reset handling is a rat's nest so maybe I'm missing something. In the case of a DPC trip the link is disabled which has the side-effect of hot-resetting the downstream device. Maybe it's fine?
Yes, in case of DPC (Fatal errors) link is already reset. So we don't need any special handling. This reset logic is mainly for non-fatal errors.
Quoted 4 lines
As an aside, why do we have both ->slot_reset() and ->reset_done() in the error handling callbacks? Seems like their roles are almost identical.
Not sure.I think reset_done() is final cleanup.
Quoted 3 lines
Oliver
Oliver O'Halloran
reviewerRe: [PATCH v1 1/1] PCI/ERR: Handle fatal error recovery for non-hotplug capable devices
Quoted 4 lines · Sathyanarayanan
Yes, in case of DPC (Fatal errors) link is already reset. So we don't need any special handling. This reset logic is mainly for non-fatal errors.
Why? In our experience most fatal errors aren't all that fatal and can be recovered by resetting the device. The base spec backs that up (see gen5 base, sec 6.2) too saying the main point of distinction between fatal and non-fatal errors is whether handling the error requires a reset or not. For EEH we always try to recover the device and only mark it as permanently failed once the devices goes over the max error threshold (5 errors per hour, by default). Doing something similar for (native) DPC would make sense IMO.
Re: [PATCH v1 1/1] PCI/ERR: Handle fatal error recovery for non-hotplug capable devices
In reply to Kuppuswamy, SathyanarayananHi,
Quoted 94 lines · Sathyanarayanan
Hi,
Quoted 85 lines · Yicong Yang
Quoted 73 lines · Sathyanarayanan
Quoted 70 lines · Yicong Yang
Quoted 68 lines · Sathyanarayanan
Quoted 54 lines · Yicong Yang
Quoted 53 lines · On 2020/5/7 11:32, [email protected] wrote:
From: Kuppuswamy Sathyanarayanan <[email protected]>
If there are non-hotplug capable devices connected to a given port, then during the fatal error recovery(triggered by DPC or AER), after calling
reset_link()function, we cannot rely on hotplug handler to detach and re-enumerate the device drivers in the affected bus. Instead, we will have to let the error recovery handler callreport_slot_reset()for all devices in the bus to notify about the reset operation. Although this is only required for non hot-plug capable devices, doing it for hotplug capable devices should not affect the functionality.Along with above issue, this fix also applicable to following issue.
Commit
6d2c89441571("PCI/ERR: Update error status afterreset_link()") added support to store status ofreset_link()call. Although this fixed the error recovery issue observed if the initial value of error status isPCI_ERS_RESULT_DISCONNECTorPCI_ERS_RESULT_NO_AER_DRIVER, it also discarded the status result fromreport_frozen_detected. This can cause a failure to recover if_NEED_RESETis returned byreport_frozen_detectedandreport_slot_resetis not invoked.Such an event can be induced for testing purposes by reducing the
Max_Payload_Sizeof a PCIe bridge to less than that of a device downstream from the bridge, and then initiating I/O through the device, resulting in oversize transactions. In the presence of DPC, this results in a containment event and attempted reset and recovery viapcie_do_recovery. After6d2c89441571report_slot_resetis not invoked, and the device does not recover.[original patch is from [email protected]][original patch link https://lore.kernel.org/linux-pci/18609.1588812972@famine/]Fixes 6d2c89441571 ("PCI/ERR: Update error status after reset_link()")Signed-off-by Jay Vosburgh <[email protected]>Signed-off-by Kuppuswamy Sathyanarayanan <[email protected]>--- drivers/pci/pcie/err.c | 19 +++++++++++++++---- 1 file changed, 15 insertions(+), 4 deletions(-)drivers/pci/pcie/err.c +2
-2@@ -165,13 +165,24 @@ pci_ers_result_t pcie_do_recovery(struct pci_dev *dev,pci_dbg (dev,"broadcast error_detected message\n" );if (state == pci_channel_io_frozen) {pci_walk_bus (bus, report_frozen_detected, &status); - status =reset_link (dev); -if (status !=PCI_ERS_RESULT_RECOVERED ) { + status =PCI_ERS_RESULT_NEED_RESET ; + }else {Quoted 8 lines
drivers/pci/pcie/err.c pci_ers_result_t pcie_do_recovery(struct pci_dev *dev, · line 175
Quoted 6 lines
drivers/pci/pcie/err.c +6
-0+pci_walk_bus (bus, report_normal_detected, &status); + } + +if (status ==PCI_ERS_RESULT_NEED_RESET ) { +if (reset_link) { +if (reset_link (dev) !=PCI_ERS_RESULT_RECOVERED )we'll call
reset_link()only if link is frozen. so it may have problem here.you mean before this change right?
After this change,reset_link()will be called as long as status isPCI_ERS_RESULT_NEED_RESET.Quoted 8 lines
Quoted 2 lines
Yes. I think we should reset the link only if the io is blocked as before. There's no reason to reset a normal link.
Currently, only AER and DPC driver uses
pcie_do_recovery()call. So the possiblereset_linkoptions aredpc_reset_link()andaer_root_reset().In
dpc_reset_link()case, the link is already disabled and hence we don't need to do another reset. In case ofaer_root_reset()it usespci_bus_error_reset()to reset the slot.Quoted 5 lines
Not exactly. In
pci_bus_error_reset(), we callpci_slot_reset()only if it's hotpluggable. But we always callpci_bus_reset()to perform a secondary bus reset for the bridge. That's what I think is unnecessary for a normal link,
and that's what reset link indicates us to do. The slot reset is introduced
in the process only to solve side effects. (c4eed62a2143, PCI/ERR: Use slot reset if available)
Quoted 8 lines
IIUC,
pci_bus_reset()will do slot reset if its supported (hot-plug capable slots). If its not supported then it will attempt secondary bus reset. So secondary bus reset will be attempted only if slot reset is not supported.Since
reported_error_detected()requests us to do reset, we will have to attempt some kind of reset before we call ->slot_reset() right? What is the side effect in calling secondary bus reset?
I agree we should do a slot reset if driver required. The question is if we apply the patch, think of a situation that the io is normal, the slot is not hotpluggable but driver reports a reset, then:
-->aer_root_reset()
----->pci_bus_error_reset()
---------> pci_bridge_secondary_bus_reset() // Is it necessary to reset if the link is not blocked?
Before commit (c4eed62a2143, PCI/ERR: Use slot reset if available), the reset_link() for aer is -->aer_root_reset()
----->pci_bridge_secondary_bus_reset()
As mentioned by the commit c4eed62a2143 "The secondary bus reset may have link side effects that a hotplug capable port may incorrectly react to. Use the slot specific reset for hotplug ports, fixing the undesirable link down-up handling during error recovering." So I assume it use hotplug slot reset rather than secondary bus reset to recover the link. If the link is normal, it's unnecessary to do so. so we should add a check before reset the link in the patch:
+ if (status == PCI_ERS_RESULT_NEED_RESET &&
+ state == pci_channel_io_frozen) {
We should do slot reset if driver required, but it's different from the slot reset in pci_bus_error_reset(). Previously we don't do a slot reset and call ->slot_reset() directly, I don't know the certain reason.
Thanks,
Yicong
Quoted 58 lines
Quoted 6 lines
PCI_ERS_RESULT_NEED_RESETindicates that the driver
wants a platform-dependent slot reset and its ->slot_reset() method to be called then. I don't think it's same as slot reset mentioned above, which is only for hotpluggable ones.What you think is the correct reset implementation ? Is it something like this?
if (hotplug capable)
try_slot_reset()
else
do_nothing()Quoted 43 lines
Previously, if link is normal and the driver reports
PCI_ERS_RESULT_NEED_RESET, we'll only call ->slot_reset() without slot reset inreset_link(). Maybe it's better to perform just like before.Thanks.
Quoted 34 lines
Quoted 32 lines
Furthermore,
PCI_ERS_RESULT_NEED_RESETmeans device driver requires a slot reset rather than a link reset, so it maybe improper to use it to judge whether a link reset is needed. We decide whether to do a link reset only by the io state.Thanks,
YicongQuoted 22 lines
Quoted 20 lines
Thanks,
YicongQuoted 14 lines
drivers/pci/pcie/err.c +7
-2+ status =PCI_ERS_RESULT_DISCONNECT ; + }else { +if (pci_bus_error_reset (dev)) + status =PCI_ERS_RESULT_DISCONNECT ; + } + +if (status ==PCI_ERS_RESULT_DISCONNECT ) {pci_warn (dev,"link reset failed\n" ); goto failed; } - }else { -pci_walk_bus (bus, report_normal_detected, &status); }if (status ==PCI_ERS_RESULT_CAN_RECOVER ) {.
.
.
Re: [PATCH v1 1/1] PCI/ERR: Handle fatal error recovery for non-hotplug capable devices
Quoted 124 lines · Yicong Yang
Hi,
Quoted 94 lines · Sathyanarayanan
Hi,
Quoted 85 lines · Yicong Yang
Quoted 73 lines · Sathyanarayanan
Quoted 70 lines · Yicong Yang
Quoted 68 lines · Sathyanarayanan
Quoted 54 lines · Yicong Yang
Quoted 53 lines · On 2020/5/7 11:32, [email protected] wrote:
From: Kuppuswamy Sathyanarayanan <[email protected]>
If there are non-hotplug capable devices connected to a given port, then during the fatal error recovery(triggered by DPC or AER), after calling
reset_link()function, we cannot rely on hotplug handler to detach and re-enumerate the device drivers in the affected bus. Instead, we will have to let the error recovery handler callreport_slot_reset()for all devices in the bus to notify about the reset operation. Although this is only required for non hot-plug capable devices, doing it for hotplug capable devices should not affect the functionality.Along with above issue, this fix also applicable to following issue.
Commit
6d2c89441571("PCI/ERR: Update error status afterreset_link()") added support to store status ofreset_link()call. Although this fixed the error recovery issue observed if the initial value of error status isPCI_ERS_RESULT_DISCONNECTorPCI_ERS_RESULT_NO_AER_DRIVER, it also discarded the status result fromreport_frozen_detected. This can cause a failure to recover if_NEED_RESETis returned byreport_frozen_detectedandreport_slot_resetis not invoked.Such an event can be induced for testing purposes by reducing the
Max_Payload_Sizeof a PCIe bridge to less than that of a device downstream from the bridge, and then initiating I/O through the device, resulting in oversize transactions. In the presence of DPC, this results in a containment event and attempted reset and recovery viapcie_do_recovery. After6d2c89441571report_slot_resetis not invoked, and the device does not recover.[original patch is from [email protected]][original patch link https://lore.kernel.org/linux-pci/18609.1588812972@famine/]Fixes 6d2c89441571 ("PCI/ERR: Update error status after reset_link()")Signed-off-by Jay Vosburgh <[email protected]>Signed-off-by Kuppuswamy Sathyanarayanan <[email protected]>--- drivers/pci/pcie/err.c | 19 +++++++++++++++---- 1 file changed, 15 insertions(+), 4 deletions(-)drivers/pci/pcie/err.c +2
-2@@ -165,13 +165,24 @@ pci_ers_result_t pcie_do_recovery(struct pci_dev *dev,pci_dbg (dev,"broadcast error_detected message\n" );if (state == pci_channel_io_frozen) {pci_walk_bus (bus, report_frozen_detected, &status); - status =reset_link (dev); -if (status !=PCI_ERS_RESULT_RECOVERED ) { + status =PCI_ERS_RESULT_NEED_RESET ; + }else {Quoted 8 lines
drivers/pci/pcie/err.c pci_ers_result_t pcie_do_recovery(struct pci_dev *dev, · line 175
Quoted 6 lines
drivers/pci/pcie/err.c +6
-0+pci_walk_bus (bus, report_normal_detected, &status); + } + +if (status ==PCI_ERS_RESULT_NEED_RESET ) { +if (reset_link) { +if (reset_link (dev) !=PCI_ERS_RESULT_RECOVERED )we'll call
reset_link()only if link is frozen. so it may have problem here.you mean before this change right?
After this change,reset_link()will be called as long as status isPCI_ERS_RESULT_NEED_RESET.Quoted 8 lines
Quoted 2 lines
Yes. I think we should reset the link only if the io is blocked as before. There's no reason to reset a normal link.
Currently, only AER and DPC driver uses
pcie_do_recovery()call. So the possiblereset_linkoptions aredpc_reset_link()andaer_root_reset().In
dpc_reset_link()case, the link is already disabled and hence we don't need to do another reset. In case ofaer_root_reset()it usespci_bus_error_reset()to reset the slot.Quoted 5 lines
Not exactly. In
pci_bus_error_reset(), we callpci_slot_reset()only if it's hotpluggable. But we always callpci_bus_reset()to perform a secondary bus reset for the bridge. That's what I think is unnecessary for a normal link,
and that's what reset link indicates us to do. The slot reset is introduced
in the process only to solve side effects. (c4eed62a2143, PCI/ERR: Use slot reset if available)Quoted 8 lines
IIUC,
pci_bus_reset()will do slot reset if its supported (hot-plug capable slots). If its not supported then it will attempt secondary bus reset. So secondary bus reset will be attempted only if slot reset is not supported.Since
reported_error_detected()requests us to do reset, we will have to attempt some kind of reset before we call ->slot_reset() right? What is the side effect in calling secondary bus reset?I agree we should do a slot reset if driver required. The question is if we apply the patch, think of a situation that the io is normal, the slot is not hotpluggable but driver reports a reset, then:
-->aer_root_reset()
----->pci_bus_error_reset()
--------->pci_bridge_secondary_bus_reset()// Is it necessary to reset if the link is not blocked?Before commit (
c4eed62a2143, PCI/ERR: Use slot reset if available), thereset_link()for aer is -->aer_root_reset()
----->pci_bridge_secondary_bus_reset()As mentioned by the commit
c4eed62a2143"The secondary bus reset may have link side effects that a hotplug capable port may incorrectly react to. Use the slot specific reset for hotplug ports, fixing the undesirable link down-up handling during error recovering." So I assume it use hotplug slot reset rather than secondary bus reset to recover the link. If the link is normal, it's unnecessary to do so. so we should add a check before reset the link in the patch:
Quoted 6 lines
+ if (status ==
PCI_ERS_RESULT_NEED_RESET&&
+ state ==pci_channel_io_frozen) {We should do slot reset if driver required, but it's different from the
slot resetinpci_bus_error_reset(). Previously we don't do a slot reset and call ->slot_reset() directly, I don't know the certain reason.
IIUC, your concern is whether it is correct to trigger reset for pci_channel_io_normal case right ? Please correct me if my assumption is incorrect.
If its true, then why would report_error_detected() will return PCI_ERS_*_NEED_RESET for pci_channel_io_normal case ? If report_error_detected() requests reset in pci_channel_io_normal case then I think we should give preference to it.
Let me know your comments.
Quoted 62 lines
Thanks,
YicongQuoted 58 lines
Quoted 6 lines
PCI_ERS_RESULT_NEED_RESETindicates that the driver
wants a platform-dependent slot reset and its ->slot_reset() method to be called then. I don't think it's same as slot reset mentioned above, which is only for hotpluggable ones.What you think is the correct reset implementation ? Is it something like this?
if (hotplug capable)
try_slot_reset()
else
do_nothing()Quoted 43 lines
Previously, if link is normal and the driver reports
PCI_ERS_RESULT_NEED_RESET, we'll only call ->slot_reset() without slot reset inreset_link(). Maybe it's better to perform just like before.Thanks.
Quoted 34 lines
Quoted 32 lines
Furthermore,
PCI_ERS_RESULT_NEED_RESETmeans device driver requires a slot reset rather than a link reset, so it maybe improper to use it to judge whether a link reset is needed. We decide whether to do a link reset only by the io state.Thanks,
YicongQuoted 22 lines
Quoted 20 lines
Thanks,
YicongQuoted 14 lines
drivers/pci/pcie/err.c +7
-2+ status =PCI_ERS_RESULT_DISCONNECT ; + }else { +if (pci_bus_error_reset (dev)) + status =PCI_ERS_RESULT_DISCONNECT ; + } + +if (status ==PCI_ERS_RESULT_DISCONNECT ) {pci_warn (dev,"link reset failed\n" ); goto failed; } - }else { -pci_walk_bus (bus, report_normal_detected, &status); }if (status ==PCI_ERS_RESULT_CAN_RECOVER ) {.
.
.
Re: [PATCH v1 1/1] PCI/ERR: Handle fatal error recovery for non-hotplug capable devices
Quoted 131 lines · Sathyanarayanan
Quoted 128 lines · Yicong Yang
Hi,
Quoted 94 lines · Sathyanarayanan
Hi,
Quoted 85 lines · Yicong Yang
Quoted 73 lines · Sathyanarayanan
Quoted 70 lines · Yicong Yang
Quoted 68 lines · Sathyanarayanan
Quoted 54 lines · Yicong Yang
Quoted 53 lines · On 2020/5/7 11:32, [email protected] wrote:
From: Kuppuswamy Sathyanarayanan <[email protected]>
If there are non-hotplug capable devices connected to a given port, then during the fatal error recovery(triggered by DPC or AER), after calling
reset_link()function, we cannot rely on hotplug handler to detach and re-enumerate the device drivers in the affected bus. Instead, we will have to let the error recovery handler callreport_slot_reset()for all devices in the bus to notify about the reset operation. Although this is only required for non hot-plug capable devices, doing it for hotplug capable devices should not affect the functionality.Along with above issue, this fix also applicable to following issue.
Commit
6d2c89441571("PCI/ERR: Update error status afterreset_link()") added support to store status ofreset_link()call. Although this fixed the error recovery issue observed if the initial value of error status isPCI_ERS_RESULT_DISCONNECTorPCI_ERS_RESULT_NO_AER_DRIVER, it also discarded the status result fromreport_frozen_detected. This can cause a failure to recover if_NEED_RESETis returned byreport_frozen_detectedandreport_slot_resetis not invoked.Such an event can be induced for testing purposes by reducing the
Max_Payload_Sizeof a PCIe bridge to less than that of a device downstream from the bridge, and then initiating I/O through the device, resulting in oversize transactions. In the presence of DPC, this results in a containment event and attempted reset and recovery viapcie_do_recovery. After6d2c89441571report_slot_resetis not invoked, and the device does not recover.[original patch is from [email protected]][original patch link https://lore.kernel.org/linux-pci/18609.1588812972@famine/]Fixes 6d2c89441571 ("PCI/ERR: Update error status after reset_link()")Signed-off-by Jay Vosburgh <[email protected]>Signed-off-by Kuppuswamy Sathyanarayanan <[email protected]>--- drivers/pci/pcie/err.c | 19 +++++++++++++++---- 1 file changed, 15 insertions(+), 4 deletions(-)drivers/pci/pcie/err.c +2
-2@@ -165,13 +165,24 @@ pci_ers_result_t pcie_do_recovery(struct pci_dev *dev,pci_dbg (dev,"broadcast error_detected message\n" );if (state == pci_channel_io_frozen) {pci_walk_bus (bus, report_frozen_detected, &status); - status =reset_link (dev); -if (status !=PCI_ERS_RESULT_RECOVERED ) { + status =PCI_ERS_RESULT_NEED_RESET ; + }else {Quoted 8 lines
drivers/pci/pcie/err.c pci_ers_result_t pcie_do_recovery(struct pci_dev *dev, · line 175
Quoted 6 lines
drivers/pci/pcie/err.c +6
-0+pci_walk_bus (bus, report_normal_detected, &status); + } + +if (status ==PCI_ERS_RESULT_NEED_RESET ) { +if (reset_link) { +if (reset_link (dev) !=PCI_ERS_RESULT_RECOVERED )we'll call
reset_link()only if link is frozen. so it may have problem here.you mean before this change right?
After this change,reset_link()will be called as long as status isPCI_ERS_RESULT_NEED_RESET.Quoted 8 lines
Quoted 2 lines
Yes. I think we should reset the link only if the io is blocked as before. There's no reason to reset a normal link.
Currently, only AER and DPC driver uses
pcie_do_recovery()call. So the possiblereset_linkoptions aredpc_reset_link()andaer_root_reset().In
dpc_reset_link()case, the link is already disabled and hence we don't need to do another reset. In case ofaer_root_reset()it usespci_bus_error_reset()to reset the slot.Quoted 5 lines
Not exactly. In
pci_bus_error_reset(), we callpci_slot_reset()only if it's hotpluggable. But we always callpci_bus_reset()to perform a secondary bus reset for the bridge. That's what I think is unnecessary for a normal link,
and that's what reset link indicates us to do. The slot reset is introduced
in the process only to solve side effects. (c4eed62a2143, PCI/ERR: Use slot reset if available)Quoted 8 lines
IIUC,
pci_bus_reset()will do slot reset if its supported (hot-plug capable slots). If its not supported then it will attempt secondary bus reset. So secondary bus reset will be attempted only if slot reset is not supported.Since
reported_error_detected()requests us to do reset, we will have to attempt some kind of reset before we call ->slot_reset() right? What is the side effect in calling secondary bus reset?I agree we should do a slot reset if driver required. The question is if we apply the patch, think of a situation that the io is normal, the slot is not hotpluggable but driver reports a reset, then:
-->aer_root_reset()
----->pci_bus_error_reset()
--------->pci_bridge_secondary_bus_reset()// Is it necessary to reset if the link is not blocked?Before commit (
c4eed62a2143, PCI/ERR: Use slot reset if available), thereset_link()for aer is -->aer_root_reset()
----->pci_bridge_secondary_bus_reset()As mentioned by the commit
c4eed62a2143"The secondary bus reset may have link side effects that a hotplug capable port may incorrectly react to. Use the slot specific reset for hotplug ports, fixing the undesirable link down-up handling during error recovering." So I assume it use hotplug slot reset rather than secondary bus reset to recover the link. If the link is normal, it's unnecessary to do so. so we should add a check before reset the link in the patch:+ if (status ==
PCI_ERS_RESULT_NEED_RESET&&
+ state ==pci_channel_io_frozen) {
Quoted 5 lines
Quoted 2 lines
We should do slot reset if driver required, but it's different from the
slot resetinpci_bus_error_reset(). Previously we don't do a slot reset and call ->slot_reset() directly, I don't know the certain reason.IIUC, your concern is whether it is correct to trigger reset for
pci_channel_io_normalcase right ? Please correct me if my assumption is incorrect.
right.
Quoted 5 lines
If its true, then why would
report_error_detected()will return PCI_ERS_*_NEED_RESET forpci_channel_io_normalcase ? Ifreport_error_detected()requests reset inpci_channel_io_normalcase then I think we should give preference to it.
If we get PCI_ERS_*_NEED_RESET, we should do slot reset, no matter it's a hotpluggable slot or not. And we shouldn't do it here in reset_link(), that's two separate things. The slot reset done in aer_root_reset() is only for *link reset*, as there may have some side effects to perform secondary bus reset directly for hotpluggable slot, as mentioned in commit c4eed62a2143, so it use slot reset to do the reset link things.
As for slot reset required by the driver, we should perform it later just before the ->slot_reset(). I noticed the TODO comments there and we should implement it if it's necessary.
It lies in line 183, drivers/pcie/err.c:
if (status == PCI_ERS_RESULT_NEED_RESET) {
/*
* TODO: Should call platform-specific
* functions to reset slot before calling
* drivers' slot_reset callbacks?
*/
status = PCI_ERS_RESULT_RECOVERED;
pci_dbg(dev, "broadcast slot_reset message\n");
pci_walk_bus(bus, report_slot_reset, &status);
}Quoted 66 lines
Let me know your comments.
Quoted 62 lines
Thanks,
YicongQuoted 58 lines
Quoted 6 lines
PCI_ERS_RESULT_NEED_RESETindicates that the driver
wants a platform-dependent slot reset and its ->slot_reset() method to be called then. I don't think it's same as slot reset mentioned above, which is only for hotpluggable ones.What you think is the correct reset implementation ? Is it something like this?
if (hotplug capable)
try_slot_reset()
else
do_nothing()Quoted 43 lines
Previously, if link is normal and the driver reports
PCI_ERS_RESULT_NEED_RESET, we'll only call ->slot_reset() without slot reset inreset_link(). Maybe it's better to perform just like before.Thanks.
Quoted 34 lines
Quoted 32 lines
Furthermore,
PCI_ERS_RESULT_NEED_RESETmeans device driver requires a slot reset rather than a link reset, so it maybe improper to use it to judge whether a link reset is needed. We decide whether to do a link reset only by the io state.Thanks,
YicongQuoted 22 lines
Quoted 20 lines
Thanks,
YicongQuoted 14 lines
drivers/pci/pcie/err.c +7
-2+ status =PCI_ERS_RESULT_DISCONNECT ; + }else { +if (pci_bus_error_reset (dev)) + status =PCI_ERS_RESULT_DISCONNECT ; + } + +if (status ==PCI_ERS_RESULT_DISCONNECT ) {pci_warn (dev,"link reset failed\n" ); goto failed; } - }else { -pci_walk_bus (bus, report_normal_detected, &status); }if (status ==PCI_ERS_RESULT_CAN_RECOVER ) {.
.
.
.
Re: [PATCH v1 1/1] PCI/ERR: Handle fatal error recovery for non-hotplug capable devices
Quoted 5 lines · Yicong Yang
Quoted 5 lines
Quoted 2 lines
We should do slot reset if driver required, but it's different from the
slot resetinpci_bus_error_reset(). Previously we don't do a slot reset and call ->slot_reset() directly, I don't know the certain reason.IIUC, your concern is whether it is correct to trigger reset for
pci_channel_io_normalcase right ? Please correct me if my assumption is incorrect.
Quoted 8 lines
right.
Quoted 4 lines
If its true, then why would
report_error_detected()will return PCI_ERS_*_NEED_RESET forpci_channel_io_normalcase ? Ifreport_error_detected()requests reset inpci_channel_io_normalcase then I think we should give preference to it.If we get PCI_ERS_*_NEED_RESET, we should do slot reset, no matter it's a hotpluggable slot or not.
pci_slot_reset() function itself has dependency on hotplug ops. So what kind of slot reset is needed for non-hotplug case?
static int pci_slot_reset(struct pci_slot *slot, int probe)
{
int rc;
if (!slot || !pci_slot_resetable(slot))
return -ENOTTY;
if (!probe)
pci_slot_lock(slot);
might_sleep();
rc = pci_reset_hotplug_slot(slot->hotplug, probe);
if (!probe)
pci_slot_unlock(slot);
return rc;
}static int pci_reset_hotplug_slot(struct hotplug_slot *hotplug, int probe)
{
int rc = -ENOTTY;
if (!hotplug || !try_module_get(hotplug->owner))
return rc;
if (hotplug->ops->reset_slot)
rc = hotplug->ops->reset_slot(hotplug, probe);
module_put(hotplug->owner);
return rc;
} And we shouldn't do it here in reset_link(), that's
Quoted 8 lines
two separate things. The
slot resetdone inaer_root_reset()is only for *link reset*, as there may have some side effects to perform secondary bus reset directly for hotpluggable slot, as mentioned in commitc4eed62a2143, so it use slot reset to do the reset link things.As for slot reset required by the driver, we should perform it later just before the ->slot_reset(). I noticed the TODO comments there and we should implement it if it's necessary.
I agree.
Quoted 15 lines
It lies in line 183,
drivers/pcie/err.c:if (status == PCI_ERS_RESULT_NEED_RESET) { /* * TODO: Should call platform-specific * functions to reset slot before calling * drivers' slot_reset callbacks? */ status = PCI_ERS_RESULT_RECOVERED; pci_dbg(dev, "broadcast slot_reset message\n"); pci_walk_bus(bus, report_slot_reset, &status); }