powerpc/eeh: Only dump stack once if an MMIO loop is detected
Subsystems (2)
Oliver O'Halloran
reviewerPATCH 1/1 powerpc/eeh: Only dump stack once if an MMIO loop is detected
Many drivers don't check for errors when they get a 0xFFs response from an MMIO load. As a result after an EEH event occurs a driver can get stuck in a polling loop unless it some kind of internal timeout logic.
Currently EEH tries to detect and report stuck drivers by dumping a stack trace after eeh_dev_check_failure() is called EEH_MAX_FAILS times on an already frozen PE. The value of EEH_MAX_FAILS was chosen so that a dump would occur every few seconds if the driver was spinning in a loop. This results in a lot of spurious stack traces in the kernel log.
Fix this by limiting it to printing one stack trace for each PE freeze. If the driver is truely stuck the kernel's hung task detector is better suited to reporting the probelm anyway.
--- arch/powerpc/kernel/eeh.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/powerpc/kernel/eeh.c +1 -1
@@ -503,7 +503,7 @@ int eeh_dev_check_failure(struct eeh_dev *edev) rc =1 ;if (pe->state &EEH_PE_ISOLATED ) { pe->check_count++; -if (pe->check_count %EEH_MAX_FAILS ==0 ) { +if (pe->check_count ==EEH_MAX_FAILS ) { dn =pci_device_to_OF_node (dev);if (dn) location = of_get_property(dn, "ibm,loc-code",
Signature · 1 line
2.21.0
Quoted 10 lines · Oliver O'Halloran
Many drivers don't check for errors when they get a 0xFFs response from an MMIO load. As a result after an EEH event occurs a driver can get stuck in a polling loop unless it some kind of internal timeout logic.
Currently EEH tries to detect and report stuck drivers by dumping a stack trace after
eeh_dev_check_failure()is calledEEH_MAX_FAILStimes on an already frozen PE. The value ofEEH_MAX_FAILSwas chosen so that a dump would occur every few seconds if the driver was spinning in a loop. This results in a lot of spurious stack traces in the kernel log.
Quoted 3 lines
Fix this by limiting it to printing one stack trace for each PE freeze. If the driver is truely stuck the kernel's hung task detector is better suited to reporting the probelm anyway.
problem
Quoted 3 lines
Cc Sam Bobroff <[email protected]>Signed-off-by Oliver O'Halloran <[email protected]>
Looks good to me (especially because if it's stuck in a loop the stack trace is going to be pretty much the same every time). I tested it by recovering a device that uses the mlx5_core driver.
Quoted 20 lines
--- arch/powerpc/kernel/eeh.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)arch/powerpc/kernel/eeh.c +1
-1@@ -503,7 +503,7 @@ int eeh_dev_check_failure(struct eeh_dev *edev) rc =1 ;if (pe->state &EEH_PE_ISOLATED ) { pe->check_count++; -if (pe->check_count %EEH_MAX_FAILS ==0 ) { +if (pe->check_count ==EEH_MAX_FAILS ) { dn =pci_device_to_OF_node (dev);if (dn) location = of_get_property(dn, "ibm,loc-code",Signature · 1 line
2.21.0
Michael Ellerman
AppliedQuoted 10 lines · Oliver O'Halloran
Many drivers don't check for errors when they get a 0xFFs response from an MMIO load. As a result after an EEH event occurs a driver can get stuck in a polling loop unless it some kind of internal timeout logic.
Currently EEH tries to detect and report stuck drivers by dumping a stack trace after
eeh_dev_check_failure()is calledEEH_MAX_FAILStimes on an already frozen PE. The value ofEEH_MAX_FAILSwas chosen so that a dump would occur every few seconds if the driver was spinning in a loop. This results in a lot of spurious stack traces in the kernel log.
Quoted 6 lines
Fix this by limiting it to printing one stack trace for each PE freeze. If the driver is truely stuck the kernel's hung task detector is better suited to reporting the probelm anyway.
Cc Sam Bobroff <[email protected]>Signed-off-by Oliver O'Halloran <[email protected]>
Applied to powerpc next, thanks.
https://git.kernel.org/powerpc/c/4e0942c0302b5ad76b228b1a7b8c09f658a1d58a
cheers