watchdog: touch_nmi_watchdog should only touch local cpu not every one
PATCH v2 1/1 watchdog: touch_nmi_watchdog should only touch local cpu not every one
I ran into a scenario where while one cpu was stuck and should have panic'd because of the NMI watchdog, it didn't. The reason was another cpu was spewing stack dumps on to the console. Upon investigation, I noticed that when writing to the console and also when dumping the stack, the watchdog is touched.
This causes all the cpus to reset their NMI watchdog flags and the 'stuck' cpu just spins forever.
This change causes the semantics of touch_nmi_watchdog to be changed slightly. Previously, I accidentally changed the semantics and we noticed there was a codepath in which touch_nmi_watchdog could be touched from a preemtible area. That caused a BUG() to happen when CONFIG_DEBUG_PREEMPT was enabled. I believe it was the acpi code.
My attempt here re-introduces the change to have the touch_nmi_watchdog() code only touch the local cpu instead of all of the cpus. But instead of using __get_cpu_var(), I use the __raw_get_cpu_var() version.
This avoids the preemption problem. However my reasoning wasn't because I was trying to be lazy. Instead I rationalized it as, well if preemption is enabled then interrupts should be enabled to and the NMI watchdog will have no reason to trigger. So it won't matter if the wrong cpu is touched because the percpu interrupt counters the NMI watchdog uses should still be incrementing.
V2: remove touch_all_nmi_watchdog code
--- kernel/watchdog.c | 17 ++++++++++++++++- 1 files changed, 16 insertions(+), 1 deletions(-)
kernel/watchdog.c +16 -1
@@ -141,6 +141,21 @@ void touch_all_softlockup_watchdogs(void)#ifdef CONFIG_HARDLOCKUP_DETECTOR void touch_nmi_watchdog (void ) { +/* +* Using __raw here because some code paths have +* preemption enabled. If preemption is enabled +* then interrupts should be enabled too, in which +* case we shouldn't have to worry about the watchdog +* going off. +*/ +__raw_get_cpu_var (watchdog_nmi_touch) = true; + +touch_softlockup_watchdog (); +} +EXPORT_SYMBOL (touch_nmi_watchdog); + +void touch_all_nmi_watchdogs (void ) +{if (watchdog_enabled) {unsigned cpu;
@@ -151,7 +166,7 @@ void touch_nmi_watchdog(void) }touch_softlockup_watchdog (); } -EXPORT_SYMBOL (touch_nmi_watchdog); +EXPORT_SYMBOL (touch_all_nmi_watchdogs);#endif
Signature · 1 line
1.7.2.3
Quoted 18 lines · Don Zickus
I ran into a scenario where while one cpu was stuck and should have panic'd because of the NMI watchdog, it didn't. The reason was another cpu was spewing stack dumps on to the console. Upon investigation, I noticed that when writing to the console and also when dumping the stack, the watchdog is touched.
This causes all the cpus to reset their NMI watchdog flags and the 'stuck' cpu just spins forever.
This change causes the semantics of
touch_nmi_watchdogto be changed slightly. Previously, I accidentally changed the semantics and we noticed there was a codepath in whichtouch_nmi_watchdogcould be touched from a preemtible area. That caused aBUG()to happen whenCONFIG_DEBUG_PREEMPTwas enabled. I believe it was the acpi code.My attempt here re-introduces the change to have the
touch_nmi_watchdog()code only touch the local cpu instead of all of the cpus. But instead of using__get_cpu_var(), I use the__raw_get_cpu_var()version.
Quoted 7 lines
This avoids the preemption problem. However my reasoning wasn't because I was trying to be lazy. Instead I rationalized it as, well if preemption is enabled then interrupts should be enabled to and the NMI watchdog will have no reason to trigger. So it won't matter if the wrong cpu is touched because the percpu interrupt counters the NMI watchdog uses should still be incrementing.
V2: remove
touch_all_nmi_watchdogcode
Are you sure you did? :)
Quoted 17 lines
kernel/watchdog.c +3
-1+void touch_all_nmi_watchdogs (void ) +{if (watchdog_enabled) {unsigned cpu;@@ -151,7 +166,7 @@ void touch_nmi_watchdog(void) }touch_softlockup_watchdog (); } -EXPORT_SYMBOL (touch_nmi_watchdog); +EXPORT_SYMBOL (touch_all_nmi_watchdogs);#endif Signature · 1 line
1.7.2.3
Quoted 25 lines · Frederic Weisbecker
Quoted 24 lines · Don Zickus
I ran into a scenario where while one cpu was stuck and should have panic'd because of the NMI watchdog, it didn't. The reason was another cpu was spewing stack dumps on to the console. Upon investigation, I noticed that when writing to the console and also when dumping the stack, the watchdog is touched.
This causes all the cpus to reset their NMI watchdog flags and the 'stuck' cpu just spins forever.
This change causes the semantics of
touch_nmi_watchdogto be changed slightly. Previously, I accidentally changed the semantics and we noticed there was a codepath in whichtouch_nmi_watchdogcould be touched from a preemtible area. That caused aBUG()to happen whenCONFIG_DEBUG_PREEMPTwas enabled. I believe it was the acpi code.My attempt here re-introduces the change to have the
touch_nmi_watchdog()code only touch the local cpu instead of all of the cpus. But instead of using__get_cpu_var(), I use the__raw_get_cpu_var()version.This avoids the preemption problem. However my reasoning wasn't because I was trying to be lazy. Instead I rationalized it as, well if preemption is enabled then interrupts should be enabled to and the NMI watchdog will have no reason to trigger. So it won't matter if the wrong cpu is touched because the percpu interrupt counters the NMI watchdog uses should still be incrementing.
Quoted 4 lines
Quoted 1 line
V2: remove
touch_all_nmi_watchdogcodeAre you sure you did? :)
Hmm.. Odd. I remember making and committing the changes. But now I can't find them.
Thanks for catching that! I'll send out a version 3.
Cheers,
Don