Skip to contentSkip to conversation

watchdog: touch_nmi_watchdog should only touch local cpu not every one

PATCH Superseded Don Zickus 4 participants 6 messages Nov 5, 2010 – Nov 8, 2010

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Don Zickus

author
#0 lore raw

PATCH 1/1 watchdog: touch_nmi_watchdog should only touch local cpu not every one

I ran into a scenario where while one cpu was stuck and should have panic'd because of the NMI watchdog, it didn't. The reason was another cpu was spewing stack dumps on to the console. Upon investigation, I noticed that when writing to the console and also when dumping the stack, the watchdog is touched.

This causes all the cpus to reset their NMI watchdog flags and the 'stuck' cpu just spins forever.

This change causes the semantics of touch_nmi_watchdog to be changed slightly. Previously, I accidentally changed the semantics and we noticed there was a codepath in which touch_nmi_watchdog could be touched from a preemtible area. That caused a BUG() to happen when CONFIG_DEBUG_PREEMPT was enabled. I believe it was the acpi code.

My attempt here re-introduces the change to have the touch_nmi_watchdog() code only touch the local cpu instead of all of the cpus. But instead of using __get_cpu_var(), I use the __raw_get_cpu_var() version.

This avoids the preemption problem. However my reasoning wasn't because I was trying to be lazy. Instead I rationalized it as, well if preemption is enabled then interrupts should be enabled to and the NMI watchdog will have no reason to trigger. So it won't matter if the wrong cpu is touched because the percpu interrupt counters the NMI watchdog uses should still be incrementing.

---
 kernel/watchdog.c |   17 ++++++++++++++++-
 1 files changed, 16 insertions(+), 1 deletions(-)
kernel/watchdog.c +16 -1
index dc8e168..dd0c140 100644
@@ -141,6 +141,21 @@ void touch_all_softlockup_watchdogs(void)
 #ifdef CONFIG_HARDLOCKUP_DETECTOR
 void touch_nmi_watchdog(void)
 {
+	/*
+	 * Using __raw here because some code paths have
+	 * preemption enabled.  If preemption is enabled
+	 * then interrupts should be enabled too, in which
+	 * case we shouldn't have to worry about the watchdog
+	 * going off.
+	 */
+	__raw_get_cpu_var(watchdog_nmi_touch) = true;
+
+	touch_softlockup_watchdog();
+}
+EXPORT_SYMBOL(touch_nmi_watchdog);
+
+void touch_all_nmi_watchdogs(void)
+{
 	if (watchdog_enabled) {
 		unsigned cpu;
 
@@ -151,7 +166,7 @@ void touch_nmi_watchdog(void)
 	}
 	touch_softlockup_watchdog();
 }
-EXPORT_SYMBOL(touch_nmi_watchdog);
+EXPORT_SYMBOL(touch_all_nmi_watchdogs);
 
 #endif
 
Signature · 1 line
1.7.2.3
#1 lore raw
Quoted 21 lines · Don Zickus

void touch_nmi_watchdog(void)
{

kernel/watchdog.c +15 -0
+	/*
+	 * Using __raw here because some code paths have
+	 * preemption enabled.  If preemption is enabled
+	 * then interrupts should be enabled too, in which
+	 * case we shouldn't have to worry about the watchdog
+	 * going off.
+	 */
+	__raw_get_cpu_var(watchdog_nmi_touch) = true;
+
+	touch_softlockup_watchdog();
+}
+EXPORT_SYMBOL(touch_nmi_watchdog);
+
+void touch_all_nmi_watchdogs(void)
+{
 	if (watchdog_enabled) {
 		unsigned cpu;

kernel/watchdog.c void touch_nmi_watchdog(void) · line 170

Quoted 6 lines
kernel/watchdog.c +1 -1
@@ -151,7 +166,7 @@ void touch_nmi_watchdog(void)
 	}
 	touch_softlockup_watchdog();
 }
-EXPORT_SYMBOL(touch_nmi_watchdog);
+EXPORT_SYMBOL(touch_all_nmi_watchdogs);
 

Hello,
Seems like no one is actually calling touch_all_nmi_watchdogs, as for now. Right?

Minor nit

	touch_all_nmi_watchdogs:
	...
	for_each_present_cpu(cpu) {
		if (per_cpu(watchdog_nmi_touch, cpu) != true)
			per_cpu(watchdog_nmi_touch, cpu) = true;
	}

which is, I belive, could be simplified to

	for_each_present_cpu(cpu) {
		per_cpu(watchdog_nmi_touch, cpu) = true;
	}



	Sergey
#2 lore raw
Quoted 21 lines · On Fri, 5 Nov 2010 15:51:18 +0200 Sergey Senozhatsky…
Quoted 20 lines · Don Zickus

void touch_nmi_watchdog(void)
{

kernel/watchdog.c +15 -0
+	/*
+	 * Using __raw here because some code paths have
+	 * preemption enabled.  If preemption is enabled
+	 * then interrupts should be enabled too, in which
+	 * case we shouldn't have to worry about the watchdog
+	 * going off.
+	 */
+	__raw_get_cpu_var(watchdog_nmi_touch) = true;
+
+	touch_softlockup_watchdog();
+}
+EXPORT_SYMBOL(touch_nmi_watchdog);
+
+void touch_all_nmi_watchdogs(void)
+{
 	if (watchdog_enabled) {
 		unsigned cpu;
Quoted 11 lines

kernel/watchdog.c void touch_nmi_watchdog(void) · line 170

Quoted 7 lines
kernel/watchdog.c +1 -1
@@ -151,7 +166,7 @@ void touch_nmi_watchdog(void)
 	}
 	touch_softlockup_watchdog();
 }
-EXPORT_SYMBOL(touch_nmi_watchdog);
+EXPORT_SYMBOL(touch_all_nmi_watchdogs);
 

Hello,
Seems like no one is actually calling touch_all_nmi_watchdogs, as for now. Right?

Yes, there doesn't seem a lot of point in adding the interface unless we have callers.

Quoted 11 lines

Minor nit

	touch_all_nmi_watchdogs:
	...
	for_each_present_cpu(cpu) {
		if (per_cpu(watchdog_nmi_touch, cpu) != true)
			per_cpu(watchdog_nmi_touch, cpu) = true;
	}
Quoted 4 lines

which is, I belive, could be simplified to

	for_each_present_cpu(cpu) {
		per_cpu(watchdog_nmi_touch, cpu) = true;
	}

We sometimes do this trick to avoid dirtying lots of cachelines which already held the correct value. It'll be extra-benefical when dealing with other CPU's data, I expect.

author
#3 lore raw
Quoted 36 lines · Andrew Morton
Quoted 21 lines · On Fri, 5 Nov 2010 15:51:18 +0200 Sergey Senozhatsky…
Quoted 20 lines · Don Zickus

void touch_nmi_watchdog(void)
{

kernel/watchdog.c +15 -0
+	/*
+	 * Using __raw here because some code paths have
+	 * preemption enabled.  If preemption is enabled
+	 * then interrupts should be enabled too, in which
+	 * case we shouldn't have to worry about the watchdog
+	 * going off.
+	 */
+	__raw_get_cpu_var(watchdog_nmi_touch) = true;
+
+	touch_softlockup_watchdog();
+}
+EXPORT_SYMBOL(touch_nmi_watchdog);
+
+void touch_all_nmi_watchdogs(void)
+{
 	if (watchdog_enabled) {
 		unsigned cpu;
Quoted 11 lines

kernel/watchdog.c void touch_nmi_watchdog(void) · line 170

Quoted 7 lines
kernel/watchdog.c +1 -1
@@ -151,7 +166,7 @@ void touch_nmi_watchdog(void)
 	}
 	touch_softlockup_watchdog();
 }
-EXPORT_SYMBOL(touch_nmi_watchdog);
+EXPORT_SYMBOL(touch_all_nmi_watchdogs);
 

Hello,
Seems like no one is actually calling touch_all_nmi_watchdogs, as for now. Right?

Quoted 2 lines

Yes, there doesn't seem a lot of point in adding the interface unless we have callers.

Yeah I wasn't sure how to deal with that. It didn't seem like any of the callers was relying on the fact that touch_nmi_watchdog() touched everyone. I just provided it as an option in case I misread someone's use of the touch_nmi_watchdog.

I'll repost and remove it then.

Thanks for the feedback.

Cheers,
Don

#4 lore raw
In reply to Don Zickus
Quoted 39 lines · Don Zickus

I ran into a scenario where while one cpu was stuck and should have panic'd because of the NMI watchdog, it didn't. The reason was another cpu was spewing stack dumps on to the console. Upon investigation, I noticed that when writing to the console and also when dumping the stack, the watchdog is touched.

This causes all the cpus to reset their NMI watchdog flags and the 'stuck' cpu just spins forever.

This change causes the semantics of touch_nmi_watchdog to be changed slightly. Previously, I accidentally changed the semantics and we noticed there was a codepath in which touch_nmi_watchdog could be touched from a preemtible area. That caused a BUG() to happen when CONFIG_DEBUG_PREEMPT was enabled. I believe it was the acpi code.

My attempt here re-introduces the change to have the touch_nmi_watchdog() code only touch the local cpu instead of all of the cpus. But instead of using __get_cpu_var(), I use the __raw_get_cpu_var() version.

This avoids the preemption problem. However my reasoning wasn't because I was trying to be lazy. Instead I rationalized it as, well if preemption is enabled then interrupts should be enabled to and the NMI watchdog will have no reason to trigger. So it won't matter if the wrong cpu is touched because the percpu interrupt counters the NMI watchdog uses should still be incrementing.

---
 kernel/watchdog.c |   17 ++++++++++++++++-
 1 files changed, 16 insertions(+), 1 deletions(-)
kernel/watchdog.c +2 -0
index dc8e168..dd0c140 100644
@@ -141,6 +141,21 @@ void touch_all_softlockup_watchdogs(void)
 #ifdef CONFIG_HARDLOCKUP_DETECTOR
 void touch_nmi_watchdog(void)
 {
+	/*
+	 * Using __raw here because some code paths have

kernel/watchdog.c void touch_all_softlockup_watchdogs(void) · line 155

Quoted 10 lines
kernel/watchdog.c +10 -0
+	 * preemption enabled.  If preemption is enabled
+	 * then interrupts should be enabled too, in which
+	 * case we shouldn't have to worry about the watchdog
+	 * going off.
+	 */
+	__raw_get_cpu_var(watchdog_nmi_touch) = true;
+
+	touch_softlockup_watchdog();
+}
+EXPORT_SYMBOL(touch_nmi_watchdog);

Did the old watchdog also touched every CPUs?

That doesn't appear to be a good thing, we may indeed miss hardlockups because of that.

And it seems you can drop touch_all_nmi_watchdogs() as, like others pointed out, there are no users of it.

author
#5 lore raw
Quoted 8 lines · Frederic Weisbecker
Quoted 8 lines
kernel/watchdog.c +8 -0
+	/*
+	 * Using __raw here because some code paths have
+	 * preemption enabled.  If preemption is enabled
+	 * then interrupts should be enabled too, in which
+	 * case we shouldn't have to worry about the watchdog
+	 * going off.
+	 */
+	__raw_get_cpu_var(watchdog_nmi_touch) = true;
Quoted 8 lines

kernel/watchdog.c void touch_all_softlockup_watchdogs(void) · line 155

Quoted 4 lines
kernel/watchdog.c +4 -0
+
+	touch_softlockup_watchdog();
+}
+EXPORT_SYMBOL(touch_nmi_watchdog);

Did the old watchdog also touched every CPUs?

Yeah, unfortunately.

Cheers,
Don