Skip to contentSkip to conversation

watchdog: touch_nmi_watchdog should only touch local cpu not every one

PATCH In review Don Zickus 3 participants 8 messages Nov 8, 2010 – Nov 15, 2010 Documentation

lore
Share & tools
lore

Don Zickus

author
#0 lore raw

PATCH v3 1/1 watchdog: touch_nmi_watchdog should only touch local cpu not every one

I ran into a scenario where while one cpu was stuck and should have panic'd because of the NMI watchdog, it didn't. The reason was another cpu was spewing stack dumps on to the console. Upon investigation, I noticed that when writing to the console and also when dumping the stack, the watchdog is touched.

This causes all the cpus to reset their NMI watchdog flags and the 'stuck' cpu just spins forever.

This change causes the semantics of touch_nmi_watchdog to be changed slightly. Previously, I accidentally changed the semantics and we noticed there was a codepath in which touch_nmi_watchdog could be touched from a preemtible area. That caused a BUG() to happen when CONFIG_DEBUG_PREEMPT was enabled. I believe it was the acpi code.

My attempt here re-introduces the change to have the touch_nmi_watchdog() code only touch the local cpu instead of all of the cpus. But instead of using __get_cpu_var(), I use the __raw_get_cpu_var() version.

This avoids the preemption problem. However my reasoning wasn't because I was trying to be lazy. Instead I rationalized it as, well if preemption is enabled then interrupts should be enabled to and the NMI watchdog will have no reason to trigger. So it won't matter if the wrong cpu is touched because the percpu interrupt counters the NMI watchdog uses should still be incrementing.

V3: Really remove touch_all_nmi_watchdogs()
V2: Remove touch_all_nmi_watchdogs()

---
 kernel/watchdog.c |   17 +++++++++--------
 1 files changed, 9 insertions(+), 8 deletions(-)
kernel/watchdog.c +8 -7
index dc8e168..09fddd7 100644
@@ -141,14 +141,15 @@ void touch_all_softlockup_watchdogs(void)
 #ifdef CONFIG_HARDLOCKUP_DETECTOR
 void touch_nmi_watchdog(void)
 {
-	if (watchdog_enabled) {
-		unsigned cpu;
+	/*
+	 * Using __raw here because some code paths have
+	 * preemption enabled.  If preemption is enabled
+	 * then interrupts should be enabled too, in which
+	 * case we shouldn't have to worry about the watchdog
+	 * going off.
+	 */
+	__raw_get_cpu_var(watchdog_nmi_touch) = true;
 
-		for_each_present_cpu(cpu) {
-			if (per_cpu(watchdog_nmi_touch, cpu) != true)
-				per_cpu(watchdog_nmi_touch, cpu) = true;
-		}
-	}
 	touch_softlockup_watchdog();
 }
 EXPORT_SYMBOL(touch_nmi_watchdog);
Signature · 1 line
1.7.2.3
#1 lore raw
Quoted 7 lines · Don Zickus

I ran into a scenario where while one cpu was stuck and should have panic'd because of the NMI watchdog, it didn't. The reason was another cpu was spewing stack dumps on to the console. Upon investigation, I noticed that when writing to the console and also when dumping the stack, the watchdog is touched.

This causes all the cpus to reset their NMI watchdog flags and the 'stuck' cpu just spins forever.

Hm, the flip side is that if a CPU is stuck spewing backtraces, we will now make all the other CPUs a lot more noisy - which might only 'lock up' because this CPU is stuck spewing oopses, right?

Andrew, what would be your preference?

Thanks,

Ingo

#2 lore raw
Quoted 8 lines · On Wed, 10 Nov 2010 08:49:41 +0100 Ingo Molnar <[email protected]> wrote:
Quoted 5 lines · Don Zickus

I ran into a scenario where while one cpu was stuck and should have panic'd because of the NMI watchdog, it didn't. The reason was another cpu was spewing stack dumps on to the console. Upon investigation, I noticed that when writing to the console and also when dumping the stack, the watchdog is touched.

Quoted 9 lines
Quoted 2 lines

This causes all the cpus to reset their NMI watchdog flags and the 'stuck' cpu just spins forever.

Hm, the flip side is that if a CPU is stuck spewing backtraces, we will now make all the other CPUs a lot more noisy - which might only 'lock up' because this CPU is stuck spewing oopses, right?

Andrew, what would be your preference?

I saw you were cc'ed and felt rather pleased that I wouldn't have to decide ;)

I guess the NMI really is a cpu-local concept rather than a machine-wide one. And touch_nmi_watchdog() says "this CPU isn't stuck", rather than "this CPU and all those others aren't stuck", yes?

author
#3 lore raw
In reply to Ingo Molnar
Quoted 8 lines · Ingo Molnar
Quoted 5 lines · Don Zickus

I ran into a scenario where while one cpu was stuck and should have panic'd because of the NMI watchdog, it didn't. The reason was another cpu was spewing stack dumps on to the console. Upon investigation, I noticed that when writing to the console and also when dumping the stack, the watchdog is touched.

Quoted 6 lines
Quoted 2 lines

This causes all the cpus to reset their NMI watchdog flags and the 'stuck' cpu just spins forever.

Hm, the flip side is that if a CPU is stuck spewing backtraces, we will now make all the other CPUs a lot more noisy - which might only 'lock up' because this CPU is stuck spewing oopses, right?

When you say the other CPUs will be a lot more noisy, is that because they are busy processing backtraces for the first cpu to spew? I guess I don't understand how the other CPUs could have their interrupts off the whole time while the first cpu is spewing a backtrace (just trying to educate myself).

Cheers,
Don

#4 lore raw
Quoted 8 lines · Don Zickus
Quoted 3 lines

Hm, the flip side is that if a CPU is stuck spewing backtraces, we will now make all the other CPUs a lot more noisy - which might only 'lock up' because this CPU is stuck spewing oopses, right?

When you say the other CPUs will be a lot more noisy, is that because they are busy processing backtraces for the first cpu to spew? I guess I don't understand how the other CPUs could have their interrupts off the whole time while the first cpu is spewing a backtrace (just trying to educate myself).

Say the kernel crashes on a CPU and keeps spewing new oopses, while write-holding tasklist_lock.

Any other CPU that delivers a signal from IRQ context, trying to take the tasklist_lock, will loop indefinitely until that crashing CPU releases the lock.

In that case the 'secondary' NMI warnings from all other CPUs (eventually every CPU gets stuck in such a scenario) will start spewing NMI lockup messages.

Dunno. Maybe we should do your change - but also have an option to 'shut up' the kernel after the first hard oops [not warning]. That would silence the secondary NMI watchdog messages as well.

Thanks,

Ingo

author
#5 lore raw
Quoted 18 lines · Ingo Molnar
Quoted 8 lines · Don Zickus
Quoted 3 lines

Hm, the flip side is that if a CPU is stuck spewing backtraces, we will now make all the other CPUs a lot more noisy - which might only 'lock up' because this CPU is stuck spewing oopses, right?

When you say the other CPUs will be a lot more noisy, is that because they are busy processing backtraces for the first cpu to spew? I guess I don't understand how the other CPUs could have their interrupts off the whole time while the first cpu is spewing a backtrace (just trying to educate myself).

Say the kernel crashes on a CPU and keeps spewing new oopses, while write-holding tasklist_lock.

Any other CPU that delivers a signal from IRQ context, trying to take the tasklist_lock, will loop indefinitely until that crashing CPU releases the lock.

Quoted 6 lines

In that case the 'secondary' NMI warnings from all other CPUs (eventually every CPU gets stuck in such a scenario) will start spewing NMI lockup messages.

Dunno. Maybe we should do your change - but also have an option to 'shut up' the kernel after the first hard oops [not warning]. That would silence the secondary NMI watchdog messages as well.

You mean inside the panic() routine? like a ratelimit?

Cheers,
Don

#6 lore raw
Quoted 19 lines · Don Zickus
Quoted 18 lines · Ingo Molnar
Quoted 8 lines · Don Zickus
Quoted 3 lines

Hm, the flip side is that if a CPU is stuck spewing backtraces, we will now make all the other CPUs a lot more noisy - which might only 'lock up' because this CPU is stuck spewing oopses, right?

When you say the other CPUs will be a lot more noisy, is that because they are busy processing backtraces for the first cpu to spew? I guess I don't understand how the other CPUs could have their interrupts off the whole time while the first cpu is spewing a backtrace (just trying to educate myself).

Say the kernel crashes on a CPU and keeps spewing new oopses, while write-holding tasklist_lock.

Any other CPU that delivers a signal from IRQ context, trying to take the tasklist_lock, will loop indefinitely until that crashing CPU releases the lock.

Quoted 8 lines
Quoted 6 lines

In that case the 'secondary' NMI warnings from all other CPUs (eventually every CPU gets stuck in such a scenario) will start spewing NMI lockup messages.

Dunno. Maybe we should do your change - but also have an option to 'shut up' the kernel after the first hard oops [not warning]. That would silence the secondary NMI watchdog messages as well.

You mean inside the panic() routine? like a ratelimit?

A ratelimit, but some really serious one - like only one crash displayed per 10 minutes, or so - to give the user time to make a picture of the first crash, if it's still visible on the screen or so.

Thanks,

Ingo

author
#7 lore raw
Quoted 9 lines · Ingo Molnar
Quoted 5 lines
Quoted 3 lines

Dunno. Maybe we should do your change - but also have an option to 'shut up' the kernel after the first hard oops [not warning]. That would silence the secondary NMI watchdog messages as well.

You mean inside the panic() routine? like a ratelimit?

A ratelimit, but some really serious one - like only one crash displayed per 10 minutes, or so - to give the user time to make a picture of the first crash, if it's still visible on the screen or so.

Well I hacked up something. It seems to work, but I am not sure I put the check in the right spot.

Comments welcome.

-----------8<---cut---8<-----

From: Don Zickus <[email protected]>
Date: Mon, 15 Nov 2010 13:09:31 -0500
Subject: [PATCH] panic: ratelimit panic messages

Sometimes when things go bad, so much spew is coming on the console it is hard to figure out what happened. This patch allows you to ratelimit the panic messages with the intent that the first panic message will provide the info we need to figure out what happened.

Adds new kernel param 'panic_ratelimit=on/<integer in seconds>'

---
 Documentation/kernel-parameters.txt |    6 ++++++
 kernel/panic.c                      |   30 ++++++++++++++++++++++++++++++
 2 files changed, 36 insertions(+), 0 deletions(-)
Documentation/kernel-parameters.txt +6 -0
index ed45e98..5d69064 100644
@@ -1815,6 +1815,12 @@ and is between 256 and 4096 characters. It is defined in the file
 	panic=		[KNL] Kernel behaviour on panic
 			Format: <timeout>
 
+	panic_ratelimit=	[KNL] ratelimit the panic messages
+			Useful for slowing down multiple panics to capture
+			the first one before it scrolls off the screen
+			Format: "on" or some integer in seconds
+			"on" defaults to 10 minutes
+
 	parkbd.port=	[HW] Parallel port number the keyboard adapter is
 			connected to, default is 0.
 			Format: <parport#>
kernel/panic.c +30 -0
index 4c13b1a..fe89e04 100644
@@ -23,6 +23,7 @@
 #include <linux/init.h>
 #include <linux/nmi.h>
 #include <linux/dmi.h>
+#include <linux/ratelimit.h>
 
 #define PANIC_TIMER_STEP 100
 #define PANIC_BLINK_SPD 18
@@ -48,6 +49,31 @@ static long no_blink(int state)
 long (*panic_blink)(int state);
 EXPORT_SYMBOL(panic_blink);
 
+/* setting default to 0 effectively disables it */
+DEFINE_RATELIMIT_STATE(panic_ratelimit_state, 0, 1);
+
+static int __init panic_ratelimit_setup(char *str)
+{
+	int interval;
+
+	if (!strncmp(str, "on", 2))
+		/* default to 10 minutes */
+		interval = 600 * HZ;
+	else
+		interval = simple_strtoul(str, NULL, 0) * HZ;
+
+	panic_ratelimit_state.interval = interval;
+	return 1;
+}
+__setup("panic_ratelimit=", panic_ratelimit_setup);
+
+static int __panic_ratelimit(const char *func)
+{
+	return ___ratelimit(&panic_ratelimit_state, func);
+}
+
+#define panic_ratelimit() __panic_ratelimit(__func__)
+
 /**
  *	panic - halt the system
  *	@fmt: The text string to print
@@ -63,6 +89,10 @@ NORET_TYPE void panic(const char * fmt, ...)
 	long i, i_next = 0;
 	int state = 0;
 
+	if (!panic_ratelimit())
+		for(;;)
+			cpu_relax();
+
 	/*
 	 * It's possible to come here directly from a panic-assertion and
 	 * not have preempt disabled. Some functions called from here want
Signature · 1 line
1.7.2.3