watchdog: touch_nmi_watchdog should only touch local cpu not every one
PATCH v3 1/1 watchdog: touch_nmi_watchdog should only touch local cpu not every one
I ran into a scenario where while one cpu was stuck and should have panic'd because of the NMI watchdog, it didn't. The reason was another cpu was spewing stack dumps on to the console. Upon investigation, I noticed that when writing to the console and also when dumping the stack, the watchdog is touched.
This causes all the cpus to reset their NMI watchdog flags and the 'stuck' cpu just spins forever.
This change causes the semantics of touch_nmi_watchdog to be changed slightly. Previously, I accidentally changed the semantics and we noticed there was a codepath in which touch_nmi_watchdog could be touched from a preemtible area. That caused a BUG() to happen when CONFIG_DEBUG_PREEMPT was enabled. I believe it was the acpi code.
My attempt here re-introduces the change to have the touch_nmi_watchdog() code only touch the local cpu instead of all of the cpus. But instead of using __get_cpu_var(), I use the __raw_get_cpu_var() version.
This avoids the preemption problem. However my reasoning wasn't because I was trying to be lazy. Instead I rationalized it as, well if preemption is enabled then interrupts should be enabled to and the NMI watchdog will have no reason to trigger. So it won't matter if the wrong cpu is touched because the percpu interrupt counters the NMI watchdog uses should still be incrementing.
V3: Really remove touch_all_nmi_watchdogs()
V2: Remove touch_all_nmi_watchdogs()
--- kernel/watchdog.c | 17 +++++++++-------- 1 files changed, 9 insertions(+), 8 deletions(-)
kernel/watchdog.c +8 -7
@@ -141,14 +141,15 @@ void touch_all_softlockup_watchdogs(void)#ifdef CONFIG_HARDLOCKUP_DETECTOR void touch_nmi_watchdog (void ) { -if (watchdog_enabled) { -unsigned cpu; +/* +* Using __raw here because some code paths have +* preemption enabled. If preemption is enabled +* then interrupts should be enabled too, in which +* case we shouldn't have to worry about the watchdog +* going off. +*/ +__raw_get_cpu_var (watchdog_nmi_touch) = true; -for_each_present_cpu (cpu) { -if (per_cpu (watchdog_nmi_touch, cpu) != true) -per_cpu (watchdog_nmi_touch, cpu) = true; - } - }touch_softlockup_watchdog (); }EXPORT_SYMBOL (touch_nmi_watchdog);
Signature · 1 line
1.7.2.3
Quoted 7 lines · Don Zickus
I ran into a scenario where while one cpu was stuck and should have panic'd because of the NMI watchdog, it didn't. The reason was another cpu was spewing stack dumps on to the console. Upon investigation, I noticed that when writing to the console and also when dumping the stack, the watchdog is touched.
This causes all the cpus to reset their NMI watchdog flags and the 'stuck' cpu just spins forever.
Hm, the flip side is that if a CPU is stuck spewing backtraces, we will now make all the other CPUs a lot more noisy - which might only 'lock up' because this CPU is stuck spewing oopses, right?
Andrew, what would be your preference?
Thanks,
Ingo
Quoted 8 lines · On Wed, 10 Nov 2010 08:49:41 +0100 Ingo Molnar <[email protected]> wrote:
Quoted 5 lines · Don Zickus
I ran into a scenario where while one cpu was stuck and should have panic'd because of the NMI watchdog, it didn't. The reason was another cpu was spewing stack dumps on to the console. Upon investigation, I noticed that when writing to the console and also when dumping the stack, the watchdog is touched.
Quoted 9 lines
Quoted 2 lines
This causes all the cpus to reset their NMI watchdog flags and the 'stuck' cpu just spins forever.
Hm, the flip side is that if a CPU is stuck spewing backtraces, we will now make all the other CPUs a lot more noisy - which might only 'lock up' because this CPU is stuck spewing oopses, right?
Andrew, what would be your preference?
I saw you were cc'ed and felt rather pleased that I wouldn't have to decide ;)
I guess the NMI really is a cpu-local concept rather than a machine-wide one. And touch_nmi_watchdog() says "this CPU isn't stuck", rather than "this CPU and all those others aren't stuck", yes?
Quoted 8 lines · Ingo Molnar
Quoted 5 lines · Don Zickus
I ran into a scenario where while one cpu was stuck and should have panic'd because of the NMI watchdog, it didn't. The reason was another cpu was spewing stack dumps on to the console. Upon investigation, I noticed that when writing to the console and also when dumping the stack, the watchdog is touched.
Quoted 6 lines
Quoted 2 lines
This causes all the cpus to reset their NMI watchdog flags and the 'stuck' cpu just spins forever.
Hm, the flip side is that if a CPU is stuck spewing backtraces, we will now make all the other CPUs a lot more noisy - which might only 'lock up' because this CPU is stuck spewing oopses, right?
When you say the other CPUs will be a lot more noisy, is that because they are busy processing backtraces for the first cpu to spew? I guess I don't understand how the other CPUs could have their interrupts off the whole time while the first cpu is spewing a backtrace (just trying to educate myself).
Cheers,
Don
Quoted 8 lines · Don Zickus
Quoted 3 lines
Hm, the flip side is that if a CPU is stuck spewing backtraces, we will now make all the other CPUs a lot more noisy - which might only 'lock up' because this CPU is stuck spewing oopses, right?
When you say the other CPUs will be a lot more noisy, is that because they are busy processing backtraces for the first cpu to spew? I guess I don't understand how the other CPUs could have their interrupts off the whole time while the first cpu is spewing a backtrace (just trying to educate myself).
Say the kernel crashes on a CPU and keeps spewing new oopses, while write-holding tasklist_lock.
Any other CPU that delivers a signal from IRQ context, trying to take the tasklist_lock, will loop indefinitely until that crashing CPU releases the lock.
In that case the 'secondary' NMI warnings from all other CPUs (eventually every CPU gets stuck in such a scenario) will start spewing NMI lockup messages.
Dunno. Maybe we should do your change - but also have an option to 'shut up' the kernel after the first hard oops [not warning]. That would silence the secondary NMI watchdog messages as well.
Thanks,
Ingo
Quoted 18 lines · Ingo Molnar
Quoted 8 lines · Don Zickus
Quoted 3 lines
Hm, the flip side is that if a CPU is stuck spewing backtraces, we will now make all the other CPUs a lot more noisy - which might only 'lock up' because this CPU is stuck spewing oopses, right?
When you say the other CPUs will be a lot more noisy, is that because they are busy processing backtraces for the first cpu to spew? I guess I don't understand how the other CPUs could have their interrupts off the whole time while the first cpu is spewing a backtrace (just trying to educate myself).
Say the kernel crashes on a CPU and keeps spewing new oopses, while write-holding
tasklist_lock.Any other CPU that delivers a signal from IRQ context, trying to take the
tasklist_lock, will loop indefinitely until that crashing CPU releases the lock.
Quoted 6 lines
In that case the 'secondary' NMI warnings from all other CPUs (eventually every CPU gets stuck in such a scenario) will start spewing NMI lockup messages.
Dunno. Maybe we should do your change - but also have an option to 'shut up' the kernel after the first hard oops [not warning]. That would silence the secondary NMI watchdog messages as well.
You mean inside the panic() routine? like a ratelimit?
Cheers,
Don
Quoted 19 lines · Don Zickus
Quoted 18 lines · Ingo Molnar
Quoted 8 lines · Don Zickus
Quoted 3 lines
Hm, the flip side is that if a CPU is stuck spewing backtraces, we will now make all the other CPUs a lot more noisy - which might only 'lock up' because this CPU is stuck spewing oopses, right?
When you say the other CPUs will be a lot more noisy, is that because they are busy processing backtraces for the first cpu to spew? I guess I don't understand how the other CPUs could have their interrupts off the whole time while the first cpu is spewing a backtrace (just trying to educate myself).
Say the kernel crashes on a CPU and keeps spewing new oopses, while write-holding
tasklist_lock.Any other CPU that delivers a signal from IRQ context, trying to take the
tasklist_lock, will loop indefinitely until that crashing CPU releases the lock.
Quoted 8 lines
Quoted 6 lines
In that case the 'secondary' NMI warnings from all other CPUs (eventually every CPU gets stuck in such a scenario) will start spewing NMI lockup messages.
Dunno. Maybe we should do your change - but also have an option to 'shut up' the kernel after the first hard oops [not warning]. That would silence the secondary NMI watchdog messages as well.
You mean inside the
panic()routine? like a ratelimit?
A ratelimit, but some really serious one - like only one crash displayed per 10 minutes, or so - to give the user time to make a picture of the first crash, if it's still visible on the screen or so.
Thanks,
Ingo
Quoted 9 lines · Ingo Molnar
Quoted 5 lines
Quoted 3 lines
Dunno. Maybe we should do your change - but also have an option to 'shut up' the kernel after the first hard oops [not warning]. That would silence the secondary NMI watchdog messages as well.
You mean inside the
panic()routine? like a ratelimit?A ratelimit, but some really serious one - like only one crash displayed per 10 minutes, or so - to give the user time to make a picture of the first crash, if it's still visible on the screen or so.
Well I hacked up something. It seems to work, but I am not sure I put the check in the right spot.
Comments welcome.
-----------8<---cut---8<-----
From: Don Zickus <[email protected]>
Date: Mon, 15 Nov 2010 13:09:31 -0500
Subject: [PATCH] panic: ratelimit panic messages
Sometimes when things go bad, so much spew is coming on the console it is hard to figure out what happened. This patch allows you to ratelimit the panic messages with the intent that the first panic message will provide the info we need to figure out what happened.
Adds new kernel param 'panic_ratelimit=on/<integer in seconds>'
--- Documentation/kernel-parameters.txt | 6 ++++++ kernel/panic.c | 30 ++++++++++++++++++++++++++++++ 2 files changed, 36 insertions(+), 0 deletions(-)
Documentation/kernel-parameters.txt +6 -0
@@ -1815,6 +1815,12 @@ and is between 256 and 4096 characters. It is defined in the file panic= [KNL] Kernel behaviour on panic Format: <timeout> + panic_ratelimit= [KNL] ratelimit the panic messages + Useful for slowing down multiple panics to capture + the first one before it scrolls off the screen + Format: "on" or some integer in seconds + "on" defaults to 10 minutes + parkbd.port= [HW] Parallel port number the keyboard adapter is connected to, default is 0. Format: <parport#>
kernel/panic.c +30 -0
@@ -23,6 +23,7 @@#include <linux/init.h> #include <linux/nmi.h> #include <linux/dmi.h> +#include <linux/ratelimit.h> #define PANIC_TIMER_STEP 100#define PANIC_BLINK_SPD 18
@@ -48,6 +49,31 @@ static long no_blink(int state)long (*panic_blink)(int state);EXPORT_SYMBOL (panic_blink); +/* setting default to 0 effectively disables it */ +DEFINE_RATELIMIT_STATE (panic_ratelimit_state,0 ,1 ); + +static int __init panic_ratelimit_setup (char *str) +{ +int interval; + +if (!strncmp (str,"on" ,2 )) +/* default to 10 minutes */ + interval =600 *HZ ; +else + interval =simple_strtoul (str,NULL ,0 ) *HZ ; + + panic_ratelimit_state.interval = interval; +return 1 ; +} +__setup ("panic_ratelimit=" , panic_ratelimit_setup); + +static int __panic_ratelimit (const char *func) +{ +return ___ratelimit (&panic_ratelimit_state, func); +} + +#define panic_ratelimit () __panic_ratelimit(__func__) +/** * panic - halt the system * @fmt: The text string to print
@@ -63,6 +89,10 @@ NORET_TYPE void panic(const char * fmt, ...)long i, i_next =0 ;int state =0 ; +if (!panic_ratelimit ()) +for (;;) +cpu_relax (); +/* * It's possible to come here directly from a panic-assertion and * not have preempt disabled. Some functions called from here want
Signature · 1 line
1.7.2.3