NMI watch dog notify patch
PATCH 1/1 NMI watch dog notify patch
Andrew,
I have been doing some work on kgdb to pull a few of it "fingers" out of various places in the kernel. This is the final location where we have a kgdb intercept not covered by a notify.
On a related issue, I feel very queasy with sending nmi interrupts and non-nmi events to the same notify code. Would you be open to a patch to create a seperate notify list for nmi events?
-
George Anzinger [email protected]
HRT (High-res-timers): http://sourceforge.net/projects/high-res-timers/
Source: MontaVista Software, Inc. George Anzinger <[email protected]>
Type: Enhancement
Description:
This patch adds a notify to the nmi watchdog to notify that the system is about to be taken down by the watchdog. If the notify is handled with a NOTIFY_STOP return, the system is given a new lease on life.
This give debug code a chance to a) catch watchdog timeouts and
b) possibly allow the system to continue, realizing that the time out may be due to debugger activities such as single stepping which is usually done with "other" cpus held.
nmi.c | 15 ++++++++++++--- 1 files changed, 12 insertions(+), 3 deletions(-)
arch/i386/kernel/nmi.c +12 -3
@@ -26,11 +26,13 @@#include <linux/nmi.h> #include <linux/sysdev.h> #include <linux/sysctl.h> +#include <linux/notifier.h> #include <asm/smp.h> #include <asm/mtrr.h> #include <asm/mpspec.h> #include <asm/nmi.h> +#include <asm/kdebug.h> #include "mach_traps.h"
@@ -494,8 +496,15 @@ void nmi_watchdog_tick (struct pt_regs ** wait a few IRQs (5 seconds) before doing the oops ... */ alert_counter[cpu]++; -if (alert_counter[cpu] ==5 *nmi_hz) -die_nmi (regs,"NMI Watchdog detected LOCKUP" ); +if (alert_counter[cpu] ==5 *nmi_hz) { +if (notify_die (DIE_NMIWATCHDOG ,"nmi_ipi_watchdog" , + regs,0 ,0 ,SIGINT ) ==NOTIFY_STOP ) { + last_irq_sums[cpu] = sum; + alert_counter[cpu] =0 ; + }else { +die_nmi (regs,"NMI Watchdog detected LOCKUP" ); + } + } }else { last_irq_sums[cpu] = sum; alert_counter[cpu] =0 ;
@@ -555,7 +564,7 @@ int proc_unknown_nmi_panic(ctl_table *tareturn -EBUSY ; }else {set_nmi_callback (unknown_nmi_panic_callback); - } + } }else {release_lapic_nmi ();unset_nmi_callback ();
Review
Quoted 5 lines · George Anzinger
This patch adds a notify to the nmi watchdog to notify that
(quoted diff) +0
-0the system is about to be taken down by the watchdog. If the notify is handled with a NOTIFY_STOP return, the system is given a new lease on life.
It looks sensible, but as there aren't actually any in-kernel uses for this I'd have thought it would be better for it to live out-of-tree?
Quoted 10 lines · Andrew Morton
Quoted 4 lines · George Anzinger
This patch adds a notify to the nmi watchdog to notify that the system is about to be taken down by the watchdog. If the notify is handled with a
NOTIFY_STOPreturn, the system is given a new lease on life.It looks sensible, but as there aren't actually any in-kernel uses for this I'd have thought it would be better for it to live out-of-tree?
I should just bundle it with the kgdb patch then?
Signature · 2 lines
George Anzinger [email protected] HRT (High-res-timers): http://sourceforge.net/projects/high-res-timers/
Quoted 10 lines · George Anzinger
Quoted 8 lines · Andrew Morton
Quoted 4 lines · George Anzinger
This patch adds a notify to the nmi watchdog to notify that the system is about to be taken down by the watchdog. If the notify is handled with a
NOTIFY_STOPreturn, the system is given a new lease on life.
Quoted 4 lines
Quoted 2 lines
It looks sensible, but as there aren't actually any in-kernel uses for this I'd have thought it would be better for it to live out-of-tree?
I should just bundle it with the kgdb patch then?
I spose so, for now. If kdb and/or nlkd could benefit from it then it might simplify life to merge it into mainline. Perhaps you could ping Keith Owens <[email protected]> and [email protected]?
Quoted 3 lines · George Anzinger
I have been doing some work on kgdb to pull a few of it "fingers" out of various places in the kernel. This is the final location where we have a kgdb intercept not covered by a notify.
I like the idea, but the hook should be in die_nmi(), not in the watchdog, using the reason that is already passed into die_nmi. die_nmi() is also called for a real NMI.
Quoted 9 lines · Keith Owens
Quoted 1 line · George Anzinger
I have been doing some work on kgdb to pull a few of it "fingers" out of various places in the kernel. This is the final location where we have a kgdb intercept not covered by a notify.
I like the idea, but the hook should be in
die_nmi(), not in the watchdog, using the reason that is already passed intodie_nmi.die_nmi()is also called for a real NMI.
I had though that too, but it does not allow recovery (i.e. lets reset the watchdog and try again).
Hmm.. just looked at traps.c. Seems die_nmi is NOT called from the nmi trap, only from the watchdog. Also, there is a notify in the path to the other nmi stuff.
Signature · 2 lines
George Anzinger [email protected] HRT (High-res-timers): http://sourceforge.net/projects/high-res-timers/
Quoted 9 lines · George Anzinger
Quoted 8 lines · Keith Owens
Quoted 3 lines · George Anzinger
I have been doing some work on kgdb to pull a few of it "fingers" out of various places in the kernel. This is the final location where we have a kgdb intercept not covered by a notify.
Quoted 6 lines
Quoted 4 lines
I like the idea, but the hook should be in
die_nmi(), not in the watchdog, using the reason that is already passed intodie_nmi.die_nmi()is also called for a real NMI.I had though that too, but it does not allow recovery (i.e. lets reset the watchdog and try again).
die_nmi() returns to nmi_watchdog_tick(), nmi_watchdog_tick does the reset and continues. Patch below.
Quoted 3 lines
Hmm.. just looked at
traps.c. Seemsdie_nmiis NOT called from the nmi trap, only from the watchdog. Also, there is a notify in the path to the other nmi stuff.
I was looking at unknown_nmi_panic_callback(), which also calls die_nmi().
traps.c already has several notify_die() calls, nmi.c has none. It is cleaner to keep all the notification in traps.c, with this small change to nmi.c to cope with die_nmi() returning.
arch/i386/kernel/nmi.c +3 -1
@@ -494,8 +494,10 @@ void nmi_watchdog_tick (struct pt_regs ** wait a few IRQs (5 seconds) before doing the oops ... */ alert_counter[cpu]++; -if (alert_counter[cpu] ==5 *nmi_hz) +if (alert_counter[cpu] ==5 *nmi_hz) {die_nmi (regs,"NMI Watchdog detected LOCKUP" ); + alert_counter[cpu] =0 ; + } }else { last_irq_sums[cpu] = sum; alert_counter[cpu] =0 ;
Quoted 18 lines · Keith Owens
Quoted 3 lines
I had though that too, but it does not allow recovery (i.e. lets reset the watchdog and try again).
die_nmi()returns tonmi_watchdog_tick(),nmi_watchdog_tickdoes the reset and continues. Patch below.Quoted 3 lines
Hmm.. just looked at
traps.c. Seemsdie_nmiis NOT called from the nmi trap, only from the watchdog. Also, there is a notify in the path to the other nmi stuff.I was looking at
unknown_nmi_panic_callback(), which also callsdie_nmi().
traps.calready has severalnotify_die()calls,nmi.chas none. It is cleaner to keep all the notification intraps.c, with this small change tonmi.cto cope withdie_nmi()returning.
Quoted 16 lines
Index:
linux/arch/i386/kernel/nmi.c
===================================================================
---linux.orig/arch/i386/kernel/nmi.c2005-07-28 17:22:06.735038510 +1000
+++linux/arch/i386/kernel/nmi.c2005-07-29 15:19:00.371196596 +1000
@@ -494,8 +494,10 @@ voidnmi_watchdog_tick(structpt_regs*
* wait a few IRQs (5 seconds) before doing the oops ... */
alert_counter[cpu]++;
- if (alert_counter[cpu] == 5*nmi_hz)
+ if (alert_counter[cpu] == 5*nmi_hz) {
die_nmi(regs, "NMI Watchdog detected LOCKUP");
+ alert_counter[cpu] = 0;
+ }
} else {
last_irq_sums[cpu] = sum;
alert_counter[cpu] = 0;
That all makes sense - let's go that way?
Quoted 29 lines · Andrew Morton
Quoted 27 lines · Keith Owens
Quoted 3 lines
I had though that too, but it does not allow recovery (i.e. lets reset
the watchdog and try again).
die_nmi()returns tonmi_watchdog_tick(),nmi_watchdog_tickdoes the reset and continues. Patch below.Quoted 1 line
Hmm.. just looked at
traps.c. Seemsdie_nmiis NOT called from the nmi trap, only from the watchdog. Also, there is a notify in the path to the other nmi stuff.I was looking at
unknown_nmi_panic_callback(), which also callsdie_nmi().
traps.calready has severalnotify_die()calls,nmi.chas none. It is cleaner to keep all the notification intraps.c, with this small change tonmi.cto cope withdie_nmi()returning.Index:
linux/arch/i386/kernel/nmi.c
===================================================================arch/i386/kernel/nmi.c +1
-1@@ -494,8 +494,10 @@ void nmi_watchdog_tick (struct pt_regs ** wait a few IRQs (5 seconds) before doing the oops ... */ alert_counter[cpu]++; -if (alert_counter[cpu] ==5 *nmi_hz) +if (alert_counter[cpu] ==5 *nmi_hz) {die_nmi (regs,"NMI Watchdog detected LOCKUP" );
Quoted 8 lines
arch/i386/kernel/nmi.c void nmi_watchdog_tick (struct pt_regs * · line 503
Quoted 5 lines
arch/i386/kernel/nmi.c +2
-0+ alert_counter[cpu] =0 ; + } }else { last_irq_sums[cpu] = sum; alert_counter[cpu] =0 ;That all makes sense - let's go that way?
Looks good to me. Trimed a bit more fat too. Here is the complete patch.
Quoted 1 line
-
-
George Anzinger [email protected]
HRT (High-res-timers): http://sourceforge.net/projects/high-res-timers/
Source: MontaVista Software, Inc. George Anzinger <[email protected]>
Type: Enhancement
Description:
This patch adds a notify to the die_nmi notify that the system is about to be taken down. If the notify is handled with a NOTIFY_STOP return, the system is given a new lease on life.
We also change the nmi watchdog to carry on if die_nmi returns.
This give debug code a chance to a) catch watchdog timeouts and
b) possibly allow the system to continue, realizing that the time out may be due to debugger activities such as single stepping which is usually done with "other" cpus held.
nmi.c | 5 ++++- traps.c | 4 ++++ 2 files changed, 8 insertions(+), 1 deletion(-)
arch/i386/kernel/nmi.c +4 -1
@@ -495,8 +495,11 @@ void nmi_watchdog_tick (struct pt_regs **/ alert_counter[cpu]++;if (alert_counter[cpu] ==5 *nmi_hz) +/* +* die_nmi will return ONLY if NOTIFY_STOP happens.. +*/ die_nmi (regs,"NMI Watchdog detected LOCKUP" ); - }else { + last_irq_sums[cpu] = sum; alert_counter[cpu] =0 ; }
arch/i386/kernel/traps.c +4 -0
@@ -555,6 +555,10 @@ static DEFINE_SPINLOCK(nmi_print_lock);void die_nmi (struct pt_regs *regs,const char *msg) { +if (notify_die (DIE_NMIWATCHDOG ,"nmi_watchdog" , regs, +0 ,0 ,SIGINT ) ==NOTIFY_STOP ) +return ; +spin_lock (&nmi_print_lock);/* * We are in trouble anyway, lets at least try
Quoted 10 lines · George Anzinger
This patch adds a notify to the
die_nminotify that the system is about to be taken down. If the notify is handled with aNOTIFY_STOPreturn, the system is given a new lease on life.void
die_nmi(structpt_regs*regs, const char *msg) {arch/i386/kernel/traps.c +4
-0+if (notify_die (DIE_NMIWATCHDOG ,"nmi_watchdog" , regs, +0 ,0 ,SIGINT ) ==NOTIFY_STOP ) +return ; +
Quoted 3 lines
spin_lock(&nmi_print_lock); /* * We are in trouble anyway, lets at least try
Minor nitpick. die_nmi() already gets a message passed in to distinguish between different types of nmi. Pass that message to notify_die(), on the off chance that the notified routines can use that difference.
Also your patch adds a trailing whitespace on the call to notify_die().
Quoted 6 lines · Keith Owens
Quoted 4 lines · George Anzinger
This patch adds a notify to the
die_nminotify that the system is about to be taken down. If the notify is handled with aNOTIFY_STOPreturn, the system is given a new lease on life.
Quoted 15 lines
Quoted 9 lines
void
die_nmi(structpt_regs*regs, const char *msg) {(quoted diff) +4
-0+ if (notify_die(DIE_NMIWATCHDOG, "nmi_watchdog", regs, + 0, 0, SIGINT) == NOTIFY_STOP) + return; +spin_lock(&nmi_print_lock); /* * We are in trouble anyway, lets at least tryMinor nitpick.
die_nmi()already gets a message passed in to distinguish between different types of nmi. Pass that message tonotify_die(), on the off chance that the notified routines can use that difference.
Excellent idea!
Quoted 3 lines
Also your patch adds a trailing whitespace on the call to
notify_die().
Fixed.
This should do it.
-
George Anzinger [email protected]
HRT (High-res-timers): http://sourceforge.net/projects/high-res-timers/
Source: MontaVista Software, Inc. George Anzinger <[email protected]>
Type: Enhancement
Description:
This patch adds a notify to the die_nmi notify that the system is about to be taken down. If the notify is handled with a NOTIFY_STOP return, the system is given a new lease on life.
We also change the nmi watchdog to carry on if die_nmi returns.
This give debug code a chance to a) catch watchdog timeouts and
b) possibly allow the system to continue, realizing that the time out may be due to debugger activities such as single stepping which is usually done with "other" cpus held.
nmi.c | 5 ++++- traps.c | 4 ++++ 2 files changed, 8 insertions(+), 1 deletion(-)
arch/i386/kernel/nmi.c +4 -1
@@ -495,8 +495,11 @@ void nmi_watchdog_tick (struct pt_regs **/ alert_counter[cpu]++;if (alert_counter[cpu] ==5 *nmi_hz) +/* +* die_nmi will return ONLY if NOTIFY_STOP happens.. +*/ die_nmi (regs,"NMI Watchdog detected LOCKUP" ); - }else { + last_irq_sums[cpu] = sum; alert_counter[cpu] =0 ; }
arch/i386/kernel/traps.c +4 -0
@@ -555,6 +555,10 @@ static DEFINE_SPINLOCK(nmi_print_lock);void die_nmi (struct pt_regs *regs,const char *msg) { +if (notify_die (DIE_NMIWATCHDOG , msg, regs,0 ,0 ,SIGINT ) == +NOTIFY_STOP ) +return ; +spin_lock (&nmi_print_lock);/* * We are in trouble anyway, lets at least try
It seems that the subject patch generates a warning (missed it on the compile). Here is a patch to eliminate the warning.
Signature · 2 lines
George Anzinger [email protected] HRT (High-res-timers): http://sourceforge.net/projects/high-res-timers/
Source: MontaVista Software, Inc. George Anzinger<[email protected]>
Type: Defect Fix
Description:
This patch eliminates the warning generated in die_nmi() when calling notify_die() by adding "const" to notify_die()'s definition.
include/asm-i386/kdebug.h +1 -1
@@ -41,7 +41,7 @@ enum die_val { DIE_PAGE_FAULT, }; -static inline int notify_die (enum die_val val,char *str,struct pt_regs *regs,long err,int trap,int sig) +static inline int notify_die (enum die_val val,const char *str,struct pt_regs *regs,long err,int trap,int sig) {struct die_args args = { .regs=regs, .str=str, .err=err, .trapnr=trap,.signr=sig };return notifier_call_chain (&i386die_chain, val, &args);