Skip to contentSkip to conversation

NMI watch dog notify patch

PATCH In review George Anzinger 4 participants 12 messages Jul 28, 2005 – Aug 2, 2005

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 NMI watch dog notify patch

Andrew,
I have been doing some work on kgdb to pull a few of it "fingers" out of various places in the kernel. This is the final location where we have a kgdb intercept not covered by a notify.

On a related issue, I feel very queasy with sending nmi interrupts and non-nmi events to the same notify code. Would you be open to a patch to create a seperate notify list for nmi events?

-
George Anzinger [email protected]
HRT (High-res-timers): http://sourceforge.net/projects/high-res-timers/

Source: MontaVista Software, Inc. George Anzinger <[email protected]>
Type: Enhancement
Description:
This patch adds a notify to the nmi watchdog to notify that the system is about to be taken down by the watchdog. If the notify is handled with a NOTIFY_STOP return, the system is given a new lease on life.

This give debug code a chance to a) catch watchdog timeouts and
b) possibly allow the system to continue, realizing that the time out may be due to debugger activities such as single stepping which is usually done with "other" cpus held.

 nmi.c |   15 ++++++++++++---
 1 files changed, 12 insertions(+), 3 deletions(-)
arch/i386/kernel/nmi.c +12 -3
Index: linux-2.6.13-rc/arch/i386/kernel/nmi.c
===================================================================
@@ -26,11 +26,13 @@
 #include <linux/nmi.h>
 #include <linux/sysdev.h>
 #include <linux/sysctl.h>
+#include <linux/notifier.h>
 
 #include <asm/smp.h>
 #include <asm/mtrr.h>
 #include <asm/mpspec.h>
 #include <asm/nmi.h>
+#include <asm/kdebug.h>
 
 #include "mach_traps.h"
 
@@ -494,8 +496,15 @@ void nmi_watchdog_tick (struct pt_regs *
 		 * wait a few IRQs (5 seconds) before doing the oops ...
 		 */
 		alert_counter[cpu]++;
-		if (alert_counter[cpu] == 5*nmi_hz)
-			die_nmi(regs, "NMI Watchdog detected LOCKUP");
+		if (alert_counter[cpu] == 5*nmi_hz) {
+			if (notify_die(DIE_NMIWATCHDOG, "nmi_ipi_watchdog", 
+				       regs, 0, 0, SIGINT) == NOTIFY_STOP) {
+				last_irq_sums[cpu] = sum;
+				alert_counter[cpu] = 0;
+			} else {
+				die_nmi(regs, "NMI Watchdog detected LOCKUP");
+			}
+		}
 	} else {
 		last_irq_sums[cpu] = sum;
 		alert_counter[cpu] = 0;
@@ -555,7 +564,7 @@ int proc_unknown_nmi_panic(ctl_table *ta
 			return -EBUSY;
 		} else {
 			set_nmi_callback(unknown_nmi_panic_callback);
-		}
+		} 
 	} else {
 		release_lapic_nmi();
 		unset_nmi_callback();
#1 lore raw

Review

Quoted 5 lines · George Anzinger

This patch adds a notify to the nmi watchdog to notify that

(quoted diff) +0 -0
 	the system is about to be taken down by the watchdog.  If the
 	notify is handled with a NOTIFY_STOP return, the system is
 	given a new lease on life.

It looks sensible, but as there aren't actually any in-kernel uses for this I'd have thought it would be better for it to live out-of-tree?

#2 lore raw
Quoted 10 lines · Andrew Morton
Quoted 4 lines · George Anzinger

This patch adds a notify to the nmi watchdog to notify that the system is about to be taken down by the watchdog. If the notify is handled with a NOTIFY_STOP return, the system is given a new lease on life.

It looks sensible, but as there aren't actually any in-kernel uses for this I'd have thought it would be better for it to live out-of-tree?

I should just bundle it with the kgdb patch then?

Signature · 2 lines
George Anzinger   [email protected]
HRT (High-res-timers):  http://sourceforge.net/projects/high-res-timers/
#3 lore raw
Quoted 10 lines · George Anzinger
Quoted 8 lines · Andrew Morton
Quoted 4 lines · George Anzinger

This patch adds a notify to the nmi watchdog to notify that the system is about to be taken down by the watchdog. If the notify is handled with a NOTIFY_STOP return, the system is given a new lease on life.

Quoted 4 lines
Quoted 2 lines

It looks sensible, but as there aren't actually any in-kernel uses for this I'd have thought it would be better for it to live out-of-tree?

I should just bundle it with the kgdb patch then?

I spose so, for now. If kdb and/or nlkd could benefit from it then it might simplify life to merge it into mainline. Perhaps you could ping Keith Owens <[email protected]> and [email protected]?

#4 lore raw
In reply to George Anzinger
Quoted 3 lines · George Anzinger

I have been doing some work on kgdb to pull a few of it "fingers" out of various places in the kernel. This is the final location where we have a kgdb intercept not covered by a notify.

I like the idea, but the hook should be in die_nmi(), not in the watchdog, using the reason that is already passed into die_nmi. die_nmi() is also called for a real NMI.

#5 lore raw
Quoted 9 lines · Keith Owens
Quoted 1 line · George Anzinger

I have been doing some work on kgdb to pull a few of it "fingers" out of various places in the kernel. This is the final location where we have a kgdb intercept not covered by a notify.

I like the idea, but the hook should be in die_nmi(), not in the watchdog, using the reason that is already passed into die_nmi. die_nmi() is also called for a real NMI.

I had though that too, but it does not allow recovery (i.e. lets reset the watchdog and try again).

Hmm.. just looked at traps.c. Seems die_nmi is NOT called from the nmi trap, only from the watchdog. Also, there is a notify in the path to the other nmi stuff.

Signature · 2 lines
George Anzinger   [email protected]
HRT (High-res-timers):  http://sourceforge.net/projects/high-res-timers/
#6 lore raw
Quoted 9 lines · George Anzinger
Quoted 8 lines · Keith Owens
Quoted 3 lines · George Anzinger

I have been doing some work on kgdb to pull a few of it "fingers" out of various places in the kernel. This is the final location where we have a kgdb intercept not covered by a notify.

Quoted 6 lines
Quoted 4 lines

I like the idea, but the hook should be in die_nmi(), not in the watchdog, using the reason that is already passed into die_nmi. die_nmi() is also called for a real NMI.

I had though that too, but it does not allow recovery (i.e. lets reset the watchdog and try again).

die_nmi() returns to nmi_watchdog_tick(), nmi_watchdog_tick does the reset and continues. Patch below.

Quoted 3 lines

Hmm.. just looked at traps.c. Seems die_nmi is NOT called from the nmi trap, only from the watchdog. Also, there is a notify in the path to the other nmi stuff.

I was looking at unknown_nmi_panic_callback(), which also calls die_nmi().

traps.c already has several notify_die() calls, nmi.c has none. It is cleaner to keep all the notification in traps.c, with this small change to nmi.c to cope with die_nmi() returning.

arch/i386/kernel/nmi.c +3 -1
Index: linux/arch/i386/kernel/nmi.c
===================================================================
@@ -494,8 +494,10 @@ void nmi_watchdog_tick (struct pt_regs *
 		 * wait a few IRQs (5 seconds) before doing the oops ...
 		 */
 		alert_counter[cpu]++;
-		if (alert_counter[cpu] == 5*nmi_hz)
+		if (alert_counter[cpu] == 5*nmi_hz) {
 			die_nmi(regs, "NMI Watchdog detected LOCKUP");
+			alert_counter[cpu] = 0;
+		}
 	} else {
 		last_irq_sums[cpu] = sum;
 		alert_counter[cpu] = 0;
#7 lore raw
Quoted 18 lines · Keith Owens
Quoted 3 lines

I had though that too, but it does not allow recovery (i.e. lets reset the watchdog and try again).

die_nmi() returns to nmi_watchdog_tick(), nmi_watchdog_tick does the reset and continues. Patch below.

Quoted 3 lines

Hmm.. just looked at traps.c. Seems die_nmi is NOT called from the nmi trap, only from the watchdog. Also, there is a notify in the path to the other nmi stuff.

I was looking at unknown_nmi_panic_callback(), which also calls die_nmi().

traps.c already has several notify_die() calls, nmi.c has none. It is cleaner to keep all the notification in traps.c, with this small change to nmi.c to cope with die_nmi() returning.

Quoted 16 lines

Index: linux/arch/i386/kernel/nmi.c
===================================================================
--- linux.orig/arch/i386/kernel/nmi.c 2005-07-28 17:22:06.735038510 +1000
+++ linux/arch/i386/kernel/nmi.c 2005-07-29 15:19:00.371196596 +1000
@@ -494,8 +494,10 @@ void nmi_watchdog_tick (struct pt_regs *
* wait a few IRQs (5 seconds) before doing the oops ... */
alert_counter[cpu]++;
- if (alert_counter[cpu] == 5*nmi_hz)
+ if (alert_counter[cpu] == 5*nmi_hz) {
die_nmi(regs, "NMI Watchdog detected LOCKUP");
+ alert_counter[cpu] = 0;
+ }
} else {
last_irq_sums[cpu] = sum;
alert_counter[cpu] = 0;

That all makes sense - let's go that way?

#8 lore raw
Quoted 29 lines · Andrew Morton
Quoted 27 lines · Keith Owens
Quoted 3 lines

I had though that too, but it does not allow recovery (i.e. lets reset

the watchdog and try again).

die_nmi() returns to nmi_watchdog_tick(), nmi_watchdog_tick does the reset and continues. Patch below.

Quoted 1 line

Hmm.. just looked at traps.c. Seems die_nmi is NOT called from the nmi trap, only from the watchdog. Also, there is a notify in the path to the other nmi stuff.

I was looking at unknown_nmi_panic_callback(), which also calls die_nmi().

traps.c already has several notify_die() calls, nmi.c has none. It is cleaner to keep all the notification in traps.c, with this small change to nmi.c to cope with die_nmi() returning.

Index: linux/arch/i386/kernel/nmi.c
===================================================================

arch/i386/kernel/nmi.c +1 -1
@@ -494,8 +494,10 @@ void nmi_watchdog_tick (struct pt_regs *
 		 * wait a few IRQs (5 seconds) before doing the oops ...
 		 */
 		alert_counter[cpu]++;
-		if (alert_counter[cpu] == 5*nmi_hz)
+		if (alert_counter[cpu] == 5*nmi_hz) {
 			die_nmi(regs, "NMI Watchdog detected LOCKUP");
Quoted 8 lines

arch/i386/kernel/nmi.c void nmi_watchdog_tick (struct pt_regs * · line 503

Quoted 5 lines
arch/i386/kernel/nmi.c +2 -0
+			alert_counter[cpu] = 0;
+		}
 	} else {
 		last_irq_sums[cpu] = sum;
 		alert_counter[cpu] = 0;

That all makes sense - let's go that way?

Looks good to me. Trimed a bit more fat too. Here is the complete patch.

Quoted 1 line

-

-
George Anzinger [email protected]
HRT (High-res-timers): http://sourceforge.net/projects/high-res-timers/

Source: MontaVista Software, Inc. George Anzinger <[email protected]>
Type: Enhancement
Description:

This patch adds a notify to the die_nmi notify that the system is about to be taken down. If the notify is handled with a NOTIFY_STOP return, the system is given a new lease on life.

We also change the nmi watchdog to carry on if die_nmi returns.

This give debug code a chance to a) catch watchdog timeouts and
b) possibly allow the system to continue, realizing that the time out may be due to debugger activities such as single stepping which is usually done with "other" cpus held.

 nmi.c   |    5 ++++-
 traps.c |    4 ++++
 2 files changed, 8 insertions(+), 1 deletion(-)
arch/i386/kernel/nmi.c +4 -1
Index: linux-2.6.13-rc/arch/i386/kernel/nmi.c
===================================================================
@@ -495,8 +495,11 @@ void nmi_watchdog_tick (struct pt_regs *
 		 */
 		alert_counter[cpu]++;
 		if (alert_counter[cpu] == 5*nmi_hz)
+			/*
+			 * die_nmi will return ONLY if NOTIFY_STOP happens..
+			 */
 			die_nmi(regs, "NMI Watchdog detected LOCKUP");
-	} else {
+
 		last_irq_sums[cpu] = sum;
 		alert_counter[cpu] = 0;
 	}
arch/i386/kernel/traps.c +4 -0
Index: linux-2.6.13-rc/arch/i386/kernel/traps.c
===================================================================
@@ -555,6 +555,10 @@ static DEFINE_SPINLOCK(nmi_print_lock);
 
 void die_nmi (struct pt_regs *regs, const char *msg)
 {
+	if (notify_die(DIE_NMIWATCHDOG, "nmi_watchdog", regs, 
+		       0, 0, SIGINT) == NOTIFY_STOP)
+		return;
+
 	spin_lock(&nmi_print_lock);
 	/*
 	* We are in trouble anyway, lets at least try
#9 lore raw
Quoted 10 lines · George Anzinger

This patch adds a notify to the die_nmi notify that the system is about to be taken down. If the notify is handled with a NOTIFY_STOP return, the system is given a new lease on life.

void die_nmi (struct pt_regs *regs, const char *msg) {

arch/i386/kernel/traps.c +4 -0
+	if (notify_die(DIE_NMIWATCHDOG, "nmi_watchdog", regs, 
+		       0, 0, SIGINT) == NOTIFY_STOP)
+		return;
+
Quoted 3 lines
	spin_lock(&nmi_print_lock);
	/*
	* We are in trouble anyway, lets at least try

Minor nitpick. die_nmi() already gets a message passed in to distinguish between different types of nmi. Pass that message to notify_die(), on the off chance that the notified routines can use that difference.

Also your patch adds a trailing whitespace on the call to notify_die().

#10 lore raw
Quoted 6 lines · Keith Owens
Quoted 4 lines · George Anzinger

This patch adds a notify to the die_nmi notify that the system is about to be taken down. If the notify is handled with a NOTIFY_STOP return, the system is given a new lease on life.

Quoted 15 lines
Quoted 9 lines

void die_nmi (struct pt_regs *regs, const char *msg) {

(quoted diff) +4 -0
+	if (notify_die(DIE_NMIWATCHDOG, "nmi_watchdog", regs, 
+		       0, 0, SIGINT) == NOTIFY_STOP)
+		return;
+
	spin_lock(&nmi_print_lock);
	/*
	* We are in trouble anyway, lets at least try

Minor nitpick. die_nmi() already gets a message passed in to distinguish between different types of nmi. Pass that message to notify_die(), on the off chance that the notified routines can use that difference.

Excellent idea!

Quoted 3 lines

Also your patch adds a trailing whitespace on the call to notify_die().

Fixed.

This should do it.
-
George Anzinger [email protected]
HRT (High-res-timers): http://sourceforge.net/projects/high-res-timers/

Source: MontaVista Software, Inc. George Anzinger <[email protected]>
Type: Enhancement
Description:

This patch adds a notify to the die_nmi notify that the system is about to be taken down. If the notify is handled with a NOTIFY_STOP return, the system is given a new lease on life.

We also change the nmi watchdog to carry on if die_nmi returns.

This give debug code a chance to a) catch watchdog timeouts and
b) possibly allow the system to continue, realizing that the time out may be due to debugger activities such as single stepping which is usually done with "other" cpus held.

 nmi.c   |    5 ++++-
 traps.c |    4 ++++
 2 files changed, 8 insertions(+), 1 deletion(-)
arch/i386/kernel/nmi.c +4 -1
Index: linux-2.6.13-rc/arch/i386/kernel/nmi.c
===================================================================
@@ -495,8 +495,11 @@ void nmi_watchdog_tick (struct pt_regs *
 		 */
 		alert_counter[cpu]++;
 		if (alert_counter[cpu] == 5*nmi_hz)
+			/*
+			 * die_nmi will return ONLY if NOTIFY_STOP happens..
+			 */
 			die_nmi(regs, "NMI Watchdog detected LOCKUP");
-	} else {
+
 		last_irq_sums[cpu] = sum;
 		alert_counter[cpu] = 0;
 	}
arch/i386/kernel/traps.c +4 -0
Index: linux-2.6.13-rc/arch/i386/kernel/traps.c
===================================================================
@@ -555,6 +555,10 @@ static DEFINE_SPINLOCK(nmi_print_lock);
 
 void die_nmi (struct pt_regs *regs, const char *msg)
 {
+	if (notify_die(DIE_NMIWATCHDOG, msg, regs, 0, 0, SIGINT) ==
+	    NOTIFY_STOP)
+		return;
+
 	spin_lock(&nmi_print_lock);
 	/*
 	* We are in trouble anyway, lets at least try
#11 lore raw

It seems that the subject patch generates a warning (missed it on the compile). Here is a patch to eliminate the warning.

Signature · 2 lines
George Anzinger   [email protected]
HRT (High-res-timers):  http://sourceforge.net/projects/high-res-timers/

Source: MontaVista Software, Inc. George Anzinger<[email protected]>
Type: Defect Fix

Description:
This patch eliminates the warning generated in die_nmi() when calling notify_die() by adding "const" to notify_die()'s definition.

include/asm-i386/kdebug.h +1 -1
Index: linux-2.6.13-rc/include/asm-i386/kdebug.h
===================================================================
@@ -41,7 +41,7 @@ enum die_val {
 	DIE_PAGE_FAULT,
 };
 
-static inline int notify_die(enum die_val val,char *str,struct pt_regs *regs,long err,int trap, int sig)
+static inline int notify_die(enum die_val val, const char *str,struct pt_regs *regs,long err,int trap, int sig)
 {
 	struct die_args args = { .regs=regs, .str=str, .err=err, .trapnr=trap,.signr=sig };
 	return notifier_call_chain(&i386die_chain, val, &args);