Skip to contentSkip to conversation

Inhibit NMI watchdog when Alt-SysRq-T operation is underway.

PATCH In review Konrad Rzeszutek 3 participants 5 messages Jul 9, 2007 – Jul 20, 2007

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 Inhibit NMI watchdog when Alt-SysRq-T operation is underway.

On large memory configuration with not so fast CPUs the NMI watchdog is triggered when memory addresses are being gathered and printed. The code paths for Alt-SysRq-t are sprinkled with touch_nmi_watchdog in various places but not in this routine (or in the loop that utilizes this function). The patch has been tested for regression on large CPU+memory configuration (128 logical CPUs + 224 GB) and 1,2,4,16-CPU sockets with various memory sizes (1,2,4,6,20).

#1 lore raw
Quoted 7 lines · Konrad Rzeszutek

On large memory configuration with not so fast CPUs the NMI watchdog is triggered when memory addresses are being gathered and printed. The code paths for Alt-SysRq-t are sprinkled with touch_nmi_watchdog in various places but not in this routine (or in the loop that utilizes this function). The patch has been tested for regression on large CPU+memory configuration (128 logical CPUs + 224 GB) and 1,2,4,16-CPU sockets with various memory sizes (1,2,4,6,20).

And the patch:

arch/x86_64/kernel/traps.c +4 -0
index aac1c0b..b6e7d67 100644
@@ -330,6 +330,10 @@ static int print_trace_stack(void *data,
 
 static void print_trace_address(void *data, unsigned long addr)
 {
+	static int i = 0;		
+	if (i && ((i % 8) == 0)) 
+		touch_nmi_watchdog();
+	i++;
 	printk_address(addr);
 }
 
#2 lore raw

Review

Quoted 8 lines · On Mon, 9 Jul 2007 11:53:02 -0400 [email protected] wrote:

static void print_trace_address(void *data, unsigned long addr) {

(quoted diff) +4 -0
+	static int i = 0;		
+	if (i && ((i % 8) == 0)) 
+		touch_nmi_watchdog();
+	i++;
 	printk_address(addr);
 }

I doubt if the "% 8" thing is really needed? printk_address() is pretty slow and touch_nmi_watchdog is _reasonably_ fast. It could be made heaps faster by:

From: Andrew Morton <[email protected]>

Avoid dirtying remote cpu's memory if it already has the correct value.

Cc Andi Kleen <[email protected]>
Cc Konrad Rzeszutek <[email protected]>

---

 arch/i386/kernel/nmi.c |    8 +++++---
 x86_64/kernel/nmi.c    |    0 
 2 files changed, 5 insertions(+), 3 deletions(-)
arch/i386/kernel/nmi.c renamed from arch/i386/kernel/nmi.c~i386-speedup-touch_nmi_watchdog +5 -3
diff -puN arch/i386/kernel/nmi.c~i386-speedup-touch_nmi_watchdog arch/i386/kernel/nmi.c
@@ -298,7 +298,7 @@ static unsigned int
 	last_irq_sums [NR_CPUS],
 	alert_counter [NR_CPUS];
 
-void touch_nmi_watchdog (void)
+void touch_nmi_watchdog(void)
 {
 	if (nmi_watchdog > 0) {
 		unsigned cpu;
@@ -307,8 +307,10 @@ void touch_nmi_watchdog (void)
 		 * Just reset the alert counters, (other CPUs might be
 		 * spinning on locks we hold):
 		 */
-		for_each_present_cpu (cpu)
-			alert_counter[cpu] = 0;
+		for_each_present_cpu(cpu) {
+			if (alert_counter[cpu])
+				alert_counter[cpu] = 0;
+		}
 	}
 
 	/*

So I'd be inclined to simplify your patch to a bare

From: Konrad Rzeszutek <[email protected]>

On large memory configuration with not so fast CPUs the NMI watchdog is triggered when memory addresses are being gathered and printed. The code paths for Alt-SysRq-t are sprinkled with touch_nmi_watchdog in various places but not in this routine (or in the loop that utilizes this function). The patch has been tested for regression on large CPU+memory configuration (128 logical CPUs + 224 GB) and 1,2,4,16-CPU sockets with various memory sizes (1,2,4,6,20).

---

 arch/x86_64/kernel/traps.c |    1 +
 1 files changed, 1 insertion(+)
arch/x86_64/kernel/traps.c renamed from arch/x86_64/kernel/traps.c~inhibit-nmi-watchdog-when-alt-sysrq-t-operation-is-underway +1 -0
diff -puN arch/x86_64/kernel/traps.c~inhibit-nmi-watchdog-when-alt-sysrq-t-operation-is-underway arch/x86_64/kernel/traps.c
@@ -397,6 +397,7 @@ static int print_trace_stack(void *data,
 
 static void print_trace_address(void *data, unsigned long addr)
 {
+	touch_nmi_watchdog();
 	printk_address(addr);
 }
 

_

OK?

#3 lore raw

Hey Andrew,

I tested your patch along with mine and found two things out:

1). Missing this patch (for i386 platform)

arch/i386/kernel/traps.c +1 -0
index 90da057..9f3a7ff 100644
@@ -207,6 +207,7 @@ static void print_trace_address(void *da
 {
 	printk("%s [<%08lx>] ", (char *)data, addr);
 	print_symbol("%s\n", addr);
+	touch_nmi_watchdog();
 }
 
 static struct stacktrace_ops print_trace_ops = {

2). If I run Alt-SysRq-t about 5000 times in a loop, the slow down
with this change is about 5%. Is this a big issue? (This was testing both i686 and x86_64).

Quoted 89 lines · Andrew Morton

Review

Quoted 8 lines · On Mon, 9 Jul 2007 11:53:02 -0400 [email protected] wrote:

static void print_trace_address(void *data, unsigned long addr) {

(quoted diff) +4 -0
+	static int i = 0;		
+	if (i && ((i % 8) == 0)) 
+		touch_nmi_watchdog();
+	i++;
 	printk_address(addr);
 }

I doubt if the "% 8" thing is really needed? printk_address() is pretty slow and touch_nmi_watchdog is _reasonably_ fast. It could be made heaps faster by:

From: Andrew Morton <[email protected]>

Avoid dirtying remote cpu's memory if it already has the correct value.

Cc Andi Kleen <[email protected]>
Cc Konrad Rzeszutek <[email protected]>

---

 arch/i386/kernel/nmi.c |    8 +++++---
 x86_64/kernel/nmi.c    |    0 
 2 files changed, 5 insertions(+), 3 deletions(-)

diff -puN arch/i386/kernel/nmi.c~i386-speedup-touch_nmi_watchdog arch/i386/kernel/nmi.c

arch/i386/kernel/nmi.c renamed from arch/i386/kernel/nmi.c~i386-speedup-touch_nmi_watchdog +5 -3
@@ -298,7 +298,7 @@ static unsigned int
 	last_irq_sums [NR_CPUS],
 	alert_counter [NR_CPUS];
 
-void touch_nmi_watchdog (void)
+void touch_nmi_watchdog(void)
 {
 	if (nmi_watchdog > 0) {
 		unsigned cpu;
@@ -307,8 +307,10 @@ void touch_nmi_watchdog (void)
 		 * Just reset the alert counters, (other CPUs might be
 		 * spinning on locks we hold):
 		 */
-		for_each_present_cpu (cpu)
-			alert_counter[cpu] = 0;
+		for_each_present_cpu(cpu) {
+			if (alert_counter[cpu])
+				alert_counter[cpu] = 0;
+		}
 	}
 
 	/*

So I'd be inclined to simplify your patch to a bare

From: Konrad Rzeszutek <[email protected]>

On large memory configuration with not so fast CPUs the NMI watchdog is triggered when memory addresses are being gathered and printed. The code paths for Alt-SysRq-t are sprinkled with touch_nmi_watchdog in various places but not in this routine (or in the loop that utilizes this function). The patch has been tested for regression on large CPU+memory configuration (128 logical CPUs + 224 GB) and 1,2,4,16-CPU sockets with various memory sizes (1,2,4,6,20).

---

 arch/x86_64/kernel/traps.c |    1 +
 1 files changed, 1 insertion(+)

diff -puN arch/x86_64/kernel/traps.c~inhibit-nmi-watchdog-when-alt-sysrq-t-operation-is-underway arch/x86_64/kernel/traps.c

arch/x86_64/kernel/traps.c renamed from arch/x86_64/kernel/traps.c~inhibit-nmi-watchdog-when-alt-sysrq-t-operation-is-underway +1 -0
@@ -397,6 +397,7 @@ static int print_trace_stack(void *data,
 
 static void print_trace_address(void *data, unsigned long addr)
 {
+	touch_nmi_watchdog();
 	printk_address(addr);
 }
 

_

OK?

#4 lore raw
Quoted 9 lines · On Fri, 20 Jul 2007 07:18:23 -0400 Konrad Rzeszutek <[email protected]> wrote:

I tested your patch along with mine and found two things out:

1). Missing this patch (for i386 platform)

arch/i386/kernel/traps.c +0 -0
index 90da057..9f3a7ff 100644

arch/i386/kernel/traps.c static void print_trace_address(void *da · line 211

Quoted 5 lines
arch/i386/kernel/traps.c +1 -0
@@ -207,6 +207,7 @@ static void print_trace_address(void *da
 {
 	printk("%s [<%08lx>] ", (char *)data, addr);
 	print_symbol("%s\n", addr);
+	touch_nmi_watchdog();
 }

ok...

Quoted 7 lines
arch/i386/kernel/traps.c +0 -0
 static struct stacktrace_ops print_trace_ops = {
 

2). If I run Alt-SysRq-t about 5000 times in a loop, the slow down
with this change is about 5%. Is this a big issue? (This was testing both i686 and x86_64).

I'm surprised. I assume this was when it was printing to a high-speed device? console or netconsole?

I don't think we need to spend too much time optimising sysrq-T performance ;)