Skip to contentSkip to conversation

new nmi_watchdog using perf events

RFC PATCH Superseded Don Zickus 4 participants 15 messages Jan 27, 2010 – Feb 2, 2010

Subsystems (2)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

Don Zickus

author
#0 lore raw

cover new nmi_watchdog using perf events

This patch series tries to implement a new nmi_watchdog using the perf events subsystem. I am posting this series early for feedback on the approach. It isn't feature compatible with the old nmi_watchdog yet, nor does it have all the old workarounds either.

The basic design is to create an in-kernel performance counter that goes off every few seconds and checks for cpu lockups. It is fairly straight forward. Some of the quirks are making sure the cpu lockup detection works correctly.

It has been lightly tested for now. Once people are ok with the approach, I'll expand testing to more machines in our lab.

I tried taking a generic approach so all arches could use it if they want and then implement some per arch specific hooks. I believe this is what Ingo was suggesting. The initial work is based off of kernel/softlockup.c.

Any feedback would be great.

Cheers,
Don

--
Sorry if this spams people again. My mailer was broke and some mail wasn't sent correctly.

Don Zickus (3):
[RFC][x86] move notify_die from nmi.c to traps.c [RFC] nmi_watchdog: new implementation using perf events [RFC] nmi_watchdog: config option to enable new nmi_watchdog

 arch/x86/kernel/apic/Makefile |    7 ++-
 arch/x86/kernel/apic/hw_nmi.c |  114 ++++++++++++++++++++++++
 arch/x86/kernel/apic/nmi.c    |    7 --
 arch/x86/kernel/traps.c       |    7 ++
 include/linux/nmi.h           |    4 +
 kernel/Makefile               |    1 +
 kernel/nmi_watchdog.c         |  196 +++++++++++++++++++++++++++++++++++++++++
 lib/Kconfig.debug             |   13 +++
 8 files changed, 341 insertions(+), 8 deletions(-)
 create mode 100644 arch/x86/kernel/apic/hw_nmi.c
 create mode 100644 kernel/nmi_watchdog.c
author
#1 lore raw

RFC PATCH 1/3 move notify_die from nmi.c to traps.c

In order to handle a new nmi_watchdog approach, I need to move the notify_die() routine out of nmi_watchdog_tick() and into default_do_nmi(). This lets me easily swap out the old nmi_watchdog with the new one with just a config change.

The change probably makes sense from a high level perspective because the nmi_watchdog shouldn't be handling notify_die routines anyway. However, this move does change the semantics a little bit. Instead of checking on every nmi interrupt if the cpus are stuck, only check them on the nmi_watchdog interrupts.

---
 arch/x86/kernel/apic/nmi.c |    7 -------
 arch/x86/kernel/traps.c    |    5 +++++
 2 files changed, 5 insertions(+), 7 deletions(-)
arch/x86/kernel/apic/nmi.c +0 -7
index 0159a69..5d47682 100644
@@ -400,13 +400,6 @@ nmi_watchdog_tick(struct pt_regs *regs, unsigned reason)
 	int cpu = smp_processor_id();
 	int rc = 0;
 
-	/* check for other users first */
-	if (notify_die(DIE_NMI, "nmi", regs, reason, 2, SIGINT)
-			== NOTIFY_STOP) {
-		rc = 1;
-		touched = 1;
-	}
-
 	sum = get_timer_irqs(cpu);
 
 	if (__get_cpu_var(nmi_touch)) {
arch/x86/kernel/traps.c +5 -0
index 3339917..3b98dd3 100644
@@ -400,6 +400,11 @@ static notrace __kprobes void default_do_nmi(struct pt_regs *regs)
 		if (notify_die(DIE_NMI_IPI, "nmi_ipi", regs, reason, 2, SIGINT)
 								== NOTIFY_STOP)
 			return;
+
+	        if (notify_die(DIE_NMI, "nmi", regs, reason, 2, SIGINT)
+	        			                == NOTIFY_STOP)
+	                return;
+
 #ifdef CONFIG_X86_LOCAL_APIC
 		/*
 		 * Ok, so this is none of the documented NMI sources,
Signature · 1 line
1.6.5.2
#2 lore raw

Re: [PATCH 1/3] [RFC][x86] move notify_die from nmi.c to traps.c

On Wed, Jan 27, 2010 at 03:03:40PM -0500, Don Zickus wrote:
...

Quoted 7 lines
arch/x86/kernel/traps.c +0 -0
index 3339917..3b98dd3 100644
@@ -400,6 +400,11 @@ static notrace __kprobes void default_do_nmi(struct pt_regs *regs)
 		if (notify_die(DIE_NMI_IPI, "nmi_ipi", regs, reason, 2, SIGINT)
 								== NOTIFY_STOP)

arch/x86/kernel/traps.c static notrace __kprobes void default_do_nmi(struct pt_regs *regs)

Quoted 10 lines
arch/x86/kernel/traps.c +5 -1
 			return;
+
+	        if (notify_die(DIE_NMI, "nmi", regs, reason, 2, SIGINT)
+	        			                == NOTIFY_STOP)
+	                return;
+
 #ifdef CONFIG_X86_LOCAL_APIC
 		/*
 		 * Ok, so this is none of the documented NMI sources,
--

Hi Don, I suppose this notify_die should be in CONFIG_X86_LOCAL_APIC section?

-- Cyrill

author
#3 lore raw

Re: [PATCH 1/3] [RFC][x86] move notify_die from nmi.c to traps.c

Quoted 14 lines · Cyrill Gorcunov

On Wed, Jan 27, 2010 at 03:03:40PM -0500, Don Zickus wrote:
...

Quoted 12 lines
arch/x86/kernel/traps.c +4 -0
index 3339917..3b98dd3 100644
@@ -400,6 +400,11 @@ static notrace __kprobes void default_do_nmi(struct pt_regs *regs)
 		if (notify_die(DIE_NMI_IPI, "nmi_ipi", regs, reason, 2, SIGINT)
 								== NOTIFY_STOP)
 			return;
+
+	        if (notify_die(DIE_NMI, "nmi", regs, reason, 2, SIGINT)
+	        			                == NOTIFY_STOP)
+	                return;
Quoted 8 lines

arch/x86/kernel/traps.c static notrace __kprobes void default_do_nmi(struct pt_regs *regs)

Quoted 5 lines
arch/x86/kernel/traps.c +1 -1
+
 #ifdef CONFIG_X86_LOCAL_APIC
 		/*
 		 * Ok, so this is none of the documented NMI sources,
--

Hi Don, I suppose this notify_die should be in CONFIG_X86_LOCAL_APIC section?

To maintain old behaviour I suppose, yes. Personally I don't think notify_die has anything to do with CONFIG_X86_LOCAL_APIC so I put it in above the #define.

Cheers,
Don

#4 lore raw

Re: [PATCH 1/3] [RFC][x86] move notify_die from nmi.c to traps.c

Quoted 24 lines · Don Zickus
Quoted 14 lines · Cyrill Gorcunov

On Wed, Jan 27, 2010 at 03:03:40PM -0500, Don Zickus wrote:
...

Quoted 12 lines
arch/x86/kernel/traps.c +4 -0
index 3339917..3b98dd3 100644
@@ -400,6 +400,11 @@ static notrace __kprobes void default_do_nmi(struct pt_regs *regs)
 		if (notify_die(DIE_NMI_IPI, "nmi_ipi", regs, reason, 2, SIGINT)
 								== NOTIFY_STOP)
 			return;
+
+	        if (notify_die(DIE_NMI, "nmi", regs, reason, 2, SIGINT)
+	        			                == NOTIFY_STOP)
+	                return;
Quoted 8 lines

arch/x86/kernel/traps.c static notrace __kprobes void default_do_nmi(struct pt_regs *regs)

Quoted 5 lines
arch/x86/kernel/traps.c +1 -1
+
 #ifdef CONFIG_X86_LOCAL_APIC
 		/*
 		 * Ok, so this is none of the documented NMI sources,
--

Hi Don, I suppose this notify_die should be in CONFIG_X86_LOCAL_APIC section?

Quoted 4 lines

To maintain old behaviour I suppose, yes. Personally I don't think notify_die has anything to do with CONFIG_X86_LOCAL_APIC so I put it in above the #define.

I think it is. It becomes that if some (possible buggy in future) code notify default_do_nmi via NOTIFY_STOP we may loose unknown_nmi_error for non-apic configs. And I reckon that even DIE_NMI_IPI is a bit "weird" by not being under apic here, but this one should stay there in a sake of kgdb I guess.

All-in-one: I would better not change old behaviour.

Though, it is possbile that I just miss something obvious :)

Quoted 3 lines

Cheers,
Don

-- Cyrill

author
#5 lore raw

Re: [PATCH 1/3] [RFC][x86] move notify_die from nmi.c to traps.c

Quoted 12 lines
Quoted 12 lines
Quoted 11 lines

arch/x86/kernel/traps.c static notrace __kprobes void default_do_nmi(struct pt_regs *regs)

Quoted 8 lines
arch/x86/kernel/traps.c +4 -1
+	        if (notify_die(DIE_NMI, "nmi", regs, reason, 2, SIGINT)
+	        			                == NOTIFY_STOP)
+	                return;
+
 #ifdef CONFIG_X86_LOCAL_APIC
 		/*
 		 * Ok, so this is none of the documented NMI sources,
--

Hi Don, I suppose this notify_die should be in CONFIG_X86_LOCAL_APIC section?

Quoted 7 lines
Quoted 4 lines

To maintain old behaviour I suppose, yes. Personally I don't think notify_die has anything to do with CONFIG_X86_LOCAL_APIC so I put it in above the #define.

I think it is. It becomes that if some (possible buggy in future) code notify default_do_nmi via NOTIFY_STOP we may loose unknown_nmi_error

How is that different if the code is under the #define?

Quoted 3 lines

for non-apic configs. And I reckon that even DIE_NMI_IPI is a bit "weird" by not being under apic here, but this one should stay there in a sake of kgdb I guess.

So you are saying the only way to get NMIs on x86 is through the local apic? That seems odd.

I really don't care either way, I was just trying to cleanup the code to make it easier to understand. Putting it under the #define didn't seem to make sense (though that doesn't mean there is a valid reason).

Cheers,
Don

#6 lore raw

Re: [PATCH 1/3] [RFC][x86] move notify_die from nmi.c to traps.c

Quoted 17 lines · Don Zickus
Quoted 17 lines
Quoted 16 lines
Quoted 11 lines

arch/x86/kernel/traps.c static notrace __kprobes void default_do_nmi(struct pt_regs *regs)

Quoted 8 lines
arch/x86/kernel/traps.c +4 -1
+	        if (notify_die(DIE_NMI, "nmi", regs, reason, 2, SIGINT)
+	        			                == NOTIFY_STOP)
+	                return;
+
 #ifdef CONFIG_X86_LOCAL_APIC
 		/*
 		 * Ok, so this is none of the documented NMI sources,
--

Hi Don, I suppose this notify_die should be in CONFIG_X86_LOCAL_APIC section?

To maintain old behaviour I suppose, yes. Personally I don't think notify_die has anything to do with CONFIG_X86_LOCAL_APIC so I put it in above the #define.

Quoted 5 lines
Quoted 2 lines

I think it is. It becomes that if some (possible buggy in future) code notify default_do_nmi via NOTIFY_STOP we may loose unknown_nmi_error

How is that different if the code is under the #define?

It matters if code is compiled without CONFIG_X86_LOCAL_APIC.

Quoted 6 lines
Quoted 3 lines

for non-apic configs. And I reckon that even DIE_NMI_IPI is a bit "weird" by not being under apic here, but this one should stay there in a sake of kgdb I guess.

So you are saying the only way to get NMIs on x86 is through the local apic? That seems odd.

No, I didn't say that. I said that semantic of DIE_NMI_IPI is somehow weird for me. There is no IPI if no apic present (at least in official way). But in a sake of kgdb which checks for this notification we should save it here.

Quoted 5 lines

I really don't care either way, I was just trying to cleanup the code to make it easier to understand. Putting it under the #define didn't seem to make sense (though that doesn't mean there is a valid reason).

Don, please don't get me wrong. At moment I don't see problems with putting DIE_NMI in a place you had put it at. But something worrying me, can't explain what exactly, perhaps that is how paranoia happens :)

Quoted 4 lines

Cheers,
Don

-- Cyrill

author
#7 lore raw

RFC PATCH 2/3 nmi_watchdog: new implementation using perf events

In reply to Don Zickus

This is a new generic nmi_watchdog implementation using the perf events infrastructure as suggested by Ingo.

The implementation is simple, just create an in-kernel perf event and register an overflow handler to check for cpu lockups. I created a generic implementation that lives in kernel/ and the hardware specific part that for now lives in arch/x86.

I have done light testing to make sure the framework works correctly and it does. Correctly detecting hardlocks is still a work in progress, though it should work, I am having trouble proving it through testing.

---
 arch/x86/kernel/apic/hw_nmi.c |  114 ++++++++++++++++++++++++
 kernel/nmi_watchdog.c         |  196 +++++++++++++++++++++++++++++++++++++++++
 2 files changed, 310 insertions(+), 0 deletions(-)
 create mode 100644 arch/x86/kernel/apic/hw_nmi.c
 create mode 100644 kernel/nmi_watchdog.c
arch/x86/kernel/apic/hw_nmi.c new file +114 -0
new file mode 100644
index 0000000..8c0e6a4
@@ -0,0 +1,114 @@
+/*
+ *  HW NMI watchdog support
+ *
+ *  started by Don Zickus, Copyright (C) 2010 Red Hat, Inc.
+ *
+ *  Arch specific calls to support NMI watchdog
+ *
+ *  Bits copied from original nmi.c file
+ *
+ */
+
+#include <asm/apic.h>
+#include <linux/smp.h>
+#include <linux/cpumask.h>
+#include <linux/sched.h>
+#include <linux/percpu.h>
+#include <linux/cpumask.h>
+#include <linux/kernel_stat.h>
+#include <asm/mce.h>
+
+#include <linux/nmi.h>
+#include <linux/module.h>
+
+/* For reliability, we're prepared to waste bits here. */
+static DECLARE_BITMAP(backtrace_mask, NR_CPUS) __read_mostly;
+
+static DEFINE_PER_CPU(unsigned, last_irq_sum);
+
+/*
+ * Take the local apic timer and PIT/HPET into account. We don't
+ * know which one is active, when we have highres/dyntick on
+ */
+static inline unsigned int get_timer_irqs(int cpu)
+{
+        return per_cpu(irq_stat, cpu).apic_timer_irqs +
+                per_cpu(irq_stat, cpu).irq0_irqs;
+}
+
+static inline int mce_in_progress(void)
+{
+#if defined(CONFIG_X86_MCE)
+        return atomic_read(&mce_entry) > 0;
+#endif
+        return 0;
+}
+
+int hw_nmi_is_cpu_stuck(struct pt_regs *regs)
+{
+	unsigned int sum;
+	int cpu = smp_processor_id();
+
+	/* FIXME: cheap hack for this check, probably should get its own
+	 * die_notifier handler
+	 */
+	if (cpumask_test_cpu(cpu, to_cpumask(backtrace_mask))) {
+		static DEFINE_SPINLOCK(lock);	/* Serialise the printks */
+
+		spin_lock(&lock);
+		printk(KERN_WARNING "NMI backtrace for cpu %d\n", cpu);
+		show_regs(regs);
+		dump_stack();
+		spin_unlock(&lock);
+		cpumask_clear_cpu(cpu, to_cpumask(backtrace_mask));
+	}
+
+	/* if we are doing an mce, just assume the cpu is not stuck */
+        /* Could check oops_in_progress here too, but it's safer not to */
+        if (mce_in_progress())
+                return 0;
+
+	/* We determine if the cpu is stuck by checking whether any
+	 * interrupts have happened since we last checked.  Of course
+	 * an nmi storm could create false positives, but the higher
+	 * level logic should account for that
+	 */
+	sum = get_timer_irqs(cpu);
+	if (__get_cpu_var(last_irq_sum) == sum) {
+		return 1;
+	} else {
+		__get_cpu_var(last_irq_sum) = sum;
+		return 0;
+	}
+}
+
+void arch_trigger_all_cpu_backtrace(void)
+{
+	int i;
+
+	cpumask_copy(to_cpumask(backtrace_mask), cpu_online_mask);
+
+	printk(KERN_INFO "sending NMI to all CPUs:\n");
+	apic->send_IPI_all(NMI_VECTOR);
+
+	/* Wait for up to 10 seconds for all CPUs to do the backtrace */
+	for (i = 0; i < 10 * 1000; i++) {
+		if (cpumask_empty(to_cpumask(backtrace_mask)))
+			break;
+		mdelay(1);
+	}
+}
+
+/* STUB calls to mimic old nmi_watchdog behaviour */
+unsigned int nmi_watchdog = NMI_NONE;
+EXPORT_SYMBOL(nmi_watchdog);
+atomic_t nmi_active = ATOMIC_INIT(0);           /* oprofile uses this */
+EXPORT_SYMBOL(nmi_active);
+int nmi_watchdog_enabled;
+int unknown_nmi_panic;
+void cpu_nmi_set_wd_enabled(void) { return; }
+void acpi_nmi_enable(void) { return; }
+void acpi_nmi_disable(void) { return; }
+void stop_apic_nmi_watchdog(void *unused) { return; }
+void setup_apic_nmi_watchdog(void *unused) { return; }
+int __init check_nmi_watchdog(void) { return 0; }
kernel/nmi_watchdog.c new file +196 -0

196 lines not on this page · raw message

Signature · 1 line
1.6.5.2
author
#8 lore raw

RFC PATCH 3/3 nmi_watchdog: config option to enable new nmi_watchdog

In reply to Don Zickus

These are the bits that enable the new nmi_watchdog and safely isolate the old nmi_watchdog. Only one or the other can run, not both at the same time.

---
 arch/x86/kernel/apic/Makefile |    7 ++++++-
 arch/x86/kernel/traps.c       |    2 ++
 include/linux/nmi.h           |    4 ++++
 kernel/Makefile               |    1 +
 lib/Kconfig.debug             |   13 +++++++++++++
 5 files changed, 26 insertions(+), 1 deletions(-)
arch/x86/kernel/apic/Makefile +6 -1
index 565c1bf..1a4512e 100644
@@ -2,7 +2,12 @@
 # Makefile for local APIC drivers and for the IO-APIC code
 #
 
-obj-$(CONFIG_X86_LOCAL_APIC)	+= apic.o apic_noop.o probe_$(BITS).o ipi.o nmi.o
+obj-$(CONFIG_X86_LOCAL_APIC)	+= apic.o apic_noop.o probe_$(BITS).o ipi.o
+ifneq ($(CONFIG_NMI_WATCHDOG),y)
+obj-$(CONFIG_X86_LOCAL_APIC)	+= nmi.o
+endif
+obj-$(CONFIG_NMI_WATCHDOG)	+= hw_nmi.o
+
 obj-$(CONFIG_X86_IO_APIC)	+= io_apic.o
 obj-$(CONFIG_SMP)		+= ipi.o
 
arch/x86/kernel/traps.c +2 -0
index 3b98dd3..eee1dc8 100644
@@ -406,6 +406,7 @@ static notrace __kprobes void default_do_nmi(struct pt_regs *regs)
 	                return;
 
 #ifdef CONFIG_X86_LOCAL_APIC
+#ifndef CONFIG_NMI_WATCHDOG
 		/*
 		 * Ok, so this is none of the documented NMI sources,
 		 * so it must be the NMI watchdog.
@@ -413,6 +414,7 @@ static notrace __kprobes void default_do_nmi(struct pt_regs *regs)
 		if (nmi_watchdog_tick(regs, reason))
 			return;
 		if (!do_nmi_callback(regs, cpu))
+#endif /* !CONFIG_NMI_WATCHDOG */
 			unknown_nmi_error(reason, regs);
 #else
 		unknown_nmi_error(reason, regs);
include/linux/nmi.h +4 -0
index b752e80..a42ff0b 100644
@@ -47,4 +47,8 @@ static inline bool trigger_all_cpu_backtrace(void)
 }
 #endif
 
+#ifdef CONFIG_NMI_WATCHDOG
+int hw_nmi_is_cpu_stuck(struct pt_regs *);
+#endif
+
 #endif
kernel/Makefile +1 -0
index 864ff75..8a5abe5 100644
@@ -76,6 +76,7 @@ obj-$(CONFIG_AUDIT_TREE) += audit_tree.o
 obj-$(CONFIG_KPROBES) += kprobes.o
 obj-$(CONFIG_KGDB) += kgdb.o
 obj-$(CONFIG_DETECT_SOFTLOCKUP) += softlockup.o
+obj-$(CONFIG_NMI_WATCHDOG) += nmi_watchdog.o
 obj-$(CONFIG_DETECT_HUNG_TASK) += hung_task.o
 obj-$(CONFIG_GENERIC_HARDIRQS) += irq/
 obj-$(CONFIG_SECCOMP) += seccomp.o
lib/Kconfig.debug +13 -0
index 25c3ed5..04a43a2 100644
@@ -170,6 +170,19 @@ config DETECT_SOFTLOCKUP
 	   can be detected via the NMI-watchdog, on platforms that
 	   support it.)
 
+config NMI_WATCHDOG
+	bool "Detect Hard Lockups with an NMI Watchdog"
+	depends on DEBUG_KERNEL && PERF_EVENTS
+	default y
+	help
+	  Say Y here to enable the kernel to use the NMI as a watchdog
+	  to detect hard lockups.  This is useful when a cpu hangs for no
+	  reason but can still respond to NMIs.  A backtrace is displayed
+	  for reviewing and reporting.
+
+	  The overhead should be minimal, just an extra NMI every few 
+	  seconds.
+
 config BOOTPARAM_SOFTLOCKUP_PANIC
 	bool "Panic (Reboot) On Soft Lockups"
 	depends on DETECT_SOFTLOCKUP
Signature · 1 line
1.6.5.2
#9 lore raw

Re: [PATCH 3/3] [RFC] nmi_watchdog: config option to enable new nmi_watchdog

Quoted 3 lines · Don Zickus

These are the bits that enable the new nmi_watchdog and safely isolate the old nmi_watchdog. Only one or the other can run, not both at the same time.

perf disables the lapic watchdog when it wants the pmu, so there shouldn't be a problem having both built in.

author
#10 lore raw

Re: [PATCH 3/3] [RFC] nmi_watchdog: config option to enable new nmi_watchdog

Quoted 7 lines · Peter Zijlstra
Quoted 3 lines · Don Zickus

These are the bits that enable the new nmi_watchdog and safely isolate the old nmi_watchdog. Only one or the other can run, not both at the same time.

perf disables the lapic watchdog when it wants the pmu, so there shouldn't be a problem having both built in.

Yes it does disable but does not prevent nmi_watchdog_tick from running nor the /proc interface from being loaded. So perhaps my description isn't very good. The idea with the new watchdog was to re-use some of the bits of the old one, but having them both compiled in seemed to stomp on each other. That is what I was trying to prevent.

I can certainly change the behaviour, just makes the code a little more messy I think.

Cheers,
Don

#11 lore raw

Re: [PATCH 3/3] [RFC] nmi_watchdog: config option to enable new nmi_watchdog

Quoted 9 lines · Don Zickus
Quoted 7 lines · Peter Zijlstra
Quoted 3 lines · Don Zickus

These are the bits that enable the new nmi_watchdog and safely isolate the old nmi_watchdog. Only one or the other can run, not both at the same time.

perf disables the lapic watchdog when it wants the pmu, so there shouldn't be a problem having both built in.

Quoted 8 lines

Yes it does disable but does not prevent nmi_watchdog_tick from running nor the /proc interface from being loaded. So perhaps my description isn't very good. The idea with the new watchdog was to re-use some of the bits of the old one, but having them both compiled in seemed to stomp on each other. That is what I was trying to prevent.

I can certainly change the behaviour, just makes the code a little more messy I think.

I think that's a good idea - and i think we want to be bold and just have the new code run seemlessly. (and fix bugs, if any.)

In fact we want to be even bolder: how about enabling the NMI watchdog by default again?

The problem with the old one was its fragility - but now if we have a PMU driver active and perf events enabled we might as well use your brand new NMI watchdog code as a testing facility as well: if there's _any_ problem with NMIs then regular 'perf' use would trigger it too - except that not all people run perf while an always-enabled NMI watchdog would.

And it would detect hard hangs too.

What do you think?

Ingo

author
#12 lore raw

Re: [PATCH 3/3] [RFC] nmi_watchdog: config option to enable new nmi_watchdog

Quoted 18 lines · Ingo Molnar
Quoted 15 lines · Don Zickus
Quoted 7 lines · Peter Zijlstra
Quoted 3 lines · Don Zickus

These are the bits that enable the new nmi_watchdog and safely isolate the old nmi_watchdog. Only one or the other can run, not both at the same time.

perf disables the lapic watchdog when it wants the pmu, so there shouldn't be a problem having both built in.

Yes it does disable but does not prevent nmi_watchdog_tick from running nor the /proc interface from being loaded. So perhaps my description isn't very good. The idea with the new watchdog was to re-use some of the bits of the old one, but having them both compiled in seemed to stomp on each other. That is what I was trying to prevent.

Quoted 5 lines
Quoted 2 lines

I can certainly change the behaviour, just makes the code a little more messy I think.

I think that's a good idea - and i think we want to be bold and just have the new code run seemlessly. (and fix bugs, if any.)

Ok. I guess I am confused what you are suggesting here, to do as Peter suggested and run both at the same time?

Quoted 3 lines

In fact we want to be even bolder: how about enabling the NMI watchdog by default again?

I won't complain. :-)

Quoted 6 lines

The problem with the old one was its fragility - but now if we have a PMU driver active and perf events enabled we might as well use your brand new NMI watchdog code as a testing facility as well: if there's _any_ problem with NMIs then regular 'perf' use would trigger it too - except that not all people run perf while an always-enabled NMI watchdog would.

Agreed.

Quoted 4 lines

And it would detect hard hangs too.

What do you think?

I will need to give you an updated patch that properly sets the frequency of the NMI and I probably should still implement a code path that uses the software perf counters in the cases where the hardware perf counters are not available.

It seems like you are ok with my approach. If that is so, I can test on more machines to iron out some more bugs. Or did you want to take my patches as is and have me throw fixes on top?

Cheers,
Don

Quoted 2 lines

Ingo

#13 lore raw

Re: [PATCH 3/3] [RFC] nmi_watchdog: config option to enable new nmi_watchdog

Quoted 20 lines · Don Zickus
Quoted 19 lines · Ingo Molnar
Quoted 16 lines · Don Zickus
Quoted 8 lines · Peter Zijlstra
Quoted 4 lines · Don Zickus

These are the bits that enable the new nmi_watchdog and safely isolate the old nmi_watchdog. Only one or the other can run, not both at the same time.

perf disables the lapic watchdog when it wants the pmu, so there shouldn't be a problem having both built in.

Yes it does disable but does not prevent nmi_watchdog_tick from running nor the /proc interface from being loaded. So perhaps my description isn't very good. The idea with the new watchdog was to re-use some of the bits of the old one, but having them both compiled in seemed to stomp on each other. That is what I was trying to prevent.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

I can certainly change the behaviour, just makes the code a little more messy I think.

I think that's a good idea - and i think we want to be bold and just have the new code run seemlessly. (and fix bugs, if any.)

Ok. I guess I am confused what you are suggesting here, to do as Peter suggested and run both at the same time?

I dont think we want to run old and new code at once, the old NMI watchdog code is really a hardcoded minimal PMU driver generating a cycles based NMI tick once per second.

Quoted 10 lines
Quoted 1 line

What do you think?

I will need to give you an updated patch that properly sets the frequency of the NMI and I probably should still implement a code path that uses the software perf counters in the cases where the hardware perf counters are not available.

It seems like you are ok with my approach. If that is so, I can test on more machines to iron out some more bugs. Or did you want to take my patches as is and have me throw fixes on top?

Well, all known bugs/showstoppers should be fixed - but otherwise if you think it works fine we can certainly apply it and then iterate it from that point on to increase coverage and add features.

Ingo

author
#14 lore raw

Re: [PATCH 3/3] [RFC] nmi_watchdog: config option to enable new nmi_watchdog

Quoted 7 lines · Ingo Molnar
Quoted 3 lines

Ok. I guess I am confused what you are suggesting here, to do as Peter suggested and run both at the same time?

I dont think we want to run old and new code at once, the old NMI watchdog code is really a hardcoded minimal PMU driver generating a cycles based NMI tick once per second.

Ok. Agreed.

Quoted 8 lines
Quoted 8 lines
Quoted 2 lines

What do you think?

I will need to give you an updated patch that properly sets the frequency of the NMI and I probably should still implement a code path that uses the software perf counters in the cases where the hardware perf counters are not available.

Quoted 7 lines
Quoted 3 lines

It seems like you are ok with my approach. If that is so, I can test on more machines to iron out some more bugs. Or did you want to take my patches as is and have me throw fixes on top?

Well, all known bugs/showstoppers should be fixed - but otherwise if you think it works fine we can certainly apply it and then iterate it from that point on to increase coverage and add features.

I have a small cleanup to my second patch that makes it functional. I'll run some tests on some more machines and post version two tomorrow. We can probably build from there.

Cheers,
Don