new nmi_watchdog using perf events
Subsystems (2)
v2 cover new nmi_watchdog using perf events
This patch series tries to implement a new nmi_watchdog using the perf events subsystem. I am posting this series early for feedback on the approach. It isn't feature compatible with the old nmi_watchdog yet, nor does it have all the old workarounds either.
The basic design is to create an in-kernel performance counter that goes off every few seconds and checks for cpu lockups. It is fairly straight forward. Some of the quirks are making sure the cpu lockup detection works correctly.
It has been lightly tested for now. Once people are ok with the approach, I'll expand testing to more machines in our lab.
I tried taking a generic approach so all arches could use it if they want and then implement some per arch specific hooks. I believe this is what Ingo was suggesting. The initial work is based off of kernel/softlockup.c.
Any feedback would be great.
v2:
- moved a notify_die call into a #ifdef block
- used better default values for configuring the nmi_watchdog based on the old nmi_watchdog values
Cheers,
Don
--
damn it forgot to cc lkml
Don Zickus (3):
[RFC][x86] move notify_die from nmi.c to traps.c [RFC] nmi_watchdog: new implementation using perf events [RFC] nmi_watchdog: config option to enable new nmi_watchdog
arch/x86/kernel/apic/Makefile | 7 ++- arch/x86/kernel/apic/hw_nmi.c | 114 ++++++++++++++++++++++++ arch/x86/kernel/apic/nmi.c | 7 -- arch/x86/kernel/traps.c | 7 ++ include/linux/nmi.h | 4 + kernel/Makefile | 1 + kernel/nmi_watchdog.c | 196 +++++++++++++++++++++++++++++++++++++++++ lib/Kconfig.debug | 13 +++ 8 files changed, 341 insertions(+), 8 deletions(-) create mode 100644 arch/x86/kernel/apic/hw_nmi.c create mode 100644 kernel/nmi_watchdog.c
PATCH v2 1/3 move notify_die from nmi.c to traps.c
In order to handle a new nmi_watchdog approach, I need to move the notify_die() routine out of nmi_watchdog_tick() and into default_do_nmi(). This lets me easily swap out the old nmi_watchdog with the new one with just a config change.
The change probably makes sense from a high level perspective because the nmi_watchdog shouldn't be handling notify_die routines anyway. However, this move does change the semantics a little bit. Instead of checking on every nmi interrupt if the cpus are stuck, only check them on the nmi_watchdog interrupts.
v2: move notify_die call into #idef block
--- arch/x86/kernel/apic/nmi.c | 7 ------- arch/x86/kernel/traps.c | 5 +++++ 2 files changed, 5 insertions(+), 7 deletions(-)
arch/x86/kernel/apic/nmi.c +0 -7
@@ -400,13 +400,6 @@ nmi_watchdog_tick(struct pt_regs *regs, unsigned reason)int cpu =smp_processor_id ();int rc =0 ; -/* check for other users first */ -if (notify_die (DIE_NMI ,"nmi" , regs, reason,2 ,SIGINT ) - ==NOTIFY_STOP ) { - rc =1 ; - touched =1 ; - } - sum =get_timer_irqs (cpu);if (__get_cpu_var (nmi_touch)) {
arch/x86/kernel/traps.c +5 -0
@@ -400,7 +400,12 @@ static notrace __kprobes void default_do_nmi(struct pt_regs *regs)if (notify_die (DIE_NMI_IPI ,"nmi_ipi" , regs, reason,2 ,SIGINT ) ==NOTIFY_STOP )return ; +#ifdef CONFIG_X86_LOCAL_APIC +if (notify_die (DIE_NMI ,"nmi" , regs, reason,2 ,SIGINT ) + ==NOTIFY_STOP ) +return ; +/* * Ok, so this is none of the documented NMI sources, * so it must be the NMI watchdog.
Signature · 1 line
1.6.6.83.gc9a2
[tip:perf/nmi] x86: Move notify_die from nmi.c to traps.c
Commit-ID: e40b17208b6805be50ffe891878662b6076206b9 Gitweb: http://git.kernel.org/tip/e40b17208b6805be50ffe891878662b6076206b9 Author: Don Zickus <[email protected]> AuthorDate: Fri, 5 Feb 2010 21:47:03 -0500 Committer: Ingo Molnar <[email protected]> CommitDate: Mon, 8 Feb 2010 08:29:02 +0100
x86: Move notify_die from nmi.c to traps.c
In order to handle a new nmi_watchdog approach, I need to move the notify_die() routine out of nmi_watchdog_tick() and into default_do_nmi(). This lets me easily swap out the old nmi_watchdog with the new one with just a config change.
The change probably makes sense from a high level perspective because the nmi_watchdog shouldn't be handling notify_die routines anyway. However, this move does change the semantics a little bit. Instead of checking on every nmi interrupt if the cpus are stuck, only check them on the nmi_watchdog interrupts.
v2: Move notify_die call into #idef block
LKML-Reference: <[email protected]>
--- arch/x86/kernel/apic/nmi.c | 7 ------- arch/x86/kernel/traps.c | 5 +++++ 2 files changed, 5 insertions(+), 7 deletions(-)
arch/x86/kernel/apic/nmi.c +0 -7
@@ -400,13 +400,6 @@ nmi_watchdog_tick(struct pt_regs *regs, unsigned reason)int cpu =smp_processor_id ();int rc =0 ; -/* check for other users first */ -if (notify_die (DIE_NMI ,"nmi" , regs, reason,2 ,SIGINT ) - ==NOTIFY_STOP ) { - rc =1 ; - touched =1 ; - } - sum =get_timer_irqs (cpu);if (__get_cpu_var (nmi_touch)) {
arch/x86/kernel/traps.c +5 -0
@@ -400,7 +400,12 @@ static notrace __kprobes void default_do_nmi(struct pt_regs *regs)if (notify_die (DIE_NMI_IPI ,"nmi_ipi" , regs, reason,2 ,SIGINT ) ==NOTIFY_STOP )return ; +#ifdef CONFIG_X86_LOCAL_APIC +if (notify_die (DIE_NMI ,"nmi" , regs, reason,2 ,SIGINT ) + ==NOTIFY_STOP ) +return ; +/* * Ok, so this is none of the documented NMI sources, * so it must be the NMI watchdog.
PATCH v2 2/3 nmi_watchdog: new implementation using perf events
In reply to Don ZickusThis is a new generic nmi_watchdog implementation using the perf events infrastructure as suggested by Ingo.
The implementation is simple, just create an in-kernel perf event and register an overflow handler to check for cpu lockups. I created a generic implementation that lives in kernel/ and the hardware specific part that for now lives in arch/x86.
I have done light testing to make sure the framework works correctly and it does.
v2: sets the correct timeout values based on the old nmi watchdog
--- arch/x86/kernel/apic/hw_nmi.c | 114 ++++++++++++++++++++++++ kernel/nmi_watchdog.c | 191 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 305 insertions(+), 0 deletions(-) create mode 100644 arch/x86/kernel/apic/hw_nmi.c create mode 100644 kernel/nmi_watchdog.c
arch/x86/kernel/apic/hw_nmi.c new file +114 -0
@@ -0,0 +1,114 @@ +/* +* HW NMI watchdog support +* +* started by Don Zickus, Copyright (C) 2010 Red Hat, Inc. +* +* Arch specific calls to support NMI watchdog +* +* Bits copied from original nmi.c file +* +*/ + +#include <asm/apic.h> +#include <linux/smp.h> +#include <linux/cpumask.h> +#include <linux/sched.h> +#include <linux/percpu.h> +#include <linux/cpumask.h> +#include <linux/kernel_stat.h> +#include <asm/mce.h> + +#include <linux/nmi.h> +#include <linux/module.h> + +/* For reliability, we're prepared to waste bits here. */ +static DECLARE_BITMAP(backtrace_mask, NR_CPUS) __read_mostly; + +static DEFINE_PER_CPU(unsigned, last_irq_sum); + +/* +* Take the local apic timer and PIT/HPET into account. We don't +* know which one is active, when we have highres/dyntick on +*/ +static inline unsigned int get_timer_irqs (int cpu) +{ +return per_cpu (irq_stat, cpu).apic_timer_irqs + +per_cpu (irq_stat, cpu).irq0_irqs; +} + +static inline int mce_in_progress (void ) +{ +#if defined(CONFIG_X86_MCE ) +return atomic_read (&mce_entry) >0 ; +#endif +return 0 ; +} + +int hw_nmi_is_cpu_stuck (struct pt_regs *regs) +{ +unsigned int sum; +int cpu =smp_processor_id (); + +/* FIXME: cheap hack for this check, probably should get its own +* die_notifier handler +*/ +if (cpumask_test_cpu (cpu,to_cpumask (backtrace_mask))) { +static DEFINE_SPINLOCK (lock);/* Serialise the printks */ + +spin_lock (&lock); +printk (KERN_WARNING "NMI backtrace for cpu %d\n" , cpu); +show_regs (regs); +dump_stack (); +spin_unlock (&lock); +cpumask_clear_cpu (cpu,to_cpumask (backtrace_mask)); + } + +/* if we are doing an mce, just assume the cpu is not stuck */ +/* Could check oops_in_progress here too, but it's safer not to */ +if (mce_in_progress ()) +return 0 ; + +/* We determine if the cpu is stuck by checking whether any +* interrupts have happened since we last checked. Of course +* an nmi storm could create false positives, but the higher +* level logic should account for that +*/ + sum =get_timer_irqs (cpu); +if (__get_cpu_var (last_irq_sum) == sum) { +return 1 ; + }else { +__get_cpu_var (last_irq_sum) = sum; +return 0 ; + } +} + +void arch_trigger_all_cpu_backtrace (void ) +{ +int i; + +cpumask_copy (to_cpumask (backtrace_mask), cpu_online_mask); + +printk (KERN_INFO "sending NMI to all CPUs:\n" ); + apic->send_IPI_all (NMI_VECTOR ); + +/* Wait for up to 10 seconds for all CPUs to do the backtrace */ +for (i =0 ; i <10 *1000 ; i++) { +if (cpumask_empty (to_cpumask (backtrace_mask))) +break ; +mdelay (1 ); + } +} + +/* STUB calls to mimic old nmi_watchdog behaviour */ +unsigned int nmi_watchdog =NMI_NONE ; +EXPORT_SYMBOL (nmi_watchdog); +atomic_t nmi_active =ATOMIC_INIT (0 );/* oprofile uses this */ +EXPORT_SYMBOL (nmi_active); +int nmi_watchdog_enabled; +int unknown_nmi_panic; +void cpu_nmi_set_wd_enabled (void ) {return ; } +void acpi_nmi_enable (void ) {return ; } +void acpi_nmi_disable (void ) {return ; } +void stop_apic_nmi_watchdog (void *unused) {return ; } +void setup_apic_nmi_watchdog (void *unused) {return ; } +int __init check_nmi_watchdog (void ) {return 0 ; }
kernel/nmi_watchdog.c new file +191 -0
191 lines not on this page · raw message
Signature · 1 line
1.6.6.83.gc9a2
[tip:perf/nmi] nmi_watchdog: Add new, generic implementation, using perf events
Commit-ID: 1fb9d6ad2766a1dd70d167552988375049a97f21 Gitweb: http://git.kernel.org/tip/1fb9d6ad2766a1dd70d167552988375049a97f21 Author: Don Zickus <[email protected]> AuthorDate: Fri, 5 Feb 2010 21:47:04 -0500 Committer: Ingo Molnar <[email protected]> CommitDate: Mon, 8 Feb 2010 08:29:02 +0100
nmi_watchdog: Add new, generic implementation, using perf events
This is a new generic nmi_watchdog implementation using the perf events infrastructure as suggested by Ingo.
The implementation is simple, just create an in-kernel perf event and register an overflow handler to check for cpu lockups.
I created a generic implementation that lives in kernel/ and the hardware specific part that for now lives in arch/x86.
This approach has a number of advantages:
- It simplifies the x86 PMU implementation in the long run, in that it removes the hardcoded low-level PMU implementation that was the NMI watchdog before.
- It allows new NMI watchdog features to be added in a central place.
- It allows other architectures to enable the NMI watchdog, as long as they have perf events (that provide NMIs) implemented.
- It also allows for more graceful co-existence of existing perf events apps and the NMI watchdog - before these changes the relationship was exclusive. (The NMI watchdog will 'spend' a perf event when enabled. In later iterations we might be able to piggyback from an existing NMI event without having to allocate a hardware event for the NMI watchdog - turning this into a no-hardware-cost feature.)
As for compatibility, we'll keep the old NMI watchdog code as well until the new one can 100% replace it on all CPUs, old and new alike. That might take some time as the NMI watchdog has been ported to many CPU models.
I have done light testing to make sure the framework works correctly and it does.
v2: Set the correct timeout values based on the old nmi
watchdog
LKML-Reference: <[email protected]>
--- arch/x86/kernel/apic/hw_nmi.c | 114 ++++++++++++++++++++++++ kernel/nmi_watchdog.c | 191 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 305 insertions(+), 0 deletions(-)
arch/x86/kernel/apic/hw_nmi.c new file +114 -0
@@ -0,0 +1,114 @@ +/* +* HW NMI watchdog support +* +* started by Don Zickus, Copyright (C) 2010 Red Hat, Inc. +* +* Arch specific calls to support NMI watchdog +* +* Bits copied from original nmi.c file +* +*/ + +#include <asm/apic.h> +#include <linux/smp.h> +#include <linux/cpumask.h> +#include <linux/sched.h> +#include <linux/percpu.h> +#include <linux/cpumask.h> +#include <linux/kernel_stat.h> +#include <asm/mce.h> + +#include <linux/nmi.h> +#include <linux/module.h> + +/* For reliability, we're prepared to waste bits here. */ +static DECLARE_BITMAP(backtrace_mask, NR_CPUS) __read_mostly; + +static DEFINE_PER_CPU(unsigned, last_irq_sum); + +/* +* Take the local apic timer and PIT/HPET into account. We don't +* know which one is active, when we have highres/dyntick on +*/ +static inline unsigned int get_timer_irqs (int cpu) +{ +return per_cpu (irq_stat, cpu).apic_timer_irqs + +per_cpu (irq_stat, cpu).irq0_irqs; +} + +static inline int mce_in_progress (void ) +{ +#if defined(CONFIG_X86_MCE ) +return atomic_read (&mce_entry) >0 ; +#endif +return 0 ; +} + +int hw_nmi_is_cpu_stuck (struct pt_regs *regs) +{ +unsigned int sum; +int cpu =smp_processor_id (); + +/* FIXME: cheap hack for this check, probably should get its own +* die_notifier handler +*/ +if (cpumask_test_cpu (cpu,to_cpumask (backtrace_mask))) { +static DEFINE_SPINLOCK (lock);/* Serialise the printks */ + +spin_lock (&lock); +printk (KERN_WARNING "NMI backtrace for cpu %d\n" , cpu); +show_regs (regs); +dump_stack (); +spin_unlock (&lock); +cpumask_clear_cpu (cpu,to_cpumask (backtrace_mask)); + } + +/* if we are doing an mce, just assume the cpu is not stuck */ +/* Could check oops_in_progress here too, but it's safer not to */ +if (mce_in_progress ()) +return 0 ; + +/* We determine if the cpu is stuck by checking whether any +* interrupts have happened since we last checked. Of course +* an nmi storm could create false positives, but the higher +* level logic should account for that +*/ + sum =get_timer_irqs (cpu); +if (__get_cpu_var (last_irq_sum) == sum) { +return 1 ; + }else { +__get_cpu_var (last_irq_sum) = sum; +return 0 ; + } +} + +void arch_trigger_all_cpu_backtrace (void ) +{ +int i; + +cpumask_copy (to_cpumask (backtrace_mask), cpu_online_mask); + +printk (KERN_INFO "sending NMI to all CPUs:\n" ); + apic->send_IPI_all (NMI_VECTOR ); + +/* Wait for up to 10 seconds for all CPUs to do the backtrace */ +for (i =0 ; i <10 *1000 ; i++) { +if (cpumask_empty (to_cpumask (backtrace_mask))) +break ; +mdelay (1 ); + } +} + +/* STUB calls to mimic old nmi_watchdog behaviour */ +unsigned int nmi_watchdog =NMI_NONE ; +EXPORT_SYMBOL (nmi_watchdog); +atomic_t nmi_active =ATOMIC_INIT (0 );/* oprofile uses this */ +EXPORT_SYMBOL (nmi_active); +int nmi_watchdog_enabled; +int unknown_nmi_panic; +void cpu_nmi_set_wd_enabled (void ) {return ; } +void acpi_nmi_enable (void ) {return ; } +void acpi_nmi_disable (void ) {return ; } +void stop_apic_nmi_watchdog (void *unused) {return ; } +void setup_apic_nmi_watchdog (void *unused) {return ; } +int __init check_nmi_watchdog (void ) {return 0 ; }
kernel/nmi_watchdog.c new file +191 -0
191 lines not on this page · raw message
PATCH v2 3/3 nmi_watchdog: config option to enable new nmi_watchdog
In reply to Don ZickusThese are the bits that enable the new nmi_watchdog and safely isolate the old nmi_watchdog. Only one or the other can run, not both at the same time.
--- arch/x86/kernel/apic/Makefile | 7 ++++++- arch/x86/kernel/traps.c | 2 ++ include/linux/nmi.h | 4 ++++ kernel/Makefile | 1 + lib/Kconfig.debug | 13 +++++++++++++ 5 files changed, 26 insertions(+), 1 deletions(-)
arch/x86/kernel/apic/Makefile +6 -1
@@ -2,7 +2,12 @@ # Makefile for local APIC drivers and for the IO-APIC code # -obj-$(CONFIG_X86_LOCAL_APIC) += apic.o apic_noop.o probe_$(BITS).o ipi.o nmi.o +obj-$(CONFIG_X86_LOCAL_APIC) += apic.o apic_noop.o probe_$(BITS).o ipi.o +ifneq ($(CONFIG_NMI_WATCHDOG),y) +obj-$(CONFIG_X86_LOCAL_APIC) += nmi.o +endif +obj-$(CONFIG_NMI_WATCHDOG) += hw_nmi.o + obj-$(CONFIG_X86_IO_APIC) += io_apic.o obj-$(CONFIG_SMP) += ipi.o
arch/x86/kernel/traps.c +2 -0
@@ -406,6 +406,7 @@ static notrace __kprobes void default_do_nmi(struct pt_regs *regs) == NOTIFY_STOP)return ; +#ifndef CONFIG_NMI_WATCHDOG /* * Ok, so this is none of the documented NMI sources, * so it must be the NMI watchdog.
@@ -413,6 +414,7 @@ static notrace __kprobes void default_do_nmi(struct pt_regs *regs)if (nmi_watchdog_tick (regs, reason))return ; if (!do_nmi_callback(regs, cpu)) +#endif /* !CONFIG_NMI_WATCHDOG */ unknown_nmi_error (reason, regs);#else unknown_nmi_error (reason, regs);
include/linux/nmi.h +4 -0
@@ -47,4 +47,8 @@ static inline bool trigger_all_cpu_backtrace(void) }#endif +#ifdef CONFIG_NMI_WATCHDOG +int hw_nmi_is_cpu_stuck (struct pt_regs *); +#endif +#endif
kernel/Makefile +1 -0
@@ -76,6 +76,7 @@ obj-$(CONFIG_AUDIT_TREE) += audit_tree.o obj-$(CONFIG_KPROBES) += kprobes.o obj-$(CONFIG_KGDB) += kgdb.o obj-$(CONFIG_DETECT_SOFTLOCKUP) += softlockup.o +obj-$(CONFIG_NMI_WATCHDOG) += nmi_watchdog.o obj-$(CONFIG_DETECT_HUNG_TASK) += hung_task.o obj-$(CONFIG_GENERIC_HARDIRQS) += irq/ obj-$(CONFIG_SECCOMP) += seccomp.o
lib/Kconfig.debug +13 -0
@@ -170,6 +170,19 @@ config DETECT_SOFTLOCKUP can be detected via the NMI-watchdog, on platforms that support it.) +config NMI_WATCHDOG + bool "Detect Hard Lockups with an NMI Watchdog" + depends on DEBUG_KERNEL && PERF_EVENTS + default y + help + Say Y here to enable the kernel to use the NMI as a watchdog + to detect hard lockups. This is useful when a cpu hangs for no + reason but can still respond to NMIs. A backtrace is displayed + for reviewing and reporting. + + The overhead should be minimal, just an extra NMI every few + seconds. + config BOOTPARAM_SOFTLOCKUP_PANIC bool "Panic (Reboot) On Soft Lockups" depends on DETECT_SOFTLOCKUP
Signature · 1 line
1.6.6.83.gc9a2
Re: [PATCH 3/3 v2] nmi_watchdog: config option to enable new nmi_watchdog
lib/Kconfig.debug config DETECT_SOFTLOCKUP · line 184
Quoted 12 lines · Don Zickus
lib/Kconfig.debug +12
-0+config NMI_WATCHDOG + bool "Detect Hard Lockups with an NMI Watchdog" + depends on DEBUG_KERNEL && PERF_EVENTS + default y + help + Say Y here to enable the kernel to use the NMI as a watchdog + to detect hard lockups. This is useful when a cpu hangs for no + reason but can still respond to NMIs. A backtrace is displayed + for reviewing and reporting. + + The overhead should be minimal, just an extra NMI every few + seconds.
Thought for later patches: I think an architecture should be able to express via a Kconfig switch that it actually _has_ NMI events. There's architectures which dont have a PMU driver and only have software events. There's also architectures that have a PMU driver but no NMIs.
Something like ARCH_HAS_NMI_PERF_EVENTS?
Also, i havent checked, but what is the practical effect of the new generic watchdog on x86 CPUs that does not have a native PMU driver yet - such as P4s?
Anyway, i'll create a tip:perf/nmi topic branch for these patches, it certainly looks like a useful generalization and a new architecture that has perf could easily enable it, without having to write its own NMI watchdog implementation. It's also useful for any new watchdog features that people might want to add. Plus it makes the x86 PMU code cleaner in the long run as well.
Thanks,
Ingo
Re: [PATCH 3/3 v2] nmi_watchdog: config option to enable new nmi_watchdog
Quoted 22 lines · Ingo Molnar
lib/Kconfig.debug config DETECT_SOFTLOCKUP · line 184
Quoted 12 lines · Don Zickus
lib/Kconfig.debug +12
-0+config NMI_WATCHDOG + bool "Detect Hard Lockups with an NMI Watchdog" + depends on DEBUG_KERNEL && PERF_EVENTS + default y + help + Say Y here to enable the kernel to use the NMI as a watchdog + to detect hard lockups. This is useful when a cpu hangs for no + reason but can still respond to NMIs. A backtrace is displayed + for reviewing and reporting. + + The overhead should be minimal, just an extra NMI every few + seconds.Thought for later patches: I think an architecture should be able to express via a
Kconfigswitch that it actually _has_ NMI events. There's architectures
which dont have a PMU driver and only have software events. There's also architectures that have a PMU driver but no NMIs.
Quoted 6 lines
Something like
ARCH_HAS_NMI_PERF_EVENTS?Also, i havent checked, but what is the practical effect of the new generic watchdog on x86 CPUs that does not have a native PMU driver yet - such as P4s?
p4 pmu is not yet implemented. I'll try to post on lkml the thnigs i've done for it today evening, though it's pretty ugly i would say.
Quoted 11 lines
Anyway, i'll create a tip:perf/nmi topic branch for these patches, it certainly looks like a useful generalization and a new architecture that has perf could easily enable it, without having to write its own NMI watchdog implementation. It's also useful for any new watchdog features that people might want to add. Plus it makes the x86 PMU code cleaner in the long run as well.
Thanks,
Ingo
Re: [PATCH 3/3 v2] nmi_watchdog: config option to enable new nmi_watchdog
In reply to Ingo MolnarQuoted 21 lines · Ingo Molnar
lib/Kconfig.debug config DETECT_SOFTLOCKUP · line 184
Quoted 12 lines · Don Zickus
lib/Kconfig.debug +12
-0+config NMI_WATCHDOG + bool "Detect Hard Lockups with an NMI Watchdog" + depends on DEBUG_KERNEL && PERF_EVENTS + default y + help + Say Y here to enable the kernel to use the NMI as a watchdog + to detect hard lockups. This is useful when a cpu hangs for no + reason but can still respond to NMIs. A backtrace is displayed + for reviewing and reporting. + + The overhead should be minimal, just an extra NMI every few + seconds.Thought for later patches: I think an architecture should be able to express via a
Kconfigswitch that it actually _has_ NMI events. There's architectures which dont have a PMU driver and only have software events. There's also architectures that have a PMU driver but no NMIs.
Quoted 1 line
Something like
ARCH_HAS_NMI_PERF_EVENTS?
I guess I assumed the perf event subsystem would take care of that which is why I made the config option dependent on PERF_EVENTS. I am open to suggestions on enhance it.
Quoted 4 lines
Also, i havent checked, but what is the practical effect of the new generic watchdog on x86 CPUs that does not have a native PMU driver yet - such as P4s?
I believe the call to perf_event_create_kernel_counter would fail, which then prevents the cpu from coming online. Probably not the smartest thing to do. I was looking at adding code to fall back to trying PERF_TYPE_SOFTWARE. Let me dig up a P4 box and see what happens.
Quoted 7 lines
Anyway, i'll create a tip:perf/nmi topic branch for these patches, it certainly looks like a useful generalization and a new architecture that has perf could easily enable it, without having to write its own NMI watchdog implementation. It's also useful for any new watchdog features that people might want to add. Plus it makes the x86 PMU code cleaner in the long run as well.
Agreed.
Cheers,
Don
[tip:perf/nmi] nmi_watchdog: Config option to enable new nmi_watchdog
In reply to Don ZickusCommit-ID: 84e478c6f1eb9c4bfa1fff2f8108e9a061b46428 Gitweb: http://git.kernel.org/tip/84e478c6f1eb9c4bfa1fff2f8108e9a061b46428 Author: Don Zickus <[email protected]> AuthorDate: Fri, 5 Feb 2010 21:47:05 -0500 Committer: Ingo Molnar <[email protected]> CommitDate: Mon, 8 Feb 2010 08:29:03 +0100
nmi_watchdog: Config option to enable new nmi_watchdog
These are the bits that enable the new nmi_watchdog and safely isolate the old nmi_watchdog. Only one or the other can run, not both at the same time.
LKML-Reference: <[email protected]>
--- arch/x86/kernel/apic/Makefile | 7 ++++++- arch/x86/kernel/traps.c | 2 ++ include/linux/nmi.h | 4 ++++ kernel/Makefile | 1 + lib/Kconfig.debug | 13 +++++++++++++ 5 files changed, 26 insertions(+), 1 deletions(-)
arch/x86/kernel/apic/Makefile +6 -1
@@ -2,7 +2,12 @@ # Makefile for local APIC drivers and for the IO-APIC code # -obj-$(CONFIG_X86_LOCAL_APIC) += apic.o apic_noop.o probe_$(BITS).o ipi.o nmi.o +obj-$(CONFIG_X86_LOCAL_APIC) += apic.o apic_noop.o probe_$(BITS).o ipi.o +ifneq ($(CONFIG_NMI_WATCHDOG),y) +obj-$(CONFIG_X86_LOCAL_APIC) += nmi.o +endif +obj-$(CONFIG_NMI_WATCHDOG) += hw_nmi.o + obj-$(CONFIG_X86_IO_APIC) += io_apic.o obj-$(CONFIG_SMP) += ipi.o
arch/x86/kernel/traps.c +2 -0
@@ -406,6 +406,7 @@ static notrace __kprobes void default_do_nmi(struct pt_regs *regs) == NOTIFY_STOP)return ; +#ifndef CONFIG_NMI_WATCHDOG /* * Ok, so this is none of the documented NMI sources, * so it must be the NMI watchdog.
@@ -413,6 +414,7 @@ static notrace __kprobes void default_do_nmi(struct pt_regs *regs)if (nmi_watchdog_tick (regs, reason))return ; if (!do_nmi_callback(regs, cpu)) +#endif /* !CONFIG_NMI_WATCHDOG */ unknown_nmi_error (reason, regs);#else unknown_nmi_error (reason, regs);
include/linux/nmi.h +4 -0
@@ -47,4 +47,8 @@ static inline bool trigger_all_cpu_backtrace(void) }#endif +#ifdef CONFIG_NMI_WATCHDOG +int hw_nmi_is_cpu_stuck (struct pt_regs *); +#endif +#endif
kernel/Makefile +1 -0
@@ -76,6 +76,7 @@ obj-$(CONFIG_AUDIT_TREE) += audit_tree.o obj-$(CONFIG_KPROBES) += kprobes.o obj-$(CONFIG_KGDB) += kgdb.o obj-$(CONFIG_DETECT_SOFTLOCKUP) += softlockup.o +obj-$(CONFIG_NMI_WATCHDOG) += nmi_watchdog.o obj-$(CONFIG_DETECT_HUNG_TASK) += hung_task.o obj-$(CONFIG_GENERIC_HARDIRQS) += irq/ obj-$(CONFIG_SECCOMP) += seccomp.o
lib/Kconfig.debug +13 -0
@@ -170,6 +170,19 @@ config DETECT_SOFTLOCKUP can be detected via the NMI-watchdog, on platforms that support it.) +config NMI_WATCHDOG + bool "Detect Hard Lockups with an NMI Watchdog" + depends on DEBUG_KERNEL && PERF_EVENTS + default y + help + Say Y here to enable the kernel to use the NMI as a watchdog + to detect hard lockups. This is useful when a cpu hangs for no + reason but can still respond to NMIs. A backtrace is displayed + for reviewing and reporting. + + The overhead should be minimal, just an extra NMI every few + seconds. + config BOOTPARAM_SOFTLOCKUP_PANIC bool "Panic (Reboot) On Soft Lockups" depends on DETECT_SOFTLOCKUP
[tip:perf/nmi] nmi_watchdog: Only enable on x86 for now
In reply to Don ZickusCommit-ID: 8e7672cdb413af859086ffceaed68f7e1e8ea4c2 Gitweb: http://git.kernel.org/tip/8e7672cdb413af859086ffceaed68f7e1e8ea4c2 Author: Ingo Molnar <[email protected]> AuthorDate: Tue, 9 Feb 2010 06:11:00 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Tue, 9 Feb 2010 06:11:00 +0100
nmi_watchdog: Only enable on x86 for now
It wont even build on other platforms just yet - so restrict it to x86 for now.
Cc: Don Zickus <[email protected]>
Cc: [email protected]
Cc: [email protected]
Cc: [email protected]
LKML-Reference: <[email protected]>
--- lib/Kconfig.debug | 1 + 1 files changed, 1 insertions(+), 0 deletions(-)
lib/Kconfig.debug +1 -0
@@ -173,6 +173,7 @@ config DETECT_SOFTLOCKUP config NMI_WATCHDOG bool "Detect Hard Lockups with an NMI Watchdog" depends on DEBUG_KERNEL && PERF_EVENTS + depends on X86 default y help Say Y here to enable the kernel to use the NMI as a watchdog