NULL pointer crash in early NMI handler
I'm hitting this bug in latest Linus tree:
[ 0.161089] Setting APIC routing to flat [ 0.171346] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=0 pin2=0 [ 0.180001] BUG: unable to handle kernel NULL pointer dereference at (null) [ 0.180001] IP: [<ffffffff8063f8a6>] nmi_watchdog_tick+0xd0/0x27d
[...]
[ 0.180001] Call Trace: [ 0.180001] <NMI> <0> [<ffffffff8063e9b7>] do_nmi+0x12e/0x3af [ 0.180001] [<ffffffff8063e59a>] nmi+0x1a/0x2c [ 0.180001] [<ffffffff806415c2>] ? add_preempt_count+0xdc/0x18b [ 0.180001] <<EOE>> <0> [<ffffffff8040944c>] delay_tsc+0xa7/0x13b [ 0.180001] [<ffffffff804092df>] __delay+0xf/0x11 [ 0.180001] [<ffffffff80409322>] __const_udelay+0x41/0x43 [ 0.180001] [<ffffffff80f18539>] timer_irq_works+0x4e/0xb0 [ 0.180001] [<ffffffff80f18ad4>] setup_IO_APIC+0x539/0xb26 [ 0.180001] [<ffffffff8041b840>] ? debug_smp_processor_id+0x38/0x170 [ 0.180001] [<ffffffff80226152>] ? setup_apic_nmi_watchdog+0xb8/0xdb [ 0.180001] [<ffffffff80f14231>] native_smp_prepare_cpus+0x606/0x6be [ 0.180001] [<ffffffff80f05a30>] kernel_init+0x56/0x1fc [ 0.180001] [<ffffffff8020d7fa>] child_rip+0xa/0x20 [ 0.180001] [<ffffffff8020d1c0>] ? restore_args+0x0/0x30 [ 0.180001] [<ffffffff80f059da>] ? kernel_init+0x0/0x1fc [ 0.180001] [<ffffffff8020d7f0>] ? child_rip+0x0/0x20
Looking into exactly where it crashed, it seems to be when it accesses the CPU mask variable backtrace_mask.
When the APIC routing is set to flat, it somehow starts triggering the NMI watchdog. This happens before we run "check_nmi_watchdog" which is what allocates the backtrace_mask cpu mask.
Yes I have CONFIG_CPUMASK_OFFSTACK=y.
When I disable it, the box boots up fine.
-- Steve
PATCH 1/1 x86: fix boot crash in NMI watchdog with CONFIG_CPUMASK_OFFSTACK=y and flat APIC
fcef8576d8a64fc603e719c97d423f9f6d4e0e8b converted backtrace_mask to a cpumask_var_t, and assumed check_nmi_watchdog was called before nmi_watchdog_tick was ever called. Steven's oops shows I was wrong.
This is something of a bandaid: I'm not sure we *should* be calling nmi_watchdog_tick before check_nmi_watchdog. Note that gcc eliminates this test for the CONFIG_CPUMASK_OFFSTACK=n case.
LKML Message-ID: <[email protected]>
--- arch/x86/kernel/apic/nmi.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
arch/x86/kernel/apic/nmi.c +2 -1
@@ -414,7 +414,8 @@ nmi_watchdog_tick(struct pt_regs *regs, touched =1 ; } -if (cpumask_test_cpu (cpu, backtrace_mask)) { +/* We can be called before check_nmi_watchdog, hence NULL check. */ +if (backtrace_mask !=NULL &&cpumask_test_cpu (cpu, backtrace_mask)) {static DEFINE_SPINLOCK (lock);/* Serialise the printks */ spin_lock (&lock);
PATCH 1/1 x86: avoid theoretical spurious NMI backtraces with CONFIG_CPUMASK_OFFSTACK=y
In theory (though not shown in practice) alloc_cpumask_var() doesn't zero memory, so CPUs might print an "NMI backtrace for cpu %d" once on boot.
(Bug introduced in fcef8576d8a64fc603e719c97d423f9f6d4e0e8b).
--- arch/x86/kernel/apic/nmi.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kernel/apic/nmi.c +1 -1
@@ -138,7 +138,7 @@ int __init check_nmi_watchdog(void)if (!prev_nmi_count) goto error; -alloc_cpumask_var (&backtrace_mask,GFP_KERNEL ); +alloc_cpumask_var (&backtrace_mask,GFP_KERNEL |__GFP_ZERO);printk (KERN_INFO "Testing NMI watchdog ... " );#ifdef CONFIG_SMP
[tip:x86/urgent] x86: fix boot crash in NMI watchdog with CONFIG_CPUMASK_OFFSTACK=y and flat APIC
In reply to Steven RostedtCommit-ID: 2f537a9f8e82f55c241b002c8cfbf34303b45ada Gitweb: http://git.kernel.org/tip/2f537a9f8e82f55c241b002c8cfbf34303b45ada Author: Rusty Russell <[email protected]> AuthorDate: Tue, 21 Apr 2009 16:00:15 +0930 Committer: Ingo Molnar <[email protected]> CommitDate: Tue, 21 Apr 2009 10:09:49 +0200
x86: fix boot crash in NMI watchdog with CONFIG_CPUMASK_OFFSTACK=y and flat APIC
fcef8576d8a64fc603e719c97d423f9f6d4e0e8b converted backtrace_mask to a cpumask_var_t, and assumed check_nmi_watchdog was called before nmi_watchdog_tick was ever called. Steven's oops shows I was wrong.
This is something of a bandaid: I'm not sure we *should* be calling nmi_watchdog_tick before check_nmi_watchdog. Note that gcc eliminates this test for the CONFIG_CPUMASK_OFFSTACK=n case.
[ Impact: fix boot crash in rare configs ]
LKML-Reference: <[email protected]>
--- arch/x86/kernel/apic/nmi.c | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-)
arch/x86/kernel/apic/nmi.c +2 -1
@@ -414,7 +414,8 @@ nmi_watchdog_tick(struct pt_regs *regs, unsigned reason) touched =1 ; } -if (cpumask_test_cpu (cpu, backtrace_mask)) { +/* We can be called before check_nmi_watchdog, hence NULL check. */ +if (backtrace_mask !=NULL &&cpumask_test_cpu (cpu, backtrace_mask)) {static DEFINE_SPINLOCK (lock);/* Serialise the printks */ spin_lock (&lock);
[tip:x86/urgent] x86: avoid theoretical spurious NMI backtraces with CONFIG_CPUMASK_OFFSTACK=y
In reply to Steven RostedtCommit-ID: fcc5c4a2feea3886dc058498b28508b2731720d5 Gitweb: http://git.kernel.org/tip/fcc5c4a2feea3886dc058498b28508b2731720d5 Author: Rusty Russell <[email protected]> AuthorDate: Tue, 21 Apr 2009 16:03:41 +0930 Committer: Ingo Molnar <[email protected]> CommitDate: Tue, 21 Apr 2009 10:09:50 +0200
x86: avoid theoretical spurious NMI backtraces with CONFIG_CPUMASK_OFFSTACK=y
In theory (though not shown in practice) alloc_cpumask_var() doesn't zero memory, so CPUs might print an "NMI backtrace for cpu %d" once on boot.
(Bug introduced in fcef8576d8a64fc603e719c97d423f9f6d4e0e8b).
[ Impact: avoid theoretical syslog noise in rare configs ]
LKML-Reference: <[email protected]>
--- arch/x86/kernel/apic/nmi.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
arch/x86/kernel/apic/nmi.c +1 -1
@@ -138,7 +138,7 @@ int __init check_nmi_watchdog(void)if (!prev_nmi_count) goto error; -alloc_cpumask_var (&backtrace_mask,GFP_KERNEL ); +alloc_cpumask_var (&backtrace_mask,GFP_KERNEL |__GFP_ZERO);printk (KERN_INFO "Testing NMI watchdog ... " );#ifdef CONFIG_SMP