x86 NMI: Be smarter about invoking panic() inside NMI handler.
PATCH 1/1 x86 NMI: Be smarter about invoking panic() inside NMI handler.
If two (or more) unknown NMIs arrive on different CPUs, there is a large chance both CPUs will wind up inside panic(). This is fine, unless you want to enter KDB - KDB cannot round up all CPUs, because some of them are stuck inside panic_smp_self_stop with NMI latched. This is
easy to replicate with QEMU. Boot with -smp 4 and send NMI using the monitor.
Solution for this - attempt to enter panic() from NMI handler. If panic() is already active in the system, just exit out of the NMI handler. This lets KDB round up CPUs.
--- arch/x86/kernel/nmi.c | 6 ++-- include/linux/kernel.h | 1 + kernel/panic.c | 82 +++++++++++++++++++++++++++++++++++++----------- 3 files changed, 67 insertions(+), 22 deletions(-)
arch/x86/kernel/nmi.c +3 -3
@@ -227,7 +227,7 @@ pci_serr_error(unsigned char reason, struct pt_regs *regs)#endif if (panic_on_unrecovered_nmi) -panic ("NMI: Not continuing" ); +try_panic ("NMI: Not continuing" );pr_emerg ("Dazed and confused, but trying to continue\n" );
@@ -247,7 +247,7 @@ io_check_error(unsigned char reason, struct pt_regs *regs)show_registers (regs);if (panic_on_io_nmi) -panic ("NMI IOCK error: Not continuing" ); +try_panic ("NMI IOCK error: Not continuing" );/* Re-enable the IOCK line, wait for a few seconds */ reason = (reason &NMI_REASON_CLEAR_MASK ) |NMI_REASON_CLEAR_IOCHK ;
@@ -297,7 +297,7 @@ unknown_nmi_error(unsigned char reason, struct pt_regs *regs)pr_emerg ("Do you have a strange power saving mode enabled?\n" );if (unknown_nmi_panic || panic_on_unrecovered_nmi) -panic ("NMI: Not continuing" ); +try_panic ("NMI: Not continuing" );pr_emerg ("Dazed and confused, but trying to continue\n" ); }
include/linux/kernel.h +1 -0
@@ -188,6 +188,7 @@ extern long (*panic_blink)(int state); __printf(1, 2)void panic (const char *fmt, ...)__noreturn __cold ; +void try_panic (const char *fmt, ...)__cold ;extern void oops_enter (void );extern void oops_exit (void );void print_oops_end_marker (void );
kernel/panic.c +63 -19
111 lines not on this page · raw message
Signature · 1 line
1.7.9.2
Hi,
2012/3/1 Andrei Warkentin <[email protected]>:
Quoted 8 lines
If two (or more) unknown NMIs arrive on different CPUs, there is a large chance both CPUs will wind up inside
panic(). This is fine, unless you want to enter KDB - KDB cannot round up all CPUs, because some of them are stuck insidepanic_smp_self_stopwith NMI latched. This is
easy to replicate with QEMU. Boot with -smp 4 and send NMI using the monitor.
Quoted 7 lines
Solution for this - attempt to enter
panic()from NMI handler. Ifpanic()is already active in the system, just exit out of the NMI handler. This lets KDB round up CPUs.Signed-off-by Andrei Warkentin <[email protected]>---
Any feedback on this? Who are the right maintainers to bug about this?
A
Quoted 16 lines · Andrei E. Warkentin
Hi,
2012/3/1 Andrei Warkentin <[email protected]>:
Quoted 13 lines
If two (or more) unknown NMIs arrive on different CPUs, there is a large chance both CPUs will wind up inside
panic(). This is fine, unless you want to enter KDB - KDB cannot round up all CPUs, because some of them are stuck insidepanic_smp_self_stopwith NMI latched. This is
easy to replicate with QEMU. Boot with -smp 4 and send NMI using the monitor.Solution for this - attempt to enter
panic()from NMI handler. Ifpanic()is already active in the system, just exit out of the NMI handler. This lets KDB round up CPUs.
Quoted 4 lines
Quoted 2 lines
Signed-off-by Andrei Warkentin <[email protected]>---
Any feedback on this? Who are the right maintainers to bug about this?
Hmm, if try_panic fails, then the cpu continues on executing code. This might further corrupt an already broken system. So I don't think this patch will work as is.
Perhaps instead of panic'ing in the NMI context, we use irq_work and panic in an interrupt context instead. We still get the system to stop (though it might still execute some interrupts) and it will be out of the NMI context.
However, you will still run into a similar problem when in the panic/reboot case we shutdown all the remote cpus and have them sitting in a similar cpu_relax loop in the NMI context, while the panic'ing cpu cleans things up.
Cheers,
Don
Hi Don,
Thank you for your feedback!
2012/3/27 Don Zickus <[email protected]>:
Quoted 5 lines
Hmm, if
try_panicfails, then the cpu continues on executing code. This might further corrupt an already broken system. So I don't think this patch will work as is.
I see what you are saying. I could make the argument that this kind of system corruption could occur anyway even if you did panic inside an IRQ context instead, but I would tend to agree that your proposed solution is much better than adding another panic interface.
Quoted 10 lines
Perhaps instead of panic'ing in the NMI context, we use
irq_workand panic in an interrupt context instead. We still get the system to stop (though it might still execute some interrupts) and it will be out of the NMI context.However, you will still run into a similar problem when in the panic/reboot case we shutdown all the remote cpus and have them sitting in a similar
cpu_relaxloop in the NMI context, while the panic'ing cpu cleans things up.
Sorry, could you clarify what you mean? How does this affect KDB usage?
A
Quoted 21 lines · Andrei E. Warkentin
Hi Don,
Thank you for your feedback!
2012/3/27 Don Zickus <[email protected]>:
Quoted 5 lines
Hmm, if
try_panicfails, then the cpu continues on executing code. This might further corrupt an already broken system. So I don't think this patch will work as is.I see what you are saying. I could make the argument that this kind of system corruption could occur anyway even if you did panic inside an IRQ context instead, but I would tend to agree that your proposed solution is much better than adding another panic interface.
Quoted 5 lines
Perhaps instead of panic'ing in the NMI context, we use
irq_workand panic in an interrupt context instead. We still get the system to stop (though it might still execute some interrupts) and it will be out of the NMI context.
Quoted 7 lines
Quoted 5 lines
However, you will still run into a similar problem when in the panic/reboot case we shutdown all the remote cpus and have them sitting in a similar
cpu_relaxloop in the NMI context, while the panic'ing cpu cleans things up.Sorry, could you clarify what you mean? How does this affect KDB usage?
I figured it would affect it the same way you described in your panic scenario. The machine panics and you are trying to break in with KDB. The above issue just says the other cpus could block KDB from stopping all the cpus much like your original issue.
But I will admit I didn't fully understand the original problem you were trying to solve.
Cheers,
Don
Quoted 2 lines
A