Skip to contentSkip to conversation

x86 NMI: Be smarter about invoking panic() inside NMI handler.

PATCH In review Andrei Warkentin 2 participants 5 messages Mar 1, 2012 – Mar 29, 2012 x86

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 x86 NMI: Be smarter about invoking panic() inside NMI handler.

If two (or more) unknown NMIs arrive on different CPUs, there is a large chance both CPUs will wind up inside panic(). This is fine, unless you want to enter KDB - KDB cannot round up all CPUs, because some of them are stuck inside panic_smp_self_stop with NMI latched. This is
easy to replicate with QEMU. Boot with -smp 4 and send NMI using the monitor.

Solution for this - attempt to enter panic() from NMI handler. If panic() is already active in the system, just exit out of the NMI handler. This lets KDB round up CPUs.

---
 arch/x86/kernel/nmi.c  |    6 ++--
 include/linux/kernel.h |    1 +
 kernel/panic.c         |   82 +++++++++++++++++++++++++++++++++++++-----------
 3 files changed, 67 insertions(+), 22 deletions(-)
arch/x86/kernel/nmi.c +3 -3
index 47acaf3..9e6a69a 100644
@@ -227,7 +227,7 @@ pci_serr_error(unsigned char reason, struct pt_regs *regs)
 #endif
 
 	if (panic_on_unrecovered_nmi)
-		panic("NMI: Not continuing");
+		try_panic("NMI: Not continuing");
 
 	pr_emerg("Dazed and confused, but trying to continue\n");
 
@@ -247,7 +247,7 @@ io_check_error(unsigned char reason, struct pt_regs *regs)
 	show_registers(regs);
 
 	if (panic_on_io_nmi)
-		panic("NMI IOCK error: Not continuing");
+		try_panic("NMI IOCK error: Not continuing");
 
 	/* Re-enable the IOCK line, wait for a few seconds */
 	reason = (reason & NMI_REASON_CLEAR_MASK) | NMI_REASON_CLEAR_IOCHK;
@@ -297,7 +297,7 @@ unknown_nmi_error(unsigned char reason, struct pt_regs *regs)
 
 	pr_emerg("Do you have a strange power saving mode enabled?\n");
 	if (unknown_nmi_panic || panic_on_unrecovered_nmi)
-		panic("NMI: Not continuing");
+		try_panic("NMI: Not continuing");
 
 	pr_emerg("Dazed and confused, but trying to continue\n");
 }
include/linux/kernel.h +1 -0
index 01ab0aa..82983f9 100644
@@ -188,6 +188,7 @@ extern long (*panic_blink)(int state);
 __printf(1, 2)
 void panic(const char *fmt, ...)
 	__noreturn __cold;
+void try_panic(const char *fmt, ...) __cold;
 extern void oops_enter(void);
 extern void oops_exit(void);
 void print_oops_end_marker(void);
kernel/panic.c +63 -19

111 lines not on this page · raw message

Signature · 1 line
1.7.9.2
#1 lore raw

Hi,

2012/3/1 Andrei Warkentin <[email protected]>:

Quoted 8 lines

If two (or more) unknown NMIs arrive on different CPUs, there is a large chance both CPUs will wind up inside panic(). This is fine, unless you want to enter KDB - KDB cannot round up all CPUs, because some of them are stuck inside panic_smp_self_stop with NMI latched. This is
easy to replicate with QEMU. Boot with -smp 4 and send NMI using the monitor.

Quoted 7 lines

Solution for this - attempt to enter panic() from NMI handler. If panic() is already active in the system, just exit out of the NMI handler. This lets KDB round up CPUs.

---

Any feedback on this? Who are the right maintainers to bug about this?

A

#2 lore raw
Quoted 16 lines · Andrei E. Warkentin

Hi,

2012/3/1 Andrei Warkentin <[email protected]>:

Quoted 13 lines

If two (or more) unknown NMIs arrive on different CPUs, there is a large chance both CPUs will wind up inside panic(). This is fine, unless you want to enter KDB - KDB cannot round up all CPUs, because some of them are stuck inside panic_smp_self_stop with NMI latched. This is
easy to replicate with QEMU. Boot with -smp 4 and send NMI using the monitor.

Solution for this - attempt to enter panic() from NMI handler. If panic() is already active in the system, just exit out of the NMI handler. This lets KDB round up CPUs.

Quoted 4 lines
Quoted 2 lines

---

Any feedback on this? Who are the right maintainers to bug about this?

Hmm, if try_panic fails, then the cpu continues on executing code. This might further corrupt an already broken system. So I don't think this patch will work as is.

Perhaps instead of panic'ing in the NMI context, we use irq_work and panic in an interrupt context instead. We still get the system to stop (though it might still execute some interrupts) and it will be out of the NMI context.

However, you will still run into a similar problem when in the panic/reboot case we shutdown all the remote cpus and have them sitting in a similar cpu_relax loop in the NMI context, while the panic'ing cpu cleans things up.

Cheers,
Don

#3 lore raw

Hi Don,

Thank you for your feedback!

2012/3/27 Don Zickus <[email protected]>:

Quoted 5 lines

Hmm, if try_panic fails, then the cpu continues on executing code.  This might further corrupt an already broken system.  So I don't think this patch will work as is.

I see what you are saying. I could make the argument that this kind of system corruption could occur anyway even if you did panic inside an IRQ context instead, but I would tend to agree that your proposed solution is much better than adding another panic interface.

Quoted 10 lines

Perhaps instead of panic'ing in the NMI context, we use irq_work and panic in an interrupt context instead.  We still get the system to stop (though it might still execute some interrupts) and it will be out of the NMI context.

However, you will still run into a similar problem when in the panic/reboot case we shutdown all the remote cpus and have them sitting in a similar cpu_relax loop in the NMI context, while the panic'ing cpu cleans things up.

Sorry, could you clarify what you mean? How does this affect KDB usage?

A

#4 lore raw
Quoted 21 lines · Andrei E. Warkentin

Hi Don,

Thank you for your feedback!

2012/3/27 Don Zickus <[email protected]>:

Quoted 5 lines

Hmm, if try_panic fails, then the cpu continues on executing code.  This might further corrupt an already broken system.  So I don't think this patch will work as is.

I see what you are saying. I could make the argument that this kind of system corruption could occur anyway even if you did panic inside an IRQ context instead, but I would tend to agree that your proposed solution is much better than adding another panic interface.

Quoted 5 lines

Perhaps instead of panic'ing in the NMI context, we use irq_work and panic in an interrupt context instead.  We still get the system to stop (though it might still execute some interrupts) and it will be out of the NMI context.

Quoted 7 lines
Quoted 5 lines

However, you will still run into a similar problem when in the panic/reboot case we shutdown all the remote cpus and have them sitting in a similar cpu_relax loop in the NMI context, while the panic'ing cpu cleans things up.

Sorry, could you clarify what you mean? How does this affect KDB usage?

I figured it would affect it the same way you described in your panic scenario. The machine panics and you are trying to break in with KDB. The above issue just says the other cpus could block KDB from stopping all the cpus much like your original issue.

But I will admit I didn't fully understand the original problem you were trying to solve.

Cheers,
Don

Quoted 2 lines

A