kernel/printk: prevent deadlock at calling kmsg_dump from NMI context
PATCH 1/1 kernel/printk: prevent deadlock at calling kmsg_dump from NMI context
Function kmsg_dump could be invoked from NMI context intentionally or accidentally because it is called at various oops/panic paths. Kernel message dumpers are not ready to work in NMI context right now. They could deadlock on lockbuf_lock or break internal structures.
Theoretically some dumpers could be fixed and marked as NMI-safe. Right now it's safer to print warning once and give up.
--- kernel/printk/printk.c | 3 +++ 1 file changed, 3 insertions(+)
kernel/printk/printk.c +3 -0
@@ -3104,6 +3104,9 @@ void kmsg_dump(enum kmsg_dump_reason reason)struct kmsg_dumper *dumper;unsigned long flags; +if (WARN_ON_ONCE (in_nmi ())) +return ; +if ((reason >KMSG_DUMP_OOPS ) && !always_kmsg_dump)return ;
Quoted 4 lines · Konstantin Khlebnikov
Function
kmsg_dumpcould be invoked from NMI context intentionally or accidentally because it is called at various oops/panic paths. Kernel message dumpers are not ready to work in NMI context right now. They could deadlock onlockbuf_lockor break internal structures.
Hmm.
printk()-s from NMI go through per-CPU printk_safe/nmi - a bunch of lockless buffers which is supposed to deal with printk() deadlocks, including NMI printk()-s.
include/linux/hardirq.h
#define nmi_enter() ... printk_nmi_enter(); ...
#define nmi_exit() ... printk_nmi_exit(); ...
So we are not really supposed to deadlock.
-ss
Quoted 19 lines · Sergey Senozhatsky
Quoted 4 lines · Konstantin Khlebnikov
Function
kmsg_dumpcould be invoked from NMI context intentionally or accidentally because it is called at various oops/panic paths. Kernel message dumpers are not ready to work in NMI context right now. They could deadlock onlockbuf_lockor break internal structures.Hmm.
printk()-s from NMI go through per-CPU printk_safe/nmi - a bunch of lockless buffers which is supposed to deal withprintk()deadlocks, including NMI printk()-s.
include/linux/hardirq.h#define nmi_enter() ... printk_nmi_enter(); ...
Quoted 6 lines
#define nmi_exit() ... printk_nmi_exit(); ...So we are not really supposed to deadlock.
Yep printk() can deal with NMI, but kmsg_dump() is a different beast. It reads printk buffer and saves content into persistent storage like ACPI ERST.
Quoted 2 lines
-ssQuoted 4 lines · Konstantin Khlebnikov
Quoted 1 line · Konstantin Khlebnikov
Yep
printk()can deal with NMI, butkmsg_dump()is a different beast. It reads printk buffer and saves content into persistent storage like ACPI ERST.
Ah, sorry! I misread your patch. Yeah, I see what you are doing.
OK. So, I guess that for kmsg_dump(KMSG_DUMP_PANIC) we should be fine in general.
We call kmsg_dump(KMSG_DUMP_PANIC) after smp_send_stop() and after printk_safe_flush_on_panic(). printk_safe_flush_on_panic() resets the state of logbuf_lock, so logbuf_lock, in general case, should be unlocked by the time we call kmsg_dump(KMSG_DUMP_PANIC). Even for nested contexts.
CPU0 printk() logbuf_lock_irqsave(flags) -> NMI panic() smp_send_stop() printk_safe_flush_on_panic() raw_spin_lock_init(&logbuf_lock) << reinit >> kmsg_dump(KMSG_DUMP_PANIC) logbuf_lock_irqsave(flags) << expected to be OK >>
So do we have strong reasons to disable NMI->panic->kmsg_dump(DUMP_PANIC)?
Other kmsg_dump(), maybe, can experience some troubles sometimes, need to check that.
-ss
Quoted 29 lines · Sergey Senozhatsky
Quoted 4 lines · Konstantin Khlebnikov
Quoted 1 line · Konstantin Khlebnikov
Yep
printk()can deal with NMI, butkmsg_dump()is a different beast. It reads printk buffer and saves content into persistent storage like ACPI ERST.Ah, sorry! I misread your patch. Yeah, I see what you are doing.
OK. So, I guess that for
kmsg_dump(KMSG_DUMP_PANIC)we should be fine in general.We call
kmsg_dump(KMSG_DUMP_PANIC)aftersmp_send_stop()and afterprintk_safe_flush_on_panic().printk_safe_flush_on_panic()resets the state oflogbuf_lock, sologbuf_lock, in general case, should be unlocked by the time we callkmsg_dump(KMSG_DUMP_PANIC). Even for nested contexts.CPU0 printk() logbuf_lock_irqsave(flags) -> NMI panic() smp_send_stop() printk_safe_flush_on_panic() raw_spin_lock_init(&logbuf_lock) << reinit >> kmsg_dump(KMSG_DUMP_PANIC) logbuf_lock_irqsave(flags) << expected to be OK >>
Quoted 4 lines
So do we have strong reasons to disable
NMI->panic->kmsg_dump(DUMP_PANIC)?Other
kmsg_dump(), maybe, can experience some troubles sometimes, need to check that.
Indeed, panic is especially handled and looks fine.
Sanity check in my patch could be relaxed:
if (WARN_ON_ONCE(reason != KMSG_DUMP_PANIC && in_nmi()))
return;Quoted 2 lines
-ssQuoted 22 lines · Konstantin Khlebnikov
Quoted 17 lines
We call
kmsg_dump(KMSG_DUMP_PANIC)aftersmp_send_stop()and afterprintk_safe_flush_on_panic().printk_safe_flush_on_panic()resets the state oflogbuf_lock, sologbuf_lock, in general case, should be unlocked by the time we callkmsg_dump(KMSG_DUMP_PANIC). Even for nested contexts.CPU0 printk() logbuf_lock_irqsave(flags) -> NMI panic() smp_send_stop() printk_safe_flush_on_panic() raw_spin_lock_init(&logbuf_lock) << reinit >> kmsg_dump(KMSG_DUMP_PANIC) logbuf_lock_irqsave(flags) << expected to be OK >>Quoted 4 lines
So do we have strong reasons to disable
NMI->panic->kmsg_dump(DUMP_PANIC)?Other
kmsg_dump(), maybe, can experience some troubles sometimes, need to check that.
Quoted 6 lines
Indeed, panic is especially handled and looks fine.
Sanity check in my patch could be relaxed:
if (WARN_ON_ONCE(reason != KMSG_DUMP_PANIC && in_nmi())) return;
How critical kmsg_dump() is? We deadlock only if NMI->kmsg_dump() happens on the CPU which already holds the logbuf_lock; in any other case logbuf_lock is owned by another CPU which is expected to unlock it eventually. So it doesn't look like disabling all NMI->kmsg_dump() is the only way to fix it.
When we lock logbuf_lock we increment per-CPU printk_context (PRINTK_SAFE_CONTEXT_MASK bits); when we unlock logbuf_lock we decrement printk_context. Thus we always can tell if the logbuf_lock was locked on the very same CPU - this_cpu printk_context has PRINTK_SAFE_CONTEXT_MASK bits sets - and we are about to deadlock in a nested context (NMI), or the lock was locked on another CPU and it's "safe" to spin on logbuf_lock and wait for logbuf_lock to become available.
-ss
Quoted 36 lines · Sergey Senozhatsky
Quoted 22 lines · Konstantin Khlebnikov
Quoted 17 lines
We call
kmsg_dump(KMSG_DUMP_PANIC)aftersmp_send_stop()and afterprintk_safe_flush_on_panic().printk_safe_flush_on_panic()resets the state oflogbuf_lock, sologbuf_lock, in general case, should be unlocked by the time we callkmsg_dump(KMSG_DUMP_PANIC). Even for nested contexts.CPU0 printk() logbuf_lock_irqsave(flags) -> NMI panic() smp_send_stop() printk_safe_flush_on_panic() raw_spin_lock_init(&logbuf_lock) << reinit >> kmsg_dump(KMSG_DUMP_PANIC) logbuf_lock_irqsave(flags) << expected to be OK >>Quoted 4 lines
So do we have strong reasons to disable
NMI->panic->kmsg_dump(DUMP_PANIC)?Other
kmsg_dump(), maybe, can experience some troubles sometimes, need to check that.Quoted 6 lines
Indeed, panic is especially handled and looks fine.
Sanity check in my patch could be relaxed:
if (WARN_ON_ONCE(reason != KMSG_DUMP_PANIC && in_nmi())) return;How critical
kmsg_dump()is? We deadlock only ifNMI->kmsg_dump()happens on the CPU which already holds thelogbuf_lock; in any other caselogbuf_lockis owned by another CPU which is expected to unlock it eventually. So it doesn't look like disabling allNMI->kmsg_dump()is the only way to fix it.
Quoted 8 lines
When we lock
logbuf_lockwe increment per-CPUprintk_context(PRINTK_SAFE_CONTEXT_MASKbits); when we unlocklogbuf_lockwe decrementprintk_context. Thus we always can tell if thelogbuf_lockwas locked on the very same CPU -this_cpuprintk_contexthasPRINTK_SAFE_CONTEXT_MASKbits sets - and we are about to deadlock in a nested context (NMI), or the lock was locked on another CPU and it's "safe" to spin onlogbuf_lockand wait forlogbuf_lockto become available.
I see no users who dumps dmesg from NMI context except panic. This shouldn't happen. But might happen is something goes very wrong.
So, this trickery is not required.
Also spinning in NMI for handling non-fatal cases is a bad idea.
Quoted 22 lines · Konstantin Khlebnikov
Quoted 13 lines
Quoted 6 lines
Indeed, panic is especially handled and looks fine.
Sanity check in my patch could be relaxed:
if (WARN_ON_ONCE(reason != KMSG_DUMP_PANIC && in_nmi())) return;How critical
kmsg_dump()is? We deadlock only ifNMI->kmsg_dump()happens on the CPU which already holds thelogbuf_lock; in any other caselogbuf_lockis owned by another CPU which is expected to unlock it eventually. So it doesn't look like disabling allNMI->kmsg_dump()is the only way to fix it.Quoted 8 lines
When we lock
logbuf_lockwe increment per-CPUprintk_context(PRINTK_SAFE_CONTEXT_MASKbits); when we unlocklogbuf_lockwe decrementprintk_context. Thus we always can tell if thelogbuf_lockwas locked on the very same CPU -this_cpuprintk_contexthasPRINTK_SAFE_CONTEXT_MASKbits sets - and we are about to deadlock in a nested context (NMI), or the lock was locked on another CPU and it's "safe" to spin onlogbuf_lockand wait forlogbuf_lockto become available.
Quoted 2 lines
I see no users who dumps dmesg from NMI context except panic. This shouldn't happen. But might happen is something goes very wrong.
I see. OK, if NMI->kmsg_dump() never happens (except for panic) then we are fine, I guess.
-ss
Petr Mladek
maintainerQuoted 4 lines · Sergey Senozhatsky
Quoted 3 lines · Konstantin Khlebnikov
Quoted 1 line
We call
kmsg_dump(KMSG_DUMP_PANIC)aftersmp_send_stop()Indeed, panic is especially handled and looks fine.
Just to get a picture. What other situations are we talking about, please?
oops_exit() is one candidate. The other callers seem to be working in normal context.
Quoted 11 lines
Quoted 4 lines
Sanity check in my patch could be relaxed:
if (WARN_ON_ONCE(reason != KMSG_DUMP_PANIC && in_nmi())) return;How critical
kmsg_dump()is? We deadlock only ifNMI->kmsg_dump()happens on the CPU which already holds thelogbuf_lock; in any other caselogbuf_lockis owned by another CPU which is expected to unlock it eventually. So it doesn't look like disabling allNMI->kmsg_dump()is the only way to fix it.
Quoted 8 lines
When we lock
logbuf_lockwe increment per-CPUprintk_context(PRINTK_SAFE_CONTEXT_MASKbits); when we unlocklogbuf_lockwe decrementprintk_context. Thus we always can tell if thelogbuf_lockwas locked on the very same CPU -this_cpuprintk_contexthasPRINTK_SAFE_CONTEXT_MASKbits sets - and we are about to deadlock in a nested context (NMI), or the lock was locked on another CPU and it's "safe" to spin onlogbuf_lockand wait forlogbuf_lockto become available.
This sounds familiar. I think that we did not consider it safe in the end, see the commit 03fc7f9c99c1e7ae29 ("printk/nmi: Prevent deadlock when accessing the main log buffer in NMI").
If the problem is only with Oops then the 2nd propose might be acceptable. The system will either try to continue or it will end up in panic() anyway.
Well, WARN() looks like an overkill, especially if there is only one possible caller that prints the stack anyway. I would personally do not print any message and do just:
/* * Prevent deadlock on logbuf_lock. It is safe only * in panic() after smp_send_stop() and resetting * the lock. */ if (in_nmi() && reason != KMSG_DUMP_PANIC) return;
Best Regards,
Petr
Quoted 11 lines · Petr Mladek
Quoted 4 lines · Sergey Senozhatsky
Quoted 3 lines · Konstantin Khlebnikov
Quoted 1 line
We call
kmsg_dump(KMSG_DUMP_PANIC)aftersmp_send_stop()Indeed, panic is especially handled and looks fine.
Just to get a picture. What other situations are we talking about, please?
oops_exit()is one candidate. The other callers seem to be working in normal context.
Oops in NMI mostly. Also I've screwed up several times with NMI watchdog and dumping log at setting taint.
Quoted 34 lines
Quoted 13 lines
Quoted 6 lines
Sanity check in my patch could be relaxed:
if (WARN_ON_ONCE(reason != KMSG_DUMP_PANIC && in_nmi())) return;How critical
kmsg_dump()is? We deadlock only ifNMI->kmsg_dump()happens on the CPU which already holds thelogbuf_lock; in any other caselogbuf_lockis owned by another CPU which is expected to unlock it eventually. So it doesn't look like disabling allNMI->kmsg_dump()is the only way to fix it.Quoted 8 lines
When we lock
logbuf_lockwe increment per-CPUprintk_context(PRINTK_SAFE_CONTEXT_MASKbits); when we unlocklogbuf_lockwe decrementprintk_context. Thus we always can tell if thelogbuf_lockwas locked on the very same CPU -this_cpuprintk_contexthasPRINTK_SAFE_CONTEXT_MASKbits sets - and we are about to deadlock in a nested context (NMI), or the lock was locked on another CPU and it's "safe" to spin onlogbuf_lockand wait forlogbuf_lockto become available.This sounds familiar. I think that we did not consider it safe in the end, see the commit
03fc7f9c99c1e7ae29("printk/nmi: Prevent deadlock when accessing the main log buffer in NMI").If the problem is only with Oops then the 2nd propose might be acceptable. The system will either try to continue or it will end up in
panic()anyway.Well,
WARN()looks like an overkill, especially if there is only one possible caller that prints the stack anyway. I would personally do not print any message and do just:
Quoted 8 lines
/* * Prevent deadlock on logbuf_lock. It is safe only * in panic() after smp_send_stop() and resetting * the lock. */ if (in_nmi() && reason != KMSG_DUMP_PANIC) return;
WARN_ON_ONCE is useful timesaver in debugging. It's better to know when happens something that shouldn't.
Quoted 3 lines
Best Regards,
Petr