Skip to contentSkip to conversation

kernel/printk: prevent deadlock at unexpected call kmsg_dump in NMI context

PATCH In review Konstantin Khlebnikov 2 participants 3 messages Jul 15, 2019 – Jul 16, 2019 Printk

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 kernel/printk: prevent deadlock at unexpected call kmsg_dump in NMI context

Kernel message dumper - function kmsg_dump() is called on various oops or panic paths which could happen in unpredictable context including NMI.

Panic in NMI is handled especially by stopping all other cpus with smp_send_stop() and busting locks in printk_safe_flush_on_panic().

Other less-fatal cases shouldn't happen in NMI and cannot be handled. But this might happen for example on oops in nmi context. In this case dumper could deadlock on lockbuf_lock or break internal structures.

This patch catches kmsg_dump() called in NMI context except panic and prints warning once.

---
 kernel/printk/printk.c |    7 +++++++
 1 file changed, 7 insertions(+)
kernel/printk/printk.c +7 -0
index 1888f6a3b694..e711f64a1843 100644
@@ -3104,6 +3104,13 @@ void kmsg_dump(enum kmsg_dump_reason reason)
 	struct kmsg_dumper *dumper;
 	unsigned long flags;
 
+	/*
+	 * In NMI context only panic could be handled safely:
+	 * it stops other cpus and busts logbuf lock.
+	 */
+	if (WARN_ON_ONCE(reason != KMSG_DUMP_PANIC && in_nmi()))
+		return;
+
 	if ((reason > KMSG_DUMP_OOPS) && !always_kmsg_dump)
 		return;
 
maintainer
#1 lore raw
Quoted 9 lines · Konstantin Khlebnikov

Kernel message dumper - function kmsg_dump() is called on various oops or panic paths which could happen in unpredictable context including NMI.

Panic in NMI is handled especially by stopping all other cpus with smp_send_stop() and busting locks in printk_safe_flush_on_panic().

Other less-fatal cases shouldn't happen in NMI and cannot be handled. But this might happen for example on oops in nmi context. In this case dumper could deadlock on lockbuf_lock or break internal structures.

If I get it correctly than this patch could really prevent a deadlock in at least:

+ oops_end()
+ oops_exit()
+ kmsg_dump(KMSG_DUMP_OOPS)

If it is called in NMI, it should end up with panic(). Then the dump will be called later after stopping CPUs...

Or am I wrong?

Otherwise, the patch looks good to me. I would just mention the above scenario if it is correct.

Best Regards,
Petr

Quoted 28 lines

This patch catches kmsg_dump() called in NMI context except panic and prints warning once.

---
 kernel/printk/printk.c |    7 +++++++
 1 file changed, 7 insertions(+)
kernel/printk/printk.c +7 -0
index 1888f6a3b694..e711f64a1843 100644
@@ -3104,6 +3104,13 @@ void kmsg_dump(enum kmsg_dump_reason reason)
 	struct kmsg_dumper *dumper;
 	unsigned long flags;
 
+	/*
+	 * In NMI context only panic could be handled safely:
+	 * it stops other cpus and busts logbuf lock.
+	 */
+	if (WARN_ON_ONCE(reason != KMSG_DUMP_PANIC && in_nmi()))
+		return;
+
 	if ((reason > KMSG_DUMP_OOPS) && !always_kmsg_dump)
 		return;
 
#2 lore raw
Quoted 14 lines · Petr Mladek
Quoted 9 lines · Konstantin Khlebnikov

Kernel message dumper - function kmsg_dump() is called on various oops or panic paths which could happen in unpredictable context including NMI.

Panic in NMI is handled especially by stopping all other cpus with smp_send_stop() and busting locks in printk_safe_flush_on_panic().

Other less-fatal cases shouldn't happen in NMI and cannot be handled. But this might happen for example on oops in nmi context. In this case dumper could deadlock on lockbuf_lock or break internal structures.

If I get it correctly than this patch could really prevent a deadlock in at least:

Quoted 8 lines

+ oops_end()
+ oops_exit()
+ kmsg_dump(KMSG_DUMP_OOPS)

If it is called in NMI, it should end up with panic(). Then the dump will be called later after stopping CPUs...

Or am I wrong?

Yep. Under 'oops in nmi context' I mean exactly that case.

Quoted 35 lines

Otherwise, the patch looks good to me. I would just mention the above scenario if it is correct.

Best Regards,
Petr

Quoted 28 lines

This patch catches kmsg_dump() called in NMI context except panic and prints warning once.

---
  kernel/printk/printk.c |    7 +++++++
  1 file changed, 7 insertions(+)
kernel/printk/printk.c +7 -0
index 1888f6a3b694..e711f64a1843 100644
@@ -3104,6 +3104,13 @@ void kmsg_dump(enum kmsg_dump_reason reason)
  	struct kmsg_dumper *dumper;
  	unsigned long flags;
 
+	/*
+	 * In NMI context only panic could be handled safely:
+	 * it stops other cpus and busts logbuf lock.
+	 */
+	if (WARN_ON_ONCE(reason != KMSG_DUMP_PANIC && in_nmi()))
+		return;
+
  	if ((reason > KMSG_DUMP_OOPS) && !always_kmsg_dump)
  		return;