Skip to contentSkip to conversation

panic: move bust_spinlocks(0) after console_flush_on_panic() to avoid deadlocks

PATCH Superseded Hoeun Ryu 5 participants 5 messages Jun 4, 2018 – Jun 21, 2018

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

Hoeun Ryu

author
#0 lore raw

PATCH 1/1 panic: move bust_spinlocks(0) after console_flush_on_panic() to avoid deadlocks

From: Hoeun Ryu <[email protected]>

Many console device drivers hold the uart_port->lock spinlock with irq enabled
(using spin_lock()) while the device drivers are writing characters to their devices, but the device drivers just try to hold the spin lock (using spin_trylock()) if "oops_in_progress" is equal or greater than 1 to avoid deadlocks.

There is a case ocurring a deadlock related to the lock and oops_in_progress. A CPU
could be stopped by smp_send_stop() while it was holding the port lock because irq was enabled. Once a CPU stops, it doesn't respond interrupts anymore and the lock stays locked forever.

console_flush_on_panic() is called during panic() and it eventually holds the uart
lock but the lock is held by another stopped CPU and it is a deadlock. By moving bust_spinlocks(0) after console_flush_on_panic(), let the console device drivers think the Oops is still in progress to call spin_trylock() instead of spin_lock() to avoid the deadlock.

---
 kernel/panic.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
kernel/panic.c +2 -2
index 42e4874..b4063b6 100644
@@ -233,8 +233,6 @@ void panic(const char *fmt, ...)
 	if (_crash_kexec_post_notifiers)
 		__crash_kexec(NULL);
 
-	bust_spinlocks(0);
-
 	/*
 	 * We may have ended up stopping the CPU holding the lock (in
 	 * smp_send_stop()) while still having some valuable data in the console
@@ -246,6 +244,8 @@ void panic(const char *fmt, ...)
 	debug_locks_off();
 	console_flush_on_panic();
 
+	bust_spinlocks(0);
+
 	if (!panic_blink)
 		panic_blink = no_blink;
 
Signature · 1 line
2.1.4
#1 lore raw
Quoted 37 lines · On Mon, 4 Jun 2018 14:45:57 +0900 Hoeun Ryu <[email protected]> wrote:

From: Hoeun Ryu <[email protected]>

Many console device drivers hold the uart_port->lock spinlock with irq enabled
(using spin_lock()) while the device drivers are writing characters to their devices, but the device drivers just try to hold the spin lock (using spin_trylock()) if "oops_in_progress" is equal or greater than 1 to avoid deadlocks.

There is a case ocurring a deadlock related to the lock and oops_in_progress. A CPU
could be stopped by smp_send_stop() while it was holding the port lock because irq was enabled. Once a CPU stops, it doesn't respond interrupts anymore and the lock stays locked forever.

console_flush_on_panic() is called during panic() and it eventually holds the uart
lock but the lock is held by another stopped CPU and it is a deadlock. By moving bust_spinlocks(0) after console_flush_on_panic(), let the console device drivers think the Oops is still in progress to call spin_trylock() instead of spin_lock() to avoid the deadlock.

---
 kernel/panic.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
kernel/panic.c +0 -2
index 42e4874..b4063b6 100644
@@ -233,8 +233,6 @@ void panic(const char *fmt, ...)
 	if (_crash_kexec_post_notifiers)
 		__crash_kexec(NULL);
 
-	bust_spinlocks(0);
-
 	/*
 	 * We may have ended up stopping the CPU holding the lock (in
 	 * smp_send_stop()) while still having some valuable data in the console

kernel/panic.c void panic(const char *fmt, ...) · line 247

Quoted 4 lines
kernel/panic.c +1 -0
@@ -246,6 +244,8 @@ void panic(const char *fmt, ...)
 	debug_locks_off();
 	console_flush_on_panic();
 
+	bust_spinlocks(0);

Added a few more to Cc. This looks like it could have subtle side-effects. I'd like those that have been touching the code around here to have a look.

-- Steve

Quoted 4 lines
kernel/panic.c +1 -0
+
 	if (!panic_blink)
 		panic_blink = no_blink;
 
#2 lore raw

I misunderstood the cause of a deadlock.
I sent v2 fixing the commit message about the reason of the deadlock. Please ignore this and review v2.
Thank you.

Quoted 79 lines

-----Original Message-----
From: Steven Rostedt [mailto:[email protected]]
Sent: Tuesday, June 05, 2018 10:44 AM
To: Hoeun Ryu <[email protected]>
Cc: Andrew Morton <[email protected]>; Kees Cook <[email protected]>; Borislav Petkov <[email protected]>; Andi Kleen <[email protected]>; Josh Poimboeuf <[email protected]>; Hoeun Ryu <[email protected]>; [email protected]; Tejun Heo <[email protected]>; Vitaly Kuznetsov <[email protected]>
Subject: Re: [PATCH] panic: move bust_spinlocks(0) after console_flush_on_panic() to avoid deadlocks

Quoted 3 lines · On Mon, 4 Jun 2018 14:45:57 +0900 Hoeun Ryu <[email protected]> wrote:

From: Hoeun Ryu <[email protected]>

Many console device drivers hold the uart_port->lock spinlock with irq

enabled

Quoted 1 line

(using spin_lock()) while the device drivers are writing characters to

their devices,

Quoted 1 line

but the device drivers just try to hold the spin lock (using

spin_trylock()) if

Quoted 3 lines

"oops_in_progress" is equal or greater than 1 to avoid deadlocks.

There is a case ocurring a deadlock related to the lock and

oops_in_progress. A CPU

Quoted 1 line

could be stopped by smp_send_stop() while it was holding the port lock

because irq was

Quoted 1 line

enabled. Once a CPU stops, it doesn't respond interrupts anymore and the

lock stays

Quoted 3 lines

locked forever.

console_flush_on_panic() is called during panic() and it eventually

holds the uart

Quoted 1 line

lock but the lock is held by another stopped CPU and it is a deadlock.

By moving

Quoted 1 line

bust_spinlocks(0) after console_flush_on_panic(), let the console device

drivers

Quoted 1 line

think the Oops is still in progress to call spin_trylock() instead of

spin_lock() to

Quoted 12 lines

avoid the deadlock.

---
 kernel/panic.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
kernel/panic.c +0 -0
index 42e4874..b4063b6 100644

kernel/panic.c void panic(const char *fmt, ...) · line 238

Quoted 8 lines
kernel/panic.c +0 -2
@@ -233,8 +233,6 @@ void panic(const char *fmt, ...)
 	if (_crash_kexec_post_notifiers)
 		__crash_kexec(NULL);
 
-	bust_spinlocks(0);
-
 	/*
 	 * We may have ended up stopping the CPU holding the lock (in
 	 * smp_send_stop()) while still having some valuable data in the

console

kernel/panic.c void panic(const char *fmt, ...) · line 247

Quoted 5 lines
kernel/panic.c +1 -0
@@ -246,6 +244,8 @@ void panic(const char *fmt, ...)
 	debug_locks_off();
 	console_flush_on_panic();
 
+	bust_spinlocks(0);

Added a few more to Cc. This looks like it could have subtle side-effects. I'd like those that have been touching the code around here to have a look.

-- Steve

Quoted 4 lines
kernel/panic.c +1 -0
+
 	if (!panic_blink)
 		panic_blink = no_blink;
 
#3 lore raw
In reply to Hoeun Ryu
Quoted 12 lines · On Mon, 4 Jun 2018 14:45:57 +0900 Hoeun Ryu <[email protected]> wrote:

From: Hoeun Ryu <[email protected]>

Many console device drivers hold the uart_port->lock spinlock with irq enabled
(using spin_lock()) while the device drivers are writing characters to their devices, but the device drivers just try to hold the spin lock (using spin_trylock()) if "oops_in_progress" is equal or greater than 1 to avoid deadlocks.

There is a case ocurring a deadlock related to the lock and oops_in_progress. A CPU
could be stopped by smp_send_stop() while it was holding the port lock because irq was enabled. Once a CPU stops, it doesn't respond interrupts anymore and the lock stays locked forever.

Quoted 5 lines

console_flush_on_panic() is called during panic() and it eventually holds the uart
lock but the lock is held by another stopped CPU and it is a deadlock. By moving bust_spinlocks(0) after console_flush_on_panic(), let the console device drivers think the Oops is still in progress to call spin_trylock() instead of spin_lock() to avoid the deadlock.

hm. Sergey, is this at all related to the UART printk deadlock change which you're presently discussing in
http://lkml.kernel.org/r/[email protected]?

Quoted 20 lines
kernel/panic.c +2 -2
@@ -233,8 +233,6 @@ void panic(const char *fmt, ...)
 	if (_crash_kexec_post_notifiers)
 		__crash_kexec(NULL);
 
-	bust_spinlocks(0);
-
 	/*
 	 * We may have ended up stopping the CPU holding the lock (in
 	 * smp_send_stop()) while still having some valuable data in the console
@@ -246,6 +244,8 @@ void panic(const char *fmt, ...)
 	debug_locks_off();
 	console_flush_on_panic();
 
+	bust_spinlocks(0);
+
 	if (!panic_blink)
 		panic_blink = no_blink;
 
#4 lore raw
Quoted 18 lines · Andrew Morton
Quoted 12 lines

From: Hoeun Ryu <[email protected]>

Many console device drivers hold the uart_port->lock spinlock with irq enabled
(using spin_lock()) while the device drivers are writing characters to their devices, but the device drivers just try to hold the spin lock (using spin_trylock()) if "oops_in_progress" is equal or greater than 1 to avoid deadlocks.

There is a case ocurring a deadlock related to the lock and oops_in_progress. A CPU
could be stopped by smp_send_stop() while it was holding the port lock because irq was enabled. Once a CPU stops, it doesn't respond interrupts anymore and the lock stays locked forever.

Quoted 5 lines

console_flush_on_panic() is called during panic() and it eventually holds the uart
lock but the lock is held by another stopped CPU and it is a deadlock. By moving bust_spinlocks(0) after console_flush_on_panic(), let the console device drivers think the Oops is still in progress to call spin_trylock() instead of spin_lock() to avoid the deadlock.

Quoted 3 lines

hm. Sergey, is this at all related to the UART printk deadlock change which you're presently discussing in
http://lkml.kernel.org/r/[email protected]?

Hi Andrew,

Not exactly. The change I'm discussing is a little different - it's about re-entrant UART [+ circular locking in TTY], when UART deadlocks us because of printk()-s issued by MM/tty/WQ/sched/other core kernel stuff/etc

Example:

	IRQ -> uart -> tty -> WQ -> printk -> uart

	-ss