panic: move bust_spinlocks(0) after console_flush_on_panic() to avoid deadlocks
PATCH 1/1 panic: move bust_spinlocks(0) after console_flush_on_panic() to avoid deadlocks
From: Hoeun Ryu <[email protected]>
Many console device drivers hold the uart_port->lock spinlock with irq enabled
(using spin_lock()) while the device drivers are writing characters to their devices, but the device drivers just try to hold the spin lock (using spin_trylock()) if "oops_in_progress" is equal or greater than 1 to avoid deadlocks.
There is a case ocurring a deadlock related to the lock and oops_in_progress. A CPU
could be stopped by smp_send_stop() while it was holding the port lock because irq was enabled. Once a CPU stops, it doesn't respond interrupts anymore and the lock stays locked forever.
console_flush_on_panic() is called during panic() and it eventually holds the uart
lock but the lock is held by another stopped CPU and it is a deadlock. By moving bust_spinlocks(0) after console_flush_on_panic(), let the console device drivers think the Oops is still in progress to call spin_trylock() instead of spin_lock() to avoid the deadlock.
--- kernel/panic.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
kernel/panic.c +2 -2
@@ -233,8 +233,6 @@ void panic(const char *fmt, ...)if (_crash_kexec_post_notifiers)__crash_kexec (NULL ); -bust_spinlocks (0 ); -/* * We may have ended up stopping the CPU holding the lock (in * smp_send_stop()) while still having some valuable data in the console
@@ -246,6 +244,8 @@ void panic(const char *fmt, ...)debug_locks_off ();console_flush_on_panic (); +bust_spinlocks (0 ); +if (!panic_blink) panic_blink = no_blink;
Signature · 1 line
2.1.4
Quoted 37 lines · On Mon, 4 Jun 2018 14:45:57 +0900 Hoeun Ryu <[email protected]> wrote:
From: Hoeun Ryu <[email protected]>
Many console device drivers hold the
uart_port->lockspinlock with irq enabled
(usingspin_lock()) while the device drivers are writing characters to their devices, but the device drivers just try to hold the spin lock (usingspin_trylock()) if "oops_in_progress" is equal or greater than 1 to avoid deadlocks.There is a case ocurring a deadlock related to the lock and
oops_in_progress. A CPU
could be stopped bysmp_send_stop()while it was holding the port lock because irq was enabled. Once a CPU stops, it doesn't respond interrupts anymore and the lock stays locked forever.
console_flush_on_panic()is called duringpanic()and it eventually holds the uart
lock but the lock is held by another stopped CPU and it is a deadlock. By movingbust_spinlocks(0)afterconsole_flush_on_panic(), let the console device drivers think the Oops is still in progress to callspin_trylock()instead ofspin_lock()to avoid the deadlock.Signed-off-by Hoeun Ryu <[email protected]>--- kernel/panic.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)kernel/panic.c +0
-2@@ -233,8 +233,6 @@ void panic(const char *fmt, ...)if (_crash_kexec_post_notifiers)__crash_kexec (NULL ); -bust_spinlocks (0 ); -/* * We may have ended up stopping the CPU holding the lock (in * smp_send_stop()) while still having some valuable data in the console
kernel/panic.c void panic(const char *fmt, ...) · line 247
Quoted 4 lines
kernel/panic.c +1
-0@@ -246,6 +244,8 @@ void panic(const char *fmt, ...)debug_locks_off ();console_flush_on_panic (); +bust_spinlocks (0 );
Added a few more to Cc. This looks like it could have subtle side-effects. I'd like those that have been touching the code around here to have a look.
-- Steve
Quoted 4 lines
kernel/panic.c +1
-0+if (!panic_blink) panic_blink = no_blink;
I misunderstood the cause of a deadlock.
I sent v2 fixing the commit message about the reason of the deadlock. Please ignore this and review v2.
Thank you.
Quoted 79 lines
-----Original Message-----
From: Steven Rostedt [mailto:[email protected]]
Sent: Tuesday, June 05, 2018 10:44 AM
To: Hoeun Ryu <[email protected]>
Cc: Andrew Morton <[email protected]>; Kees Cook <[email protected]>; Borislav Petkov <[email protected]>; Andi Kleen <[email protected]>; Josh Poimboeuf <[email protected]>; Hoeun Ryu <[email protected]>; [email protected]; Tejun Heo <[email protected]>; Vitaly Kuznetsov <[email protected]>
Subject: Re: [PATCH] panic: movebust_spinlocks(0)afterconsole_flush_on_panic()to avoid deadlocksQuoted 3 lines · On Mon, 4 Jun 2018 14:45:57 +0900 Hoeun Ryu <[email protected]> wrote:
From: Hoeun Ryu <[email protected]>
Many console device drivers hold the
uart_port->lockspinlock with irqenabled
Quoted 1 line
(using
spin_lock()) while the device drivers are writing characters totheir devices,
Quoted 1 line
but the device drivers just try to hold the spin lock (using
spin_trylock()) ifQuoted 3 lines
"
oops_in_progress" is equal or greater than 1 to avoid deadlocks.There is a case ocurring a deadlock related to the lock and
oops_in_progress. A CPUQuoted 1 line
could be stopped by
smp_send_stop()while it was holding the port lockbecause irq was
Quoted 1 line
enabled. Once a CPU stops, it doesn't respond interrupts anymore and the
lock stays
Quoted 3 lines
locked forever.
console_flush_on_panic()is called duringpanic()and it eventuallyholds the uart
Quoted 1 line
lock but the lock is held by another stopped CPU and it is a deadlock.
By moving
Quoted 1 line
bust_spinlocks(0)afterconsole_flush_on_panic(), let the console devicedrivers
Quoted 1 line
think the Oops is still in progress to call
spin_trylock()instead of
spin_lock()toQuoted 12 lines
avoid the deadlock.
Signed-off-by Hoeun Ryu <[email protected]>--- kernel/panic.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)kernel/panic.c +0
-0kernel/panic.c void panic(const char *fmt, ...) · line 238
Quoted 8 lines
kernel/panic.c +0
-2@@ -233,8 +233,6 @@ void panic(const char *fmt, ...)if (_crash_kexec_post_notifiers)__crash_kexec (NULL ); -bust_spinlocks (0 ); -/* * We may have ended up stopping the CPU holding the lock (in * smp_send_stop()) while still having some valuable data in the console
kernel/panic.c void panic(const char *fmt, ...) · line 247
Quoted 5 lines
kernel/panic.c +1
-0@@ -246,6 +244,8 @@ void panic(const char *fmt, ...)debug_locks_off ();console_flush_on_panic (); +bust_spinlocks (0 );Added a few more to Cc. This looks like it could have subtle side-effects. I'd like those that have been touching the code around here to have a look.
-- Steve
Quoted 4 lines
kernel/panic.c +1
-0+if (!panic_blink) panic_blink = no_blink;
Quoted 12 lines · On Mon, 4 Jun 2018 14:45:57 +0900 Hoeun Ryu <[email protected]> wrote:
From: Hoeun Ryu <[email protected]>
Many console device drivers hold the
uart_port->lockspinlock with irq enabled
(usingspin_lock()) while the device drivers are writing characters to their devices, but the device drivers just try to hold the spin lock (usingspin_trylock()) if "oops_in_progress" is equal or greater than 1 to avoid deadlocks.There is a case ocurring a deadlock related to the lock and
oops_in_progress. A CPU
could be stopped bysmp_send_stop()while it was holding the port lock because irq was enabled. Once a CPU stops, it doesn't respond interrupts anymore and the lock stays locked forever.
Quoted 5 lines
console_flush_on_panic()is called duringpanic()and it eventually holds the uart
lock but the lock is held by another stopped CPU and it is a deadlock. By movingbust_spinlocks(0)afterconsole_flush_on_panic(), let the console device drivers think the Oops is still in progress to callspin_trylock()instead ofspin_lock()to avoid the deadlock.
hm. Sergey, is this at all related to the UART printk deadlock change which you're presently discussing in
http://lkml.kernel.org/r/[email protected]?
Quoted 20 lines
kernel/panic.c +2
-2@@ -233,8 +233,6 @@ void panic(const char *fmt, ...)if (_crash_kexec_post_notifiers)__crash_kexec (NULL ); -bust_spinlocks (0 ); -/* * We may have ended up stopping the CPU holding the lock (in * smp_send_stop()) while still having some valuable data in the console @@ -246,6 +244,8 @@ void panic(const char *fmt, ...)debug_locks_off ();console_flush_on_panic (); +bust_spinlocks (0 ); +if (!panic_blink) panic_blink = no_blink;
Quoted 18 lines · Andrew Morton
Quoted 12 lines
From: Hoeun Ryu <[email protected]>
Many console device drivers hold the
uart_port->lockspinlock with irq enabled
(usingspin_lock()) while the device drivers are writing characters to their devices, but the device drivers just try to hold the spin lock (usingspin_trylock()) if "oops_in_progress" is equal or greater than 1 to avoid deadlocks.There is a case ocurring a deadlock related to the lock and
oops_in_progress. A CPU
could be stopped bysmp_send_stop()while it was holding the port lock because irq was enabled. Once a CPU stops, it doesn't respond interrupts anymore and the lock stays locked forever.Quoted 5 lines
console_flush_on_panic()is called duringpanic()and it eventually holds the uart
lock but the lock is held by another stopped CPU and it is a deadlock. By movingbust_spinlocks(0)afterconsole_flush_on_panic(), let the console device drivers think the Oops is still in progress to callspin_trylock()instead ofspin_lock()to avoid the deadlock.
Quoted 3 lines
hm. Sergey, is this at all related to the UART printk deadlock change which you're presently discussing in
http://lkml.kernel.org/r/[email protected]?
Hi Andrew,
Not exactly. The change I'm discussing is a little different - it's about re-entrant UART [+ circular locking in TTY], when UART deadlocks us because of printk()-s issued by MM/tty/WQ/sched/other core kernel stuff/etc
Example:
IRQ -> uart -> tty -> WQ -> printk -> uart -ss