CPU hotplug: active_writer not woken up in some cases - deadlock
PATCH v3 1/1 CPU hotplug: active_writer not woken up in some cases - deadlock
Commit b2c4623dcd07 ("rcu: More on deadlock between CPU hotplug and expedited grace periods") introduced another problem that can easily be reproduced by starting/stopping cpus in a loop.
E.g.:
for i in seq 5000; do
echo 1 > /sys/devices/system/cpu/cpu1/online echo 0 > /sys/devices/system/cpu/cpu1/online
done
Will result in:
INFO: task /cpu_start_stop:1 blocked for more than 120 seconds. Call Trace: ([<00000000006a028e>] __schedule+0x406/0x91c) [<0000000000130f60>] cpu_hotplug_begin+0xd0/0xd4 [<0000000000130ff6>] _cpu_up+0x3e/0x1c4 [<0000000000131232>] cpu_up+0xb6/0xd4 [<00000000004a5720>] device_online+0x80/0xc0 [<00000000004a57f0>] online_store+0x90/0xb0 ...
And a deadlock.
Problem is that if the last ref in put_online_cpus() can't get the cpu_hotplug.lock the puts_pending count is incremented, but a sleeping active_writer might never be woken up, therefore never exiting the loop in cpu_hotplug_begin().
This fix wakes up the active_writer proactively. The writer already goes back to sleep if the ref count isn't already down to 0, so this should be fine.
In order to avoid many potential races, we have to:
- Protect current_writer by a spin lock. When holding this lock we can be sure that the writer won't vainsh or change. (use-after-free)
- Increment the cpu_hotplug.puts_pending count before we test for an active_writer. (otherwise a wakeup might get lost)
- Move setting of TASK_UNINTERRUPTIBLE in cpu_hotplug_begin() above the condition check. (otherwise a wakeup might get lost)
Can't reproduce it with this fix.
--- kernel/cpu.c | 18 ++++++++++++++++-- 1 file changed, 16 insertions(+), 2 deletions(-)
kernel/cpu.c +16 -2
@@ -58,6 +58,7 @@ static int cpu_hotplug_disabled; static struct { struct task_struct *active_writer; + spinlock_t awr_lock; /* protects active_writer from being changed */ struct mutex lock; /* Synchronizes accesses to refcount, */ /* * Also blocks the new readers during
@@ -72,6 +73,7 @@ static struct {#endif } cpu_hotplug = { .active_writer = NULL, + .awr_lock = __SPIN_LOCK_UNLOCKED(cpu_hotplug.awr_lock), .lock = __MUTEX_INITIALIZER(cpu_hotplug.lock), .refcount =0 , #ifdef CONFIG_DEBUG_LOCK_ALLOC
@@ -116,7 +118,13 @@ void put_online_cpus(void)if (cpu_hotplug.active_writer == current)return ;if (!mutex_trylock (&cpu_hotplug.lock)) { +/* inc before testing for active_writer to not lose wake ups */ atomic_inc (&cpu_hotplug.puts_pending); +spin_lock (&cpu_hotplug.awr_lock); +/* we might be the last one */ +if (unlikely (cpu_hotplug.active_writer)) +wake_up_process (cpu_hotplug.active_writer); +spin_unlock (&cpu_hotplug.awr_lock);cpuhp_lock_release ();return ; }
@@ -156,20 +164,24 @@ EXPORT_SYMBOL_GPL(put_online_cpus);*/ void cpu_hotplug_begin (void ) { +spin_lock (&cpu_hotplug.awr_lock); cpu_hotplug.active_writer = current; +spin_unlock (&cpu_hotplug.awr_lock);cpuhp_lock_acquire ();for (;;) {mutex_lock (&cpu_hotplug.lock); +__set_current_state (TASK_UNINTERRUPTIBLE );if (atomic_read (&cpu_hotplug.puts_pending)) {int delta; delta =atomic_xchg (&cpu_hotplug.puts_pending,0 ); cpu_hotplug.refcount -= delta; } -if (likely (!cpu_hotplug.refcount)) +if (likely (!cpu_hotplug.refcount)) { +__set_current_state (TASK_RUNNING );break ; -__set_current_state (TASK_UNINTERRUPTIBLE ); + }mutex_unlock (&cpu_hotplug.lock);schedule (); }
@@ -177,7 +189,9 @@ void cpu_hotplug_begin(void)void cpu_hotplug_done (void ) { +spin_lock (&cpu_hotplug.awr_lock); cpu_hotplug.active_writer =NULL ; +spin_unlock (&cpu_hotplug.awr_lock);mutex_unlock (&cpu_hotplug.lock);cpuhp_lock_release (); }
Signature · 1 line
1.8.5.5
Quoted 39 lines · David Hildenbrand
Commit
b2c4623dcd07("rcu: More on deadlock between CPU hotplug and expedited grace periods") introduced another problem that can easily be reproduced by starting/stopping cpus in a loop.E.g.:
for i inseq 5000; do
echo 1 >/sys/devices/system/cpu/cpu1/onlineecho 0 >/sys/devices/system/cpu/cpu1/online
doneWill result in:
INFO: task /cpu_start_stop:1 blocked for more than 120 seconds. Call Trace: ([<00000000006a028e>] __schedule+0x406/0x91c) [<0000000000130f60>] cpu_hotplug_begin+0xd0/0xd4 [<0000000000130ff6>] _cpu_up+0x3e/0x1c4 [<0000000000131232>] cpu_up+0xb6/0xd4 [<00000000004a5720>] device_online+0x80/0xc0 [<00000000004a57f0>] online_store+0x90/0xb0 ...And a deadlock.
Problem is that if the last ref in
put_online_cpus()can't get the cpu_hotplug.lock theputs_pendingcount is incremented, but a sleepingactive_writermight never be woken up, therefore never exiting the loop incpu_hotplug_begin().This fix wakes up the
active_writerproactively. The writer already goes back to sleep if the ref count isn't already down to 0, so this should be fine.In order to avoid many potential races, we have to:
- Protectcurrent_writerby a spin lock. When holding this lock we can be sure that the writer won't vainsh or change. (use-after-free)
- Increment the cpu_hotplug.puts_pending count before we test for anactive_writer. (otherwise a wakeup might get lost)
- Move setting ofTASK_UNINTERRUPTIBLEincpu_hotplug_begin()above the condition check. (otherwise a wakeup might get lost)
Quoted 1 line
Can't reproduce it with this fix.
Would wait_event()/wake_up() work for the wakeup-writer case?
Thanx, Paul
Quoted 79 lines
Signed-off-by David Hildenbrand <[email protected]>--- kernel/cpu.c | 18 ++++++++++++++++-- 1 file changed, 16 insertions(+), 2 deletions(-)kernel/cpu.c +16
-2@@ -58,6 +58,7 @@ static int cpu_hotplug_disabled; static struct { struct task_struct *active_writer; + spinlock_t awr_lock; /* protects active_writer from being changed */ struct mutex lock; /* Synchronizes accesses to refcount, */ /* * Also blocks the new readers during@@ -72,6 +73,7 @@ static struct {#endif } cpu_hotplug = { .active_writer = NULL, + .awr_lock = __SPIN_LOCK_UNLOCKED(cpu_hotplug.awr_lock), .lock = __MUTEX_INITIALIZER(cpu_hotplug.lock), .refcount =0 , #ifdef CONFIG_DEBUG_LOCK_ALLOC@@ -116,7 +118,13 @@ void put_online_cpus(void)if (cpu_hotplug.active_writer == current)return ;if (!mutex_trylock (&cpu_hotplug.lock)) { +/* inc before testing for active_writer to not lose wake ups */ atomic_inc (&cpu_hotplug.puts_pending); +spin_lock (&cpu_hotplug.awr_lock); +/* we might be the last one */ +if (unlikely (cpu_hotplug.active_writer)) +wake_up_process (cpu_hotplug.active_writer); +spin_unlock (&cpu_hotplug.awr_lock);cpuhp_lock_release ();return ; }@@ -156,20 +164,24 @@ EXPORT_SYMBOL_GPL(put_online_cpus);*/ void cpu_hotplug_begin (void ) { +spin_lock (&cpu_hotplug.awr_lock); cpu_hotplug.active_writer = current; +spin_unlock (&cpu_hotplug.awr_lock);cpuhp_lock_acquire ();for (;;) {mutex_lock (&cpu_hotplug.lock); +__set_current_state (TASK_UNINTERRUPTIBLE );if (atomic_read (&cpu_hotplug.puts_pending)) {int delta; delta =atomic_xchg (&cpu_hotplug.puts_pending,0 ); cpu_hotplug.refcount -= delta; } -if (likely (!cpu_hotplug.refcount)) +if (likely (!cpu_hotplug.refcount)) { +__set_current_state (TASK_RUNNING );break ; -__set_current_state (TASK_UNINTERRUPTIBLE ); + }mutex_unlock (&cpu_hotplug.lock);schedule (); }@@ -177,7 +189,9 @@ void cpu_hotplug_begin(void)void cpu_hotplug_done (void ) { +spin_lock (&cpu_hotplug.awr_lock); cpu_hotplug.active_writer =NULL ; +spin_unlock (&cpu_hotplug.awr_lock);mutex_unlock (&cpu_hotplug.lock);cpuhp_lock_release (); }Signature · 1 line
1.8.5.5
Quoted 40 lines · David Hildenbrand
Quoted 39 lines
Commit
b2c4623dcd07("rcu: More on deadlock between CPU hotplug and expedited grace periods") introduced another problem that can easily be reproduced by starting/stopping cpus in a loop.E.g.:
for i inseq 5000; do
echo 1 >/sys/devices/system/cpu/cpu1/onlineecho 0 >/sys/devices/system/cpu/cpu1/online
doneWill result in:
INFO: task /cpu_start_stop:1 blocked for more than 120 seconds. Call Trace: ([<00000000006a028e>] __schedule+0x406/0x91c) [<0000000000130f60>] cpu_hotplug_begin+0xd0/0xd4 [<0000000000130ff6>] _cpu_up+0x3e/0x1c4 [<0000000000131232>] cpu_up+0xb6/0xd4 [<00000000004a5720>] device_online+0x80/0xc0 [<00000000004a57f0>] online_store+0x90/0xb0 ...And a deadlock.
Problem is that if the last ref in
put_online_cpus()can't get the cpu_hotplug.lock theputs_pendingcount is incremented, but a sleepingactive_writermight never be woken up, therefore never exiting the loop incpu_hotplug_begin().This fix wakes up the
active_writerproactively. The writer already goes back to sleep if the ref count isn't already down to 0, so this should be fine.In order to avoid many potential races, we have to:
- Protectcurrent_writerby a spin lock. When holding this lock we can be sure that the writer won't vainsh or change. (use-after-free)
- Increment the cpu_hotplug.puts_pending count before we test for anactive_writer. (otherwise a wakeup might get lost)
- Move setting ofTASK_UNINTERRUPTIBLEincpu_hotplug_begin()above the condition check. (otherwise a wakeup might get lost)
Quoted 6 lines
Quoted 1 line
Can't reproduce it with this fix.
Would wait_event()/wake_up() work for the wakeup-writer case?
Thanx, Paul
Thanks! Was also thinking about wait queues. Will investigate if that can help to beautify this :)
David
Quoted 40 lines · David Hildenbrand
Quoted 40 lines · David Hildenbrand
Quoted 39 lines
Commit
b2c4623dcd07("rcu: More on deadlock between CPU hotplug and expedited grace periods") introduced another problem that can easily be reproduced by starting/stopping cpus in a loop.E.g.:
for i inseq 5000; do
echo 1 >/sys/devices/system/cpu/cpu1/onlineecho 0 >/sys/devices/system/cpu/cpu1/online
doneWill result in:
INFO: task /cpu_start_stop:1 blocked for more than 120 seconds. Call Trace: ([<00000000006a028e>] __schedule+0x406/0x91c) [<0000000000130f60>] cpu_hotplug_begin+0xd0/0xd4 [<0000000000130ff6>] _cpu_up+0x3e/0x1c4 [<0000000000131232>] cpu_up+0xb6/0xd4 [<00000000004a5720>] device_online+0x80/0xc0 [<00000000004a57f0>] online_store+0x90/0xb0 ...And a deadlock.
Problem is that if the last ref in
put_online_cpus()can't get the cpu_hotplug.lock theputs_pendingcount is incremented, but a sleepingactive_writermight never be woken up, therefore never exiting the loop incpu_hotplug_begin().This fix wakes up the
active_writerproactively. The writer already goes back to sleep if the ref count isn't already down to 0, so this should be fine.In order to avoid many potential races, we have to:
- Protectcurrent_writerby a spin lock. When holding this lock we can be sure that the writer won't vainsh or change. (use-after-free)
- Increment the cpu_hotplug.puts_pending count before we test for anactive_writer. (otherwise a wakeup might get lost)
- Move setting ofTASK_UNINTERRUPTIBLEincpu_hotplug_begin()above the condition check. (otherwise a wakeup might get lost)
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
Can't reproduce it with this fix.
Would wait_event()/wake_up() work for the wakeup-writer case?
Thanks! Was also thinking about wait queues. Will investigate if that can help to beautify this :)
Looking forward to seeing it. ;-)
Thanx, Paul
Quoted 2 lines · Paul E. McKenney
Would wait_event()/wake_up() work for the wakeup-writer case?
Yes, and in this case we could probably kill this puts_pending logic and avoid cpu_hotplug.lock in put_online_cpus() altogether? Can't we just make cpu_hotplug.refcount atomic_t?
Anyway, this makes me think again that this code should use percpu_rwsem. Perhaps I'll try to make a patch next week...
(we need down_write_recursive_readers(), and probably rcusync patches).
Oleg.
Quoted 7 lines · Oleg Nesterov
Quoted 2 lines · Paul E. McKenney
Would wait_event()/wake_up() work for the wakeup-writer case?
Yes, and in this case we could probably kill this
puts_pendinglogic and avoid cpu_hotplug.lock input_online_cpus()altogether? Can't we just make cpu_hotplug.refcountatomic_t?
Seems like that should be possible. That would certainly simplify the wakeup logic from put_online_cpus(). It might even be possible to avoid acquiring cpu_hotplug.lock in put_online_cpus(), though that would of course require more luck than anyone deserves.
Quoted 4 lines
Anyway, this makes me think again that this code should use
percpu_rwsem. Perhaps I'll try to make a patch next week...(we need
down_write_recursive_readers(), and probably rcusync patches).
Careful! You might end up re-introducing the deadlock that I used puts_pending to get rid of. ;-)
Thanx, Paul
(sorry if this was already discussed, I ignored most of my emails
I got this week)
kernel/cpu.c void put_online_cpus(void) · line 127
Quoted 12 lines · David Hildenbrand
kernel/cpu.c +6
-0@@ -116,7 +118,13 @@ void put_online_cpus(void)if (cpu_hotplug.active_writer == current)return ;if (!mutex_trylock (&cpu_hotplug.lock)) { +/* inc before testing for active_writer to not lose wake ups */ atomic_inc (&cpu_hotplug.puts_pending); +spin_lock (&cpu_hotplug.awr_lock); +/* we might be the last one */ +if (unlikely (cpu_hotplug.active_writer)) +wake_up_process (cpu_hotplug.active_writer); +spin_unlock (&cpu_hotplug.awr_lock);
Not sure I understand. awr_lock can only ensure that active_writer can't go away.
Why active_writer should see .puts_pending != 0 if this is called right after cpu_hotplug_begin() takes cpu_hotplug.lock but before it sets TASK_UNINTERRUPTIBLE?
IOW,
kernel/cpu.c EXPORT_SYMBOL_GPL(put_online_cpus); · line 174
Quoted 10 lines
kernel/cpu.c +3
-0void cpu_hotplug_begin (void ) { +spin_lock (&cpu_hotplug.awr_lock); cpu_hotplug.active_writer = current; +spin_unlock (&cpu_hotplug.awr_lock);cpuhp_lock_acquire ();for (;;) {mutex_lock (&cpu_hotplug.lock); +__set_current_state (TASK_UNINTERRUPTIBLE );
don't we need set_current_state() here ?
Oleg.
Quoted 5 lines
(sorry if this was already discussed, I ignored most of my emails
I got this week)Quoted 1 line · David Hildenbrand
Quoted 14 lines
kernel/cpu.c void put_online_cpus(void) · line 127
Quoted 11 lines
kernel/cpu.c +6
-0@@ -116,7 +118,13 @@ void put_online_cpus(void)if (cpu_hotplug.active_writer == current)return ;if (!mutex_trylock (&cpu_hotplug.lock)) { +/* inc before testing for active_writer to not lose wake ups */ atomic_inc (&cpu_hotplug.puts_pending); +spin_lock (&cpu_hotplug.awr_lock); +/* we might be the last one */ +if (unlikely (cpu_hotplug.active_writer)) +wake_up_process (cpu_hotplug.active_writer); +spin_unlock (&cpu_hotplug.awr_lock);Not sure I understand.
awr_lockcan only ensure thatactive_writercan't go away.
This solution is not optimal but works without races ... I'll try to get something with wait queues running and/or even change the way refcount is accessed as suggested by you.
And yes, awr_lock will only ensure that active_writer won't go away.
Quoted 4 lines
Why
active_writershould see .puts_pending != 0 if this is called right aftercpu_hotplug_begin()takes cpu_hotplug.lock but before it setsTASK_UNINTERRUPTIBLE?
get_online_cpus() increased the refcount.put_online_cpus() will increment puts_pending and trigger a wake up (if the lock is alread taken - might be by cpu_hotplug_begin() or by some other get_online_cpus()).
So refcount == 1, puts_pending == 1
cpu_hotplug_begin() gets the lock and sees refcount == 1 and puts_pending == 0 or puts_pending == 1 (race with put_online_cpus()).
If that answers your question :)
Quoted 9 lines
IOW,
Quoted 6 lines
kernel/cpu.c +2
-0void cpu_hotplug_begin (void ) { +spin_lock (&cpu_hotplug.awr_lock); cpu_hotplug.active_writer = current; +spin_unlock (&cpu_hotplug.awr_lock);
Quoted 6 lines
kernel/cpu.c EXPORT_SYMBOL_GPL(put_online_cpus); · line 174
Quoted 4 lines
kernel/cpu.c +1
-0cpuhp_lock_acquire ();for (;;) {mutex_lock (&cpu_hotplug.lock); +__set_current_state (TASK_UNINTERRUPTIBLE );don't we need
set_current_state()here ?
Hm, good question, this was only a move of existing code. But I thing the checked variant would be better.
Quoted 3 lines
Oleg.
Thanks!
David
Quoted 10 lines · David Hildenbrand
Quoted 4 lines
Why
active_writershould see .puts_pending != 0 if this is called right aftercpu_hotplug_begin()takes cpu_hotplug.lock but before it setsTASK_UNINTERRUPTIBLE?
get_online_cpus()increased the refcount.put_online_cpus()will incrementputs_pendingand trigger a wake up (if the lock is alread taken - might be bycpu_hotplug_begin()or by some otherget_online_cpus()).
Quoted 6 lines
So refcount == 1,
puts_pending== 1
cpu_hotplug_begin()gets the lock and sees refcount == 1 andputs_pending== 0 orputs_pending== 1 (race withput_online_cpus()).If that answers your question :)
Sorry for confusion ;)
I meant that without mb() cpu_hotplug_begin() can miss puts_pending != 0, so it needs set_current_state() before atomic_read().
But this doesn't matter, your v4 uses wake_up/prepare_to_wait.
Quoted 26 lines
Quoted 8 lines
IOW,
Quoted 6 lines
void
cpu_hotplug_begin(void)
{kernel/cpu.c +2
-0+spin_lock (&cpu_hotplug.awr_lock); cpu_hotplug.active_writer = current; +spin_unlock (&cpu_hotplug.awr_lock);Quoted 6 lines
kernel/cpu.c EXPORT_SYMBOL_GPL(put_online_cpus); · line 174
Quoted 4 lines
kernel/cpu.c +1
-0cpuhp_lock_acquire ();for (;;) {mutex_lock (&cpu_hotplug.lock); +__set_current_state (TASK_UNINTERRUPTIBLE );don't we need
set_current_state()here ?Hm, good question, this was only a move of existing code. But I thing the checked variant would be better.
Quoted 3 lines
Oleg.
Thanks!
David