CPU hotplug: active_reader not woken up in some cases - deadlock
PATCH 1/1 CPU hotplug: active_reader not woken up in some cases - deadlock
Commit b2c4623dcd07 ("rcu: More on deadlock between CPU hotplug and expedited grace periods") introduced another problem that can easily be reproduced by starting/stopping cpus in a loop.
E.g.:
for i in seq 5000; do
echo 1 > /sys/devices/system/cpu/cpu1/online echo 0 > /sys/devices/system/cpu/cpu1/online
done
Will result in:
INFO: task /cpu_start_stop:1 blocked for more than 120 seconds. Call Trace: ([<00000000006a028e>] __schedule+0x406/0x91c) [<0000000000130f60>] cpu_hotplug_begin+0xd0/0xd4 [<0000000000130ff6>] _cpu_up+0x3e/0x1c4 [<0000000000131232>] cpu_up+0xb6/0xd4 [<00000000004a5720>] device_online+0x80/0xc0 [<00000000004a57f0>] online_store+0x90/0xb0 ...
And a deadlock.
Problem is that if the last ref in put_online_cpus() can't get the cpu_hotplug.lock the puts_pending count is incremented, but a sleeping active_writer might never be woken up, therefore never exiting the loop in cpu_hotplug_begin().
This quick fix wakes up the active_writer proactively. The writer already goes back to sleep if the ref count isn't already down to 0, so this should be fine.
Can't reproduce the error with this fix.
--- kernel/cpu.c | 3 +++ 1 file changed, 3 insertions(+)
kernel/cpu.c +3 -0
@@ -117,6 +117,9 @@ void put_online_cpus(void)return ;if (!mutex_trylock (&cpu_hotplug.lock)) {atomic_inc (&cpu_hotplug.puts_pending); +/* we might be the last one */ +if (unlikely (cpu_hotplug.active_writer)) +wake_up_process (cpu_hotplug.active_writer);cpuhp_lock_release ();return ; }
Signature · 1 line
1.8.5.5
The title should of course say active_writer ... grml
David
Quoted 27 lines · David Hildenbrand
Commit
b2c4623dcd07("rcu: More on deadlock between CPU hotplug and expedited grace periods") introduced another problem that can easily be reproduced by starting/stopping cpus in a loop.E.g.:
for i inseq 5000; do
echo 1 >/sys/devices/system/cpu/cpu1/onlineecho 0 >/sys/devices/system/cpu/cpu1/online
doneWill result in:
INFO: task /cpu_start_stop:1 blocked for more than 120 seconds. Call Trace: ([<00000000006a028e>] __schedule+0x406/0x91c) [<0000000000130f60>] cpu_hotplug_begin+0xd0/0xd4 [<0000000000130ff6>] _cpu_up+0x3e/0x1c4 [<0000000000131232>] cpu_up+0xb6/0xd4 [<00000000004a5720>] device_online+0x80/0xc0 [<00000000004a57f0>] online_store+0x90/0xb0 ...And a deadlock.
Problem is that if the last ref in
put_online_cpus()can't get the cpu_hotplug.lock theputs_pendingcount is incremented, but a sleepingactive_writermight never be woken up, therefore never exiting the loop incpu_hotplug_begin().
Quoted 5 lines
This quick fix wakes up the
active_writerproactively. The writer already goes back to sleep if the ref count isn't already down to 0, so this should be fine.Can't reproduce the error with this fix.
Good catch!
But don't we need to use exactly the same value for the NULL check and for the wakeup? Otherwise, wouldn't it be possible for cpu_hotplug.active_writer to be non-NULL for the check but NULL for the wake_up_process()?
Thanx, Paul
Quoted 22 lines
Signed-off-by David Hildenbrand <[email protected]>--- kernel/cpu.c | 3 +++ 1 file changed, 3 insertions(+)kernel/cpu.c +3
-0@@ -117,6 +117,9 @@ void put_online_cpus(void)return ;if (!mutex_trylock (&cpu_hotplug.lock)) {atomic_inc (&cpu_hotplug.puts_pending); +/* we might be the last one */ +if (unlikely (cpu_hotplug.active_writer)) +wake_up_process (cpu_hotplug.active_writer);cpuhp_lock_release ();return ; }Signature · 1 line
1.8.5.5
Quoted 34 lines · David Hildenbrand
Quoted 27 lines
Commit
b2c4623dcd07("rcu: More on deadlock between CPU hotplug and expedited grace periods") introduced another problem that can easily be reproduced by starting/stopping cpus in a loop.E.g.:
for i inseq 5000; do
echo 1 >/sys/devices/system/cpu/cpu1/onlineecho 0 >/sys/devices/system/cpu/cpu1/online
doneWill result in:
INFO: task /cpu_start_stop:1 blocked for more than 120 seconds. Call Trace: ([<00000000006a028e>] __schedule+0x406/0x91c) [<0000000000130f60>] cpu_hotplug_begin+0xd0/0xd4 [<0000000000130ff6>] _cpu_up+0x3e/0x1c4 [<0000000000131232>] cpu_up+0xb6/0xd4 [<00000000004a5720>] device_online+0x80/0xc0 [<00000000004a57f0>] online_store+0x90/0xb0 ...And a deadlock.
Problem is that if the last ref in
put_online_cpus()can't get the cpu_hotplug.lock theputs_pendingcount is incremented, but a sleepingactive_writermight never be woken up, therefore never exiting the loop incpu_hotplug_begin().Quoted 5 lines
This quick fix wakes up the
active_writerproactively. The writer already goes back to sleep if the ref count isn't already down to 0, so this should be fine.Can't reproduce the error with this fix.
Quoted 8 lines
Good catch!
But don't we need to use exactly the same value for the NULL check and for the wakeup? Otherwise, wouldn't it be possible for cpu_hotplug.active_writer to be non-NULL for the check but NULL for the
wake_up_process()?Thanx, Paul
active_writer is cleared while holding cpuhp_lock, so this should be safe, right?
Thanks!
Quoted 41 lines · David Hildenbrand
Quoted 41 lines · David Hildenbrand
Quoted 27 lines
Commit
b2c4623dcd07("rcu: More on deadlock between CPU hotplug and expedited grace periods") introduced another problem that can easily be reproduced by starting/stopping cpus in a loop.E.g.:
for i inseq 5000; do
echo 1 >/sys/devices/system/cpu/cpu1/onlineecho 0 >/sys/devices/system/cpu/cpu1/online
doneWill result in:
INFO: task /cpu_start_stop:1 blocked for more than 120 seconds. Call Trace: ([<00000000006a028e>] __schedule+0x406/0x91c) [<0000000000130f60>] cpu_hotplug_begin+0xd0/0xd4 [<0000000000130ff6>] _cpu_up+0x3e/0x1c4 [<0000000000131232>] cpu_up+0xb6/0xd4 [<00000000004a5720>] device_online+0x80/0xc0 [<00000000004a57f0>] online_store+0x90/0xb0 ...And a deadlock.
Problem is that if the last ref in
put_online_cpus()can't get the cpu_hotplug.lock theputs_pendingcount is incremented, but a sleepingactive_writermight never be woken up, therefore never exiting the loop incpu_hotplug_begin().Quoted 5 lines
This quick fix wakes up the
active_writerproactively. The writer already goes back to sleep if the ref count isn't already down to 0, so this should be fine.Can't reproduce the error with this fix.
Good catch!
But don't we need to use exactly the same value for the NULL check and for the wakeup? Otherwise, wouldn't it be possible for cpu_hotplug.active_writer to be non-NULL for the check but NULL for the
wake_up_process()?
Quoted 4 lines
Quoted 1 line
Thanx, Paul
active_writeris cleared while holdingcpuhp_lock, so this should be safe, right?
You lost me on that one. Don't we get to that piece of code precisely because we don't hold any of the CPU-hotplug locks? If so, the writer might well hold all the locks it needs, and might well change cpu_hotplug.active_writer out from under us.
What am I missing here?
Thanx, Paul
Quoted 11 lines
Quoted 2 lines
active_writeris cleared while holdingcpuhp_lock, so this should be safe, right?You lost me on that one. Don't we get to that piece of code precisely because we don't hold any of the CPU-hotplug locks? If so, the writer might well hold all the locks it needs, and might well change cpu_hotplug.active_writer out from under us.
What am I missing here?
Thanx, Paul
I was missing that cpuhp_lock_* are simply lockdep anotations ... it's getting late :)
So you're right, we need to verify that we don't get a 0 on the second access.
Will send an updated version soon.
Thanks!
Quoted 8 lines · David Hildenbrand
Quoted 8 lines
Quoted 2 lines
active_writeris cleared while holdingcpuhp_lock, so this should be safe, right?You lost me on that one. Don't we get to that piece of code precisely because we don't hold any of the CPU-hotplug locks? If so, the writer might well hold all the locks it needs, and might well change cpu_hotplug.active_writer out from under us.
Quoted 8 lines
Quoted 3 lines
What am I missing here?
Thanx, Paul
I was missing that cpuhp_lock_* are simply lockdep anotations ... it's getting late :)
So you're right, we need to verify that we don't get a 0 on the second access.
All you should need to do is to do something like this:
awp = ACCESS_ONCE(cpu_hotplug.active_writer); if (awp) wake_up_process(cpu_hotplug.active_writer);
That way you only have one access, and the check and wake_up_process() are guaranteed to be consistent.
Quoted 1 line
Will send an updated version soon.
Sounds good!
Thanx, Paul