sched: Fix race between yield_to() and try_to_wake_up()
PATCH 1/1 sched: Fix race between yield_to() and try_to_wake_up()
We met a SCHED_WARN in set_next_buddy():
__warn_printk
set_next_buddy
yield_to_task_fair
yield_to
kvm_vcpu_yield_to [kvm]
...
After a short dig, we found the rq_lock held by yield_to() may not be exactly the rq that the target task belongs to. There is a race window against try_to_wake_up().
CPU0 target_task
blocking on CPU1
lock rq0 & rq1
double check task_rq == p_rq, ok
woken to CPU2 (lock task_pi & rq2) task_rq = rq2
yield_to_task_fair (w/o lock rq2)
In this race window, yield_to() is operating the task w/o the currect lock. Fix this by taking task pi_lock first.
--- kernel/sched/syscalls.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/sched/syscalls.c +1 -1
@@ -1433,7 +1433,7 @@ int __sched yield_to(struct task_struct *p, bool preempt)struct rq *rq, *p_rq;int yielded =0 ; -scoped_guard (irqsave) { + scoped_guard (raw_spinlock_irqsave, &p->pi_lock) { rq =this_rq (); again:
Signature · 1 line
2.39.3
Quoted 14 lines · Tianchen Ding
We met a
SCHED_WARNinset_next_buddy():__warn_printk set_next_buddy yield_to_task_fair yield_to kvm_vcpu_yield_to [kvm] ...After a short dig, we found the
rq_lockheld byyield_to()may not be exactly the rq that the target task belongs to. There is a race window againsttry_to_wake_up().CPU0
target_task
Quoted 8 lines
blocking on CPU1
lock rq0 & rq1
double checktask_rq==p_rq, ok
woken to CPU2 (locktask_pi& rq2)task_rq= rq2
yield_to_task_fair(w/o lock rq2)In this race window,
yield_to()is operating the task w/o the currect
Sorry... a typo for "correct"
Peter Zijlstra
maintainerQuoted 35 lines · Tianchen Ding
We met a
SCHED_WARNinset_next_buddy():
__warn_printk
set_next_buddy
yield_to_task_fair
yield_to
kvm_vcpu_yield_to[kvm]
...After a short dig, we found the
rq_lockheld byyield_to()may not be exactly the rq that the target task belongs to. There is a race window againsttry_to_wake_up().CPU0
target_taskblocking on CPU1
lock rq0 & rq1
double checktask_rq==p_rq, ok
woken to CPU2 (locktask_pi& rq2)task_rq= rq2
yield_to_task_fair(w/o lock rq2)In this race window,
yield_to()is operating the task w/o the currect lock. Fix this by taking taskpi_lockfirst.Fixes d95f41220065 ("sched: Add yield_to(task, preempt) functionality")Signed-off-by Tianchen Ding <[email protected]>--- kernel/sched/syscalls.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)kernel/sched/syscalls.c +0
-0
kernel/sched/syscalls.c int __sched yield_to(struct task_struct *p, bool preempt) · line 1437
Quoted 6 lines
kernel/sched/syscalls.c +1
-1@@ -1433,7 +1433,7 @@ int __sched yield_to(struct task_struct *p, bool preempt)struct rq *rq, *p_rq;int yielded =0 ; -scoped_guard (irqsave) { + scoped_guard (raw_spinlock_irqsave, &p->pi_lock) { rq =this_rq ();
Thanks!
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 5d808c78d97251af1d3a3e4f253e7d6c39fd871e Gitweb: https://git.kernel.org/tip/5d808c78d97251af1d3a3e4f253e7d6c39fd871e Author: Tianchen Ding <[email protected]> AuthorDate: Tue, 31 Dec 2024 13:50:20 +08:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Mon, 13 Jan 2025 14:10:22 +01:00
sched: Fix race between yield_to() and try_to_wake_up()
We met a SCHED_WARN in set_next_buddy():
__warn_printk
set_next_buddy
yield_to_task_fair
yield_to
kvm_vcpu_yield_to [kvm]
...
After a short dig, we found the rq_lock held by yield_to() may not be exactly the rq that the target task belongs to. There is a race window against try_to_wake_up().
CPU0 target_task
blocking on CPU1
lock rq0 & rq1
double check task_rq == p_rq, ok
woken to CPU2 (lock task_pi & rq2) task_rq = rq2
yield_to_task_fair (w/o lock rq2)
In this race window, yield_to() is operating the task w/o the correct lock. Fix this by taking task pi_lock first.
--- kernel/sched/syscalls.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/sched/syscalls.c +1 -1
@@ -1433,7 +1433,7 @@ int __sched yield_to(struct task_struct *p, bool preempt)struct rq *rq, *p_rq;int yielded =0 ; -scoped_guard (irqsave) { + scoped_guard (raw_spinlock_irqsave, &p->pi_lock) { rq =this_rq (); again: