Proxy Execution fixes for v7.1-rc
Subsystems (4)
cover Proxy Execution fixes for v7.1-rc
Hey All,
So in testing with the full Proxy Execution series,
Vineeth Pillai managed to trip some interesting bugs which initially looked to be KVM or RCU related[1], which he later diagnosed as Proxy Execution related and created a useful test driver to reproduce.
I found these same issues could be triggered with the upstream portions of Proxy Execution, so I wanted to send along these fixes for 7.1-rc
Again, a huge thanks to Vineeth for uncovering these issues that have evaded all my stress testing so far!
Thanks
-john
[1]: https://lore.kernel.org/lkml/[email protected]/
John Stultz (2):
sched: proxy-exec: Close race causing workqueue work being delayed
locking: mutex: Fix proxy-exec potentially deactivating tasks marked
TASK_RUNNING
kernel/locking/mutex.c | 1 + kernel/sched/core.c | 11 +++++++++++ 2 files changed, 12 insertions(+)
Signature · 1 line
2.54.0.545.g6539524ca2-goog
PATCH 1/2 sched: proxy-exec: Close race causing workqueue work being delayed
Vineeth reported seeing a KVM related deadlock connected to work queue lockups using the android17-6.18 tree, which has Proxy Execution enabled (using the full patch stack), but I've subsequently reproduced it on v7.1-rc1.
On further debugging he found:
- kvm-irqfd-cleanup workqueue and rcu_gp lands in a per-cpu pwq(work queue pool)
- one of kvm-irqfd-cleanup worker(say A) takes a mutex and then calls synchronize_srcu_expedited()
- one other kvm-irqfd-cleanup worker worker(Say B) tries to acquire the lock and then gets blocked
- On the way to blocking, this cpu gets an IPI and on return from IPI, it calls __schedule() and did not get to complete workqueue accounting(worker->sleeping = 0 and decrementing pool->nr_running). This is done in sched_submit_work() -> wq_worker_sleeping() called from schedule() and we got preempted before that.
- proxy execution doesn't immediately take it off run queue as p->blocked_on is set during __mutex_lock
- Next time when B is picked for running, it notices A(mutex holder) is not on a runqueue and then blocks B. find_proxy_task() -> proxy_deactivate() -> block_task()
- And things are then stuck. A is waiting for the workqueue to be run, but B can't run the workqueue as it is blocked on A.
The trouble is that with Proxy Execution, in __mutex_lock_common() we set the task state to TASK_UNINTERRUPTIBLE, and set blocked_on before calling into schedule(), where sched_submit_work() will be called.
But if an IPI comes in before we call schedule() the interrupt will call __schedule(SM_PREEMPT) directly. This causes the scheduler to see the current task as blocked_on, and deactivate it (because the owner is off the runqueue).
Since its deactivated, it wont' be run, and it won't get to call sched_submit_work().
Without proxy-execution, the SM_PREEMPT case will prevent the task from being dequeued, and it can be reselected again and run, which will allow it to finish calling into schedule() and calling sched_submit_work() before actually blocking.
So we need to make sure on the SM_PREEMPT case, if current is marked as blocked_on, we should clear the blocked_on state and mark the task RUNNABLE so the task can be selected to complete its call to schedule() -> sched_submit_work().
Now because we cleared BLOCKED_ON and set the task RUNNABLE, the task will be able to be selected and run again and loop back in __mutex_lock_common() where it can re-set the blocked_on state and call back into schedule() in order to properly be chosen as a donor.
Many thanks to Vineeth for figuring this very obscure race out and for implementing a test tool to make it easily reproducible!
---
Cc: Vineeth Pillai <[email protected]>
Cc: Sonam Sanju <[email protected]>
Cc: Sean Christopherson <[email protected]>
Cc: Kunwu Chan <[email protected]>
Cc: Tejun Heo <[email protected]>
Cc: Joel Fernandes <[email protected]>
Cc: Qais Yousef <[email protected]>
Cc: Ingo Molnar <[email protected]>
Cc: Peter Zijlstra <[email protected]>
Cc: Juri Lelli <[email protected]>
Cc: Vincent Guittot <[email protected]>
Cc: Dietmar Eggemann <[email protected]>
Cc: Valentin Schneider <[email protected]>
Cc: Steven Rostedt <[email protected]>
Cc: Will Deacon <[email protected]>
Cc: Waiman Long <[email protected]>
Cc: Boqun Feng <[email protected]>
Cc: "Paul E. McKenney" <[email protected]>
Cc: Metin Kaya <[email protected]>
Cc: Xuewen Yan <[email protected]>
Cc: K Prateek Nayak <[email protected]>
Cc: Thomas Gleixner <[email protected]>
Cc: Daniel Lezcano <[email protected]>
Cc: Suleiman Souhlal <[email protected]>
Cc: kuyo chang <[email protected]>
Cc: hupu <[email protected]>
Cc: [email protected]
--- kernel/sched/core.c | 11 +++++++++++ 1 file changed, 11 insertions(+)
kernel/sched/core.c +11 -0
@@ -7097,6 +7097,17 @@ static void __sched notrace __schedule(int sched_mode)try_to_block_task (rq, prev, &prev_state, !task_is_blocked (prev)); switch_count = &prev->nvcsw; + }else if (preempt && prev->blocked_on) { +/* +* If we are SM_PREEMPT, we may have interrupted +* after blocked_on was set, before schedule() +* was run, preventing workques from running. So +* clear blocked_on and mark task RUNNING so it +* can be reselected to run and complete its +* logic +*/ +WRITE_ONCE (prev->__state,TASK_RUNNING ); +clear_task_blocked_on (prev,NULL ); } pick_again:
Signature · 1 line
2.54.0.545.g6539524ca2-goog
PATCH 2/2 locking: mutex: Fix proxy-exec potentially deactivating tasks marked TASK_RUNNING
In reply to John StultzVineeth found came up with a test driver that could trip up workqueue stalls. After fixing one issue this test found, Vineeth reported the test was still failing.
Greatly simplified, a task that tries to take a mutex already owned by another task that is sleeping, can hit a edge case in the mutex_lock_common() case.
If the task fails to get the lock, calls into schedule, but gets a spurious wakeup, it will find that it is first waiter, and go into the mutex_optimistic_spin() logic. Though before calling mutex_optimistic_spin(), we clear task blocked_on state, since mutex_optimistic_spin() may call schedule() if need_resched() is set.
After mutex_optimistic_spin() fails, we set blocked_on again, restart the main mutex loop, try to take the lock and call into schedule_preempt_disabled().
From there, with proxy-execution, we'll see the task is blocked_on, follow the chain, see the owner is sleeping and dequeue the waiting task from the runqueue.
This all sounds fine and reasonable. But what I had missed is that in mutex_optimistic_spin(), not only do we call schedule() but we set TASK_RUNNABLE right before doing so.
This is ok for that invocation of schedule(). But when we come back we re-set the blocked_on we had just cleared, but we do not re-set the task state to TASK_INTERRUPTIBLE/UNINTERRUPTIBLE.
This means we have a task that is blocked_on & TASK_RUNNABLE, so when the proxy execution code dequeues the task, we are in trouble since future wakeups will be shortcut by the ttwu_state_match() check.
Thus, to avoid this, after mutex_optimistic_spin(), set the task state back when we set blocked_on.
Many many thanks again to Vineeth for his very useful testing driver that uncovered this long hidden bug, that I hadn't tripped in all my testing! Very impressed with the problems he's uncovered!
---
Cc: Vineeth Pillai <[email protected]>
Cc: Sonam Sanju <[email protected]>
Cc: Sean Christopherson <[email protected]>
Cc: Kunwu Chan <[email protected]>
Cc: Tejun Heo <[email protected]>
Cc: Joel Fernandes <[email protected]>
Cc: Qais Yousef <[email protected]>
Cc: Ingo Molnar <[email protected]>
Cc: Peter Zijlstra <[email protected]>
Cc: Juri Lelli <[email protected]>
Cc: Vincent Guittot <[email protected]>
Cc: Dietmar Eggemann <[email protected]>
Cc: Valentin Schneider <[email protected]>
Cc: Steven Rostedt <[email protected]>
Cc: Will Deacon <[email protected]>
Cc: Waiman Long <[email protected]>
Cc: Boqun Feng <[email protected]>
Cc: "Paul E. McKenney" <[email protected]>
Cc: Metin Kaya <[email protected]>
Cc: Xuewen Yan <[email protected]>
Cc: K Prateek Nayak <[email protected]>
Cc: Thomas Gleixner <[email protected]>
Cc: Daniel Lezcano <[email protected]>
Cc: Suleiman Souhlal <[email protected]>
Cc: kuyo chang <[email protected]>
Cc: hupu <[email protected]>
Cc: [email protected]
--- kernel/locking/mutex.c | 1 + 1 file changed, 1 insertion(+)
kernel/locking/mutex.c +1 -0
@@ -763,6 +763,7 @@ __mutex_lock_common(struct mutex *lock, unsigned int state, unsigned int subclasraw_spin_lock_irqsave (&lock->wait_lock, flags);raw_spin_lock (¤t->blocked_lock);__set_task_blocked_on (current, lock); +set_current_state (state);if (opt_acquired)break ;
Signature · 1 line
2.54.0.545.g6539524ca2-goog
Re: [PATCH 1/2] sched: proxy-exec: Close race causing workqueue work being delayed
In reply to John StultzHello John,
Quoted 26 lines · John Stultz
Vineeth reported seeing a KVM related deadlock connected to work queue lockups using the android17-6.18 tree, which has Proxy Execution enabled (using the full patch stack), but I've subsequently reproduced it on v7.1-rc1.
On further debugging he found:
- kvm-irqfd-cleanup workqueue andrcu_gplands in a per-cpu pwq(work queue pool)
- one of kvm-irqfd-cleanup worker(say A) takes a mutex and then callssynchronize_srcu_expedited()
- one other kvm-irqfd-cleanup worker worker(Say B) tries to acquire the lock and then gets blocked
- On the way to blocking, this cpu gets an IPI and on return from IPI, it calls__schedule()and did not get to complete workqueueaccounting(worker->sleeping = 0 and decrementing pool->nr_running). This is done insched_submit_work()->wq_worker_sleeping()called fromschedule()and we got preempted before that.
- proxy execution doesn't immediately take it off run queue asp->blocked_onis set during__mutex_lock
- Next time when B is picked for running, it notices A(mutex holder) is not on a runqueue and then blocks B.find_proxy_task()->proxy_deactivate()->block_task()
- And things are then stuck. A is waiting for the workqueue to be run, but B can't run the workqueue as it is blocked on A.
Quoted 4 lines
The trouble is that with Proxy Execution, in
__mutex_lock_common()we set the task state toTASK_UNINTERRUPTIBLE, and setblocked_onbefore calling intoschedule(), wheresched_submit_work()will be called.
Geez! That is an interesting race.
Quoted 25 lines
But if an IPI comes in before we call
schedule()the interrupt will call__schedule(SM_PREEMPT)directly. This causes the scheduler to see the current task asblocked_on, and deactivate it (because the owner is off the runqueue).Since its deactivated, it wont' be run, and it won't get to call
sched_submit_work().Without proxy-execution, the
SM_PREEMPTcase will prevent the task from being dequeued, and it can be reselected again and run, which will allow it to finish calling intoschedule()and callingsched_submit_work()before actually blocking.So we need to make sure on the
SM_PREEMPTcase, if current is marked asblocked_on, we should clear theblocked_onstate and mark the task RUNNABLE so the task can be selected to complete its call toschedule()->sched_submit_work().Now because we cleared
BLOCKED_ONand set the task RUNNABLE, the task will be able to be selected and run again and loop back in__mutex_lock_common()where it can re-set theblocked_onstate and call back intoschedule()in order to properly be chosen as a donor.
Quoted 6 lines
Many thanks to Vineeth for figuring this very obscure race out and for implementing a test tool to make it easily reproducible!
Reported-by Vineeth Pillai <[email protected]>Tested-by Vineeth Pillai <[email protected]>Signed-off-by John Stultz <[email protected]>
I guess it is missing a:
since that is where we began blocking a task on task_is_blocked(). I really wish there was a better way to have detected this but I cannot think of any better way at the moment so feel free to include:
Quoted 40 lines
---
Cc: Vineeth Pillai <[email protected]>
Cc: Sonam Sanju <[email protected]>
Cc: Sean Christopherson <[email protected]>
Cc: Kunwu Chan <[email protected]>
Cc: Tejun Heo <[email protected]>
Cc: Joel Fernandes <[email protected]>
Cc: Qais Yousef <[email protected]>
Cc: Ingo Molnar <[email protected]>
Cc: Peter Zijlstra <[email protected]>
Cc: Juri Lelli <[email protected]>
Cc: Vincent Guittot <[email protected]>
Cc: Dietmar Eggemann <[email protected]>
Cc: Valentin Schneider <[email protected]>
Cc: Steven Rostedt <[email protected]>
Cc: Will Deacon <[email protected]>
Cc: Waiman Long <[email protected]>
Cc: Boqun Feng <[email protected]>
Cc: "Paul E. McKenney" <[email protected]>
Cc: Metin Kaya <[email protected]>
Cc: Xuewen Yan <[email protected]>
Cc: K Prateek Nayak <[email protected]>
Cc: Thomas Gleixner <[email protected]>
Cc: Daniel Lezcano <[email protected]>
Cc: Suleiman Souhlal <[email protected]>
Cc: kuyo chang <[email protected]>
Cc: hupu <[email protected]>
Cc: [email protected]--- kernel/sched/core.c | 11 +++++++++++ 1 file changed, 11 insertions(+)kernel/sched/core.c +0
-0@@ -7097,6 +7097,17 @@ static void __sched notrace __schedule(int sched_mode)try_to_block_task (rq, prev, &prev_state, !task_is_blocked (prev)); switch_count = &prev->nvcsw;
kernel/sched/core.c static void __sched notrace __schedule(int sched_mode) · line 7109
Quoted 10 lines
kernel/sched/core.c +10
-0+ }else if (preempt && prev->blocked_on) { +/* +* If we are SM_PREEMPT, we may have interrupted +* after blocked_on was set, before schedule() +* was run, preventing workques from running. So +* clear blocked_on and mark task RUNNING so it +* can be reselected to run and complete its +* logic +*/ +WRITE_ONCE (prev->__state,TASK_RUNNING );
nit.
You probably need to update "prev_state" too for trace_sched_switch() to capture the right state down below.
Since this is on the way to schedule(), I wonder if it possible to just do a "next = prev" and goto picked ... but that adds more latency on PREEMPT_RT so that is a no go I presume.
Quoted 4 lines
kernel/sched/core.c +1
-0+clear_task_blocked_on (prev,NULL ); } pick_again:
Signature · 2 lines
Thanks and Regards, Prateek
Re: [PATCH 2/2] locking: mutex: Fix proxy-exec potentially deactivating tasks marked TASK_RUNNING
In reply to John StultzHello John,
Quoted 39 lines · John Stultz
Vineeth found came up with a test driver that could trip up workqueue stalls. After fixing one issue this test found, Vineeth reported the test was still failing.
Greatly simplified, a task that tries to take a mutex already owned by another task that is sleeping, can hit a edge case in the
mutex_lock_common()case.If the task fails to get the lock, calls into schedule, but gets a spurious wakeup, it will find that it is first waiter, and go into the
mutex_optimistic_spin()logic. Though before callingmutex_optimistic_spin(), we clear taskblocked_onstate, sincemutex_optimistic_spin()may callschedule()ifneed_resched()is set.After
mutex_optimistic_spin()fails, we setblocked_onagain, restart the main mutex loop, try to take the lock and call intoschedule_preempt_disabled().From there, with proxy-execution, we'll see the task is
blocked_on, follow the chain, see the owner is sleeping and dequeue the waiting task from the runqueue.This all sounds fine and reasonable. But what I had missed is that in
mutex_optimistic_spin(), not only do we callschedule()but we setTASK_RUNNABLEright before doing so.This is ok for that invocation of
schedule(). But when we come back we re-set theblocked_onwe had just cleared, but we do not re-set the task state to TASK_INTERRUPTIBLE/UNINTERRUPTIBLE.This means we have a task that is
blocked_on&TASK_RUNNABLE, so when the proxy execution code dequeues the task, we are in trouble since future wakeups will be shortcut by thettwu_state_match()check.Thus, to avoid this, after
mutex_optimistic_spin(), set the task state back when we setblocked_on.
Quoted 8 lines
Many many thanks again to Vineeth for his very useful testing driver that uncovered this long hidden bug, that I hadn't tripped in all my testing! Very impressed with the problems he's uncovered!
Reported-by Vineeth Pillai <[email protected]>Tested-by Vineeth Pillai <[email protected]>Signed-off-by John Stultz <[email protected]>
I think this too requires a:
With that, feel free to include:
Quoted 37 lines
---
Cc: Vineeth Pillai <[email protected]>
Cc: Sonam Sanju <[email protected]>
Cc: Sean Christopherson <[email protected]>
Cc: Kunwu Chan <[email protected]>
Cc: Tejun Heo <[email protected]>
Cc: Joel Fernandes <[email protected]>
Cc: Qais Yousef <[email protected]>
Cc: Ingo Molnar <[email protected]>
Cc: Peter Zijlstra <[email protected]>
Cc: Juri Lelli <[email protected]>
Cc: Vincent Guittot <[email protected]>
Cc: Dietmar Eggemann <[email protected]>
Cc: Valentin Schneider <[email protected]>
Cc: Steven Rostedt <[email protected]>
Cc: Will Deacon <[email protected]>
Cc: Waiman Long <[email protected]>
Cc: Boqun Feng <[email protected]>
Cc: "Paul E. McKenney" <[email protected]>
Cc: Metin Kaya <[email protected]>
Cc: Xuewen Yan <[email protected]>
Cc: K Prateek Nayak <[email protected]>
Cc: Thomas Gleixner <[email protected]>
Cc: Daniel Lezcano <[email protected]>
Cc: Suleiman Souhlal <[email protected]>
Cc: kuyo chang <[email protected]>
Cc: hupu <[email protected]>
Cc: [email protected]--- kernel/locking/mutex.c | 1 + 1 file changed, 1 insertion(+)kernel/locking/mutex.c +0
-0
kernel/locking/mutex.c __mutex_lock_common(struct mutex *lock, unsigned int state, unsigned int subclas · line 769
Quoted 7 lines
kernel/locking/mutex.c +1
-0@@ -763,6 +763,7 @@ __mutex_lock_common(struct mutex *lock, unsigned int state, unsigned int subclasraw_spin_lock_irqsave (&lock->wait_lock, flags);raw_spin_lock (¤t->blocked_lock);__set_task_blocked_on (current, lock); +set_current_state (state);if (opt_acquired)break ;
nit.
As a micro-optimization, you can probably move the __set_task_blocked_on() and set_current_state() after this break.
Signature · 2 lines
Thanks and Regards, Prateek
Peter Zijlstra
maintainerRe: [PATCH 1/2] sched: proxy-exec: Close race causing workqueue work being delayed
In reply to John StultzQuoted 8 lines · John Stultz
kernel/sched/core.c | 11 +++++++++++ 1 file changed, 11 insertions(+)kernel/sched/core.c +0
-0
kernel/sched/core.c static void __sched notrace __schedule(int sched_mode) · line 7104
Quoted 8 lines
kernel/sched/core.c +5
-0@@ -7097,6 +7097,17 @@ static void __sched notrace __schedule(int sched_mode)try_to_block_task (rq, prev, &prev_state, !task_is_blocked (prev)); switch_count = &prev->nvcsw; + }else if (preempt && prev->blocked_on) { +/* +* If we are SM_PREEMPT, we may have interrupted +* after blocked_on was set, before schedule() +* was run, preventing workques from running. So
workqueues
kernel/sched/core.c static void __sched notrace __schedule(int sched_mode) · line 7113
Quoted 9 lines
kernel/sched/core.c +6
-0+* clear blocked_on and mark task RUNNING so it +* can be reselected to run and complete its +* logic +*/ +WRITE_ONCE (prev->__state,TASK_RUNNING ); +clear_task_blocked_on (prev,NULL ); } pick_again:
*groan*, this feels wrong. Preemption should never touch state. Let me try and wake up and make sense of this.
Peter Zijlstra
maintainerRe: [PATCH 1/2] sched: proxy-exec: Close race causing workqueue work being delayed
Quoted 29 lines · Peter Zijlstra
Quoted 8 lines · John Stultz
kernel/sched/core.c | 11 +++++++++++ 1 file changed, 11 insertions(+)kernel/sched/core.c +0
-0kernel/sched/core.c static void __sched notrace __schedule(int sched_mode) · line 7104
Quoted 8 lines
kernel/sched/core.c +5
-0@@ -7097,6 +7097,17 @@ static void __sched notrace __schedule(int sched_mode)try_to_block_task (rq, prev, &prev_state, !task_is_blocked (prev)); switch_count = &prev->nvcsw; + }else if (preempt && prev->blocked_on) { +/* +* If we are SM_PREEMPT, we may have interrupted +* after blocked_on was set, before schedule() +* was run, preventing workques from running. So workqueues
Quoted 8 lines
kernel/sched/core.c +6
-0+* clear blocked_on and mark task RUNNING so it +* can be reselected to run and complete its +* logic +*/ +WRITE_ONCE (prev->__state,TASK_RUNNING ); +clear_task_blocked_on (prev,NULL ); }
Quoted 4 lines
kernel/sched/core.c static void __sched notrace __schedule(int sched_mode) · line 7113
Quoted 1 line
kernel/sched/core.c +0
-0pick_again:*groan*, this feels wrong. Preemption should never touch state. Let me try and wake up and make sense of this.
So all non-special block states *SHOULD* be in a loop and handle spurious wakeups -- I fixed a pile of offenders some many years ago, but there really isn't anything in the kernel that validates this.
[ I suppose someone could try and do a cocci test for this? ]
Any wait for non-special states that is not a loop is fundamentally broken, since many of the lock wake-up paths are explicitly racy in that they can cause spurious wakeups (which is the safe side of the race, since insufficient wakeups is bad etc.).
OTOH special states, are special, esp. because they cannot handle spurious wakeups.
Eg, consider something like:
set_current_state(TASK_FROZEN) <PREEMPT> current->__state = TASK_RUNNING </PREEMPT/ schedule();
is all sorts of broken. Now, obiously special states must never have blocked_on set, so this can be fudged about. But still, touching __state from schedule seems wrong.
Anyway, the historical distinction between a blocked task and a preempted task is that the blocked task is not on the runqueue, while the preempted task is kept on the runqueue.
Obviously PE wrecks this, and hence the problem. And yeah, amazing we never hit this before.
Something like so perhaps?
---
include/linux/sched.h +5 -2
@@ -846,7 +846,11 @@ struct task_struct {struct alloc_tag *alloc_tag;#endif -int on_cpu; +u8 on_cpu; +u8 on_rq; +u8 is_blocked; +u8 __pad; +struct __call_single_node wake_entry;unsigned int wakee_flips;unsigned long wakee_flip_decay_ts;
@@ -861,7 +865,6 @@ struct task_struct {*/ int recent_used_cpu;int wake_cpu; -int on_rq;int prio;int static_prio;
kernel/sched/core.c +10 -1
@@ -615,6 +615,13 @@ EXPORT_SYMBOL(__trace_set_current_state);* [ The astute reader will observe that it is possible for two tasks on one * CPU to have ->on_cpu = 1 at the same time. ] * +* p->is_blocked <- { 0, 1 }: +* +* is set by block_task() and cleared by ttwu_do_activate() and indicates +* this task is blocked, as opposed to runnable. Used to distinguish between +* preempted and blocked tasks for proxy exec, which keeps everything on the +* runqueue. +* * task_cpu(p): is changed by set_task_cpu(), the rules are: * * - Don't call set_task_cpu() on a blocked task:
@@ -2225,6 +2232,7 @@ void deactivate_task(struct rq *rq, struct task_struct *p, int flags)static void block_task (struct rq *rq,struct task_struct *p,int flags) { + p->is_blocked =1 ;if (dequeue_task (rq, p,DEQUEUE_SLEEP | flags))__block_task (rq, p); }
@@ -3722,6 +3730,7 @@ ttwu_do_activate(struct rq *rq, struct task_struct *p, int wake_flags,atomic_dec (&task_rq (p)->nr_iowait); } + p->is_blocked =0 ;activate_task (rq, p, en_flags);wakeup_preempt (rq, p, wake_flags);
@@ -7107,7 +7116,7 @@ static void __sched notrace __schedule(int sched_mode)struct task_struct *prev_donor = rq->donor;rq_set_donor (rq, next); -if (unlikely (next->blocked_on)) { +if (unlikely (next->is_blocked && next->blocked_on)) { next =find_proxy_task (rq, next, &rf);if (!next) {zap_balance_callbacks (rq);
K Prateek Nayak
reviewerRe: [PATCH 1/2] sched: proxy-exec: Close race causing workqueue work being delayed
Hello Peter,
Quoted 115 lines · Peter Zijlstra
Quoted 29 lines · Peter Zijlstra
Quoted 8 lines · John Stultz
kernel/sched/core.c | 11 +++++++++++ 1 file changed, 11 insertions(+)kernel/sched/core.c +0
-0kernel/sched/core.c static void __sched notrace __schedule(int sched_mode) · line 7104
Quoted 8 lines
kernel/sched/core.c +5
-0@@ -7097,6 +7097,17 @@ static void __sched notrace __schedule(int sched_mode)try_to_block_task (rq, prev, &prev_state, !task_is_blocked (prev)); switch_count = &prev->nvcsw; + }else if (preempt && prev->blocked_on) { +/* +* If we are SM_PREEMPT, we may have interrupted +* after blocked_on was set, before schedule() +* was run, preventing workques from running. So workqueues
Quoted 8 lines
kernel/sched/core.c +6
-0+* clear blocked_on and mark task RUNNING so it +* can be reselected to run and complete its +* logic +*/ +WRITE_ONCE (prev->__state,TASK_RUNNING ); +clear_task_blocked_on (prev,NULL ); }Quoted 4 lines
kernel/sched/core.c static void __sched notrace __schedule(int sched_mode) · line 7113
Quoted 1 line
kernel/sched/core.c +0
-0pick_again:*groan*, this feels wrong. Preemption should never touch state. Let me try and wake up and make sense of this.
So all non-special block states *SHOULD* be in a loop and handle spurious wakeups -- I fixed a pile of offenders some many years ago, but there really isn't anything in the kernel that validates this.
[ I suppose someone could try and do a cocci test for this? ]
Any wait for non-special states that is not a loop is fundamentally broken, since many of the lock wake-up paths are explicitly racy in that they can cause spurious wakeups (which is the safe side of the race, since insufficient wakeups is bad etc.).
OTOH special states, are special, esp. because they cannot handle spurious wakeups.
Eg, consider something like:
set_current_state(TASK_FROZEN) <PREEMPT> current->__state = TASK_RUNNING </PREEMPT/ schedule();is all sorts of broken. Now, obiously special states must never have
blocked_onset, so this can be fudged about. But still, touching__statefrom schedule seems wrong.Anyway, the historical distinction between a blocked task and a preempted task is that the blocked task is not on the runqueue, while the preempted task is kept on the runqueue.
Obviously PE wrecks this, and hence the problem. And yeah, amazing we never hit this before.
Something like so perhaps?
---
include/linux/sched.h +5
-2@@ -846,7 +846,11 @@ struct task_struct {struct alloc_tag *alloc_tag;#endif -int on_cpu; +u8 on_cpu; +u8 on_rq; +u8 is_blocked; +u8 __pad; +struct __call_single_node wake_entry;unsigned int wakee_flips;unsigned long wakee_flip_decay_ts;@@ -861,7 +865,6 @@ struct task_struct {*/ int recent_used_cpu;int wake_cpu; -int on_rq;int prio;int static_prio;kernel/sched/core.c +7
-0@@ -615,6 +615,13 @@ EXPORT_SYMBOL(__trace_set_current_state);* [ The astute reader will observe that it is possible for two tasks on one * CPU to have ->on_cpu = 1 at the same time. ] * +* p->is_blocked <- { 0, 1 }: +* +* is set by block_task() and cleared by ttwu_do_activate() and indicates +* this task is blocked, as opposed to runnable. Used to distinguish between +* preempted and blocked tasks for proxy exec, which keeps everything on the +* runqueue. +* * task_cpu(p): is changed by set_task_cpu(), the rules are: * * - Don't call set_task_cpu() on a blocked task:
kernel/sched/core.c void deactivate_task(struct rq *rq, struct task_struct *p, int flags) · line 2235
Quoted 4 lines
kernel/sched/core.c +1
-0@@ -2225,6 +2232,7 @@ void deactivate_task(struct rq *rq, struct task_struct *p, int flags)static void block_task (struct rq *rq,struct task_struct *p,int flags) { + p->is_blocked =1 ;
We never reach here with PROXY_EXEC. Instead we bail out in the caller try_to_block_task() ...
Quoted 12 lines
kernel/sched/core.c +1
-0if (dequeue_task (rq, p,DEQUEUE_SLEEP | flags))__block_task (rq, p); }@@ -3722,6 +3730,7 @@ ttwu_do_activate(struct rq *rq, struct task_struct *p, int wake_flags,atomic_dec (&task_rq (p)->nr_iowait); } + p->is_blocked =0 ;activate_task (rq, p, en_flags);wakeup_preempt (rq, p, wake_flags);
kernel/sched/core.c static void __sched notrace __schedule(int sched_mode) · line 7119
Quoted 5 lines
kernel/sched/core.c +1
-1@@ -7107,7 +7116,7 @@ static void __sched notrace __schedule(int sched_mode)struct task_struct *prev_donor = rq->donor;rq_set_donor (rq, next); -if (unlikely (next->blocked_on)) { +if (unlikely (next->is_blocked && next->blocked_on)) {
... so ->is_blocked here is always false for proxy tasks retained on the runqueue.
I was trying something like below but I'm somewhere missing a clear_task_blocked_on() for PROXY_WAKING before going back into mutex_lock_common():
include/linux/sched.h +4 -0
@@ -586,6 +586,7 @@ struct sched_entity {unsigned char sched_delayed;unsigned char rel_deadline;unsigned char custom_slice; +unsigned char sched_proxy;/* hole */ u64 exec_start;
@@ -2222,6 +2223,7 @@ static inline void __clear_task_blocked_on(struct task_struct *p, struct mutex ** clearing the relationship with a different lock. */ WARN_ON_ONCE (m && p->blocked_on && p->blocked_on != m && p->blocked_on !=PROXY_WAKING ); +WRITE_ONCE (p->se.sched_proxy,0 ); p->blocked_on =NULL ; }
@@ -2250,6 +2252,8 @@ static inline void __set_task_blocked_on_waking(struct task_struct *p, struct mu* the relationship with a different lock. */ WARN_ON_ONCE (m && p->blocked_on != m && p->blocked_on !=PROXY_WAKING ); +/* Force the task down proxy_force_return() path. */ +WRITE_ONCE (p->se.sched_proxy,1 ); p->blocked_on =PROXY_WAKING ; }
init/init_task.c +1 -0
@@ -118,6 +118,7 @@ struct task_struct init_task __aligned(L1_CACHE_BYTES) = { }, .se = { .group_node = LIST_HEAD_INIT(init_task.se.group_node), + .sched_proxy = 0, }, .rt = { .run_list = LIST_HEAD_INIT(init_task.rt.run_list),
kernel/sched/core.c +15 -5
@@ -4395,6 +4395,7 @@ static void __sched_fork(u64 clone_flags, struct task_struct *p) p->se.nr_migrations =0 ; p->se.vruntime =0 ; p->se.vlag =0 ; + p->se.sched_proxy =0 ;INIT_LIST_HEAD (&p->se.group_node);/* A delayed task cannot be in clone(). */
@@ -6535,8 +6536,13 @@ static bool try_to_block_task(struct rq *rq, struct task_struct *p,* blocked on a mutex, and we want to keep it on the runqueue * to be selectable for proxy-execution. */ -if (!should_block) +if (!should_block) { +guard (raw_spinlock)(&p->blocked_lock); +/* Stable against race */ +if (task_is_blocked (p)) +WRITE_ONCE (p->se.sched_proxy,1 );return false; + } p->sched_contributes_to_load = (task_state & TASK_UNINTERRUPTIBLE) &&
@@ -6765,11 +6771,15 @@ find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf)bool curr_in_chain = false;int this_cpu =cpu_of (rq);struct task_struct *p; -struct mutex *mutex;int owner_cpu;/* Follow blocked_on chain. */ -for (p = donor; (mutex = p->blocked_on); p = owner) { +for (p = donor;READ_ONCE (p->se.sched_proxy); p = owner) { +struct mutex *mutex = p->blocked_on; + +if (!mutex) +return NULL ; +/* if its PROXY_WAKING, do return migration or run if current */ if (mutex ==PROXY_WAKING ) {if (task_current (rq, p)) {
@@ -6787,7 +6797,7 @@ find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf)guard (raw_spinlock)(&p->blocked_lock);/* Check again that p is blocked with blocked_lock held */ -if (mutex !=__get_task_blocked_on (p)) { +if (!p->se.sched_proxy || mutex !=__get_task_blocked_on (p)) {/* * Something changed in the blocked_on chain and * we don't know if only at this level. So, let's
@@ -7044,7 +7054,7 @@ static void __sched notrace __schedule(int sched_mode)struct task_struct *prev_donor = rq->donor;rq_set_donor (rq, next); -if (unlikely (next->blocked_on)) { +if (unlikely (READ_ONCE (next->se.sched_proxy))) { next =find_proxy_task (rq, next, &rf);if (!next) {zap_balance_callbacks (rq);
---
Quoted 3 lines
kernel/sched/core.c +0
-0next =find_proxy_task (rq, next, &rf);if (!next) {zap_balance_callbacks (rq);
Signature · 2 lines
Thanks and Regards, Prateek
K Prateek Nayak
reviewerRe: [PATCH 1/2] sched: proxy-exec: Close race causing workqueue work being delayed
Quoted 3 lines · K Prateek Nayak
I was trying something like below but I'm somewhere missing a
clear_task_blocked_on()forPROXY_WAKINGbefore going back intomutex_lock_common():
And I seem to have been missing:
kernel/sched/core.c +3 -0
@@ -7046,6 +7046,9 @@ static void __sched notrace __schedule(int sched_mode) switch_count = &prev->nvcsw; } +if (!prev_state &&task_is_blocked (prev)) +clear_task_blocked_on (prev,NULL ); + pick_again:assert_balance_callbacks_empty (rq); next =pick_next_task (rq, rq->donor, &rf);
---
With that, it survives test-ww_mutex and a sched-messaging run without any splats.
Quoted 111 lines
include/linux/sched.h +4
-0@@ -586,6 +586,7 @@ struct sched_entity {unsigned char sched_delayed;unsigned char rel_deadline;unsigned char custom_slice; +unsigned char sched_proxy;/* hole */ u64 exec_start;@@ -2222,6 +2223,7 @@ static inline void __clear_task_blocked_on(struct task_struct *p, struct mutex ** clearing the relationship with a different lock. */ WARN_ON_ONCE (m && p->blocked_on && p->blocked_on != m && p->blocked_on !=PROXY_WAKING ); +WRITE_ONCE (p->se.sched_proxy,0 ); p->blocked_on =NULL ; }@@ -2250,6 +2252,8 @@ static inline void __set_task_blocked_on_waking(struct task_struct *p, struct mu* the relationship with a different lock. */ WARN_ON_ONCE (m && p->blocked_on != m && p->blocked_on !=PROXY_WAKING ); +/* Force the task down proxy_force_return() path. */ +WRITE_ONCE (p->se.sched_proxy,1 ); p->blocked_on =PROXY_WAKING ; }init/init_task.c +1
-0@@ -118,6 +118,7 @@ struct task_struct init_task __aligned(L1_CACHE_BYTES) = { }, .se = { .group_node = LIST_HEAD_INIT(init_task.se.group_node), + .sched_proxy = 0, }, .rt = { .run_list = LIST_HEAD_INIT(init_task.rt.run_list),kernel/sched/core.c +15
-6@@ -4395,6 +4395,7 @@ static void __sched_fork(u64 clone_flags, struct task_struct *p) p->se.nr_migrations =0 ; p->se.vruntime =0 ; p->se.vlag =0 ; + p->se.sched_proxy =0 ;INIT_LIST_HEAD (&p->se.group_node);/* A delayed task cannot be in clone(). */ @@ -6535,8 +6536,13 @@ static bool try_to_block_task(struct rq *rq, struct task_struct *p,* blocked on a mutex, and we want to keep it on the runqueue * to be selectable for proxy-execution. */ -if (!should_block) +if (!should_block) { +guard (raw_spinlock)(&p->blocked_lock); +/* Stable against race */ +if (task_is_blocked (p)) +WRITE_ONCE (p->se.sched_proxy,1 );return false; + } p->sched_contributes_to_load = (task_state & TASK_UNINTERRUPTIBLE) &&@@ -6765,11 +6771,15 @@ find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf)bool curr_in_chain = false;int this_cpu =cpu_of (rq);struct task_struct *p; -struct mutex *mutex;int owner_cpu;/* Follow blocked_on chain. */ -for (p = donor; (mutex = p->blocked_on); p = owner) { +for (p = donor;READ_ONCE (p->se.sched_proxy); p = owner) { +struct mutex *mutex = p->blocked_on; + +if (!mutex) +return NULL ; +/* if its PROXY_WAKING, do return migration or run if current */ if (mutex ==PROXY_WAKING ) {if (task_current (rq, p)) {@@ -6787,7 +6797,7 @@ find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf)guard (raw_spinlock)(&p->blocked_lock);/* Check again that p is blocked with blocked_lock held */ -if (mutex !=__get_task_blocked_on (p)) { +if (!p->se.sched_proxy || mutex !=__get_task_blocked_on (p)) {/* * Something changed in the blocked_on chain and * we don't know if only at this level. So, let's @@ -7044,7 +7054,7 @@ static void __sched notrace __schedule(int sched_mode)struct task_struct *prev_donor = rq->donor;rq_set_donor (rq, next); -if (unlikely (next->blocked_on)) { +if (unlikely (READ_ONCE (next->se.sched_proxy))) { next =find_proxy_task (rq, next, &rf);if (!next) {zap_balance_callbacks (rq); ---Quoted 4 lines
(quoted diff) +0
-0next = find_proxy_task(rq, next, &rf); if (!next) { zap_balance_callbacks(rq);
Signature · 2 lines
Thanks and Regards, Prateek
Peter Zijlstra
maintainerRe: [PATCH 1/2] sched: proxy-exec: Close race causing workqueue work being delayed
In reply to K Prateek NayakQuoted 69 lines · K Prateek Nayak
Quoted 47 lines
Something like so perhaps?
---
include/linux/sched.h +5
-2@@ -846,7 +846,11 @@ struct task_struct {struct alloc_tag *alloc_tag;#endif -int on_cpu; +u8 on_cpu; +u8 on_rq; +u8 is_blocked; +u8 __pad; +struct __call_single_node wake_entry;unsigned int wakee_flips;unsigned long wakee_flip_decay_ts;@@ -861,7 +865,6 @@ struct task_struct {*/ int recent_used_cpu;int wake_cpu; -int on_rq;int prio;int static_prio;kernel/sched/core.c +7
-0@@ -615,6 +615,13 @@ EXPORT_SYMBOL(__trace_set_current_state);* [ The astute reader will observe that it is possible for two tasks on one * CPU to have ->on_cpu = 1 at the same time. ] * +* p->is_blocked <- { 0, 1 }: +* +* is set by block_task() and cleared by ttwu_do_activate() and indicates +* this task is blocked, as opposed to runnable. Used to distinguish between +* preempted and blocked tasks for proxy exec, which keeps everything on the +* runqueue. +* * task_cpu(p): is changed by set_task_cpu(), the rules are: * * - Don't call set_task_cpu() on a blocked task: kernel/sched/core.c void deactivate_task(struct rq *rq, struct task_struct *p, int flags) · line 2235
Quoted 4 lines
kernel/sched/core.c +1
-0@@ -2225,6 +2232,7 @@ void deactivate_task(struct rq *rq, struct task_struct *p, int flags)static void block_task (struct rq *rq,struct task_struct *p,int flags) { + p->is_blocked =1 ;We never reach here with
PROXY_EXEC. Instead we bail out in the callertry_to_block_task()...Quoted 14 lines
kernel/sched/core.c +1
-0if (dequeue_task (rq, p,DEQUEUE_SLEEP | flags))__block_task (rq, p); }@@ -3722,6 +3730,7 @@ ttwu_do_activate(struct rq *rq, struct task_struct *p, int wake_flags,atomic_dec (&task_rq (p)->nr_iowait); } + p->is_blocked =0 ;activate_task (rq, p, en_flags);wakeup_preempt (rq, p, wake_flags);@@ -7107,7 +7116,7 @@ static void __sched notrace __schedule(int sched_mode)struct task_struct *prev_donor = rq->donor;
Quoted 6 lines
kernel/sched/core.c static void __sched notrace __schedule(int sched_mode)
Quoted 3 lines
kernel/sched/core.c +1
-1rq_set_donor (rq, next); -if (unlikely (next->blocked_on)) { +if (unlikely (next->is_blocked && next->blocked_on)) {... so ->is_blocked here is always false for proxy tasks retained on the runqueue.
Right. Also, egads, we really should fix that block/ttwu part, this is a mess.
Anyway, idea is simple even if execution turns into a bit of a mess now, set when task really is blocked and clear on wakeup.
Quoted 9 lines
I was trying something like below but I'm somewhere missing a
clear_task_blocked_on()forPROXY_WAKINGbefore going back intomutex_lock_common():include/linux/sched.h +0
-0
include/linux/sched.h struct sched_entity { · line 590
Quoted 5 lines
include/linux/sched.h +1
-0@@ -586,6 +586,7 @@ struct sched_entity {unsigned char sched_delayed;unsigned char rel_deadline;unsigned char custom_slice; +unsigned char sched_proxy;/* hole */
Should not live in sched_entity I suppose.
Re: [PATCH 2/2] locking: mutex: Fix proxy-exec potentially deactivating tasks marked TASK_RUNNING
In reply to K Prateek NayakQuoted 8 lines · K Prateek Nayak
On 4/28/2026 12:08 AM, John Stultz wrote:
I think this too requires a:Fixes be41bde4c3a8 ("sched: Add an initial sketch of the find_proxy_task() function")With that, feel free to include:
Reviewed-by K Prateek Nayak <[email protected]>
Thanks so much for looking this over!
Quoted 10 lines
Quoted 10 lines
kernel/locking/mutex.c +1
-0@@ -763,6 +763,7 @@ __mutex_lock_common(struct mutex *lock, unsigned int state, unsigned int subclasraw_spin_lock_irqsave (&lock->wait_lock, flags);raw_spin_lock (¤t->blocked_lock);__set_task_blocked_on (current, lock); +set_current_state (state);
Quoted 7 lines
kernel/locking/mutex.c __mutex_lock_common(struct mutex *lock, unsigned int state, unsigned int subclas · line 769
Quoted 2 lines
kernel/locking/mutex.c +0
-0if (opt_acquired)break ;nit.
As a micro-optimization, you can probably move the
__set_task_blocked_on()andset_current_state()after this break.
So my main reason for setting it before the break makes it symmetric with the clearing that happens just outside the loop. Feels a little cleaner to me for it to all match.
thanks
-john
Re: [PATCH 1/2] sched: proxy-exec: Close race causing workqueue work being delayed
In reply to K Prateek NayakQuoted 154 lines · K Prateek Nayak
Quoted 115 lines · Peter Zijlstra
Quoted 29 lines · Peter Zijlstra
Quoted 8 lines · John Stultz
kernel/sched/core.c | 11 +++++++++++ 1 file changed, 11 insertions(+)kernel/sched/core.c +0
-0kernel/sched/core.c static void __sched notrace __schedule(int sched_mode) · line 7104
Quoted 8 lines
kernel/sched/core.c +5
-0@@ -7097,6 +7097,17 @@ static void __sched notrace __schedule(int sched_mode)try_to_block_task (rq, prev, &prev_state, !task_is_blocked (prev)); switch_count = &prev->nvcsw; + }else if (preempt && prev->blocked_on) { +/* +* If we are SM_PREEMPT, we may have interrupted +* after blocked_on was set, before schedule() +* was run, preventing workques from running. So workqueues
Quoted 8 lines
kernel/sched/core.c +6
-0+* clear blocked_on and mark task RUNNING so it +* can be reselected to run and complete its +* logic +*/ +WRITE_ONCE (prev->__state,TASK_RUNNING ); +clear_task_blocked_on (prev,NULL ); }Quoted 4 lines
kernel/sched/core.c static void __sched notrace __schedule(int sched_mode) · line 7113
Quoted 1 line
kernel/sched/core.c +0
-0pick_again:*groan*, this feels wrong. Preemption should never touch state. Let me try and wake up and make sense of this.
So all non-special block states *SHOULD* be in a loop and handle spurious wakeups -- I fixed a pile of offenders some many years ago, but there really isn't anything in the kernel that validates this.
[ I suppose someone could try and do a cocci test for this? ]
Any wait for non-special states that is not a loop is fundamentally broken, since many of the lock wake-up paths are explicitly racy in that they can cause spurious wakeups (which is the safe side of the race, since insufficient wakeups is bad etc.).
OTOH special states, are special, esp. because they cannot handle spurious wakeups.
Eg, consider something like:
set_current_state(TASK_FROZEN) <PREEMPT> current->__state = TASK_RUNNING </PREEMPT/ schedule();is all sorts of broken. Now, obiously special states must never have
blocked_onset, so this can be fudged about. But still, touching__statefrom schedule seems wrong.Anyway, the historical distinction between a blocked task and a preempted task is that the blocked task is not on the runqueue, while the preempted task is kept on the runqueue.
Obviously PE wrecks this, and hence the problem. And yeah, amazing we never hit this before.
Something like so perhaps?
---
include/linux/sched.h +5
-2@@ -846,7 +846,11 @@ struct task_struct {struct alloc_tag *alloc_tag;#endif -int on_cpu; +u8 on_cpu; +u8 on_rq; +u8 is_blocked; +u8 __pad; +struct __call_single_node wake_entry;unsigned int wakee_flips;unsigned long wakee_flip_decay_ts;@@ -861,7 +865,6 @@ struct task_struct {*/ int recent_used_cpu;int wake_cpu; -int on_rq;int prio;int static_prio;kernel/sched/core.c +7
-0@@ -615,6 +615,13 @@ EXPORT_SYMBOL(__trace_set_current_state);* [ The astute reader will observe that it is possible for two tasks on one * CPU to have ->on_cpu = 1 at the same time. ] * +* p->is_blocked <- { 0, 1 }: +* +* is set by block_task() and cleared by ttwu_do_activate() and indicates +* this task is blocked, as opposed to runnable. Used to distinguish between +* preempted and blocked tasks for proxy exec, which keeps everything on the +* runqueue. +* * task_cpu(p): is changed by set_task_cpu(), the rules are: * * - Don't call set_task_cpu() on a blocked task: kernel/sched/core.c void deactivate_task(struct rq *rq, struct task_struct *p, int flags) · line 2235
Quoted 4 lines
kernel/sched/core.c +1
-0@@ -2225,6 +2232,7 @@ void deactivate_task(struct rq *rq, struct task_struct *p, int flags)static void block_task (struct rq *rq,struct task_struct *p,int flags) { + p->is_blocked =1 ;We never reach here with
PROXY_EXEC. Instead we bail out in the callertry_to_block_task()...Quoted 12 lines
kernel/sched/core.c +1
-0if (dequeue_task (rq, p,DEQUEUE_SLEEP | flags))__block_task (rq, p); }@@ -3722,6 +3730,7 @@ ttwu_do_activate(struct rq *rq, struct task_struct *p, int wake_flags,atomic_dec (&task_rq (p)->nr_iowait); } + p->is_blocked =0 ;activate_task (rq, p, en_flags);wakeup_preempt (rq, p, wake_flags);kernel/sched/core.c static void __sched notrace __schedule(int sched_mode) · line 7119
Quoted 5 lines
kernel/sched/core.c +1
-1@@ -7107,7 +7116,7 @@ static void __sched notrace __schedule(int sched_mode)struct task_struct *prev_donor = rq->donor;rq_set_donor (rq, next); -if (unlikely (next->blocked_on)) { +if (unlikely (next->is_blocked && next->blocked_on)) {... so ->is_blocked here is always false for proxy tasks retained on the runqueue.
I was trying something like below but I'm somewhere missing a
clear_task_blocked_on()forPROXY_WAKINGbefore going back intomutex_lock_common():include/linux/sched.h +0
-0
include/linux/sched.h struct sched_entity { · line 590
Quoted 5 lines
include/linux/sched.h +1
-0@@ -586,6 +586,7 @@ struct sched_entity {unsigned char sched_delayed;unsigned char rel_deadline;unsigned char custom_slice; +unsigned char sched_proxy;/* hole */
I feel like this is so tied to the blocked_on value, I suspect it makes the most sense to have this flag be the low bit of that pointer?
Sort of a blocked_on latch, to signal its really in effect?
Plus it gets cleared automatically on set and clear, so it looks a little cleaner.
include/linux/sched.h static bool try_to_block_task(struct rq *rq, struct task_struct *p, · line 6545
Quoted 12 lines
include/linux/sched.h +6
-1@@ -6535,8 +6536,13 @@ static bool try_to_block_task(struct rq *rq, struct task_struct *p,* blocked on a mutex, and we want to keep it on the runqueue * to be selectable for proxy-execution. */ -if (!should_block) +if (!should_block) { +guard (raw_spinlock)(&p->blocked_lock); +/* Stable against race */ +if (task_is_blocked (p)) +WRITE_ONCE (p->se.sched_proxy,1 );return false; + }
So if we double check and find the task isn't blocked anymore, we probably shouldn't return early here, no?
Let me take a stab at the bit flag approach and see how it goes.
thanks
-john
K Prateek Nayak
reviewerRe: [PATCH 1/2] sched: proxy-exec: Close race causing workqueue work being delayed
Hello John,
Quoted 14 lines · John Stultz
include/linux/sched.h struct sched_entity { · line 590
Quoted 10 lines
include/linux/sched.h +1
-0@@ -586,6 +586,7 @@ struct sched_entity {unsigned char sched_delayed;unsigned char rel_deadline;unsigned char custom_slice; +unsigned char sched_proxy;/* hole */ I feel like this is so tied to the
blocked_onvalue, I suspect it makes the most sense to have this flag be the low bit of that pointer?
Quoted 4 lines
Sort of a
blocked_onlatch, to signal its really in effect?Plus it gets cleared automatically on set and clear, so it looks a little cleaner.
I had actually started looking into that last night but sleep eventually got to me! I agree that is much cleaner to do it alongside p->blocked_on.
Quoted 18 lines
Quoted 4 lines
include/linux/sched.h +0
-0@@ -6535,8 +6536,13 @@ static bool try_to_block_task(struct rq *rq, struct task_struct *p,* blocked on a mutex, and we want to keep it on the runqueue include/linux/sched.h static bool try_to_block_task(struct rq *rq, struct task_struct *p, · line 6545
Quoted 10 lines
include/linux/sched.h +6
-1* to be selectable for proxy-execution. */ -if (!should_block) +if (!should_block) { +guard (raw_spinlock)(&p->blocked_lock); +/* Stable against race */ +if (task_is_blocked (p)) +WRITE_ONCE (p->se.sched_proxy,1 );return false; + }So if we double check and find the task isn't blocked anymore, we probably shouldn't return early here, no?
Quoted 1 line
Let me take a stab at the bit flag approach and see how it goes.
In case you want to peek at my homework ;-)
(Lightly tested with test-ww_mutex and sched-messaging)
include/linux/sched.h +40 -8
@@ -1237,7 +1237,7 @@ struct task_struct {struct rt_mutex_waiter *pi_blocked_on;#endif -struct mutex *blocked_on;/* lock we're blocked on */ +unsigned long blocked_on;/* lock we're blocked on */ raw_spinlock_t blocked_lock;#ifdef CONFIG_DETECT_HUNG_TASK_BLOCKER
@@ -2188,12 +2188,20 @@ extern int __cond_resched_rwlock_write(rwlock_t *lock) __must_hold(lock);* evaluated for return-migration before it is run. So if the task is * blocked_on PROXY_WAKING, return migrate it before running it. */ -#define PROXY_WAKING ((struct mutex *)(-1L)) +#define PROXY_WAKING ((unsigned long)(-1L)) + +#define BO_FLAG_PROXY BIT(0) +#define BO_FLAGS GENMASK(0, 0) + +static inline struct mutex *__get_blocked_on_mutex (unsigned long m) +{ +return (struct mutex *)(m & ~BO_FLAGS ); +}static inline struct mutex *__get_task_blocked_on (struct task_struct *p) {lockdep_assert_held_once (&p->blocked_lock); -return p->blocked_on ==PROXY_WAKING ?NULL : p->blocked_on; +return p->blocked_on ==PROXY_WAKING ?NULL :__get_blocked_on_mutex (p->blocked_on); }static inline void __set_task_blocked_on (struct task_struct *p,struct mutex *m)
@@ -2208,8 +2216,11 @@ static inline void __set_task_blocked_on(struct task_struct *p, struct mutex *m)* with a different mutex. Note, setting it to the same * lock repeatedly is ok. */ -WARN_ON_ONCE (p->blocked_on && p->blocked_on != m); - p->blocked_on = m; +WARN_ON_ONCE (p->blocked_on && +__get_blocked_on_mutex (p->blocked_on) != m); + +BUG_ON ((unsigned long )m &BO_FLAGS ); + p->blocked_on = (unsigned long )m & ~BO_FLAGS ; }static inline void __clear_task_blocked_on (struct task_struct *p,struct mutex *m)
@@ -2221,8 +2232,10 @@ static inline void __clear_task_blocked_on(struct task_struct *p, struct mutex ** blocked_on relationships, but make sure we are not * clearing the relationship with a different lock. */ -WARN_ON_ONCE (m && p->blocked_on && p->blocked_on != m && p->blocked_on !=PROXY_WAKING ); - p->blocked_on =NULL ; +WARN_ON_ONCE (m && p->blocked_on && +__get_blocked_on_mutex (p->blocked_on) != m && + p->blocked_on !=PROXY_WAKING ); + p->blocked_on =0 ; }static inline void clear_task_blocked_on (struct task_struct *p,struct mutex *m)
@@ -2249,7 +2262,12 @@ static inline void __set_task_blocked_on_waking(struct task_struct *p, struct mu* already set to waking, but make sure we are not changing * the relationship with a different lock. */ -WARN_ON_ONCE (m && p->blocked_on != m && p->blocked_on !=PROXY_WAKING ); +WARN_ON_ONCE (m && +__get_blocked_on_mutex (p->blocked_on) != m && + p->blocked_on !=PROXY_WAKING ); + +/* Always ensure PROXY_WAKING implies BO_FLAG_PROXY. */ +BUILD_BUG_ON (!(PROXY_WAKING &BO_FLAG_PROXY )); p->blocked_on =PROXY_WAKING ; }
@@ -2259,6 +2277,20 @@ static inline void set_task_blocked_on_waking(struct task_struct *p, struct mute __set_task_blocked_on_waking(p, m); } +static inline void set_task_blocked_on_proxy (struct task_struct *p) +{ +guard (raw_spinlock_irqsave)(&p->blocked_lock); + +if (!sched_proxy_exec ()) +return ; + +/* Don't add BO_FLAG_PROXY if blocked_on was already cleared */ +if (!p->blocked_on) +return ; + + p->blocked_on |=BO_FLAG_PROXY ; +} +#else static inline void __clear_task_blocked_on (struct task_struct *p,struct rt_mutex *m) {
kernel/fork.c +1 -1
@@ -2177,7 +2177,7 @@ __latent_entropy struct task_struct *copy_process(lockdep_init_task (p); - p->blocked_on =NULL ;/* not blocked yet */ + p->blocked_on =0 ;/* not blocked yet */ #ifdef CONFIG_BCACHE p->sequential_io =0 ;
kernel/sched/core.c +31 -13
78 lines not on this page · raw message
kernel/sched/sched.h +1 -1
@@ -2390,7 +2390,7 @@ static inline bool task_is_blocked(struct task_struct *p)if (!sched_proxy_exec ())return false; -return !!p->blocked_on; +return !!(p->blocked_on &BO_FLAG_PROXY ); }static inline int task_on_cpu (struct rq *rq,struct task_struct *p)
---
There is probably a better way to do it by inspecting the flags only in find_proxy_task() and keep task_is_blocked() as is but I wanted to see if it can be done as a part of task_ib_blocked() helper since that seemed cleaner in my head.
I'll let you fiddle with it.
Signature · 2 lines
Thanks and Regards, Prateek
Re: [PATCH 1/2] sched: proxy-exec: Close race causing workqueue work being delayed
Quoted 21 lines · K Prateek Nayak
Quoted 20 lines · John Stultz
Quoted 6 lines
include/linux/sched.h +0
-0@@ -6535,8 +6536,13 @@ static bool try_to_block_task(struct rq *rq, struct task_struct *p,* blocked on a mutex, and we want to keep it on the runqueue include/linux/sched.h static bool try_to_block_task(struct rq *rq, struct task_struct *p, · line 6545
Quoted 10 lines
include/linux/sched.h +6
-1* to be selectable for proxy-execution. */ -if (!should_block) +if (!should_block) { +guard (raw_spinlock)(&p->blocked_lock); +/* Stable against race */ +if (task_is_blocked (p)) +WRITE_ONCE (p->se.sched_proxy,1 );return false; + }So if we double check and find the task isn't blocked anymore, we probably shouldn't return early here, no?
Quoted 4 lines
Quoted 1 line
Let me take a stab at the bit flag approach and see how it goes.
In case you want to peek at my homework ;-)
Well, now it will just seem like I'm cheating! :)
kernel/sched/core.c static void __sched notrace __schedule(int sched_mode) · line 7063
Quoted 12 lines
kernel/sched/core.c +8
-0@@ -7043,8 +7053,16 @@ static void __sched notrace __schedule(int sched_mode)if (sched_proxy_exec ()) {struct task_struct *prev_donor = rq->donor; +/* +* A wakeup raced with block_task(); +* Clear blocked_on before running the task +* again. +*/ +if (unlikely (!prev_state && prev->blocked_on)) +clear_task_blocked_on (prev,NULL ); +
So similar to your previous change, I was hitting this same case, and adding in your change here was bothering me a bit that it felt like clearing things here was papering over an unexpected state change that we didn't have before.
Part of the issue is the blocked_on state machine is already a little complex:
NULL -> ptr -> PROXY_WAKING ->NULL
(with some limited shortcuting from ptr->NULL when we are dealing with current)
So adding this BO_FLAG_PROXY/"latch" bit to the mix complicates things:
NULL -> ptr:unlatched -> ptr:latched -> PROXY_WAKING -> NULL
With extra lines for:
ptr:unlatched -> NULL
ptr:latched -> NULL
ptr:unlatched -> PROXY_WAKING
I kept on seeing these unexpected transitions where before we latched the blocked_on state, we'd get a wakeup on another cpu from the owner unlocking the lock, so we'd go ptr:unlatched -> PROXY_WAKING (along with setting TASK_RUNNING). Then because we weren't latched, we might be running midway through the lock logic. If we go into schedule() we'll just come back out eventually (as we're TASK_RUNNING). Then when then try again in the lock loop to set blocked_on to ptr:unlatched again, we hit warnings because we are setting it to a ptr when it is PROXY_WAKING and not NULL.
I'm sure this was obvious to you when you wrote the snippit above, but I've been running a little slow, so let's not talk about how long it took for me to actaully understand this. :P
And yes, your snippit above avoids this, but it feels a little incidental, cleaning up after the fact. So I think it might be better to simplify the state machine a little to prevent it.
My current thought is to cut out the ptr:unlatched -> PROXY_WAKING transition. And instead in __set_task_blocked_on_waking(), instead of checking blocked_on and doing an early return if it is null, we could instead check if the latch bit was set, and if so, set blocked_on to NULL. This bascially treats ptr:unlatched the same as NULL in most cases for the state machine.
Now, earlier you made some nice explanations earlier about how some of the blocked_on checks done w/ the rq_lock and not the blocked_lock were safe because of the limited situations where we clear blocked_on. And this at first seems to put that assumption risk, but I think it still holds if we always consider ptr:unlatched equivalent to NULL. So once ptr:latched is set, the rule still holds.
That gives us:
NULL -> ptr:unlatched -> ptr:latched -> PROXY_WAKING -> NULL
Where NULL and ptr:unlatched are functionally equivalent except for the ability to transition to ptr:latched.
With:
ptr:latched -> NULL // only done on current
ptr:unlatched -> NULL // only done on current or when trying to set waking
So far my testing is doing ok with this. Let me know if you see any holes though.
I'll try to cleanup my current changes (surely copying some of your excellent work :), and send this out tomorrow (I'm shot for today) for more concrete review.
thanks
-john
Re: [PATCH 1/2] sched: proxy-exec: Close race causing workqueue work being delayed
Quoted 4 lines · John Stultz
My current thought is to cut out the ptr:unlatched ->
PROXY_WAKINGtransition. And instead in__set_task_blocked_on_waking(), instead of checkingblocked_onand doing an early return if it is null, we could instead check if the latch bit was set, and if so, setblocked_onto
Ugh! I meant "check if the latch bit was unset" in the above. Apologies for any confusion there.
Quoted 2 lines
NULL. This bascially treats ptr:unlatched the same as NULL in most cases for the state machine.
thanks
-john
K Prateek Nayak
reviewerRe: [PATCH 1/2] sched: proxy-exec: Close race causing workqueue work being delayed
In reply to John StultzHello John,
Quoted 22 lines · John Stultz
Quoted 21 lines · K Prateek Nayak
Quoted 20 lines · John Stultz
Quoted 6 lines
include/linux/sched.h +0
-0@@ -6535,8 +6536,13 @@ static bool try_to_block_task(struct rq *rq, struct task_struct *p,* blocked on a mutex, and we want to keep it on the runqueue include/linux/sched.h static bool try_to_block_task(struct rq *rq, struct task_struct *p, · line 6545
Quoted 10 lines
include/linux/sched.h +6
-1* to be selectable for proxy-execution. */ -if (!should_block) +if (!should_block) { +guard (raw_spinlock)(&p->blocked_lock); +/* Stable against race */ +if (task_is_blocked (p)) +WRITE_ONCE (p->se.sched_proxy,1 );return false; + }So if we double check and find the task isn't blocked anymore, we probably shouldn't return early here, no?
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
Let me take a stab at the bit flag approach and see how it goes.
In case you want to peek at my homework ;-)
Well, now it will just seem like I'm cheating! :)
But I can't prove I didn't copy your idea first ;-)
Quoted 30 lines
kernel/sched/core.c static void __sched notrace __schedule(int sched_mode) · line 7063
Quoted 12 lines
kernel/sched/core.c +8
-0@@ -7043,8 +7053,16 @@ static void __sched notrace __schedule(int sched_mode)if (sched_proxy_exec ()) {struct task_struct *prev_donor = rq->donor; +/* +* A wakeup raced with block_task(); +* Clear blocked_on before running the task +* again. +*/ +if (unlikely (!prev_state && prev->blocked_on)) +clear_task_blocked_on (prev,NULL ); +So similar to your previous change, I was hitting this same case, and adding in your change here was bothering me a bit that it felt like clearing things here was papering over an unexpected state change that we didn't have before.
Part of the issue is the
blocked_onstate machine is already a little complex:
NULL -> ptr ->PROXY_WAKING->NULL
(with some limited shortcuting fromptr->NULLwhen we are dealing with current)So adding this BO_FLAG_PROXY/"latch" bit to the mix complicates things:
NULL -> ptr:unlatched -> ptr:latched ->PROXY_WAKING-> NULLWith extra lines for:
ptr:unlatched -> NULL
ptr:latched -> NULL
ptr:unlatched ->PROXY_WAKING
Quoted 9 lines
I kept on seeing these unexpected transitions where before we latched the
blocked_onstate, we'd get a wakeup on another cpu from the owner unlocking the lock, so we'd go ptr:unlatched ->PROXY_WAKING(along with settingTASK_RUNNING). Then because we weren't latched, we might be running midway through the lock logic. If we go intoschedule()we'll just come back out eventually (as we'reTASK_RUNNING). Then when then try again in the lock loop to setblocked_onto ptr:unlatched again, we hit warnings because we are setting it to a ptr when it isPROXY_WAKINGand not NULL.
Yes, it was precisely that which caused the splat in my case too for the initial suggestion on Peter's diff. I'll try to add a little bit more context in those tricky comments from next time onwards. Sorry about that!
Quoted 9 lines
I'm sure this was obvious to you when you wrote the snippit above, but I've been running a little slow, so let's not talk about how long it took for me to actaully understand this. :P
And yes, your snippit above avoids this, but it feels a little incidental, cleaning up after the fact. So I think it might be better to simplify the state machine a little to prevent it.
Quoted 6 lines
My current thought is to cut out the ptr:unlatched ->
PROXY_WAKINGtransition. And instead in__set_task_blocked_on_waking(), instead of checkingblocked_onand doing an early return if it is null, we could instead check if the latch bit was set, and if so, setblocked_onto NULL. This bascially treats ptr:unlatched the same as NULL in most cases for the state machine.
That is a fantastic idea! Since the sched bits have not blocked it yet, there is no need to go through the return migration path.
Clearing it when not latched should do the trick.
Quoted 7 lines
Now, earlier you made some nice explanations earlier about how some of the
blocked_onchecks done w/ therq_lockand not theblocked_lockwere safe because of the limited situations where we clearblocked_on. And this at first seems to put that assumption risk, but I think it still holds if we always consider ptr:unlatched equivalent to NULL. So once ptr:latched is set, the rule still holds.
Yes, that is indeed true. Transitioning the task to latched state, and unlatching it, will both be done with rq_lock + blocked_on_lock so it should be okay to peek at the latched state when holding the task's rq lock in ttwu_runnable(), and in schedule().
Quoted 7 lines
That gives us:
NULL -> ptr:unlatched -> ptr:latched ->PROXY_WAKING-> NULLWhere NULL and ptr:unlatched are functionally equivalent except for the ability to transition to ptr:latched.
Quoted 6 lines
With:
ptr:latched -> NULL // only done on current
ptr:unlatched -> NULL // only done on current or when trying to set wakingSo far my testing is doing ok with this. Let me know if you see any holes though.
This looks good to me! Cannot spot any holes with that.
However, to completely get rid of that hunk in __schedule() we probably need to order setting the blocked_on wrt task's __state based on Peter's analysis in
https://lore.kernel.org/lkml/[email protected]/
As Peter suggested, I think we need:
__set_task_blocked_on(current, lock);
/*
* Pairs with smp_rmb() after ttwu_state_match() in try_to_wake_up().
* Ensures ttwu always see the correct blocked_on state for wakeups.
*/
smp_wmb();
set_current_state(state);in __mutex_lock_common() for it to be completely safe with the proxy_needs_return() bits and that should be good enough to avoid that race but I'll let Peter comment if he thinks there is still a race window open.
Quoted 4 lines
I'll try to cleanup my current changes (surely copying some of your excellent work :), and send this out tomorrow (I'm shot for today) for more concrete review.
I'll be looking forward to it :-)
Signature · 2 lines
Thanks and Regards, Prateek
Quoted 12 lines · John Stultz
Hey All,
So in testing with the full Proxy Execution series,
Vineeth Pillai managed to trip some interesting bugs which initially looked to be KVM or RCU related[1], which he later diagnosed as Proxy Execution related and created a useful test driver to reproduce.I found these same issues could be triggered with the upstream portions of Proxy Execution, so I wanted to send along these fixes for 7.1-rc
Quoted 8 lines
Again, a huge thanks to Vineeth for uncovering these issues that have evaded all my stress testing so far!
Thanks
-john
Thanks John, Vineeth.
From the SRCU side, this helps explain the earlier KVM/RCU-looking stall pattern as a downstream symptom, with trigger likely in proxy-exec scheduler/mutex interaction.
It would be good to confirm whether the original workload still reproduces (process_srcu pending + irqfd-side stalls) after applying this series, and whether there is a proxy-exec on/off comparison under the same setup.
Thanks for driving this down.
Quoted 40 lines
Cc Vineeth Pillai <[email protected]>Cc Sonam Sanju <[email protected]>Cc Sean Christopherson <[email protected]>Cc Kunwu Chan <[email protected]>Cc Tejun Heo <[email protected]>Cc Joel Fernandes <[email protected]>Cc Qais Yousef <[email protected]>Cc Ingo Molnar <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Juri Lelli <[email protected]>Cc Vincent Guittot <[email protected]>Cc Dietmar Eggemann <[email protected]>Cc Valentin Schneider <[email protected]>Cc Steven Rostedt <[email protected]>Cc Will Deacon <[email protected]>Cc Waiman Long <[email protected]>Cc Boqun Feng <[email protected]>Cc "Paul E. McKenney" <[email protected]>Cc Metin Kaya <[email protected]>Cc Xuewen Yan <[email protected]>Cc K Prateek Nayak <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Daniel Lezcano <[email protected]>Cc Suleiman Souhlal <[email protected]>Cc kuyo chang <[email protected]>Cc hupu <[email protected]>John Stultz (2):
sched: proxy-exec: Close race causing workqueue work being delayed
locking: mutex: Fix proxy-exec potentially deactivating tasks markedTASK_RUNNINGkernel/locking/mutex.c | 1 + kernel/sched/core.c | 11 +++++++++++ 2 files changed, 12 insertions(+)Signature · 1 line
2.54.0.545.g6539524ca2-goog
Re: [PATCH 1/2] sched: proxy-exec: Close race causing workqueue work being delayed
In reply to Peter ZijlstraQuoted 58 lines · Peter Zijlstra
Quoted 29 lines · Peter Zijlstra
Quoted 8 lines · John Stultz
kernel/sched/core.c | 11 +++++++++++ 1 file changed, 11 insertions(+)kernel/sched/core.c +0
-0kernel/sched/core.c static void __sched notrace __schedule(int sched_mode) · line 7104
Quoted 8 lines
kernel/sched/core.c +5
-0@@ -7097,6 +7097,17 @@ static void __sched notrace __schedule(int sched_mode)try_to_block_task (rq, prev, &prev_state, !task_is_blocked (prev)); switch_count = &prev->nvcsw; + }else if (preempt && prev->blocked_on) { +/* +* If we are SM_PREEMPT, we may have interrupted +* after blocked_on was set, before schedule() +* was run, preventing workques from running. So workqueues
Quoted 8 lines
kernel/sched/core.c +6
-0+* clear blocked_on and mark task RUNNING so it +* can be reselected to run and complete its +* logic +*/ +WRITE_ONCE (prev->__state,TASK_RUNNING ); +clear_task_blocked_on (prev,NULL ); }Quoted 4 lines
kernel/sched/core.c static void __sched notrace __schedule(int sched_mode) · line 7113
Quoted 1 line
kernel/sched/core.c +0
-0pick_again:*groan*, this feels wrong. Preemption should never touch state. Let me try and wake up and make sense of this.
So all non-special block states *SHOULD* be in a loop and handle spurious wakeups -- I fixed a pile of offenders some many years ago, but there really isn't anything in the kernel that validates this.
[ I suppose someone could try and do a cocci test for this? ]
Any wait for non-special states that is not a loop is fundamentally broken, since many of the lock wake-up paths are explicitly racy in that they can cause spurious wakeups (which is the safe side of the race, since insufficient wakeups is bad etc.).
OTOH special states, are special, esp. because they cannot handle spurious wakeups.
Eg, consider something like:
set_current_state(TASK_FROZEN) <PREEMPT> current->__state = TASK_RUNNING </PREEMPT/ schedule();
Quoted 3 lines
is all sorts of broken. Now, obiously special states must never have
blocked_onset, so this can be fudged about. But still, touching__statefrom schedule seems wrong.
Hey Peter,
Thanks again for the background here and the alternative proposal
that K Prateek and I have iterated on.
I did want to clarify one case here just for my understanding. The approach in this first proposal was sort of leaning on the similar pattern in the signal_pending_state() case in try_to_block_task(). Is that just a hard exception to the rule here?
thanks
-john
Re: [PATCH 1/2] sched: proxy-exec: Close race causing workqueue work being delayed
In reply to K Prateek NayakQuoted 15 lines · K Prateek Nayak
However, to completely get rid of that hunk in
__schedule()we probably need to order setting theblocked_onwrt task's__statebased on Peter's analysis in
https://lore.kernel.org/lkml/[email protected]/As Peter suggested, I think we need:
__set_task_blocked_on(current, lock); /* * Pairs with smp_rmb() after ttwu_state_match() in try_to_wake_up(). * Ensures ttwu always see the correct blocked_on state for wakeups. */ smp_wmb(); set_current_state(state);
Quoted 5 lines
in
__mutex_lock_common()for it to be completely safe with theproxy_needs_return()bits and that should be good enough to avoid that race but I'll let Peter comment if he thinks there is still a race window open.
Ok, it seems like we could add the smp_wmb in the bottom of __set_task_blocked_on() then?
This is probably worth its own patch and (correct me if I'm wrong) feels separate enough that I might try to look at this after the fixes for this current issue are settled?
thanks
-john
K Prateek Nayak
reviewerRe: [PATCH 1/2] sched: proxy-exec: Close race causing workqueue work being delayed
In reply to John StultzHello John,
Quoted 20 lines · John Stultz
Quoted 16 lines
Any wait for non-special states that is not a loop is fundamentally broken, since many of the lock wake-up paths are explicitly racy in that they can cause spurious wakeups (which is the safe side of the race, since insufficient wakeups is bad etc.).
OTOH special states, are special, esp. because they cannot handle spurious wakeups.
Eg, consider something like:
set_current_state(TASK_FROZEN) <PREEMPT> current->__state = TASK_RUNNING </PREEMPT/ schedule();Quoted 3 lines
is all sorts of broken. Now, obiously special states must never have
blocked_onset, so this can be fudged about. But still, touching__statefrom schedule seems wrong.
Quoted 8 lines
Hey Peter,
Thanks again for the background here and the alternative proposal
that K Prateek and I have iterated on.I did want to clarify one case here just for my understanding. The approach in this first proposal was sort of leaning on the similar pattern in the
signal_pending_state()case intry_to_block_task(). Is that just a hard exception to the rule here?
signal_pending_state() looks at the task_state only at a SM_NONE / SM_RTLOCK_WAIT preemption point before making a call and only sets to TASK_RUNNING if the state is (TASK_INTERRUPTIBLE | TASK_WAKEKILL) and there is a signal pending. All the wakeups for that are routed through
Everything outside that state and coming from a SM_PREEMPT path will leave the task state untouched in __schedule() until it hits the natural SM_NONE / SM_RTLOCK_WAIT preemption point.
Only looking at ->blocked_on, and not the prev_state to set prev to TASK_RUNNING can be dangerous. That said, it would be stupid for a task to sets a special state and decide to grab a mutex which overrides the state.
With the latch design, we are more or less aligned with something similar to signal_pending_state() that looks at the tasks state from a safe preemption point.
Signature · 2 lines
Thanks and Regards, Prateek