sched: Fix psi_dequeue for Proxy Execution
PATCH v2 1/1 sched: Fix psi_dequeue for Proxy Execution
Currently, if the sleep flag is set, psi_dequeue() doesn't change any of the psi_flags.
This is because psi_switch_task() will clear TSK_ONCPU as well as other potential flags (TSK_RUNNING), and the assumption is that a voluntary sleep always consists of a task being dequeued followed shortly there after with a psi_sched_switch() call.
Proxy Execution changes this expectation, as mutex-blocked tasks that would normally sleep stay on the runqueue. But in the case where the mutex-owning task goes to sleep, or the owner is on a remote cpu, we will then deactivate the blocked task shortly after.
In that situation, the mutex-blocked task will have had its TSK_ONCPU cleared when it was switched off the cpu, but it will stay TSK_RUNNING. Then if we later dequeue it (as currently done if we hit a case find_proxy_task() can't yet handle, such as the case of the owner being on another rq or a sleeping owner) psi_dequeue() won't change any state (leaving it TSK_RUNNING), as it incorrectly expects a psi_task_switch() call to immediately follow.
Later on when the task get woken/re-enqueued, and psi_flags are
set for TSK_RUNNING, we hit an error as the task is alreadyTSK_RUNNING:
psi: inconsistent task state! task=188:kworker/28:0 cpu=28 psi_flags=4 clear=0 set=4
To resolve this, extend the logic in psi_dequeue() so that if the sleep flag is set, we also check if psi_flags have TSK_ONCPU set (meaning the psi_task_switch is imminent) before we do the shortcut return.
If TSK_ONCPU is not set, that means we've already switched away, and this psi_dequeue call needs to clear the flags.
---
proxy-v13:
* Reworked for collision
proxy-v15:
* Fixed commit message typo noticed by Todd Kjos v1 (separate from proxy series):
* Reworded commit message in response to K Prateek pointing out this issue can affect us earlier in the full proxy series then I had anticipated.
v2 (separate from proxy series):
* Minor tweak to comment suggested by Johannes
--- kernel/sched/stats.h | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
kernel/sched/stats.h +6 -1
@@ -180,8 +180,13 @@ static inline void psi_dequeue(struct task_struct *p, int flags)* avoid walking all ancestors twice, psi_task_switch() handles * TSK_RUNNING and TSK_IOWAIT for us when it moves TSK_ONCPU. * Do nothing here. +* +* In the SCHED_PROXY_EXECUTION case we may do sleeping +* dequeues that are not followed by a task switch, so check +* TSK_ONCPU is set to ensure the task switch is imminent. +* Otherwise clear the flags as usual. */ -if (flags &DEQUEUE_SLEEP ) +if ((flags &DEQUEUE_SLEEP ) && (p->psi_flags &TSK_ONCPU ))return ;/*
Signature · 1 line
2.52.0.rc2.455.g230fcf2819-goog
Quoted 4 lines · John Stultz
Currently, if the sleep flag is set,
psi_dequeue()doesn't change any of thepsi_flags.This is because
psi_switch_task()will clearTSK_ONCPUas well
psi_task_switch()
Quoted 77 lines
as other potential flags (
TSK_RUNNING), and the assumption is that a voluntary sleep always consists of a task being dequeued followed shortly there after with apsi_sched_switch()call.Proxy Execution changes this expectation, as mutex-blocked tasks that would normally sleep stay on the runqueue. But in the case where the mutex-owning task goes to sleep, or the owner is on a remote cpu, we will then deactivate the blocked task shortly after.
In that situation, the mutex-blocked task will have had its
TSK_ONCPUcleared when it was switched off the cpu, but it will stayTSK_RUNNING. Then if we later dequeue it (as currently done if we hit a casefind_proxy_task()can't yet handle, such as the case of the owner being on another rq or a sleeping owner)psi_dequeue()won't change any state (leaving itTSK_RUNNING), as it incorrectly expects apsi_task_switch()call to immediately follow.Later on when the task get woken/re-enqueued, and
psi_flagsare set forTSK_RUNNING, we hit an error as the task is alreadyTSK_RUNNING:psi: inconsistent task state! task=188:kworker/28:0 cpu=28 psi_flags=4 clear=0 set=4To resolve this, extend the logic in
psi_dequeue()so that if the sleep flag is set, we also check ifpsi_flagshaveTSK_ONCPUset (meaning thepsi_task_switchis imminent) before we do the shortcut return.If
TSK_ONCPUis not set, that means we've already switched away, and thispsi_dequeuecall needs to clear the flags.Fixes be41bde4c3a8 ("sched: Add an initial sketch of the find_proxy_task() function")Reported-by K Prateek Nayak <[email protected]>Signed-off-by John Stultz <[email protected]>Tested-by K Prateek Nayak <[email protected]>Acked-by Johannes Weiner <[email protected]>---
proxy-v13:
* Reworked for collision
proxy-v15:
* Fixed commit message typo noticed by Todd Kjos v1 (separate from proxy series):
* Reworded commit message in response to K Prateek pointing out this issue can affect us earlier in the full proxy series then I had anticipated.
v2 (separate from proxy series):
* Minor tweak to comment suggested by JohannesCc Joel Fernandes <[email protected]>Cc Qais Yousef <[email protected]>Cc Ingo Molnar <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Juri Lelli <[email protected]>Cc Vincent Guittot <[email protected]>Cc Dietmar Eggemann <[email protected]>Cc Valentin Schneider <[email protected]>Cc Johannes Weiner <[email protected]>Cc Suren Baghdasaryan <[email protected]>Cc Steven Rostedt <[email protected]>Cc Ben Segall <[email protected]>Cc Zimuzo Ezeozue <[email protected]>Cc Mel Gorman <[email protected]>Cc Will Deacon <[email protected]>Cc Waiman Long <[email protected]>Cc Boqun Feng <[email protected]>Cc "Paul E. McKenney" <[email protected]>Cc Metin Kaya <[email protected]>Cc Xuewen Yan <[email protected]>Cc K Prateek Nayak <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Daniel Lezcano <[email protected]>Cc Suleiman Souhlal <[email protected]>Cc kuyo chang <[email protected]>Cc hupu <[email protected]>
Quoted 4 lines
--- kernel/sched/stats.h | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
Quoted 2 lines
/*
Quoted 6 lines · Haiyue Wang
Quoted 3 lines · John Stultz
Currently, if the sleep flag is set,
psi_dequeue()doesn't change any of thepsi_flags.
Quoted 6 lines
Quoted 1 line
This is because
psi_switch_task()will clearTSK_ONCPUas well
psi_task_switch()Tested-by Haiyue Wang <[email protected]>
Thanks for the testing and the sharp eye on that detail!
I'll fix up and resend shortly here.
thanks
-john