sched_ext: Fix stale direct dispatch state in ddsp_dsq_id
Subsystems (2)
Andrea Righi
reviewerPATCH 1/1 sched_ext: Fix stale direct dispatch state in ddsp_dsq_id
@p->scx.ddsp_dsq_id can be left set (non-SCX_DSQ_INVALID) in three scenarios, causing a spurious WARN_ON_ONCE() in mark_direct_dispatch() when the next wakeup's ops.select_cpu() calls scx_bpf_dsq_insert():
1. Deferred dispatch cancellation: when a task is directly dispatched to a remote CPU's local DSQ via ops.select_cpu() or ops.enqueue(), the dispatch is deferred (since we can't lock the remote rq while holding the current one). If the task is dequeued before processing the dispatch in process_ddsp_deferred_locals(), dispatch_dequeue() removes the task from the list leaving a stale direct dispatch state.
Fix: clear ddsp_dsq_id and ddsp_enq_flags in the !list_empty branch of dispatch_dequeue().
2. Holding-cpu dispatch race: when dispatch_to_local_dsq() transfers a task to another CPU's local DSQ, it sets holding_cpu and releases DISPATCHING before locking the source rq. If dequeue wins the race and clears holding_cpu, dispatch_enqueue() is never called and ddsp_dsq_id is not cleared.
Fix: clear ddsp_dsq_id and ddsp_enq_flags when clearing holding_cpu in dispatch_dequeue().
3. Cross-scheduler-instance stale state: When an SCX scheduler exits, scx_bypass() iterates over all runnable tasks to dequeue/re-enqueue them, but sleeping tasks are not on any runqueue and are not touched. If a sleeping task had a deferred dispatch in flight (ddsp_dsq_id set) at the time the scheduler exited, the state persists. When a new scheduler instance loads and calls scx_enable_task() for all tasks, it does not reset this leftover state. The next wakeup's ops.select_cpu() then sees a non-INVALID ddsp_dsq_id and triggers:
WARN_ON_ONCE(p->scx.ddsp_dsq_id != SCX_DSQ_INVALID)
Fix: clear ddsp_dsq_id and ddsp_enq_flags in scx_enable_task() before calling ops.enable(), ensuring each new scheduler instance starts with a clean direct dispatch state per task.
With all the fixes applied the SCX_DSQ_INVALID warning doesn't seem to happen anymore.
--- kernel/sched/ext.c | 23 +++++++++++++++++++---- 1 file changed, 19 insertions(+), 4 deletions(-)
kernel/sched/ext.c +19 -4
@@ -1163,20 +1163,32 @@ static void dispatch_dequeue(struct rq *rq, struct task_struct *p)if (!dsq) {/* * If !dsq && on-list, @p is on @rq's ddsp_deferred_locals. -* Unlinking is all that's needed to cancel. +* Unlink and clear the deferred dispatch state. */ -if (unlikely (!list_empty (&p->scx.dsq_list.node))) +if (unlikely (!list_empty (&p->scx.dsq_list.node))) {list_del_init (&p->scx.dsq_list.node); + p->scx.ddsp_dsq_id =SCX_DSQ_INVALID ; + p->scx.ddsp_enq_flags =0 ; + } +/* * When dispatching directly from the BPF scheduler to a local * DSQ, the task isn't associated with any DSQ but * @p->scx.holding_cpu may be set under the protection of -* %SCX_OPSS_DISPATCHING. +* %SCX_OPSS_DISPATCHING. If we win the race and clear +* holding_cpu before dispatch_to_local_dsq() completes, the +* in-flight dispatch is cancelled and dispatch_enqueue() won't +* be called. Clear the stale direct dispatch state here so the +* next wakeup starts clean. */ -if (p->scx.holding_cpu >=0 ) +if (p->scx.holding_cpu >=0 ) { p->scx.holding_cpu =-1 ; + p->scx.ddsp_dsq_id =SCX_DSQ_INVALID ; + p->scx.ddsp_enq_flags =0 ; + } +return ; }
@@ -2945,6 +2957,9 @@ static void scx_enable_task(struct task_struct *p) p->scx.weight =sched_weight_to_cgroup (weight); + p->scx.ddsp_dsq_id =SCX_DSQ_INVALID ; + p->scx.ddsp_enq_flags =0 ; +if (SCX_HAS_OP (sch, enable))SCX_CALL_OP_TASK (sch,SCX_KF_REST , enable, rq, p);scx_set_task_state (p,SCX_TASK_ENABLED );
Signature · 1 line
2.53.0
Tejun Heo
reviewerHello,
(cc'ing Patrick Somaru. This is the same issue reported on
https://github.com/sched-ext/scx/pull/3482)
Quoted 14 lines · Andrea Righi
@p->scx.ddsp_dsq_id can be left set (non-SCX_DSQ_INVALID) in three scenarios, causing a spurious
WARN_ON_ONCE()inmark_direct_dispatch()when the next wakeup'sops.select_cpu()callsscx_bpf_dsq_insert():1. Deferred dispatch cancellation: when a task is directly dispatched to a remote CPU's local DSQ via
ops.select_cpu()orops.enqueue(), the dispatch is deferred (since we can't lock the remote rq while holding the current one). If the task is dequeued before processing the dispatch inprocess_ddsp_deferred_locals(),dispatch_dequeue()removes the task from the list leaving a stale direct dispatch state.Fix: clear
ddsp_dsq_idandddsp_enq_flagsin the !list_empty branch ofdispatch_dequeue().
Quoted 8 lines
2. Holding-cpu dispatch race: when
dispatch_to_local_dsq()transfers a task to another CPU's local DSQ, it setsholding_cpuand releases DISPATCHING before locking the source rq. If dequeue wins the race and clearsholding_cpu,dispatch_enqueue()is never called andddsp_dsq_idis not cleared.Fix: clear
ddsp_dsq_idandddsp_enq_flagswhen clearingholding_cpuindispatch_dequeue().
These two just mean that dequeue need to clear it, right?
Quoted 9 lines
3. Cross-scheduler-instance stale state: When an SCX scheduler exits,
scx_bypass()iterates over all runnable tasks to dequeue/re-enqueue them, but sleeping tasks are not on any runqueue and are not touched. If a sleeping task had a deferred dispatch in flight (ddsp_dsq_idset) at the time the scheduler exited, the state persists. When a new scheduler instance loads and callsscx_enable_task()for all tasks, it does not reset this leftover state. The next wakeup'sops.select_cpu()then sees a non-INVALIDddsp_dsq_idand triggers:
Quoted 5 lines
WARN_ON_ONCE(p->scx.ddsp_dsq_id != SCX_DSQ_INVALID)Fix: clear
ddsp_dsq_idandddsp_enq_flagsinscx_enable_task()before callingops.enable(), ensuring each new scheduler instance starts with a clean direct dispatch state per task.
I don't understand this one. If we fix the missing clearing from dequeue, where would the residual ddsp_dsq_id come from? How would a sleeping task have ddsp_dsq_id set? Note that select_cpu() + enqueue() call sequence is atomic w.r.t. dequeue as both are protected by pi_lock.
It's been always a bit bothersome that ddsp_dsq_id was being cleared in dispatch_enqueue(). It was there to catch the cases where ddsp_dsq_id was overridden but it just isn't the right place. Can we do the following?
- Add clear_direct_dispatch() which clears ddsp_dsq_id and ddsp_enq_flags.
- Add clear_direct_dispatch() call under the enqueue: in do_enqueue_task() and remove ddsp clearing from dispatch_enqueue(). This should catch all cases that ignore ddsp.
- Add clear_direct_dispatch() call after dispatch_enqueue() in direct_dispatch(). This clears it for the synchronous consumption.
- Add clear_direct_dispatch() call before dispatch_to_local_dsq() call in process_ddsp_deferred_locals(). Note that the funciton has to cache and clear ddsp fields *before* calling dispatch_to_local_enq() as the function will migrate the task to another rq and we can't control what happens to it afterwrds. Even for the previous synchronous case, it may just be a better pattern to always cache dsq_id and enq_flags in local vars and clear p->scx.ddsp* before calling dispatch_enqueue().
- Add clear_direct_dispatch() call to dequeue_task_scx() after dispatch_dequeue().
I think this should capture all cases and the fields are cleared where they should be cleared (either consumed or canceled).
Thanks.
Signature · 1 line
tejun
Andrea Righi
reviewerHi Tejun,
Quoted 26 lines · Tejun Heo
Hello,
(cc'ing Patrick Somaru. This is the same issue reported on
https://github.com/sched-ext/scx/pull/3482)Quoted 20 lines · Andrea Righi
@p->scx.ddsp_dsq_id can be left set (non-SCX_DSQ_INVALID) in three scenarios, causing a spurious
WARN_ON_ONCE()inmark_direct_dispatch()when the next wakeup'sops.select_cpu()callsscx_bpf_dsq_insert():1. Deferred dispatch cancellation: when a task is directly dispatched to a remote CPU's local DSQ via
ops.select_cpu()orops.enqueue(), the dispatch is deferred (since we can't lock the remote rq while holding the current one). If the task is dequeued before processing the dispatch inprocess_ddsp_deferred_locals(),dispatch_dequeue()removes the task from the list leaving a stale direct dispatch state.Fix: clear
ddsp_dsq_idandddsp_enq_flagsin the !list_empty branch ofdispatch_dequeue().2. Holding-cpu dispatch race: when
dispatch_to_local_dsq()transfers a task to another CPU's local DSQ, it setsholding_cpuand releases DISPATCHING before locking the source rq. If dequeue wins the race and clearsholding_cpu,dispatch_enqueue()is never called andddsp_dsq_idis not cleared.
Quoted 4 lines
Quoted 2 lines
Fix: clear
ddsp_dsq_idandddsp_enq_flagswhen clearingholding_cpuindispatch_dequeue().These two just mean that dequeue need to clear it, right?
Correct, we want to clear the state where dispatch_dequeue() cancels a pending direct dispatch without calling dispatch_enqueue(), so I could have just clear the state unconditionally in the !dsq case and simplify the code.
Quoted 42 lines
Quoted 10 lines
3. Cross-scheduler-instance stale state: When an SCX scheduler exits,
scx_bypass()iterates over all runnable tasks to dequeue/re-enqueue them, but sleeping tasks are not on any runqueue and are not touched. If a sleeping task had a deferred dispatch in flight (ddsp_dsq_idset) at the time the scheduler exited, the state persists. When a new scheduler instance loads and callsscx_enable_task()for all tasks, it does not reset this leftover state. The next wakeup'sops.select_cpu()then sees a non-INVALIDddsp_dsq_idand triggers:Quoted 5 lines
WARN_ON_ONCE(p->scx.ddsp_dsq_id != SCX_DSQ_INVALID)Fix: clear
ddsp_dsq_idandddsp_enq_flagsinscx_enable_task()before callingops.enable(), ensuring each new scheduler instance starts with a clean direct dispatch state per task.I don't understand this one. If we fix the missing clearing from dequeue, where would the residual
ddsp_dsq_idcome from? How would a sleeping task haveddsp_dsq_idset? Note thatselect_cpu()+enqueue()call sequence is atomic w.r.t. dequeue as both are protected bypi_lock.It's been always a bit bothersome that
ddsp_dsq_idwas being cleared indispatch_enqueue(). It was there to catch the cases whereddsp_dsq_idwas overridden but it just isn't the right place. Can we do the following?- Add
clear_direct_dispatch()which clearsddsp_dsq_idandddsp_enq_flags.- Add
clear_direct_dispatch()call under the enqueue: indo_enqueue_task()and remove ddsp clearing fromdispatch_enqueue(). This should catch all cases that ignore ddsp.- Add
clear_direct_dispatch()call afterdispatch_enqueue()indirect_dispatch(). This clears it for the synchronous consumption.- Add
clear_direct_dispatch()call beforedispatch_to_local_dsq()call inprocess_ddsp_deferred_locals(). Note that the funciton has to cache and clear ddsp fields *before* callingdispatch_to_local_enq()as the function will migrate the task to another rq and we can't control what happens to it afterwrds. Even for the previous synchronous case, it may just be a better pattern to always cachedsq_idandenq_flagsin local vars and clear p->scx.ddsp* before callingdispatch_enqueue().
Quoted 5 lines
- Add
clear_direct_dispatch()call todequeue_task_scx()afterdispatch_dequeue().I think this should capture all cases and the fields are cleared where they should be cleared (either consumed or canceled).
I like this, it looks like a better design.
However, I tried it, but I'm still able to trigger the warning, unless I clear the direct dispatch state in __scx_enable_task(), so we're still missing a case that doesn't properly clear the state.
I think it has something to do with sleeping tasks / queued wakeups / bypass, because now I can easily reproduce the warning running a stress-ng --sleep 1 while restarting a scheduler that uses SCX_OPS_ALLOW_QUEUED_WAKEUP, like scx_cosmos. Will keep investigating...
Thanks,
-Andrea
Andrea Righi
reviewerQuoted 89 lines · Andrea Righi
Hi Tejun,
Quoted 26 lines · Tejun Heo
Hello,
(cc'ing Patrick Somaru. This is the same issue reported on
https://github.com/sched-ext/scx/pull/3482)Quoted 20 lines · Andrea Righi
@p->scx.ddsp_dsq_id can be left set (non-SCX_DSQ_INVALID) in three scenarios, causing a spurious
WARN_ON_ONCE()inmark_direct_dispatch()when the next wakeup'sops.select_cpu()callsscx_bpf_dsq_insert():1. Deferred dispatch cancellation: when a task is directly dispatched to a remote CPU's local DSQ via
ops.select_cpu()orops.enqueue(), the dispatch is deferred (since we can't lock the remote rq while holding the current one). If the task is dequeued before processing the dispatch inprocess_ddsp_deferred_locals(),dispatch_dequeue()removes the task from the list leaving a stale direct dispatch state.Fix: clear
ddsp_dsq_idandddsp_enq_flagsin the !list_empty branch ofdispatch_dequeue().2. Holding-cpu dispatch race: when
dispatch_to_local_dsq()transfers a task to another CPU's local DSQ, it setsholding_cpuand releases DISPATCHING before locking the source rq. If dequeue wins the race and clearsholding_cpu,dispatch_enqueue()is never called andddsp_dsq_idis not cleared.Quoted 4 lines
Quoted 2 lines
Fix: clear
ddsp_dsq_idandddsp_enq_flagswhen clearingholding_cpuindispatch_dequeue().These two just mean that dequeue need to clear it, right?
Correct, we want to clear the state where
dispatch_dequeue()cancels a pending direct dispatch without callingdispatch_enqueue(), so I could have just clear the state unconditionally in the !dsq case and simplify the code.Quoted 42 lines
Quoted 10 lines
3. Cross-scheduler-instance stale state: When an SCX scheduler exits,
scx_bypass()iterates over all runnable tasks to dequeue/re-enqueue them, but sleeping tasks are not on any runqueue and are not touched. If a sleeping task had a deferred dispatch in flight (ddsp_dsq_idset) at the time the scheduler exited, the state persists. When a new scheduler instance loads and callsscx_enable_task()for all tasks, it does not reset this leftover state. The next wakeup'sops.select_cpu()then sees a non-INVALIDddsp_dsq_idand triggers:Quoted 5 lines
WARN_ON_ONCE(p->scx.ddsp_dsq_id != SCX_DSQ_INVALID)Fix: clear
ddsp_dsq_idandddsp_enq_flagsinscx_enable_task()before callingops.enable(), ensuring each new scheduler instance starts with a clean direct dispatch state per task.I don't understand this one. If we fix the missing clearing from dequeue, where would the residual
ddsp_dsq_idcome from? How would a sleeping task haveddsp_dsq_idset? Note thatselect_cpu()+enqueue()call sequence is atomic w.r.t. dequeue as both are protected bypi_lock.It's been always a bit bothersome that
ddsp_dsq_idwas being cleared indispatch_enqueue(). It was there to catch the cases whereddsp_dsq_idwas overridden but it just isn't the right place. Can we do the following?- Add
clear_direct_dispatch()which clearsddsp_dsq_idandddsp_enq_flags.- Add
clear_direct_dispatch()call under the enqueue: indo_enqueue_task()and remove ddsp clearing fromdispatch_enqueue(). This should catch all cases that ignore ddsp.- Add
clear_direct_dispatch()call afterdispatch_enqueue()indirect_dispatch(). This clears it for the synchronous consumption.- Add
clear_direct_dispatch()call beforedispatch_to_local_dsq()call inprocess_ddsp_deferred_locals(). Note that the funciton has to cache and clear ddsp fields *before* callingdispatch_to_local_enq()as the function will migrate the task to another rq and we can't control what happens to it afterwrds. Even for the previous synchronous case, it may just be a better pattern to always cachedsq_idandenq_flagsin local vars and clear p->scx.ddsp* before callingdispatch_enqueue().Quoted 5 lines
- Add
clear_direct_dispatch()call todequeue_task_scx()afterdispatch_dequeue().I think this should capture all cases and the fields are cleared where they should be cleared (either consumed or canceled).
I like this, it looks like a better design.
Quoted 8 lines
However, I tried it, but I'm still able to trigger the warning, unless I clear the direct dispatch state in
__scx_enable_task(), so we're still missing a case that doesn't properly clear the state.I think it has something to do with sleeping tasks / queued wakeups / bypass, because now I can easily reproduce the warning running a
stress-ng --sleep 1while restarting a scheduler that usesSCX_OPS_ALLOW_QUEUED_WAKEUP, likescx_cosmos. Will keep investigating...
I think I see it: waking tasks may have had ddsp_dsq_id set by the outgoing scheduler's ops.select_cpu() and then been queued on a wake_list via ttwu_queue_wakelist(), when SCX_OPS_ALLOW_QUEUED_WAKEUP is set. Such tasks are not on the runqueue (on_rq == 0) and are not iterated by scx_bypass(), so their direct dispatch state won't be cleared by dequeue_task_scx().
How about clearning the direct dispatch state in scx_disable_task()? That would catch those tasks as well and it won't leave the stale state from the previous scx scheduler instance.
Thanks,
-Andrea