sched_ext: Call wakeup_preempt() in local_dsq_post_enq()
Subsystems (2)
PATCH v2 1/1 sched_ext: Call wakeup_preempt() in local_dsq_post_enq()
There are several edge cases (see linked thread) where an IMMED task can be left lingering on a local DSQ if an RT task swoops in at the wrong time. All of these edge cases are due to rq->next_class being idle even after dispatching a task to rq's local DSQ. We should bump rq->next_class to &ext_sched_class as soon as we've inserted a task into the local DSQ.
To optimize the common case of rq->next_class == &ext_sched_class, only call wakeup_preempt() if rq->next_class is below EXT. If next_class is EXT or above, wakeup_preempt() is a no-op anyway.
This lets us also simplify the preempt_curr() logic a bit since wakeup_preempt() will call preempt_curr() for us if next_class is below EXT.
---
Changes from v1:
- Elaborated the comment on why wakeup_preempt() is needed by including specific examples (Tejun Heo)
- Moved the wakeup_preempt() call before the early return on SCX_RQ_IN_BALANCE (Sashiko), see
https://sashiko.dev/#/patchset/20260424092245.3212529-1-jpiecuch%40google.com
kernel/sched/ext.c | 44 +++++++++++++++++++++++++++++++++++++++----- 1 file changed, 39 insertions(+), 5 deletions(-)
kernel/sched/ext.c +39 -5
@@ -1402,14 +1402,51 @@ static void local_dsq_post_enq(struct scx_sched *sch, struct scx_dispatch_q *dsq struct task_struct *p, u64 enq_flags) { struct rq *rq = container_of(dsq, struct rq, scx.local_dsq); -bool preempt = false;call_task_dequeue (sch, rq, p,0 ); +/* +* Note that @rq's lock may be dropped between this enqueue and @p +* actually getting on CPU. This gives higher-class tasks (e.g. RT) +* an opportunity to wake up on @rq and prevent @p from running. +* Here are some concrete examples: +* +* Example 1: +* +* We dispatch two tasks from a single ops.dispatch(): +* - First, a local task to this CPU's local DSQ; +* - Second, a local/remote task to a remote CPU's local DSQ. +* We must drop the local rq lock in order to finish the second +* dispatch. In that time, an RT task can wake up on the local rq. +* +* Example 2: +* +* We dispatch a local/remote task to a remote CPU's local DSQ. +* We must drop the remote rq lock before the dispatched task can run, +* which gives an RT task an opportunity to wake up on the remote rq. +* +* Both examples work the same if we replace dispatching with moving +* the tasks from a user-created DSQ. +* +* We must detect these wakeups so that we can re-enqueue IMMED tasks +* from @rq's local DSQ. scx_wakeup_preempt() serves exactly this +* purpose, but for it to be invoked, we must ensure that we bump +* @rq->next_class to &ext_sched_class if it's currently idle. +* +* wakeup_preempt() does the bumping, and since we only invoke it if +* @rq->next_class is below &ext_sched_class, it will also +* resched_curr(rq). +*/ +if (sched_class_above (p->sched_class, rq->next_class)) +wakeup_preempt (rq, p,0 ); +/* * If @rq is in balance, the CPU is already vacant and looking for the * next task to run. No need to preempt or trigger resched after moving * @p into its local DSQ. +* Note that the wakeup_preempt() above may have already triggered +* a resched if @rq->next_class was idle. It's harmless, since +* need_resched is cleared immediately after task pick. */ if (rq->scx.flags &SCX_RQ_IN_BALANCE )return ;
@@ -1417,11 +1454,8 @@ static void local_dsq_post_enq(struct scx_sched *sch, struct scx_dispatch_q *dsqif ((enq_flags &SCX_ENQ_PREEMPT ) && p != rq->curr && rq->curr->sched_class == &ext_sched_class) { rq->curr->scx.slice =0 ; - preempt = true; - } - -if (preempt ||sched_class_above (&ext_sched_class, rq->curr->sched_class))resched_curr (rq); + } } static void dispatch_enqueue(struct scx_sched *sch, struct rq *rq,
Signature · 1 line
2.54.0.545.g6539524ca2-goog
Hello,
Applied to sched_ext/for-7.1-fixes.
Thanks.
--
tejun