sched_ext: Fix ops.dequeue() semantics
Subsystems (4)
Andrea Righi
reviewerPATCH v8 1/1 sched_ext: Fix ops.dequeue() semantics
The callback ops.dequeue() is provided to let BPF schedulers observe when a task leaves the scheduler, either because it is dispatched or due to a task property change. However, this callback is currently unreliable and not invoked systematically, which can result in missed ops.dequeue() events.
In particular, once a task is removed from the scheduler (whether for dispatch or due to a property change) the BPF scheduler loses visibility of the task and the sched_ext core may not always trigger ops.dequeue().
This breaks accurate accounting (i.e., per-DSQ queued runtime sums) and prevents reliable tracking of task lifecycle transitions.
This patch set fixes the semantics of ops.dequeue(), by guaranteeing that each task entering the BPF scheduler's custody triggers exactly one ops.dequeue() call when it leaves that custody, whether the exit is due to a dispatch (regular or via a core scheduling pick) or to a scheduling property change (e.g., sched_setaffinity(), sched_setscheduler(), set_user_nice(), NUMA balancing, etc.).
To identify property change dequeues a new ops.dequeue() flag is
introduced: %SCX_DEQ_SCHED_CHANGE.
Together, these changes allow BPF schedulers to reliably track task ownership and maintain accurate accounting.
Changes in v8:
- Rename SCX_TASK_NEED_DEQ -> SCX_TASK_IN_CUSTODY and set/clear this flag also when ops.dequeue() is not implemented (can be used for other purposes in the future)
- Clarify ops.select_cpu() behavior: dispatch to terminal DSQs doesn't trigger ops.dequeue(), dispatch to user DSQs triggers ops.dequeue(), store to BPF-internal data structure is discouraged
- Link to v7:
https://lore.kernel.org/all/[email protected]
Changes in v7:
- Handle tasks stored to BPF internal data structures (trigger ops.dequeue())
- Add a kselftest scenario with a BPF queue to verify ops.dequeue() behavior with tasks stored in internal BPF data structures
- Link to v6:
https://lore.kernel.org/all/[email protected]
Changes in v6:
- Rename SCX_TASK_OPS_ENQUEUED -> SCX_TASK_NEED_DSQ
- Use SCX_DSQ_FLAG_BUILTIN in is_terminal_dsq() to check for all builtin DSQs (local, global, bypass)
- centralize ops.dequeue() logic in dispatch_enqueue()
- Remove "Property Change Notifications for Running Tasks" section from the documentation
- The kselftest now validates the right behavior both from ops.enqueue() and ops.select_cpu()
- Link to v5: https://lore.kernel.org/all/[email protected]
Changes in v5:
- Introduce the concept of "terminal DSQ" (when a task is dispatched to a terminal DSQ, the task leaves the BPF scheduler's custody)
- Consider SCX_DSQ_GLOBAL as a terminal DSQ
- Link to v4: https://lore.kernel.org/all/[email protected]
Changes in v4:
- Introduce the concept of "BPF scheduler custody"
- Do not trigger ops.dequeue() for direct dispatches to local DSQs
- Trigger ops.dequeue() only once; after the task leaves BPF scheduler custody, further dequeue events are not reported.
- Link to v3: https://lore.kernel.org/all/[email protected]
Changes in v3:
- Rename SCX_DEQ_ASYNC to SCX_DEQ_SCHED_CHANGE
- Handle core-sched dequeues (Kuba)
- Link to v2: https://lore.kernel.org/all/[email protected]
Changes in v2:
- Distinguish between "dispatch" dequeues and "property change" dequeues (flag SCX_DEQ_ASYNC)
- Link to v1: https://lore.kernel.org/all/[email protected]
Andrea Righi (2):
sched_ext: Fix ops.dequeue() semantics selftests/sched_ext: Add test to validate ops.dequeue() semantics
Documentation/scheduler/sched-ext.rst | 78 ++++- include/linux/sched/ext.h | 1 + kernel/sched/ext.c | 155 ++++++++-- kernel/sched/ext_internal.h | 7 + tools/sched_ext/include/scx/enum_defs.autogen.h | 1 + tools/sched_ext/include/scx/enums.autogen.bpf.h | 2 + tools/sched_ext/include/scx/enums.autogen.h | 1 + tools/testing/selftests/sched_ext/Makefile | 1 + tools/testing/selftests/sched_ext/dequeue.bpf.c | 368 ++++++++++++++++++++++++ tools/testing/selftests/sched_ext/dequeue.c | 265 +++++++++++++++++ 10 files changed, 855 insertions(+), 24 deletions(-) create mode 100644 tools/testing/selftests/sched_ext/dequeue.bpf.c create mode 100644 tools/testing/selftests/sched_ext/dequeue.c
Andrea Righi
reviewerPATCH 1/2 sched_ext: Fix ops.dequeue() semantics
Currently, ops.dequeue() is only invoked when the sched_ext core knows that a task resides in BPF-managed data structures, which causes it to miss scheduling property change events. In addition, ops.dequeue() callbacks are completely skipped when tasks are dispatched to non-local DSQs from ops.select_cpu(). As a result, BPF schedulers cannot reliably track task state.
Fix this by guaranteeing that each task entering the BPF scheduler's custody triggers exactly one ops.dequeue() call when it leaves that custody, whether the exit is due to a dispatch (regular or via a core scheduling pick) or to a scheduling property change (e.g. sched_setaffinity(), sched_setscheduler(), set_user_nice(), NUMA balancing, etc.).
BPF scheduler custody concept: a task is considered to be in the BPF scheduler's custody when the scheduler is responsible for managing its lifecycle. This includes tasks dispatched to user-created DSQs or stored in the BPF scheduler's internal data structures from ops.enqueue(). Custody ends when the task is dispatched to a terminal DSQ (such as the local DSQ or %SCX_DSQ_GLOBAL), selected by core scheduling, or removed due to a property change.
Tasks directly dispatched to terminal DSQs bypass the BPF scheduler entirely and are never in its custody. Terminal DSQs include:
- Local DSQs (%SCX_DSQ_LOCAL or %SCX_DSQ_LOCAL_ON): per-CPU queues where tasks go directly to execution.
- Global DSQ (%SCX_DSQ_GLOBAL): the built-in fallback queue where the BPF scheduler is considered "done" with the task.
As a result, ops.dequeue() is not invoked for tasks directly dispatched to terminal DSQs.
To identify dequeues triggered by scheduling property changes, introduce the new ops.dequeue() flag %SCX_DEQ_SCHED_CHANGE: when this flag is set, the dequeue was caused by a scheduling property change.
New ops.dequeue() semantics:
- ops.dequeue() is invoked exactly once when the task leaves the BPF scheduler's custody, in one of the following cases:
a) regular dispatch: a task dispatched to a user DSQ or stored in internal BPF data structures is moved to a terminal DSQ (ops.dequeue() called without any special flags set),
b) core scheduling dispatch: core-sched picks task before dispatch (ops.dequeue() called with %SCX_DEQ_CORE_SCHED_EXEC flag set),
c) property change: task properties modified before dispatch, (ops.dequeue() called with %SCX_DEQ_SCHED_CHANGE flag set).
This allows BPF schedulers to:
- reliably track task ownership and lifecycle,
- maintain accurate accounting of managed tasks,
- update internal state when tasks change properties.
--- Documentation/scheduler/sched-ext.rst | 78 ++++++++- include/linux/sched/ext.h | 1 + kernel/sched/ext.c | 155 ++++++++++++++++-- kernel/sched/ext_internal.h | 7 + .../sched_ext/include/scx/enum_defs.autogen.h | 1 + .../sched_ext/include/scx/enums.autogen.bpf.h | 2 + tools/sched_ext/include/scx/enums.autogen.h | 1 + 7 files changed, 221 insertions(+), 24 deletions(-)
Documentation/scheduler/sched-ext.rst +71 -7
@@ -229,16 +229,23 @@ The following briefly shows how a waking task is scheduled and executed. scheduler can wake up any cpu using the ``scx_bpf_kick_cpu()`` helper, using ``ops.select_cpu()`` judiciously can be simpler and more efficient. - A task can be immediately inserted into a DSQ from ``ops.select_cpu()`` - by calling ``scx_bpf_dsq_insert()``. If the task is inserted into - ``SCX_DSQ_LOCAL`` from ``ops.select_cpu()``, it will be inserted into the - local DSQ of whichever CPU is returned from ``ops.select_cpu()``. - Additionally, inserting directly from ``ops.select_cpu()`` will cause the - ``ops.enqueue()`` callback to be skipped. - Note that the scheduler core will ignore an invalid CPU selection, for example, if it's outside the allowed cpumask of the task. + A task can be immediately inserted into a DSQ from ``ops.select_cpu()`` + by calling ``scx_bpf_dsq_insert()`` or ``scx_bpf_dsq_insert_vtime()``. + + If the task is inserted into ``SCX_DSQ_LOCAL`` from + ``ops.select_cpu()``, it will be added to the local DSQ of whichever CPU + is returned from ``ops.select_cpu()``. Additionally, inserting directly + from ``ops.select_cpu()`` will cause the ``ops.enqueue()`` callback to + be skipped. + + Any other attempt to store a task in BPF-internal data structures from + ``ops.select_cpu()`` does not prevent ``ops.enqueue()`` from being + invoked. This is discouraged, as it can introduce racy or inconsistent + state. + 2. Once the target CPU is selected, ``ops.enqueue()`` is invoked (unless the task was inserted directly from ``ops.select_cpu()``). ``ops.enqueue()`` can make one of the following decisions:
@@ -252,6 +259,61 @@ The following briefly shows how a waking task is scheduled and executed. * Queue the task on the BPF side. + **Task State Tracking and ops.dequeue() Semantics** + + A task is in the "BPF scheduler's custody" when the BPF scheduler is + responsible for managing its lifecycle. A task enters custody when it is + dispatched to a user DSQ or stored in the BPF scheduler's internal data + structures. Custody is entered only from ``ops.enqueue()`` for those + operations. The only exception is dispatching to a user DSQ from + ``ops.select_cpu()``: although the task is not yet technically in BPF + scheduler custody at that point, the dispatch has the same semantic + effect as dispatching from ``ops.enqueue()`` for custody-related + semantics. + + Once ``ops.enqueue()`` is called, the task may or may not enter custody + depending on what the scheduler does: + + * **Directly dispatched to terminal DSQs** (``SCX_DSQ_LOCAL``, + ``SCX_DSQ_LOCAL_ON | cpu``, or ``SCX_DSQ_GLOBAL``): the BPF scheduler + is done with the task - it either goes straight to a CPU's local run + queue or to the global DSQ as a fallback. The task never enters (or + exits) BPF custody, and ``ops.dequeue()`` will not be called. + + * **Dispatch to user-created DSQs** (custom DSQs): the task enters the + BPF scheduler's custody. When the task later leaves BPF custody + (dispatched to a terminal DSQ, picked by core-sched, or dequeued for + sleep/property changes), ``ops.dequeue()`` will be called exactly + once. + + * **Stored in BPF data structures** (e.g., internal BPF queues): the + task is in BPF custody. ``ops.dequeue()`` will be called when it + leaves (e.g., when ``ops.dispatch()`` moves it to a terminal DSQ, or + on property change / sleep). + + When a task leaves BPF scheduler custody, ``ops.dequeue()`` is invoked. + The dequeue can happen for different reasons, distinguished by flags: + + 1. **Regular dispatch**: when a task in BPF custody is dispatched to a + terminal DSQ from ``ops.dispatch()`` (leaving BPF custody for + execution), ``ops.dequeue()`` is triggered without any special flags. + + 2. **Core scheduling pick**: when ``CONFIG_SCHED_CORE`` is enabled and + core scheduling picks a task for execution while it's still in BPF + custody, ``ops.dequeue()`` is called with the + ``SCX_DEQ_CORE_SCHED_EXEC`` flag. + + 3. **Scheduling property change**: when a task property changes (via + operations like ``sched_setaffinity()``, ``sched_setscheduler()``, + priority changes, CPU migrations, etc.) while the task is still in + BPF custody, ``ops.dequeue()`` is called with the + ``SCX_DEQ_SCHED_CHANGE`` flag set in ``deq_flags``. + + **Important**: Once a task has left BPF custody (e.g. after being + dispatched to a terminal DSQ), property changes will not trigger + ``ops.dequeue()``, since the task is no longer managed by the BPF + scheduler. + 3. When a CPU is ready to schedule, it first looks at its local DSQ. If empty, it then looks at the global DSQ. If there still isn't a task to run, ``ops.dispatch()`` is invoked which can use the following two
@@ -319,6 +381,8 @@ by a sched_ext scheduler: /* Any usable CPU becomes available */ ops.dispatch(); /* Task is moved to a local DSQ */ + + ops.dequeue(); /* Exiting BPF scheduler */ } ops.running(); /* Task starts running on its assigned CPU */ while (task->scx.slice > 0 && task is runnable)
include/linux/sched/ext.h +1 -0
@@ -84,6 +84,7 @@ struct scx_dispatch_q {/* scx_entity.flags */ enum scx_ent_flags {SCX_TASK_QUEUED =1 <<0 ,/* on ext runqueue */ +SCX_TASK_IN_CUSTODY =1 <<1 ,/* in custody, needs ops.dequeue() when leaving */ SCX_TASK_RESET_RUNNABLE_AT =1 <<2 ,/* runnable_at should be reset */ SCX_TASK_DEQD_FOR_SLEEP =1 <<3 ,/* last dequeue was for SLEEP */
kernel/sched/ext.c +138 -17
305 lines not on this page · raw message
kernel/sched/ext_internal.h +7 -0
@@ -982,6 +982,13 @@ enum scx_deq_flags {* it hasn't been dispatched yet. Dequeue from the BPF side. */ SCX_DEQ_CORE_SCHED_EXEC =1LLU <<32 , + +/* +* The task is being dequeued due to a property change (e.g., +* sched_setaffinity(), sched_setscheduler(), set_user_nice(), +* etc.). +*/ + SCX_DEQ_SCHED_CHANGE = 1LLU << 33, }; enum scx_pick_idle_cpu_flags {
tools/sched_ext/include/scx/enum_defs.autogen.h +1 -0
@@ -21,6 +21,7 @@#define HAVE_SCX_CPU_PREEMPT_UNKNOWN #define HAVE_SCX_DEQ_SLEEP #define HAVE_SCX_DEQ_CORE_SCHED_EXEC +#define HAVE_SCX_DEQ_SCHED_CHANGE #define HAVE_SCX_DSQ_FLAG_BUILTIN #define HAVE_SCX_DSQ_FLAG_LOCAL_ON #define HAVE_SCX_DSQ_INVALID
tools/sched_ext/include/scx/enums.autogen.bpf.h +2 -0
@@ -127,3 +127,5 @@ const volatile u64 __SCX_ENQ_CLEAR_OPSS __weak;const volatile u64 __SCX_ENQ_DSQ_PRIQ__weak ;#define SCX_ENQ_DSQ_PRIQ __SCX_ENQ_DSQ_PRIQ +const volatile u64 __SCX_DEQ_SCHED_CHANGE__weak ; +#define SCX_DEQ_SCHED_CHANGE __SCX_DEQ_SCHED_CHANGE
tools/sched_ext/include/scx/enums.autogen.h +1 -0
@@ -46,4 +46,5 @@SCX_ENUM_SET (skel, scx_enq_flags,SCX_ENQ_LAST ); \SCX_ENUM_SET (skel, scx_enq_flags,SCX_ENQ_CLEAR_OPSS ); \SCX_ENUM_SET (skel, scx_enq_flags,SCX_ENQ_DSQ_PRIQ ); \ +SCX_ENUM_SET (skel, scx_deq_flags,SCX_DEQ_SCHED_CHANGE ); \ }while (0 )
Signature · 1 line
2.53.0
Tejun Heo
reviewerQuoted 10 lines · Andrea Righi
kernel/sched/ext.c +10
-0+/** +* is_terminal_dsq - Check if a DSQ is terminal for ops.dequeue() purposes +* @dsq_id: DSQ ID to check +* +* Returns true if @dsq_id is a terminal/builtin DSQ where the BPF +* scheduler is considered "done" with the task. +* +* Builtin DSQs include: +* - Local DSQs (%SCX_DSQ_LOCAL or %SCX_DSQ_LOCAL_ON): per-CPU queues +* where tasks go directly to execution,
kernel/sched/ext.c static void touch_core_sched(struct rq *rq, struct task_struct *p) · line 947
Quoted 10 lines
kernel/sched/ext.c +10
-0+* - Global DSQ (%SCX_DSQ_GLOBAL): built-in fallback queue, +* - Bypass DSQ: used during bypass mode. +* +* Tasks dispatched to builtin DSQs exit BPF scheduler custody and do not +* trigger ops.dequeue() when they are later consumed. +*/ +static inline bool is_terminal_dsq (u64 dsq_id) +{ +return dsq_id &SCX_DSQ_FLAG_BUILTIN && dsq_id !=SCX_DSQ_INVALID ; +}
Please use () do clarify ordering between & and &&. It's just visually confusing. I wonder whether it'd be cleaner to make it take @dsq instead of @dsq_id and then it can just do:
return dsq->id == SCX_DSQ_LOCAL || dsq->id == SCX_DSQ_GLOBAL;
because SCX_DSQ_LOCAL_ON is only used as the designator not as actual DSQ id, and the above code positively identifies what's terminal.
kernel/sched/ext.c static void local_dsq_post_enq(struct scx_dispatch_q *dsq, struct task_struct *p · line 1034
Quoted 4 lines
kernel/sched/ext.c +2
-1-static void dispatch_enqueue (struct scx_sched *sch,struct scx_dispatch_q *dsq, +static void dispatch_enqueue (struct scx_sched *sch,struct rq *rq, +struct scx_dispatch_q *dsq,struct task_struct *p,u64 enq_flags)
While minor, this patch would be easier to read if the @rq addition were done in a separate patch.
kernel/sched/ext.c static void enqueue_task_scx(struct rq *rq, struct task_struct *p, int enq_flags · line 1564
Quoted 3 lines
kernel/sched/ext.c +3
-0+static void call_task_dequeue (struct scx_sched *sch,struct rq *rq, +struct task_struct *p,u64 deq_flags, +bool is_sched_change)
Isn't @is_sched_change a bit of misnomer given that it needs to exclude SCX_DEQ_CORE_SCHED_EXEC. I wonder whether it'd be easier if @deq_flags handling is separated out. This part is ops_dequeue() specific, right? Everyone else statically knows what DEQ flags to use. That might make ops_dequeue() calculate flags unnecessarily but ops_dequeue() is not particularly hot, so I don't think that'd matter.
kernel/sched/ext.c static void enqueue_task_scx(struct rq *rq, struct task_struct *p, int enq_flags · line 1577
Quoted 13 lines
kernel/sched/ext.c +13
-0+{ +if (SCX_HAS_OP (sch, dequeue)) { +/* +* Set %SCX_DEQ_SCHED_CHANGE when the dequeue is due to a +* property change (not sleep or core-sched pick). +*/ +if (is_sched_change && + !(deq_flags & (DEQUEUE_SLEEP |SCX_DEQ_CORE_SCHED_EXEC ))) + deq_flags |=SCX_DEQ_SCHED_CHANGE ; + +SCX_CALL_OP_TASK (sch,SCX_KF_REST , dequeue, rq, p, deq_flags); + } + p->scx.flags &= ~SCX_TASK_IN_CUSTODY ;
Let's move flag clearing to the call sites. It's a bit confusing w/ the function name.
kernel/sched/ext.c static void ops_dequeue(struct rq *rq, struct task_struct *p, u64 deq_flags) · line 1598
Quoted 13 lines
static void
ops_dequeue(struct rq *rq, struct task_struct *p, u64 deq_flags){kernel/sched/ext.c +6
-0struct scx_sched *sch = scx_root;@@ -1524,6 +1590,12 @@ static void ops_dequeue(struct rq *rq, struct task_struct *p, u64 deq_flags)switch (opss &SCX_OPSS_STATE_MASK ) {case SCX_OPSS_NONE : +/* +* If the task is still in BPF scheduler's custody +* (%SCX_TASK_IN_CUSTODY is set) call ops.dequeue(). +*/ +if (p->scx.flags &SCX_TASK_IN_CUSTODY ) +call_task_dequeue (sch, rq, p, deq_flags, true);
Hmm... why is this path necessary? Shouldn't the one that cleared OPSS be responsible for clearing IN_CUSTODY too?
Quoted 11 lines
kernel/sched/ext.c +1
-0@@ -1631,6 +1706,7 @@ static void move_local_task_to_local_dsq(struct task_struct *p, u64 enq_flags,struct scx_dispatch_q *src_dsq, struct rq *dst_rq) { + struct scx_sched *sch = scx_root;struct scx_dispatch_q *dst_dsq = &dst_rq->scx.local_dsq;/* @dsq is locked and @p is on @dst_rq */ @@ -1639,6 +1715,16 @@ static void move_local_task_to_local_dsq(struct task_struct *p, u64 enq_flags,WARN_ON_ONCE (p->scx.holding_cpu >=0 );
kernel/sched/ext.c static void move_local_task_to_local_dsq(struct task_struct *p, u64 enq_flags, · line 1726
Quoted 10 lines
kernel/sched/ext.c +9
-0+/* +* Task is moving from a non-local DSQ to a local (terminal) DSQ. +* Call ops.dequeue() if the task was in BPF custody. +*/ +if (p->scx.flags &SCX_TASK_IN_CUSTODY ) { +if (SCX_HAS_OP (sch, dequeue)) +SCX_CALL_OP_TASK (sch,SCX_KF_REST , dequeue, dst_rq, p,0 ); + p->scx.flags &= ~SCX_TASK_IN_CUSTODY ; + }
I think a better place to put this would be inside local_dsq_post_enq() so that dispatch_enqueue() and move_local_task_to_local_dsq() can share the path. This would mean breaking out local and global cases in dispatch_enqueue(). ie. at the end of dispatch_enqueue():
if (is_local) {
local_dsq_post_enq(...);
} else {
if (dsq->id == SCX_DSQ_GLOBAL)
global_dsq_post_enq(...); /* or open code with comment */
raw_spin_unlock(&dsq->lock);
}Quoted 9 lines
kernel/sched/ext.c +3
-2@@ -1801,12 +1887,19 @@ static bool unlink_dsq_and_lock_src_rq(struct task_struct *p, !WARN_ON_ONCE (src_rq !=task_rq (p)); } -static bool consume_remote_task (struct rq *this_rq,struct task_struct *p, -struct scx_dispatch_q *dsq,struct rq *src_rq) +static bool consume_remote_task (struct scx_sched *sch,struct rq *this_rq, +struct task_struct *p, +struct scx_dispatch_q *dsq,struct rq *src_rq)
kernel/sched/ext.c static bool unlink_dsq_and_lock_src_rq(struct task_struct *p, · line 1902
Quoted 10 lines
kernel/sched/ext.c +6
-0{raw_spin_rq_unlock (this_rq);if (unlink_dsq_and_lock_src_rq (p, dsq, src_rq)) { +/* +* Task is moving from a non-local DSQ to a local (terminal) DSQ. +* Call ops.dequeue() if the task was in BPF custody. +*/ +if (p->scx.flags &SCX_TASK_IN_CUSTODY ) +call_task_dequeue (sch, src_rq, p,0 , false);
and this shouldn't be necessary. move_remote_task_to_local_dsq() deactivates and reactivates the task. The deactivation invokes ops_dequeue() but that should suppress dequeue invocation as that's internal transfer (this is discernable from p->on_rq being set to TASK_ON_RQ_MIGRATING) and when it gets enqueued on the target CPU, dispatch_enqueue() on the local DSQ should trigger dequeue invocation, right?
kernel/sched/ext.c static struct rq *move_task_between_dsqs(struct scx_sched *sch, · line 1969
Quoted 11 lines
kernel/sched/ext.c +7
-0@@ -1867,6 +1960,13 @@ static struct rq *move_task_between_dsqs(struct scx_sched *sch, src_dsq, dst_rq);raw_spin_unlock (&src_dsq->lock); }else { +/* +* Moving to a local DSQ, dispatch_enqueue() is not +* used, so call ops.dequeue() here if the task was +* in BPF scheduler's custody. +*/ +if (p->scx.flags &SCX_TASK_IN_CUSTODY ) +call_task_dequeue (sch, src_rq, p,0 , false);
and then this becomes unnecessary too.
Quoted 5 lines
kernel/sched/ext.c +0
-1@@ -2014,9 +2114,16 @@ static void dispatch_to_local_dsq(struct scx_sched *sch, struct rq *rq,*/ if (src_rq == dst_rq) { p->scx.holding_cpu =-1 ; - dispatch_enqueue(sch, &dst_rq->scx.local_dsq, p,
kernel/sched/ext.c static void dispatch_to_local_dsq(struct scx_sched *sch, struct rq *rq, · line 2126
Quoted 10 lines
kernel/sched/ext.c +8
-0+dispatch_enqueue (sch, dst_rq, &dst_rq->scx.local_dsq, p, enq_flags); }else { +/* +* Moving to a local DSQ, dispatch_enqueue() is not +* used, so call ops.dequeue() here if the task was +* in BPF scheduler's custody. +*/ +if (p->scx.flags &SCX_TASK_IN_CUSTODY ) +call_task_dequeue (sch, src_rq, p,0 , false);
ditto.
Thanks.
Signature · 1 line
tejun
Andrea Righi
reviewerHi Tejun,
Quoted 26 lines · Tejun Heo
Quoted 10 lines · Andrea Righi
kernel/sched/ext.c +10
-0+/** +* is_terminal_dsq - Check if a DSQ is terminal for ops.dequeue() purposes +* @dsq_id: DSQ ID to check +* +* Returns true if @dsq_id is a terminal/builtin DSQ where the BPF +* scheduler is considered "done" with the task. +* +* Builtin DSQs include: +* - Local DSQs (%SCX_DSQ_LOCAL or %SCX_DSQ_LOCAL_ON): per-CPU queues +* where tasks go directly to execution, kernel/sched/ext.c static void touch_core_sched(struct rq *rq, struct task_struct *p) · line 947
Quoted 10 lines
kernel/sched/ext.c +10
-0+* - Global DSQ (%SCX_DSQ_GLOBAL): built-in fallback queue, +* - Bypass DSQ: used during bypass mode. +* +* Tasks dispatched to builtin DSQs exit BPF scheduler custody and do not +* trigger ops.dequeue() when they are later consumed. +*/ +static inline bool is_terminal_dsq (u64 dsq_id) +{ +return dsq_id &SCX_DSQ_FLAG_BUILTIN && dsq_id !=SCX_DSQ_INVALID ; +}Please use () do clarify ordering between & and &&. It's just visually confusing. I wonder whether it'd be cleaner to make it take @dsq instead of @dsq_id and then it can just do:
Quoted 4 lines
return dsq->id == SCX_DSQ_LOCAL || dsq->id == SCX_DSQ_GLOBAL;because
SCX_DSQ_LOCAL_ONis only used as the designator not as actual DSQ id, and the above code positively identifies what's terminal.
Ok, but we also need to include SCX_DSQ_BYPASS, in that case maybe checking SCX_DSQ_FLAG_BUILTIN is more generic?
Quoted 8 lines
kernel/sched/ext.c static void local_dsq_post_enq(struct scx_dispatch_q *dsq, struct task_struct *p · line 1034
Quoted 5 lines
kernel/sched/ext.c +2
-1-static void dispatch_enqueue (struct scx_sched *sch,struct scx_dispatch_q *dsq, +static void dispatch_enqueue (struct scx_sched *sch,struct rq *rq, +struct scx_dispatch_q *dsq,struct task_struct *p,u64 enq_flags)While minor, this patch would be easier to read if the @rq addition were done in a separate patch.
Ack. I'll split that out.
Quoted 11 lines
kernel/sched/ext.c static void enqueue_task_scx(struct rq *rq, struct task_struct *p, int enq_flags · line 1564
Quoted 4 lines
kernel/sched/ext.c +3
-0+static void call_task_dequeue (struct scx_sched *sch,struct rq *rq, +struct task_struct *p,u64 deq_flags, +bool is_sched_change)Isn't @is_sched_change a bit of misnomer given that it needs to exclude
SCX_DEQ_CORE_SCHED_EXEC. I wonder whether it'd be easier if @deq_flags handling is separated out. This part isops_dequeue()specific, right? Everyone else statically knows what DEQ flags to use. That might makeops_dequeue()calculate flags unnecessarily butops_dequeue()is not particularly hot, so I don't think that'd matter.
Ack, I'll handle deq_flags in ops_dequeue() and simplify call_task_dequeue() accordingly.
Quoted 17 lines
Quoted 4 lines
kernel/sched/ext.c +3
-0+{ +if (SCX_HAS_OP (sch, dequeue)) { +/* kernel/sched/ext.c static void enqueue_task_scx(struct rq *rq, struct task_struct *p, int enq_flags · line 1577
Quoted 10 lines
kernel/sched/ext.c +10
-0+* Set %SCX_DEQ_SCHED_CHANGE when the dequeue is due to a +* property change (not sleep or core-sched pick). +*/ +if (is_sched_change && + !(deq_flags & (DEQUEUE_SLEEP |SCX_DEQ_CORE_SCHED_EXEC ))) + deq_flags |=SCX_DEQ_SCHED_CHANGE ; + +SCX_CALL_OP_TASK (sch,SCX_KF_REST , dequeue, rq, p, deq_flags); + } + p->scx.flags &= ~SCX_TASK_IN_CUSTODY ;Let's move flag clearing to the call sites. It's a bit confusing w/ the function name.
Ack.
Quoted 6 lines
Quoted 6 lines
static void
ops_dequeue(struct rq *rq, struct task_struct *p, u64 deq_flags){kernel/sched/ext.c +0
-0struct scx_sched *sch = scx_root;@@ -1524,6 +1590,12 @@ static void ops_dequeue(struct rq *rq, struct task_struct *p, u64 deq_flags)
Quoted 11 lines
kernel/sched/ext.c static void ops_dequeue(struct rq *rq, struct task_struct *p, u64 deq_flags) · line 1598
Quoted 8 lines
kernel/sched/ext.c +6
-0switch (opss &SCX_OPSS_STATE_MASK ) {case SCX_OPSS_NONE : +/* +* If the task is still in BPF scheduler's custody +* (%SCX_TASK_IN_CUSTODY is set) call ops.dequeue(). +*/ +if (p->scx.flags &SCX_TASK_IN_CUSTODY ) +call_task_dequeue (sch, rq, p, deq_flags, true);Hmm... why is this path necessary? Shouldn't the one that cleared OPSS be responsible for clearing
IN_CUSTODYtoo?
The path that clears OPSS to NONE doesn't always clear IN_CUSTODY: in dispatch_to_local_dsq(), when we're moving a task that was in DISPATCHING to a remote CPU's local DSQ, we only set ops_state to NONE, so a concurrent dequeue can proceed, but we only clear IN_CUSTODY when we later enqueue or move the task. So we can see NONE + IN_CUSTODY here and need to handle it. And we can't clear IN_CUSTODY at the same time we set NONE there, because we don't hold the task's rq lock yet and we can't trigger ops.dequeue().
Quoted 28 lines
Quoted 12 lines
kernel/sched/ext.c +1
-0@@ -1631,6 +1706,7 @@ static void move_local_task_to_local_dsq(struct task_struct *p, u64 enq_flags,struct scx_dispatch_q *src_dsq, struct rq *dst_rq) { + struct scx_sched *sch = scx_root;struct scx_dispatch_q *dst_dsq = &dst_rq->scx.local_dsq;/* @dsq is locked and @p is on @dst_rq */ @@ -1639,6 +1715,16 @@ static void move_local_task_to_local_dsq(struct task_struct *p, u64 enq_flags,WARN_ON_ONCE (p->scx.holding_cpu >=0 );kernel/sched/ext.c static void move_local_task_to_local_dsq(struct task_struct *p, u64 enq_flags, · line 1726
Quoted 10 lines
kernel/sched/ext.c +9
-0+/* +* Task is moving from a non-local DSQ to a local (terminal) DSQ. +* Call ops.dequeue() if the task was in BPF custody. +*/ +if (p->scx.flags &SCX_TASK_IN_CUSTODY ) { +if (SCX_HAS_OP (sch, dequeue)) +SCX_CALL_OP_TASK (sch,SCX_KF_REST , dequeue, dst_rq, p,0 ); + p->scx.flags &= ~SCX_TASK_IN_CUSTODY ; + }I think a better place to put this would be inside
local_dsq_post_enq()so thatdispatch_enqueue()andmove_local_task_to_local_dsq()can share the path. This would mean breaking out local and global cases indispatch_enqueue(). ie. at the end ofdispatch_enqueue():
Quoted 7 lines
if (is_local) {
local_dsq_post_enq(...);
} else {
if (dsq->id == SCX_DSQ_GLOBAL)
global_dsq_post_enq(...); /* or open code with comment */
raw_spin_unlock(&dsq->lock);
}Agreed, I'll move this into local_dsq_post_enq() and introduce a global_dsq_post_enq().
Quoted 13 lines
Quoted 13 lines
kernel/sched/ext.c +3
-2@@ -1801,12 +1887,19 @@ static bool unlink_dsq_and_lock_src_rq(struct task_struct *p, !WARN_ON_ONCE (src_rq !=task_rq (p)); } -static bool consume_remote_task (struct rq *this_rq,struct task_struct *p, -struct scx_dispatch_q *dsq,struct rq *src_rq) +static bool consume_remote_task (struct scx_sched *sch,struct rq *this_rq, +struct task_struct *p, +struct scx_dispatch_q *dsq,struct rq *src_rq) {raw_spin_rq_unlock (this_rq);
Quoted 14 lines
kernel/sched/ext.c static bool unlink_dsq_and_lock_src_rq(struct task_struct *p, · line 1902
Quoted 7 lines
kernel/sched/ext.c +6
-0if (unlink_dsq_and_lock_src_rq (p, dsq, src_rq)) { +/* +* Task is moving from a non-local DSQ to a local (terminal) DSQ. +* Call ops.dequeue() if the task was in BPF custody. +*/ +if (p->scx.flags &SCX_TASK_IN_CUSTODY ) +call_task_dequeue (sch, src_rq, p,0 , false);and this shouldn't be necessary.
move_remote_task_to_local_dsq()deactivates and reactivates the task. The deactivation invokesops_dequeue()but that should suppress dequeue invocation as that's internal transfer (this is discernable fromp->on_rqbeing set toTASK_ON_RQ_MIGRATING) and when it gets enqueued on the target CPU,dispatch_enqueue()on the local DSQ should trigger dequeue invocation, right?
Should we trigger ops.dequeue() when the task is dequeued inside move_remote_task_to_local_dsq() (in ops_dequeue() on the path triggered by deactivate_task() there) instead of suppressing it and invoking on the target in local_dsq_post_enq()?
That way the BPF sees dequeue on the source and then enqueue on the target, we avoid special-casing SCX_TASK_IN_CUSTODY in do_enqueue_task() and the "when to call dequeue" logic stays consistent in ops_dequeue and the terminal local/global post_enq paths.
Does it make sense or would you rather suppress it and only invoke on the target when the task lands on the local DSQ??
Quoted 14 lines
kernel/sched/ext.c static struct rq *move_task_between_dsqs(struct scx_sched *sch, · line 1969
Quoted 12 lines
kernel/sched/ext.c +7
-0@@ -1867,6 +1960,13 @@ static struct rq *move_task_between_dsqs(struct scx_sched *sch, src_dsq, dst_rq);raw_spin_unlock (&src_dsq->lock); }else { +/* +* Moving to a local DSQ, dispatch_enqueue() is not +* used, so call ops.dequeue() here if the task was +* in BPF scheduler's custody. +*/ +if (p->scx.flags &SCX_TASK_IN_CUSTODY ) +call_task_dequeue (sch, src_rq, p,0 , false);and then this becomes unnecessary too.
Ack + same comment about consume_remote_task().
Quoted 10 lines
Quoted 10 lines
kernel/sched/ext.c +2
-1@@ -2014,9 +2114,16 @@ static void dispatch_to_local_dsq(struct scx_sched *sch, struct rq *rq,*/ if (src_rq == dst_rq) { p->scx.holding_cpu =-1 ; -dispatch_enqueue (sch, &dst_rq->scx.local_dsq, p, +dispatch_enqueue (sch, dst_rq, &dst_rq->scx.local_dsq, p, enq_flags); }else { +/*
Quoted 8 lines
kernel/sched/ext.c static struct rq *move_task_between_dsqs(struct scx_sched *sch, · line 1969
Quoted 6 lines
kernel/sched/ext.c +6
-0+* Moving to a local DSQ, dispatch_enqueue() is not +* used, so call ops.dequeue() here if the task was +* in BPF scheduler's custody. +*/ +if (p->scx.flags &SCX_TASK_IN_CUSTODY ) +call_task_dequeue (sch, src_rq, p,0 , false);ditto.
Ack + same as above.
Thanks,
-Andrea
Tejun Heo
reviewerHello,
On Wed, Feb 11, 2026 at 05:06:20PM +0100, Andrea Righi wrote:
...
Quoted 11 lines
Quoted 8 lines
Please use () do clarify ordering between & and &&. It's just visually confusing. I wonder whether it'd be cleaner to make it take @dsq instead of @dsq_id and then it can just do:
return dsq->id == SCX_DSQ_LOCAL || dsq->id == SCX_DSQ_GLOBAL;because
SCX_DSQ_LOCAL_ONis only used as the designator not as actual DSQ id, and the above code positively identifies what's terminal.Ok, but we also need to include
SCX_DSQ_BYPASS, in that case maybe checkingSCX_DSQ_FLAG_BUILTINis more generic?
Ah, forgot about that. Hmm... we can do:
switch (dsq->id) {
case SCX_DSQ_LOCAL:
case SCX_DSQ_GLOBAL:
case SCX_DSQ_BYPASS:
return true;
default:
return false;
}I just feel iffy about not being specific. Easier to make mistakes in the future and more difficult to notice after doing so, but I think this point is kinda moot. If we break up LOCAL and GLOBAL/BYPASS handling into separate paths in dispatch_enqueue(), we won't need this function anyway.
Quoted 14 lines
Quoted 13 lines
kernel/sched/ext.c static void ops_dequeue(struct rq *rq, struct task_struct *p, u64 deq_flags) · line 1598
Quoted 10 lines
kernel/sched/ext.c +6
-0@@ -1524,6 +1590,12 @@ static void ops_dequeue(struct rq *rq, struct task_struct *p, u64 deq_flags)switch (opss &SCX_OPSS_STATE_MASK ) {case SCX_OPSS_NONE : +/* +* If the task is still in BPF scheduler's custody +* (%SCX_TASK_IN_CUSTODY is set) call ops.dequeue(). +*/ +if (p->scx.flags &SCX_TASK_IN_CUSTODY ) +call_task_dequeue (sch, rq, p, deq_flags, true);Hmm... why is this path necessary? Shouldn't the one that cleared OPSS be responsible for clearing
IN_CUSTODYtoo?
Quoted 7 lines
The path that clears OPSS to NONE doesn't always clear
IN_CUSTODY: indispatch_to_local_dsq(), when we're moving a task that was in DISPATCHING to a remote CPU's local DSQ, we only setops_stateto NONE, so a concurrent dequeue can proceed, but we only clearIN_CUSTODYwhen we later enqueue or move the task. So we can see NONE +IN_CUSTODYhere and need to handle it. And we can't clearIN_CUSTODYat the same time we set NONE there, because we don't hold the task's rq lock yet and we can't triggerops.dequeue().
I see. Can you please add a comment with the above?
...
Quoted 13 lines
Quoted 5 lines
I think a better place to put this would be inside
local_dsq_post_enq()so thatdispatch_enqueue()andmove_local_task_to_local_dsq()can share the path. This would mean breaking out local and global cases indispatch_enqueue(). ie. at the end ofdispatch_enqueue():Quoted 7 lines
if (is_local) { local_dsq_post_enq(...); } else { if (dsq->id == SCX_DSQ_GLOBAL) global_dsq_post_enq(...); /* or open code with comment */ raw_spin_unlock(&dsq->lock); }
Quoted 2 lines
Agreed, I'll move this into
local_dsq_post_enq()and introduce aglobal_dsq_post_enq().
Yeah, and as you pointed out, BYPASS.
Quoted 26 lines
Quoted 6 lines
Quoted 6 lines
kernel/sched/ext.c +3
-0+static bool consume_remote_task (struct scx_sched *sch,struct rq *this_rq, +struct task_struct *p, +struct scx_dispatch_q *dsq,struct rq *src_rq) {raw_spin_rq_unlock (this_rq);Quoted 14 lines
kernel/sched/ext.c static bool unlink_dsq_and_lock_src_rq(struct task_struct *p, · line 1902
Quoted 7 lines
kernel/sched/ext.c +6
-0if (unlink_dsq_and_lock_src_rq (p, dsq, src_rq)) { +/* +* Task is moving from a non-local DSQ to a local (terminal) DSQ. +* Call ops.dequeue() if the task was in BPF custody. +*/ +if (p->scx.flags &SCX_TASK_IN_CUSTODY ) +call_task_dequeue (sch, src_rq, p,0 , false);and this shouldn't be necessary.
move_remote_task_to_local_dsq()deactivates and reactivates the task. The deactivation invokesops_dequeue()but that should suppress dequeue invocation as that's internal transfer (this is discernable fromp->on_rqbeing set toTASK_ON_RQ_MIGRATING) and when it gets enqueued on the target CPU,dispatch_enqueue()on the local DSQ should trigger dequeue invocation, right?Should we trigger
ops.dequeue()when the task is dequeued insidemove_remote_task_to_local_dsq()(inops_dequeue()on the path triggered bydeactivate_task()there) instead of suppressing it and invoking on the target inlocal_dsq_post_enq()?
Quoted 7 lines
That way the BPF sees dequeue on the source and then enqueue on the target, we avoid special-casing
SCX_TASK_IN_CUSTODYindo_enqueue_task()and the "when to call dequeue" logic stays consistent inops_dequeueand the terminal local/globalpost_enqpaths.Does it make sense or would you rather suppress it and only invoke on the target when the task lands on the local DSQ??
The end result is about the same because whenever we migrate we're sending it to the local DSQ of the destination CPU, so whether we generate the event on deactivation of the source CPU or activation on the destination doesn't make *whole* lot of difference. However, conceptually, migrations are internal events. There isn't anything actionable for the BPF scheduler. The reason why ops.dequeue() should be emitted is not because the task is changing CPUs (which caused the deactivation) but the fact that it ends up in a local DSQ afterwards. I think it'll be cleaner both conceptually and code-wise to emit ops.dequeue() only from dispatch_enqueue() and dequeue paths.
Thanks.
Signature · 1 line
tejun
Andrea Righi
reviewerQuoted 27 lines · Tejun Heo
Hello,
On Wed, Feb 11, 2026 at 05:06:20PM +0100, Andrea Righi wrote:
...Quoted 11 lines
Quoted 8 lines
Please use () do clarify ordering between & and &&. It's just visually confusing. I wonder whether it'd be cleaner to make it take @dsq instead of @dsq_id and then it can just do:
return dsq->id == SCX_DSQ_LOCAL || dsq->id == SCX_DSQ_GLOBAL;because
SCX_DSQ_LOCAL_ONis only used as the designator not as actual DSQ id, and the above code positively identifies what's terminal.Ok, but we also need to include
SCX_DSQ_BYPASS, in that case maybe checkingSCX_DSQ_FLAG_BUILTINis more generic?Ah, forgot about that. Hmm... we can do:
switch (dsq->id) { case SCX_DSQ_LOCAL: case SCX_DSQ_GLOBAL: case SCX_DSQ_BYPASS: return true; default: return false; }
Quoted 4 lines
I just feel iffy about not being specific. Easier to make mistakes in the future and more difficult to notice after doing so, but I think this point is kinda moot. If we break up LOCAL and GLOBAL/BYPASS handling into separate paths in
dispatch_enqueue(), we won't need this function anyway.
Ack, makes sense.
Quoted 23 lines
Quoted 15 lines
Quoted 14 lines
kernel/sched/ext.c static void ops_dequeue(struct rq *rq, struct task_struct *p, u64 deq_flags) · line 1598
Quoted 11 lines
kernel/sched/ext.c +6
-0@@ -1524,6 +1590,12 @@ static void ops_dequeue(struct rq *rq, struct task_struct *p, u64 deq_flags)switch (opss &SCX_OPSS_STATE_MASK ) {case SCX_OPSS_NONE : +/* +* If the task is still in BPF scheduler's custody +* (%SCX_TASK_IN_CUSTODY is set) call ops.dequeue(). +*/ +if (p->scx.flags &SCX_TASK_IN_CUSTODY ) +call_task_dequeue (sch, rq, p, deq_flags, true);Hmm... why is this path necessary? Shouldn't the one that cleared OPSS be responsible for clearing
IN_CUSTODYtoo?Quoted 7 lines
The path that clears OPSS to NONE doesn't always clear
IN_CUSTODY: indispatch_to_local_dsq(), when we're moving a task that was in DISPATCHING to a remote CPU's local DSQ, we only setops_stateto NONE, so a concurrent dequeue can proceed, but we only clearIN_CUSTODYwhen we later enqueue or move the task. So we can see NONE +IN_CUSTODYhere and need to handle it. And we can't clearIN_CUSTODYat the same time we set NONE there, because we don't hold the task's rq lock yet and we can't triggerops.dequeue().
Quoted 1 line
I see. Can you please add a comment with the above?
Ok.
Quoted 15 lines
...
Quoted 13 lines
Quoted 12 lines
I think a better place to put this would be inside
local_dsq_post_enq()so thatdispatch_enqueue()andmove_local_task_to_local_dsq()can share the path. This would mean breaking out local and global cases indispatch_enqueue(). ie. at the end ofdispatch_enqueue():if (is_local) { local_dsq_post_enq(...); } else { if (dsq->id == SCX_DSQ_GLOBAL) global_dsq_post_enq(...); /* or open code with comment */ raw_spin_unlock(&dsq->lock); }
Quoted 4 lines
Quoted 2 lines
Agreed, I'll move this into
local_dsq_post_enq()and introduce aglobal_dsq_post_enq().Yeah, and as you pointed out, BYPASS.
Ok.
Quoted 35 lines
Quoted 27 lines
Quoted 7 lines
Quoted 7 lines
kernel/sched/ext.c +3
-0+static bool consume_remote_task (struct scx_sched *sch,struct rq *this_rq, +struct task_struct *p, +struct scx_dispatch_q *dsq,struct rq *src_rq) {raw_spin_rq_unlock (this_rq);Quoted 14 lines
kernel/sched/ext.c static bool unlink_dsq_and_lock_src_rq(struct task_struct *p, · line 1902
Quoted 7 lines
kernel/sched/ext.c +6
-0if (unlink_dsq_and_lock_src_rq (p, dsq, src_rq)) { +/* +* Task is moving from a non-local DSQ to a local (terminal) DSQ. +* Call ops.dequeue() if the task was in BPF custody. +*/ +if (p->scx.flags &SCX_TASK_IN_CUSTODY ) +call_task_dequeue (sch, src_rq, p,0 , false);and this shouldn't be necessary.
move_remote_task_to_local_dsq()deactivates and reactivates the task. The deactivation invokesops_dequeue()but that should suppress dequeue invocation as that's internal transfer (this is discernable fromp->on_rqbeing set toTASK_ON_RQ_MIGRATING) and when it gets enqueued on the target CPU,dispatch_enqueue()on the local DSQ should trigger dequeue invocation, right?Should we trigger
ops.dequeue()when the task is dequeued insidemove_remote_task_to_local_dsq()(inops_dequeue()on the path triggered bydeactivate_task()there) instead of suppressing it and invoking on the target inlocal_dsq_post_enq()?Quoted 7 lines
That way the BPF sees dequeue on the source and then enqueue on the target, we avoid special-casing
SCX_TASK_IN_CUSTODYindo_enqueue_task()and the "when to call dequeue" logic stays consistent inops_dequeueand the terminal local/globalpost_enqpaths.Does it make sense or would you rather suppress it and only invoke on the target when the task lands on the local DSQ??
Quoted 10 lines
The end result is about the same because whenever we migrate we're sending it to the local DSQ of the destination CPU, so whether we generate the event on deactivation of the source CPU or activation on the destination doesn't make *whole* lot of difference. However, conceptually, migrations are internal events. There isn't anything actionable for the BPF scheduler. The reason why
ops.dequeue()should be emitted is not because the task is changing CPUs (which caused the deactivation) but the fact that it ends up in a local DSQ afterwards. I think it'll be cleaner both conceptually and code-wise to emitops.dequeue()only fromdispatch_enqueue()and dequeue paths.
Does this include core scheduler migrations or just SCX-initiated migrations (move_remote_task_to_local_dsq())?
Because with core scheduler migrations we trigger ops.enqueue(), so we should also trigger ops.dequeue(). Or we need to send the task straight to local to prevent calling ops.enqueue().
Thanks,
-Andrea
Tejun Heo
reviewerHello,
Quoted 11 lines · Andrea Righi
Quoted 10 lines
The end result is about the same because whenever we migrate we're sending it to the local DSQ of the destination CPU, so whether we generate the event on deactivation of the source CPU or activation on the destination doesn't make *whole* lot of difference. However, conceptually, migrations are internal events. There isn't anything actionable for the BPF scheduler. The reason why
ops.dequeue()should be emitted is not because the task is changing CPUs (which caused the deactivation) but the fact that it ends up in a local DSQ afterwards. I think it'll be cleaner both conceptually and code-wise to emitops.dequeue()only fromdispatch_enqueue()and dequeue paths.
Quoted 6 lines
Does this include core scheduler migrations or just SCX-initiated migrations (
move_remote_task_to_local_dsq())?Because with core scheduler migrations we trigger
ops.enqueue(), so we should also triggerops.dequeue(). Or we need to send the task straight to local to prevent callingops.enqueue().
I'm a bit lost. Can you elaborate on core scheduler migrations triggering ops.enqueue()?
Thanks.
Signature · 1 line
tejun
Andrea Righi
reviewerQuoted 17 lines · Tejun Heo
Hello,
Quoted 14 lines · Andrea Righi
Quoted 10 lines
The end result is about the same because whenever we migrate we're sending it to the local DSQ of the destination CPU, so whether we generate the event on deactivation of the source CPU or activation on the destination doesn't make *whole* lot of difference. However, conceptually, migrations are internal events. There isn't anything actionable for the BPF scheduler. The reason why
ops.dequeue()should be emitted is not because the task is changing CPUs (which caused the deactivation) but the fact that it ends up in a local DSQ afterwards. I think it'll be cleaner both conceptually and code-wise to emitops.dequeue()only fromdispatch_enqueue()and dequeue paths.Does this include core scheduler migrations or just SCX-initiated migrations (
move_remote_task_to_local_dsq())?
Quoted 6 lines
Quoted 3 lines
Because with core scheduler migrations we trigger
ops.enqueue(), so we should also triggerops.dequeue(). Or we need to send the task straight to local to prevent callingops.enqueue().I'm a bit lost. Can you elaborate on core scheduler migrations triggering
ops.enqueue()?
Nevermind, just ignore that comment, we clearly want to trigger ops.dequeue/enqueue() in that case, it's the whole point of SCX_DEQ_SCHED_CHANGE. I should probably go to bed and get some sleep. :)
-Andrea
Andrea Righi
reviewerQuoted 17 lines · Tejun Heo
Hello,
Quoted 14 lines · Andrea Righi
Quoted 10 lines
The end result is about the same because whenever we migrate we're sending it to the local DSQ of the destination CPU, so whether we generate the event on deactivation of the source CPU or activation on the destination doesn't make *whole* lot of difference. However, conceptually, migrations are internal events. There isn't anything actionable for the BPF scheduler. The reason why
ops.dequeue()should be emitted is not because the task is changing CPUs (which caused the deactivation) but the fact that it ends up in a local DSQ afterwards. I think it'll be cleaner both conceptually and code-wise to emitops.dequeue()only fromdispatch_enqueue()and dequeue paths.Does this include core scheduler migrations or just SCX-initiated migrations (
move_remote_task_to_local_dsq())?
Quoted 6 lines
Quoted 3 lines
Because with core scheduler migrations we trigger
ops.enqueue(), so we should also triggerops.dequeue(). Or we need to send the task straight to local to prevent callingops.enqueue().I'm a bit lost. Can you elaborate on core scheduler migrations triggering
ops.enqueue()?
Alright, let me re-elaborate more on this with a (slightly) fresher brain.
We have two main classes of migrations:
1) Internal SCX-initiated migrations: e.g.,
dispatch_to_local_dsq() -> move_remote_task_to_local_dsq(), or
consume_remote_task() -> move_remote_task_to_local_dsq(), these
are completely internal to SCX and shouldn't trigger
ops.dequeue/enqueue() 2) Core scheduler migrations
- CPU affinity: sched_setaffinity, cpuset/cgroup mask change, etc. affine_move_task -> move_queued_task migrates it -> we trigger ops.dequeue(SCX_DEQ_SCHED_CHANGE) on the source and ops.enqueue() on the target.
- Core scheduling (CONFIG_SCHED_CORE): two different cases:
- Migration (task moved between runqueues via move_queued_task_locked() to satisfy core cookie)
- NUMA balancing: migrate_task_to() can move an SCX task to another CPU
- CPU hotplug: on CPU down, runnable tasks are pushed off via
__balance_push_cpu_stop() -> __migrate_task()
If we want to skip ops.dequeue() only for internal SCX migrations (and maybe also for NUMA and hotplug?), then only checking task_on_rq_migrating(p) is not enough, because that's true for every migration listed above and we'd skip all of them.
So, we need a way to mark "this migration is internal to SCX", like a new SCX_TASK_MIGRATING_INTERNAL flag?
The alternative is to always trigger ops.dequeue/enqueue() on every migration (no flag): even for internal SCX migrations the BPF scheduler could use it to track task movements, though there's nothing it can do. That way we don't need the additional flag.
Does one of these directions fit better with what you have in mind?
Thanks,
-Andrea
Quoted 58 lines · Andrea Righi
Quoted 17 lines · Tejun Heo
Hello,
Quoted 14 lines · Andrea Righi
Quoted 10 lines
The end result is about the same because whenever we migrate we're sending it to the local DSQ of the destination CPU, so whether we generate the event on deactivation of the source CPU or activation on the destination doesn't make *whole* lot of difference. However, conceptually, migrations are internal events. There isn't anything actionable for the BPF scheduler. The reason why
ops.dequeue()should be emitted is not because the task is changing CPUs (which caused the deactivation) but the fact that it ends up in a local DSQ afterwards. I think it'll be cleaner both conceptually and code-wise to emitops.dequeue()only fromdispatch_enqueue()and dequeue paths.Does this include core scheduler migrations or just SCX-initiated migrations (
move_remote_task_to_local_dsq())?Quoted 6 lines
Quoted 3 lines
Because with core scheduler migrations we trigger
ops.enqueue(), so we should also triggerops.dequeue(). Or we need to send the task straight to local to prevent callingops.enqueue().I'm a bit lost. Can you elaborate on core scheduler migrations triggering
ops.enqueue()?Alright, let me re-elaborate more on this with a (slightly) fresher brain.
We have two main classes of migrations:
1) Internal SCX-initiated migrations: e.g.,
dispatch_to_local_dsq() -> move_remote_task_to_local_dsq(), or consume_remote_task() -> move_remote_task_to_local_dsq(), these are completely internal to SCX and shouldn't trigger ops.dequeue/enqueue()2) Core scheduler migrations
- CPU affinity:sched_setaffinity, cpuset/cgroup mask change, etc.affine_move_task->move_queued_taskmigrates it -> we triggerops.dequeue(SCX_DEQ_SCHED_CHANGE)on the source andops.enqueue()on the target.- Core scheduling (
CONFIG_SCHED_CORE): two different cases:
- Migration (task moved between runqueues viamove_queued_task_locked()to satisfy core cookie)- NUMA balancing:
migrate_task_to()can move an SCX task to another CPU- CPU hotplug: on CPU down, runnable tasks are pushed off via
__balance_push_cpu_stop() -> __migrate_task()If we want to skip
ops.dequeue()only for internal SCX migrations (and maybe also for NUMA and hotplug?), then only checkingtask_on_rq_migrating(p)is not enough, because that's true for every migration listed above and we'd skip all of them.So, we need a way to mark "this migration is internal to SCX", like a new
SCX_TASK_MIGRATING_INTERNALflag?
Quoted 6 lines
The alternative is to always trigger ops.dequeue/enqueue() on every migration (no flag): even for internal SCX migrations the BPF scheduler could use it to track task movements, though there's nothing it can do. That way we don't need the additional flag.
Does one of these directions fit better with what you have in mind?
IIUC one example might sway your opinion (or not):
Note that not receiving a ops.dequeue() for tasks leaving one LOCAL_DSQ (and maybe being enqueued at another) prevents e.g. accurate PELT load tracking on the BPF side.
Regular utilization tracking works through ops.running() and ops.stopping() but load I don't think load can be implemented accurately.
Andrea Righi
reviewerQuoted 64 lines · Christian Loehle
Quoted 63 lines · Andrea Righi
Quoted 17 lines · Tejun Heo
Hello,
Quoted 14 lines · Andrea Righi
Quoted 10 lines
The end result is about the same because whenever we migrate we're sending it to the local DSQ of the destination CPU, so whether we generate the event on deactivation of the source CPU or activation on the destination doesn't make *whole* lot of difference. However, conceptually, migrations are internal events. There isn't anything actionable for the BPF scheduler. The reason why
ops.dequeue()should be emitted is not because the task is changing CPUs (which caused the deactivation) but the fact that it ends up in a local DSQ afterwards. I think it'll be cleaner both conceptually and code-wise to emitops.dequeue()only fromdispatch_enqueue()and dequeue paths.Does this include core scheduler migrations or just SCX-initiated migrations (
move_remote_task_to_local_dsq())?Quoted 6 lines
Quoted 3 lines
Because with core scheduler migrations we trigger
ops.enqueue(), so we should also triggerops.dequeue(). Or we need to send the task straight to local to prevent callingops.enqueue().I'm a bit lost. Can you elaborate on core scheduler migrations triggering
ops.enqueue()?Alright, let me re-elaborate more on this with a (slightly) fresher brain.
We have two main classes of migrations:
1) Internal SCX-initiated migrations: e.g.,
dispatch_to_local_dsq() -> move_remote_task_to_local_dsq(), or consume_remote_task() -> move_remote_task_to_local_dsq(), these are completely internal to SCX and shouldn't trigger ops.dequeue/enqueue()2) Core scheduler migrations
- CPU affinity:sched_setaffinity, cpuset/cgroup mask change, etc.affine_move_task->move_queued_taskmigrates it -> we triggerops.dequeue(SCX_DEQ_SCHED_CHANGE)on the source andops.enqueue()on the target.- Core scheduling (
CONFIG_SCHED_CORE): two different cases:
- Migration (task moved between runqueues viamove_queued_task_locked()to satisfy core cookie)- NUMA balancing:
migrate_task_to()can move an SCX task to another CPU- CPU hotplug: on CPU down, runnable tasks are pushed off via
__balance_push_cpu_stop() -> __migrate_task()If we want to skip
ops.dequeue()only for internal SCX migrations (and maybe also for NUMA and hotplug?), then only checkingtask_on_rq_migrating(p)is not enough, because that's true for every migration listed above and we'd skip all of them.So, we need a way to mark "this migration is internal to SCX", like a new
SCX_TASK_MIGRATING_INTERNALflag?The alternative is to always trigger ops.dequeue/enqueue() on every migration (no flag): even for internal SCX migrations the BPF scheduler could use it to track task movements, though there's nothing it can do. That way we don't need the additional flag.
Quoted 7 lines
Quoted 1 line
Does one of these directions fit better with what you have in mind?
IIUC one example might sway your opinion (or not):
Note that not receiving aops.dequeue()for tasks leaving oneLOCAL_DSQ(and maybe being enqueued at another) prevents e.g. accurate PELT load tracking on the BPF side.
Regular utilization tracking works throughops.running()andops.stopping()but load I don't think load can be implemented accurately.
It makes sense to me and I think it's actually valid reason to prefer the "always trigger" way.
We have DSQs and potentially BPF can have its own queues, but to implement accurate PELT (runnable contribution to a runqueue, possibly with decay), we'd also need to know exactly when a task leaves one runqueue and joins another.
Essentially we could get the full task lifecyle in BPF:
- runnable lifecycle:
- ops.dequeue(): task leaves runqueue, source CPU = scx_bpf_task_cpu(p),
- ops.enqueue(): task wants to run, curr CPU = scx_bpf_task_cpu(p),
- running lifecycle:
- ops.running(p): task starts running on scx_bpf_task_cpu(p),
- ops.stopping(p): task stops running on scx_bpf_task_cpu(p).
A potential concern could be about introducing more overhead, but I don't think it matters much, especially since schedulers that don't implement ops.dequeue() effectively pay no cost for these events.
Thanks,
-Andrea
Tejun Heo
reviewerHello,
Quoted 2 lines · Andrea Righi
Quoted 2 lines
Quoted 2 lines
So, we need a way to mark "this migration is internal to SCX", like a new
SCX_TASK_MIGRATING_INTERNALflag?
Yeah, I think this is what we should do. That's the only ops.dequeue() without matching ops.enqueue(), right?
...
Quoted 9 lines
Quoted 6 lines
IIUC one example might sway your opinion (or not):
Note that not receiving aops.dequeue()for tasks leaving oneLOCAL_DSQ(and maybe being enqueued at another) prevents e.g. accurate PELT load tracking on the BPF side.
Regular utilization tracking works throughops.running()andops.stopping()but load I don't think load can be implemented accurately.It makes sense to me and I think it's actually valid reason to prefer the "always trigger" way.
I don't think this is a valid argument. PELT is done that way because the association of the task and the CPU is meaningful for in-kernel schedulers. The queues are actually per-CPU. For SCX scheds, the relationship is not known to the kernel. Only the BPF scheduler itself knows, if it wants to attribute per-task load to a specific CPU, which CPU it should be attributed to. What's the point of following in-kernel association for PELT if the task was going to be hot migrated to another CPU on execution?
Thanks.
Signature · 1 line
tejun
Andrea Righi
reviewerQuoted 8 lines · Tejun Heo
Hello,
Quoted 2 lines · Andrea Righi
Quoted 2 lines
Quoted 2 lines
So, we need a way to mark "this migration is internal to SCX", like a new
SCX_TASK_MIGRATING_INTERNALflag?Yeah, I think this is what we should do. That's the only
ops.dequeue()without matchingops.enqueue(), right?
Correct.
Quoted 12 lines
...
Quoted 9 lines
Quoted 6 lines
IIUC one example might sway your opinion (or not):
Note that not receiving aops.dequeue()for tasks leaving oneLOCAL_DSQ(and maybe being enqueued at another) prevents e.g. accurate PELT load tracking on the BPF side.
Regular utilization tracking works throughops.running()andops.stopping()but load I don't think load can be implemented accurately.It makes sense to me and I think it's actually valid reason to prefer the "always trigger" way.
Quoted 7 lines
I don't think this is a valid argument. PELT is done that way because the association of the task and the CPU is meaningful for in-kernel schedulers. The queues are actually per-CPU. For SCX scheds, the relationship is not known to the kernel. Only the BPF scheduler itself knows, if it wants to attribute per-task load to a specific CPU, which CPU it should be attributed to. What's the point of following in-kernel association for PELT if the task was going to be hot migrated to another CPU on execution?
I see, let me elaborate more on this to make sure we're on the same page.
In ops.enqueue() the BPF scheduler doesn't necessarily pick a target CPU:
it can put the task on an arbitrary DSQ or even in some internal BPF data structures. The task is still associated with a runqueue, but only to satisfy a kernel requirement, for sched_ext that association isn't meaningful, because the task isn't really "on" that CPU (in fact in ops.dispatch() can do the "last minute" migration).
Therefore, keeping accurate per-CPU information from the kernel's perspective doesn't buy us much, given that the BPF scheduler can keep tasks in its own queues or structures.
Accurate PELT is still doable: the BPF scheduler can track where it puts each task in its own state, updates runnable load when it places the task in a DSQ / data structure and when the task leaves (dequeue). And it can use ops.running() / ops.stopping() for utilization.
And with a proper ops.dequeue() semantics, PELT can be driven by the BPF scheduler's own placement and the scx callbacks, not by the specific rq a task is on.
If all of the above makes sense for everyone, I agree that we don't need to notify all the internal migrations.
Thanks,
-Andrea
Tejun Heo
reviewerHello, Andrea.
On Thu, Feb 12, 2026 at 07:14:13PM +0100, Andrea Righi wrote:
...
Quoted 6 lines
In
ops.enqueue()the BPF scheduler doesn't necessarily pick a target CPU:
it can put the task on an arbitrary DSQ or even in some internal BPF data structures. The task is still associated with a runqueue, but only to satisfy a kernel requirement, forsched_extthat association isn't meaningful, because the task isn't really "on" that CPU (in fact inops.dispatch()can do the "last minute" migration).
Yes.
Quoted 8 lines
Therefore, keeping accurate per-CPU information from the kernel's perspective doesn't buy us much, given that the BPF scheduler can keep tasks in its own queues or structures.
Accurate PELT is still doable: the BPF scheduler can track where it puts each task in its own state, updates runnable load when it places the task in a DSQ / data structure and when the task leaves (dequeue). And it can use
ops.running()/ops.stopping()for utilization.
And the BPF sched might choose to do load aggregation at a differnt level too - e.g. maybe per-CPU load metric doesn't make sense given the machine and scheduler and only per-LLC level aggregation would be meaningful, which would be true for multiple of the current SCX schedulers given the per-LLC DSQ usage.
Quoted 6 lines
And with a proper
ops.dequeue()semantics, PELT can be driven by the BPF scheduler's own placement and the scx callbacks, not by the specific rq a task is on.If all of the above makes sense for everyone, I agree that we don't need to notify all the internal migrations.
Yeah, I think we're on the same page. BTW, I wonder whether we could use p->scx.sticky_cpu to detect internal migrations. It's only used for internal migrations, so maybe it can be used for detection.
Thanks.
Signature · 1 line
tejun
Andrea Righi
reviewerQuoted 32 lines · Tejun Heo
Hello, Andrea.
On Thu, Feb 12, 2026 at 07:14:13PM +0100, Andrea Righi wrote:
...Quoted 6 lines
In
ops.enqueue()the BPF scheduler doesn't necessarily pick a target CPU:
it can put the task on an arbitrary DSQ or even in some internal BPF data structures. The task is still associated with a runqueue, but only to satisfy a kernel requirement, forsched_extthat association isn't meaningful, because the task isn't really "on" that CPU (in fact inops.dispatch()can do the "last minute" migration).Yes.
Quoted 8 lines
Therefore, keeping accurate per-CPU information from the kernel's perspective doesn't buy us much, given that the BPF scheduler can keep tasks in its own queues or structures.
Accurate PELT is still doable: the BPF scheduler can track where it puts each task in its own state, updates runnable load when it places the task in a DSQ / data structure and when the task leaves (dequeue). And it can use
ops.running()/ops.stopping()for utilization.And the BPF sched might choose to do load aggregation at a differnt level too - e.g. maybe per-CPU load metric doesn't make sense given the machine and scheduler and only per-LLC level aggregation would be meaningful, which would be true for multiple of the current SCX schedulers given the per-LLC DSQ usage.
Quoted 4 lines
And with a proper
ops.dequeue()semantics, PELT can be driven by the BPF scheduler's own placement and the scx callbacks, not by the specific rq a task is on.
Quoted 6 lines
Quoted 2 lines
If all of the above makes sense for everyone, I agree that we don't need to notify all the internal migrations.
Yeah, I think we're on the same page. BTW, I wonder whether we could use
p->scx.sticky_cputo detect internal migrations. It's only used for internal migrations, so maybe it can be used for detection.
Perfect. And yes, I think if we set p->scx.sticky_cpu before deactivate_task() in move_remote_task_to_local_dsq(), then in ops_dequeue() we should be able to catch the internal migrations checking task_on_rq_migrating(p) && p->scx.sticky_cpu >= 0.
I'll run some tests with that.
Thanks,
-Andrea
Andrea Righi
reviewerQuoted 40 lines · Andrea Righi
Quoted 32 lines · Tejun Heo
Hello, Andrea.
On Thu, Feb 12, 2026 at 07:14:13PM +0100, Andrea Righi wrote:
...Quoted 6 lines
In
ops.enqueue()the BPF scheduler doesn't necessarily pick a target CPU:
it can put the task on an arbitrary DSQ or even in some internal BPF data structures. The task is still associated with a runqueue, but only to satisfy a kernel requirement, forsched_extthat association isn't meaningful, because the task isn't really "on" that CPU (in fact inops.dispatch()can do the "last minute" migration).Yes.
Quoted 8 lines
Therefore, keeping accurate per-CPU information from the kernel's perspective doesn't buy us much, given that the BPF scheduler can keep tasks in its own queues or structures.
Accurate PELT is still doable: the BPF scheduler can track where it puts each task in its own state, updates runnable load when it places the task in a DSQ / data structure and when the task leaves (dequeue). And it can use
ops.running()/ops.stopping()for utilization.And the BPF sched might choose to do load aggregation at a differnt level too - e.g. maybe per-CPU load metric doesn't make sense given the machine and scheduler and only per-LLC level aggregation would be meaningful, which would be true for multiple of the current SCX schedulers given the per-LLC DSQ usage.
Quoted 4 lines
And with a proper
ops.dequeue()semantics, PELT can be driven by the BPF scheduler's own placement and the scx callbacks, not by the specific rq a task is on.Quoted 6 lines
Quoted 2 lines
If all of the above makes sense for everyone, I agree that we don't need to notify all the internal migrations.
Yeah, I think we're on the same page. BTW, I wonder whether we could use
p->scx.sticky_cputo detect internal migrations. It's only used for internal migrations, so maybe it can be used for detection.
Quoted 6 lines
Perfect. And yes, I think if we set
p->scx.sticky_cpubeforedeactivate_task()inmove_remote_task_to_local_dsq(), then inops_dequeue()we should be able to catch the internal migrations checkingtask_on_rq_migrating(p)&&p->scx.sticky_cpu>= 0.I'll run some tests with that.
I ran more tests and I don't think we can simply rely on p->scx.sticky_cpu.
In particular, I don't see how to handle this scenario using only p->scx.sticky_cpu: a task starts an internal migration, a sched_change occurs, and ops.dequeue() gets skipped because p->scx.sticky_cpu >= 0.
So I'm back to the idea of introducing an SCX_TASK_MIGRATING_INTERNAL flag...
-Andrea
Tejun Heo
reviewerHello, Andrea.
Quoted 5 lines · Andrea Righi
I ran more tests and I don't think we can simply rely on
p->scx.sticky_cpu.In particular, I don't see how to handle this scenario using only
p->scx.sticky_cpu: a task starts an internal migration, asched_changeoccurs, andops.dequeue()gets skipped becausep->scx.sticky_cpu>= 0.
Oh, that shouldn't happen, so move_remote_task_to_local_dsq() does the following:
deactivate_task(src_rq, p, 0);
set_task_cpu(p, cpu_of(dst_rq));
p->scx.sticky_cpu = cpu_of(dst_rq);
raw_spin_rq_unlock(src_rq);
raw_spin_rq_lock(dst_rq);
...
activate_task(dst_rq, p, 0);It *looks* like something get can get while the locks are switched; however, the above deactivate_task() does WRITE_ONCE(p->on_rq, TASK_ON_RQ_MIGRATING) and task_rq_lock() does the following:
for (;;) {
raw_spin_lock_irqsave(&p->pi_lock, rf->flags);
rq = task_rq(p);
raw_spin_rq_lock(rq);
/*
* move_queued_task() task_rq_lock()
*
* ACQUIRE (rq->lock)
* [S] ->on_rq = MIGRATING [L] rq = task_rq()
* WMB (__set_task_cpu()) ACQUIRE (rq->lock);
* [S] ->cpu = new_cpu [L] task_rq()
* [L] ->on_rq
* RELEASE (rq->lock)
*
* If we observe the old CPU in task_rq_lock(), the acquire of
* the old rq->lock will fully serialize against the stores.
*
* If we observe the new CPU in task_rq_lock(), the address
* dependency headed by '[L] rq = task_rq()' and the acquire
* will pair with the WMB to ensure we then also see migrating.
*/
if (likely(rq == task_rq(p) && !task_on_rq_migrating(p))) {
rq_pin_lock(rq, rf);
return rq;
}
raw_spin_rq_unlock(rq);
raw_spin_unlock_irqrestore(&p->pi_lock, rf->flags);
while (unlikely(task_on_rq_migrating(p)))
cpu_relax();
}ie. TASK_ON_RQ_MIGRATING works like a separate lock that protects the task while it's switching the RQs, so any operations that use task_rq_lock() which includes any property changes can't get inbetween.
Thanks.
Signature · 1 line
tejun
Andrea Righi
reviewerQuoted 57 lines · Tejun Heo
Hello, Andrea.
Quoted 5 lines · Andrea Righi
I ran more tests and I don't think we can simply rely on
p->scx.sticky_cpu.In particular, I don't see how to handle this scenario using only
p->scx.sticky_cpu: a task starts an internal migration, asched_changeoccurs, andops.dequeue()gets skipped becausep->scx.sticky_cpu>= 0.Oh, that shouldn't happen, so
move_remote_task_to_local_dsq()does the following:deactivate_task(src_rq, p, 0); set_task_cpu(p, cpu_of(dst_rq)); p->scx.sticky_cpu = cpu_of(dst_rq); raw_spin_rq_unlock(src_rq); raw_spin_rq_lock(dst_rq); ... activate_task(dst_rq, p, 0);It *looks* like something get can get while the locks are switched; however, the above
deactivate_task()doesWRITE_ONCE(p->on_rq, TASK_ON_RQ_MIGRATING)andtask_rq_lock()does the following:for (;;) { raw_spin_lock_irqsave(&p->pi_lock, rf->flags); rq = task_rq(p); raw_spin_rq_lock(rq); /* * move_queued_task() task_rq_lock() * * ACQUIRE (rq->lock) * [S] ->on_rq = MIGRATING [L] rq = task_rq() * WMB (__set_task_cpu()) ACQUIRE (rq->lock); * [S] ->cpu = new_cpu [L] task_rq() * [L] ->on_rq * RELEASE (rq->lock) * * If we observe the old CPU in task_rq_lock(), the acquire of * the old rq->lock will fully serialize against the stores. * * If we observe the new CPU in task_rq_lock(), the address * dependency headed by '[L] rq = task_rq()' and the acquire * will pair with the WMB to ensure we then also see migrating. */ if (likely(rq == task_rq(p) && !task_on_rq_migrating(p))) { rq_pin_lock(rq, rf); return rq; } raw_spin_rq_unlock(rq); raw_spin_unlock_irqrestore(&p->pi_lock, rf->flags); while (unlikely(task_on_rq_migrating(p))) cpu_relax(); }
Quoted 3 lines
ie.
TASK_ON_RQ_MIGRATINGworks like a separate lock that protects the task while it's switching the RQs, so any operations that usetask_rq_lock()which includes any property changes can't get inbetween.
Yeah, that makes sense, so the scenario I was thinking it was happening can't happen. I guess I'm missing some ops.dequeue() events then or there's a race somewhere, because I can see tasks being enqueued without a corresponding ops.dequeue(). I'll add some debugging and keep investigating.
Thanks!
-Andrea
Tejun Heo
reviewerOne more comment.
kernel/sched/ext.c static void do_enqueue_task(struct rq *rq, struct task_struct *p, u64 enq_flags, · line 1454
Quoted 10 lines · Andrea Righi
kernel/sched/ext.c +6
-0@@ -1407,13 +1446,19 @@ static void do_enqueue_task(struct rq *rq, struct task_struct *p, u64 enq_flags,* dequeue may be waiting. The store_release matches their load_acquire. */ atomic_long_set_release (&p->scx.ops_state,SCX_OPSS_QUEUED | qseq); + +/* +* Task is now in BPF scheduler's custody. Set %SCX_TASK_IN_CUSTODY +* so ops.dequeue() is called when it leaves custody. +*/ + p->scx.flags |=SCX_TASK_IN_CUSTODY ;
As this is protected by task's rq lock, doing it here is okay but can you move this above atomic_long_set_release()? That's conceptually more straightforward as that set_release() is supposed to be the "I'm done with this task" point.
Thanks.
Signature · 1 line
tejun
Andrea Righi
reviewerQuoted 18 lines · Tejun Heo
One more comment.
kernel/sched/ext.c static void do_enqueue_task(struct rq *rq, struct task_struct *p, u64 enq_flags, · line 1454
Quoted 10 lines · Andrea Righi
kernel/sched/ext.c +6
-0@@ -1407,13 +1446,19 @@ static void do_enqueue_task(struct rq *rq, struct task_struct *p, u64 enq_flags,* dequeue may be waiting. The store_release matches their load_acquire. */ atomic_long_set_release (&p->scx.ops_state,SCX_OPSS_QUEUED | qseq); + +/* +* Task is now in BPF scheduler's custody. Set %SCX_TASK_IN_CUSTODY +* so ops.dequeue() is called when it leaves custody. +*/ + p->scx.flags |=SCX_TASK_IN_CUSTODY ;As this is protected by task's rq lock, doing it here is okay but can you move this above
atomic_long_set_release()? That's conceptually more straightforward as thatset_release()is supposed to be the "I'm done with this task" point.
Agreed, it definitely looks more correct to move this before the atomic_long_set_release().
Thanks,
-Andrea
Andrea Righi
reviewerPATCH 2/2 selftests/sched_ext: Add test to validate ops.dequeue() semantics
In reply to Andrea RighiAdd a new kselftest to validate that the new ops.dequeue() semantics work correctly for all task lifecycle scenarios, including the distinction between terminal DSQs (where BPF scheduler is done with the task), user DSQs (where BPF scheduler manages the task lifecycle) and BPF data structures, regardless of which event performs the dispatch.
The test validates the following scenarios:
- From ops.select_cpu():
- scenario 0 (local DSQ): tasks dispatched to the local DSQ bypass the BPF scheduler entirely; they never enter BPF custody, so ops.dequeue() is not called,
- scenario 1 (global DSQ): tasks dispatched to SCX_DSQ_GLOBAL also bypass the BPF scheduler, like the local DSQ; ops.dequeue() is not called,
- scenario 2 (user DSQ): tasks dispatched to user DSQs from ops.select_cpu(): tasks enter BPF scheduler's custody with full enqueue/dequeue lifecycle tracking and state machine validation, expects 1:1 enqueue/dequeue pairing,
- From ops.enqueue():
- scenario 3 (local DSQ): same behavior as scenario 0,
- scenario 4 (global DSQ): same behavior as scenario 1,
- scenario 5 (user DSQ): same behavior as scenario 2,
- scenario 6 (BPF internal queue): tasks are stored in a BPF queue from ops.enqueue() and consumed from ops.dispatch(); similarly to scenario 5, tasks enter BPF scheduler's custody with full lifecycle tracking and 1:1 enqueue/dequeue validation.
This verifies that:
- terminal DSQ dispatch (local, global) don't trigger ops.dequeue(),
- tasks dispatched to user DSQs, either from ops.select_cpu() or ops.enqueue(), enter BPF scheduler's custody and have exact 1:1 enqueue/dequeue pairing,
- tasks stored to internal BPF data structures from ops.enqueue() enter BPF scheduler's custody and have exact 1:1 enqueue/dequeue pairing,
- dispatch dequeues have no flags (normal workflow),
- property change dequeues have the %SCX_DEQ_SCHED_CHANGE flag set,
- no duplicate enqueues or invalid state transitions are happening.
--- tools/testing/selftests/sched_ext/Makefile | 1 + .../testing/selftests/sched_ext/dequeue.bpf.c | 368 ++++++++++++++++++ tools/testing/selftests/sched_ext/dequeue.c | 265 +++++++++++++ 3 files changed, 634 insertions(+) create mode 100644 tools/testing/selftests/sched_ext/dequeue.bpf.c create mode 100644 tools/testing/selftests/sched_ext/dequeue.c
tools/testing/selftests/sched_ext/Makefile +1 -0
@@ -161,6 +161,7 @@ all_test_bpfprogs := $(foreach prog,$(wildcard *.bpf.c),$(INCLUDE_DIR)/$(patsubs auto-test-targets := \ create_dsq \ + dequeue \ enq_last_no_enq_fails \ ddsp_bogus_dsq_fail \ ddsp_vtimelocal_fail \
tools/testing/selftests/sched_ext/dequeue.bpf.c new file +368 -0
368 lines not on this page · raw message
tools/testing/selftests/sched_ext/dequeue.c new file +265 -0
265 lines not on this page · raw message
Signature · 1 line
2.53.0
Re: [PATCH 2/2] selftests/sched_ext: Add test to validate ops.dequeue() semantics
Quoted 213 lines · Andrea Righi
Add a new kselftest to validate that the new
ops.dequeue()semantics work correctly for all task lifecycle scenarios, including the distinction between terminal DSQs (where BPF scheduler is done with the task), user DSQs (where BPF scheduler manages the task lifecycle) and BPF data structures, regardless of which event performs the dispatch.The test validates the following scenarios:
- From
ops.select_cpu():
- scenario 0 (local DSQ): tasks dispatched to the local DSQ bypass the BPF scheduler entirely; they never enter BPF custody, soops.dequeue()is not called,
- scenario 1 (global DSQ): tasks dispatched toSCX_DSQ_GLOBALalso bypass the BPF scheduler, like the local DSQ;ops.dequeue()is not called,
- scenario 2 (user DSQ): tasks dispatched to user DSQs fromops.select_cpu(): tasks enter BPF scheduler's custody with full enqueue/dequeue lifecycle tracking and state machine validation, expects 1:1 enqueue/dequeue pairing,- From
ops.enqueue():
- scenario 3 (local DSQ): same behavior as scenario 0,
- scenario 4 (global DSQ): same behavior as scenario 1,
- scenario 5 (user DSQ): same behavior as scenario 2,
- scenario 6 (BPF internal queue): tasks are stored in a BPF queue fromops.enqueue()and consumed fromops.dispatch(); similarly to scenario 5, tasks enter BPF scheduler's custody with full lifecycle tracking and 1:1 enqueue/dequeue validation.This verifies that:
- terminal DSQ dispatch (local, global) don't triggerops.dequeue(),
- tasks dispatched to user DSQs, either fromops.select_cpu()orops.enqueue(), enter BPF scheduler's custody and have exact 1:1 enqueue/dequeue pairing,
- tasks stored to internal BPF data structures fromops.enqueue()enter BPF scheduler's custody and have exact 1:1 enqueue/dequeue pairing,
- dispatch dequeues have no flags (normal workflow),
- property change dequeues have the %SCX_DEQ_SCHED_CHANGE flag set,
- no duplicate enqueues or invalid state transitions are happening.Cc Tejun Heo <[email protected]>Cc Emil Tsalapatis <[email protected]>Cc Kuba Piecuch <[email protected]>Signed-off-by Andrea Righi <[email protected]>--- tools/testing/selftests/sched_ext/Makefile | 1 + .../testing/selftests/sched_ext/dequeue.bpf.c | 368 ++++++++++++++++++ tools/testing/selftests/sched_ext/dequeue.c | 265 +++++++++++++ 3 files changed, 634 insertions(+) create mode 100644 tools/testing/selftests/sched_ext/dequeue.bpf.c create mode 100644 tools/testing/selftests/sched_ext/dequeue.ctools/testing/selftests/sched_ext/Makefile +1
-0@@ -161,6 +161,7 @@ all_test_bpfprogs := $(foreach prog,$(wildcard *.bpf.c),$(INCLUDE_DIR)/$(patsubs auto-test-targets := \ create_dsq \ + dequeue \ enq_last_no_enq_fails \ ddsp_bogus_dsq_fail \ ddsp_vtimelocal_fail \tools/testing/selftests/sched_ext/dequeue.bpf.c new file +143
-0@@ -0,0 +1,368 @@ +// SPDX-License-Identifier: GPL-2.0 +/* +* A scheduler that validates ops.dequeue() is called correctly: +* - Tasks dispatched to terminal DSQs (local, global) bypass the BPF +* scheduler entirely: no ops.dequeue() should be called +* - Tasks dispatched to user DSQs from ops.enqueue() enter BPF custody: +* ops.dequeue() must be called when they leave custody +* - Every ops.enqueue() dispatch to non-terminal DSQs is followed by +* exactly one ops.dequeue() (validate 1:1 pairing and state machine) +* +* Copyright (c) 2026 NVIDIA Corporation. +*/ + +#include <scx/common.bpf.h> + +#define SHARED_DSQ 0 + +/* +* BPF internal queue. +* +* Tasks are stored here and consumed from ops.dispatch(), validating that +* tasks on BPF internal structures still get ops.dequeue() when they +* leave. +*/ +struct { + __uint(type, BPF_MAP_TYPE_QUEUE); + __uint(max_entries, 32768); + __type(value, s32); +} global_queue SEC(".maps"); + +char _license[] SEC("license") = "GPL"; + +UEI_DEFINE (uei); + +/* +* Counters to track the lifecycle of tasks: +* - enqueue_cnt: Number of times ops.enqueue() was called +* - dequeue_cnt: Number of times ops.dequeue() was called (any type) +* - dispatch_dequeue_cnt: Number of regular dispatch dequeues (no flag) +* - change_dequeue_cnt: Number of property change dequeues +* - bpf_queue_full: Number of times the BPF internal queue was full +*/ +u64 enqueue_cnt, dequeue_cnt, dispatch_dequeue_cnt, change_dequeue_cnt, bpf_queue_full; + +/* +* Test scenarios: +* 0) Dispatch to local DSQ from ops.select_cpu() (terminal DSQ, bypasses BPF +* scheduler, no dequeue callbacks) +* 1) Dispatch to global DSQ from ops.select_cpu() (terminal DSQ, bypasses BPF +* scheduler, no dequeue callbacks) +* 2) Dispatch to shared user DSQ from ops.select_cpu() (enters BPF scheduler, +* dequeue callbacks expected) +* 3) Dispatch to local DSQ from ops.enqueue() (terminal DSQ, bypasses BPF +* scheduler, no dequeue callbacks) +* 4) Dispatch to global DSQ from ops.enqueue() (terminal DSQ, bypasses BPF +* scheduler, no dequeue callbacks) +* 5) Dispatch to shared user DSQ from ops.enqueue() (enters BPF scheduler, +* dequeue callbacks expected) +* 6) BPF internal queue from ops.enqueue(): store task PIDs in ops.enqueue(), +* consume in ops.dispatch() and dispatch to local DSQ (validates dequeue +* for tasks stored in internal BPF data structures) +*/ +u32 test_scenario; + +/* +* Per-task state to track lifecycle and validate workflow semantics. +* State transitions: +* NONE -> ENQUEUED (on enqueue) +* ENQUEUED -> DISPATCHED (on dispatch dequeue) +* DISPATCHED -> NONE (on property change dequeue or re-enqueue) +* ENQUEUED -> NONE (on property change dequeue before dispatch) +*/ +enum task_state { +TASK_NONE =0 , +TASK_ENQUEUED , +TASK_DISPATCHED , +}; + +struct task_ctx { +enum task_state state;/* Current state in the workflow */ +u64 enqueue_seq;/* Sequence number for debugging */ +}; + +struct { + __uint(type, BPF_MAP_TYPE_TASK_STORAGE); + __uint(map_flags, BPF_F_NO_PREALLOC); + __type(key, int); + __type(value, struct task_ctx); +} task_ctx_stor SEC(".maps"); + +static struct task_ctx *try_lookup_task_ctx (struct task_struct *p) +{ +return bpf_task_storage_get (&task_ctx_stor, p,0 ,0 ); +} + +s32 BPF_STRUCT_OPS (dequeue_select_cpu ,struct task_struct *p, +s32 prev_cpu,u64 wake_flags) +{ +struct task_ctx *tctx; +s32 pid = p->pid; +> + tctx = try_lookup_task_ctx(p); +if (!tctx) +return prev_cpu; + +switch (test_scenario) { +case 0 : +/* +* Direct dispatch to the local DSQ. +* +* Task bypasses BPF scheduler entirely: no enqueue +* tracking, no ops.dequeue() callbacks. +*/ +scx_bpf_dsq_insert (p,SCX_DSQ_LOCAL ,SCX_SLICE_DFL ,0 ); + tctx->state =TASK_DISPATCHED ; +break ; +case 1 : +/* +* Direct dispatch to the global DSQ. +* +* Task bypasses BPF scheduler entirely: no enqueue +* tracking, no ops.dequeue() callbacks. +*/ +scx_bpf_dsq_insert (p,SCX_DSQ_GLOBAL ,SCX_SLICE_DFL ,0 ); + tctx->state =TASK_DISPATCHED ; +break ; +case 2 : +/* +* Dispatch to shared a user DSQ. +* +* Task enters BPF scheduler management: track +* enqueue/dequeue lifecycle and validate state +* transitions. +*/ +if (tctx->state ==TASK_ENQUEUED ) +scx_bpf_error ("%d (%s): enqueue while in ENQUEUED state seq=%llu" , + p->pid, p->comm, tctx->enqueue_seq); + +scx_bpf_dsq_insert (p,SHARED_DSQ ,SCX_SLICE_DFL ,0 ); + +__sync_fetch_and_add (&enqueue_cnt,1 ); + + tctx->state =TASK_ENQUEUED ; + tctx->enqueue_seq++;
tools/testing/selftests/sched_ext/dequeue.bpf.c line 153
Quoted 10 lines
tools/testing/selftests/sched_ext/dequeue.bpf.c new file +10
-0+break ; + } + +return prev_cpu; +} + +void BPF_STRUCT_OPS (dequeue_enqueue ,struct task_struct *p,u64 enq_flags) +{ +struct task_ctx *tctx; +s32 pid = p->pid;
unused
Quoted 322 lines
tools/testing/selftests/sched_ext/dequeue.bpf.c new file +214
-0+ + tctx =try_lookup_task_ctx (p); +if (!tctx) +return ; + +switch (test_scenario) { +case 3 : +/* +* Direct dispatch to the local DSQ. +* +* Task bypasses BPF scheduler entirely: no enqueue +* tracking, no ops.dequeue() callbacks. +*/ +scx_bpf_dsq_insert (p,SCX_DSQ_LOCAL ,SCX_SLICE_DFL , enq_flags); +break ; +case 4 : +/* +* Direct dispatch to the global DSQ. +* +* Task bypasses BPF scheduler entirely: no enqueue +* tracking, no ops.dequeue() callbacks. +*/ +scx_bpf_dsq_insert (p,SCX_DSQ_GLOBAL ,SCX_SLICE_DFL , enq_flags); +break ; +case 5 : +/* +* Dispatch to shared user DSQ. +* +* Task enters BPF scheduler management: track +* enqueue/dequeue lifecycle and validate state +* transitions. +*/ +if (tctx->state ==TASK_ENQUEUED ) +scx_bpf_error ("%d (%s): enqueue while in ENQUEUED state seq=%llu" , + p->pid, p->comm, tctx->enqueue_seq); + +scx_bpf_dsq_insert (p,SHARED_DSQ ,SCX_SLICE_DFL , enq_flags); + +__sync_fetch_and_add (&enqueue_cnt,1 ); + + tctx->state =TASK_ENQUEUED ; + tctx->enqueue_seq++; +break ; +case 6 : +/* +* Store task in BPF internal queue. +* +* Task enters BPF scheduler management: track +* enqueue/dequeue lifecycle and validate state +* transitions. +*/ +if (tctx->state ==TASK_ENQUEUED ) +scx_bpf_error ("%d (%s): enqueue while in ENQUEUED state seq=%llu" , + p->pid, p->comm, tctx->enqueue_seq); + +if (bpf_map_push_elem (&global_queue, &pid,0 )) { +scx_bpf_dsq_insert (p,SCX_DSQ_GLOBAL ,SCX_SLICE_DFL , enq_flags); +__sync_fetch_and_add (&bpf_queue_full,1 ); + + tctx->state =TASK_DISPATCHED ; + }else { +__sync_fetch_and_add (&enqueue_cnt,1 ); + + tctx->state =TASK_ENQUEUED ; + tctx->enqueue_seq++; + } +break ; +default : +/* For all other scenarios, dispatch to the global DSQ */ +scx_bpf_dsq_insert (p,SCX_DSQ_GLOBAL ,SCX_SLICE_DFL , enq_flags); + tctx->state =TASK_DISPATCHED ; +break ; + } + +scx_bpf_kick_cpu (scx_bpf_task_cpu (p),SCX_KICK_IDLE ); +} + +void BPF_STRUCT_OPS (dequeue_dequeue ,struct task_struct *p,u64 deq_flags) +{ +struct task_ctx *tctx; + +__sync_fetch_and_add (&dequeue_cnt,1 ); + + tctx =try_lookup_task_ctx (p); +if (!tctx) +return ; + +/* +* For scenarios 0, 1, 3, and 4 (terminal DSQs: local and global), +* ops.dequeue() should never be called because tasks bypass the +* BPF scheduler entirely. If we get here, it's a kernel bug. +*/ +if (test_scenario ==0 || test_scenario ==3 ) { +scx_bpf_error ("%d (%s): dequeue called for local DSQ scenario" , + p->pid, p->comm); +return ; + } + +if (test_scenario ==1 || test_scenario ==4 ) { +scx_bpf_error ("%d (%s): dequeue called for global DSQ scenario" , + p->pid, p->comm); +return ; + } + +if (deq_flags &SCX_DEQ_SCHED_CHANGE ) { +/* +* Property change interrupting the workflow. Valid from +* both ENQUEUED and DISPATCHED states. Transitions task +* back to NONE state. +*/ +__sync_fetch_and_add (&change_dequeue_cnt,1 ); + +/* Validate state transition */ +if (tctx->state !=TASK_ENQUEUED && tctx->state !=TASK_DISPATCHED ) +scx_bpf_error ("%d (%s): invalid property change dequeue state=%d seq=%llu" , + p->pid, p->comm, tctx->state, tctx->enqueue_seq); + +/* Transition back to NONE: task outside scheduler control */ + tctx->state =TASK_NONE ; + }else { +/* +* Regular dispatch dequeue: normal workflow step. Valid +* only from ENQUEUED state (after enqueue, before dispatch +* dequeue). Transitions to DISPATCHED state. +*/ +__sync_fetch_and_add (&dispatch_dequeue_cnt,1 ); + +/* +* Dispatch dequeue should not have %SCX_DEQ_SCHED_CHANGE +* flag. +*/ +if (deq_flags &SCX_DEQ_SCHED_CHANGE ) +scx_bpf_error ("%d (%s): SCX_DEQ_SCHED_CHANGE in dispatch dequeue seq=%llu" , + p->pid, p->comm, tctx->enqueue_seq); + +/* +* Must be in ENQUEUED state. +*/ +if (tctx->state !=TASK_ENQUEUED ) +scx_bpf_error ("%d (%s): dispatch dequeue from state %d seq=%llu" , + p->pid, p->comm, tctx->state, tctx->enqueue_seq); + +/* +* Transition to DISPATCHED: normal cycle completed +* dispatch. +*/ + tctx->state =TASK_DISPATCHED ; + } +} + +void BPF_STRUCT_OPS (dequeue_dispatch ,s32 cpu,struct task_struct *prev) +{ +if (test_scenario ==6 ) { +struct task_struct *p; +s32 pid; + +if (bpf_map_pop_elem (&global_queue, &pid)) +return ; + + p =bpf_task_from_pid (pid); +if (!p) +return ; + +if (!bpf_cpumask_test_cpu (cpu, p->cpus_ptr)) + cpu =scx_bpf_task_cpu (p); + +scx_bpf_dsq_insert (p,SCX_DSQ_LOCAL_ON | cpu,SCX_SLICE_DFL ,0 ); +bpf_task_release (p); + }else { +scx_bpf_dsq_move_to_local (SHARED_DSQ ); + } +} + +s32 BPF_STRUCT_OPS (dequeue_init_task ,struct task_struct *p, +struct scx_init_task_args *args) +{ +struct task_ctx *tctx; + + tctx =bpf_task_storage_get (&task_ctx_stor, p,0 , +BPF_LOCAL_STORAGE_GET_F_CREATE ); +if (!tctx) +return -ENOMEM ; + +return 0 ; +} + +s32 BPF_STRUCT_OPS_SLEEPABLE (dequeue_init ) +{ +s32 ret; + + ret =scx_bpf_create_dsq (SHARED_DSQ ,-1 ); +if (ret) +return ret; + +return 0 ; +} + +void BPF_STRUCT_OPS (dequeue_exit ,struct scx_exit_info *ei) +{ +UEI_RECORD (uei, ei); +} + +SEC(".struct_ops.link") +struct sched_ext_ops dequeue_ops = { + .select_cpu = (void *)dequeue_select_cpu, + .enqueue = (void *)dequeue_enqueue, + .dequeue = (void *)dequeue_dequeue, + .dispatch = (void *)dequeue_dispatch, + .init_task = (void *)dequeue_init_task, + .init = (void *)dequeue_init, + .exit = (void *)dequeue_exit, + .timeout_ms =5000 , + .name ="dequeue_test" , +};tools/testing/selftests/sched_ext/dequeue.c new file +102
-0@@ -0,0 +1,265 @@ +// SPDX-License-Identifier: GPL-2.0 +/* +* Copyright (c) 2025 NVIDIA Corporation. +*/ +#define _GNU_SOURCE +#include <stdio.h> +#include <unistd.h> +#include <signal.h> +#include <time.h> +#include <bpf/bpf.h> +#include <scx/common.h> +#include <sys/wait.h> +#include <sched.h> +#include <pthread.h> +#include "scx_test.h" +#include "dequeue.bpf.skel.h" + +#define NUM_WORKERS 8 +#define AFFINITY_HAMMER_MS 50 + +/* +* Worker function that creates enqueue/dequeue events via CPU work and +* sleeping. Property-change dequeues are triggered by the affinity hammer +* thread (external sched_setaffinity on worker PIDs). +*/ +static void worker_fn (int id) +{ +int i; +volatile int sum =0 ; + +for (i =0 ; i <1000 ; i++) { +int j; + +/* Do some work to trigger scheduling events */ +for (j =0 ; j <10000 ; j++) + sum += j; + +/* Sleep to trigger dequeue */ +usleep (1000 + (id *100 )); + } + +exit (0 ); +} + +/* +* Property-change dequeues only happen when a task gets a property change +* while still in the queue. This thread changes workers' affinity from +* outside so that some changes hit tasks while they are still in the +* queue. +*/ +static void *affinity_hammer_fn (void *arg) +{ +pid_t *pids = arg; +cpu_set_t cpuset; +int i, n =NUM_WORKERS ; +struct timespec ts = { .tv_sec =0 , .tv_nsec =1000000 };/* 1ms */ + +for (i =0 ; i < (AFFINITY_HAMMER_MS *1000 /100 ); i++) { +int w = i % n; +int cpu = (i / n) %4 ; + +CPU_ZERO (&cpuset); +CPU_SET (cpu, &cpuset); +sched_setaffinity (pids[w],sizeof (cpuset), &cpuset); +nanosleep (&ts,NULL ); + } + +return NULL ; +} + +static enum scx_test_status run_scenario (struct dequeue *skel,u32 scenario, +const char *scenario_name) +{ +struct bpf_link *link; +pid_t pids[NUM_WORKERS ]; +pthread_t hammer; + +int i, status; +u64 enq_start, deq_start, + dispatch_deq_start, change_deq_start, bpf_queue_full_start; +u64 enq_delta, deq_delta, + dispatch_deq_delta, change_deq_delta, bpf_queue_full_delta; + +/* Set the test scenario */ + skel->bss->test_scenario = scenario; + +/* Record starting counts */ + enq_start = skel->bss->enqueue_cnt; + deq_start = skel->bss->dequeue_cnt; + dispatch_deq_start = skel->bss->dispatch_dequeue_cnt; + change_deq_start = skel->bss->change_dequeue_cnt; + bpf_queue_full_start = skel->bss->bpf_queue_full; + + link =bpf_map__attach_struct_ops (skel->maps.dequeue_ops); +SCX_FAIL_IF (!link,"Failed to attach struct_ops for scenario %s" , scenario_name); + +/* Fork worker processes to generate enqueue/dequeue events */ +for (i =0 ; i <NUM_WORKERS ; i++) { + pids[i] =fork (); +SCX_FAIL_IF (pids[i] <0 ,"Failed to fork worker %d" , i); + +if (pids[i] ==0 ) {
tools/testing/selftests/sched_ext/dequeue.c line 112
Quoted 10 lines
tools/testing/selftests/sched_ext/dequeue.c new file +10
-0+worker_fn (i); +/* Should not reach here */ +exit (1 ); + } + } + +/* +* Run an "affinity hammer" so that some property changes hit tasks +* while they are still in BPF custody (e.g. in user DSQ or BPF queue), +* triggering SCX_DEQ_SCHED_CHANGE dequeues in scenarios 2, 3, 6 and 7.
Not true for 3, right?
Quoted 153 lines
tools/testing/selftests/sched_ext/dequeue.c new file +153
-0+*/ +SCX_FAIL_IF (pthread_create (&hammer,NULL , affinity_hammer_fn, pids) !=0 , +"Failed to create affinity hammer thread" ); +pthread_join (hammer,NULL ); + +/* Wait for all workers to complete */ +for (i =0 ; i <NUM_WORKERS ; i++) { +SCX_FAIL_IF (waitpid (pids[i], &status,0 ) != pids[i], +"Failed to wait for worker %d" , i); +SCX_FAIL_IF (status !=0 ,"Worker %d exited with status %d" , i, status); + } + +bpf_link__destroy (link); + +SCX_EQ (skel->data->uei.kind,EXIT_KIND (SCX_EXIT_UNREG )); + +/* Calculate deltas */ + enq_delta = skel->bss->enqueue_cnt - enq_start; + deq_delta = skel->bss->dequeue_cnt - deq_start; + dispatch_deq_delta = skel->bss->dispatch_dequeue_cnt - dispatch_deq_start; + change_deq_delta = skel->bss->change_dequeue_cnt - change_deq_start; + bpf_queue_full_delta = skel->bss->bpf_queue_full - bpf_queue_full_start; + +printf ("%s:\n" , scenario_name); +printf (" enqueues: %lu\n" , (unsigned long )enq_delta); +printf (" dequeues: %lu (dispatch: %lu, property_change: %lu)\n" , + (unsigned long )deq_delta, + (unsigned long )dispatch_deq_delta, + (unsigned long )change_deq_delta); +printf (" BPF queue full: %lu\n" , (unsigned long )bpf_queue_full_delta); + +/* +* Validate enqueue/dequeue lifecycle tracking. +* +* For scenarios 0, 1, 3, 4 (local and global DSQs from +* ops.select_cpu() and ops.enqueue()), both enqueues and dequeues +* should be 0 because tasks bypass the BPF scheduler entirely: +* tasks never enter BPF scheduler's custody. +* +* For scenarios 2, 5, 6 (user DSQ or BPF internal queue) we expect +* both enqueues and dequeues. +* +* The BPF code does strict state machine validation with +* scx_bpf_error() to ensure the workflow semantics are correct. +* +* If we reach this point without errors, the semantics are +* validated correctly. +*/ +if (scenario ==0 || scenario ==1 || + scenario ==3 || scenario ==4 ) { +/* Tasks bypass BPF scheduler completely */ +SCX_EQ (enq_delta,0 ); +SCX_EQ (deq_delta,0 ); +SCX_EQ (dispatch_deq_delta,0 ); +SCX_EQ (change_deq_delta,0 ); + }else { +/* +* User DSQ from ops.enqueue() or ops.select_cpu(): tasks +* enter BPF scheduler's custody. +* +* Also validate 1:1 enqueue/dequeue pairing. +*/ +SCX_GT (enq_delta,0 ); +SCX_GT (deq_delta,0 ); +SCX_EQ (enq_delta, deq_delta); + } + +return SCX_TEST_PASS ; +} + +static enum scx_test_status setup (void **ctx) +{ +struct dequeue *skel; + + skel =dequeue__open (); +SCX_FAIL_IF (!skel,"Failed to open skel" ); +SCX_ENUM_INIT (skel); +SCX_FAIL_IF (dequeue__load (skel),"Failed to load skel" ); + + *ctx = skel; + +return SCX_TEST_PASS ; +} + +static enum scx_test_status run (void *ctx) +{ +struct dequeue *skel = ctx; +enum scx_test_status status; + + status =run_scenario (skel,0 ,"Scenario 0: Local DSQ from ops.select_cpu()" ); +if (status !=SCX_TEST_PASS ) +return status; + + status =run_scenario (skel,1 ,"Scenario 1: Global DSQ from ops.select_cpu()" ); +if (status !=SCX_TEST_PASS ) +return status; + + status =run_scenario (skel,2 ,"Scenario 2: User DSQ from ops.select_cpu()" ); +if (status !=SCX_TEST_PASS ) +return status; + + status =run_scenario (skel,3 ,"Scenario 3: Local DSQ from ops.enqueue()" ); +if (status !=SCX_TEST_PASS ) +return status; + + status =run_scenario (skel,4 ,"Scenario 4: Global DSQ from ops.enqueue()" ); +if (status !=SCX_TEST_PASS ) +return status; + + status =run_scenario (skel,5 ,"Scenario 5: User DSQ from ops.enqueue()" ); +if (status !=SCX_TEST_PASS ) +return status; + + status =run_scenario (skel,6 ,"Scenario 6: BPF queue from ops.enqueue()" ); +if (status !=SCX_TEST_PASS ) +return status; + +printf ("\n=== Summary ===\n" ); +printf ("Total enqueues: %lu\n" , (unsigned long )skel->bss->enqueue_cnt); +printf ("Total dequeues: %lu\n" , (unsigned long )skel->bss->dequeue_cnt); +printf (" Dispatch dequeues: %lu (no flag, normal workflow)\n" , + (unsigned long )skel->bss->dispatch_dequeue_cnt); +printf (" Property change dequeues: %lu (SCX_DEQ_SCHED_CHANGE flag)\n" , + (unsigned long )skel->bss->change_dequeue_cnt); +printf (" BPF queue full: %lu\n" , + (unsigned long )skel->bss->bpf_queue_full); +printf ("\nAll scenarios passed - no state machine violations detected\n" ); +printf ("-> Validated: Local DSQ dispatch bypasses BPF scheduler\n" ); +printf ("-> Validated: Global DSQ dispatch bypasses BPF scheduler\n" ); +printf ("-> Validated: User DSQ dispatch triggers ops.dequeue() callbacks\n" ); +printf ("-> Validated: Dispatch dequeues have no flags (normal workflow)\n" ); +printf ("-> Validated: Property change dequeues have SCX_DEQ_SCHED_CHANGE flag\n" ); +printf ("-> Validated: No duplicate enqueues or invalid state transitions\n" ); + +return SCX_TEST_PASS ; +} + +static void cleanup (void *ctx) +{ +struct dequeue *skel = ctx; + +dequeue__destroy (skel); +} + +struct scx_test dequeue_test = { + .name ="dequeue" , + .description ="Verify ops.dequeue() semantics" , + .setup = setup, + .run = run, + .cleanup = cleanup, +}; + +REGISTER_SCX_TEST (&dequeue_test)
Andrea Righi
reviewerRe: [PATCH 2/2] selftests/sched_ext: Add test to validate ops.dequeue() semantics
Quoted 1 line · Christian Loehle
On 2/10/26 21:26, Andrea Righi wrote:
...
Quoted 6 lines
tools/testing/selftests/sched_ext/dequeue.bpf.c line 154
Quoted 4 lines
tools/testing/selftests/sched_ext/dequeue.bpf.c new file +4
-0+void BPF_STRUCT_OPS (dequeue_enqueue ,struct task_struct *p,u64 enq_flags) +{ +struct task_ctx *tctx; +s32 pid = p->pid;unused
This one is used, but the one in dequeue_select_cpu() is not. I'll remove that. :)
Quoted 36 lines
Quoted 36 lines
tools/testing/selftests/sched_ext/dequeue.c new file +36
-0+static enum scx_test_status run_scenario (struct dequeue *skel,u32 scenario, +const char *scenario_name) +{ +struct bpf_link *link; +pid_t pids[NUM_WORKERS ]; +pthread_t hammer; + +int i, status; +u64 enq_start, deq_start, + dispatch_deq_start, change_deq_start, bpf_queue_full_start; +u64 enq_delta, deq_delta, + dispatch_deq_delta, change_deq_delta, bpf_queue_full_delta; + +/* Set the test scenario */ + skel->bss->test_scenario = scenario; + +/* Record starting counts */ + enq_start = skel->bss->enqueue_cnt; + deq_start = skel->bss->dequeue_cnt; + dispatch_deq_start = skel->bss->dispatch_dequeue_cnt; + change_deq_start = skel->bss->change_dequeue_cnt; + bpf_queue_full_start = skel->bss->bpf_queue_full; + + link =bpf_map__attach_struct_ops (skel->maps.dequeue_ops); +SCX_FAIL_IF (!link,"Failed to attach struct_ops for scenario %s" , scenario_name); + +/* Fork worker processes to generate enqueue/dequeue events */ +for (i =0 ; i <NUM_WORKERS ; i++) { + pids[i] =fork (); +SCX_FAIL_IF (pids[i] <0 ,"Failed to fork worker %d" , i); + +if (pids[i] ==0 ) { +worker_fn (i); +/* Should not reach here */ +exit (1 ); + }
Quoted 8 lines
tools/testing/selftests/sched_ext/dequeue.c line 112
Quoted 6 lines
tools/testing/selftests/sched_ext/dequeue.c new file +6
-0+ } + +/* +* Run an "affinity hammer" so that some property changes hit tasks +* while they are still in BPF custody (e.g. in user DSQ or BPF queue), +* triggering SCX_DEQ_SCHED_CHANGE dequeues in scenarios 2, 3, 6 and 7. Not true for 3, right?
Oh yes, this selftest has been changed so many times that I was sure I forgot to update some comments (also, scenario 7 doesn't exist anymore).
Thanks!
-Andrea