sched_ext: Fix ops.dequeue() semantics
Subsystems (4)
Andrea Righi
reviewerPATCH v10 1/1 sched_ext: Fix ops.dequeue() semantics
The callback ops.dequeue() is provided to let BPF schedulers observe when a task leaves the scheduler, either because it is dispatched or due to a task property change. However, this callback is currently unreliable and not invoked systematically, which can result in missed ops.dequeue() events.
In particular, once a task is removed from the scheduler (whether for dispatch or due to a property change) the BPF scheduler loses visibility of the task and the sched_ext core may not always trigger ops.dequeue().
This breaks accurate accounting (i.e., per-DSQ queued runtime sums) and prevents reliable tracking of task lifecycle transitions.
This patch set fixes the semantics of ops.dequeue(), by guaranteeing that each task entering the BPF scheduler's custody triggers exactly one ops.dequeue() call when it leaves that custody, whether the exit is due to a dispatch (regular or via a core scheduling pick) or to a scheduling property change (e.g., sched_setaffinity(), sched_setscheduler(), set_user_nice(), NUMA balancing, etc.).
To identify property change dequeues a new ops.dequeue() flag is
introduced: %SCX_DEQ_SCHED_CHANGE.
Together, these changes allow BPF schedulers to reliably track task ownership and maintain accurate accounting.
This patchset is also available in the following git branch:
git://git.kernel.org/pub/scm/linux/kernel/git/arighi/linux.git scx-dequeue
Changes in v10:
- Rely only on p->scx.sticky_cpu to detect in-progress SCX internal migrations
- Set SCX_TASK_IN_CUSTODY only on the non‑local path and only for non‑terminal DSQs
- Centralize "call ops.dequeue() when leaving custody" in one place, call_task_dequeue(), after the task state switch
- Fixed a p->scx.flags race between dispatch_enqueue() and dequeue_task_scx()
- Add proper synchronization in the dequeue kselftest to validate the usage of SCX_DSQ_LOCAL_ON from ops.dispatch()
- Link to v9:
https://lore.kernel.org/all/[email protected]
Changes in v9:
- Ignore internal SCX migrations (do not notify BPF schedulers for internal enqueue/dequeue events)
- Rely on sticky_cpu to determine when a task is doing an internal migration
- Trigger ops.dequeue() consistently from ops_dequeue() or when directly dispatching to terminal DSQs
- Add preliminary patches to refactor dispatch_enqueue() and mark internal migrations using sticky_cpu
- Link to v8:
https://lore.kernel.org/all/[email protected]
Changes in v8:
- Rename SCX_TASK_NEED_DEQ -> SCX_TASK_IN_CUSTODY and set/clear this flag also when ops.dequeue() is not implemented (can be used for other purposes in the future)
- Clarify ops.select_cpu() behavior: dispatch to terminal DSQs doesn't trigger ops.dequeue(), dispatch to user DSQs triggers ops.dequeue(), store to BPF-internal data structure is discouraged
- Link to v7:
https://lore.kernel.org/all/[email protected]
Changes in v7:
- Handle tasks stored to BPF internal data structures (trigger ops.dequeue())
- Add a kselftest scenario with a BPF queue to verify ops.dequeue() behavior with tasks stored in internal BPF data structures
- Link to v6:
https://lore.kernel.org/all/[email protected]
Changes in v6:
- Rename SCX_TASK_OPS_ENQUEUED -> SCX_TASK_NEED_DSQ
- Use SCX_DSQ_FLAG_BUILTIN in is_terminal_dsq() to check for all builtin DSQs (local, global, bypass)
- centralize ops.dequeue() logic in dispatch_enqueue()
- Remove "Property Change Notifications for Running Tasks" section from the documentation
- The kselftest now validates the right behavior both from ops.enqueue() and ops.select_cpu()
- Link to v5: https://lore.kernel.org/all/[email protected]
Changes in v5:
- Introduce the concept of "terminal DSQ" (when a task is dispatched to a terminal DSQ, the task leaves the BPF scheduler's custody)
- Consider SCX_DSQ_GLOBAL as a terminal DSQ
- Link to v4: https://lore.kernel.org/all/[email protected]
Changes in v4:
- Introduce the concept of "BPF scheduler custody"
- Do not trigger ops.dequeue() for direct dispatches to local DSQs
- Trigger ops.dequeue() only once; after the task leaves BPF scheduler custody, further dequeue events are not reported.
- Link to v3: https://lore.kernel.org/all/[email protected]
Changes in v3:
- Rename SCX_DEQ_ASYNC to SCX_DEQ_SCHED_CHANGE
- Handle core-sched dequeues (Kuba)
- Link to v2: https://lore.kernel.org/all/[email protected]
Changes in v2:
- Distinguish between "dispatch" dequeues and "property change" dequeues (flag SCX_DEQ_ASYNC)
- Link to v1: https://lore.kernel.org/all/[email protected]
Andrea Righi (4):
sched_ext: Properly mark SCX-internal migrations via sticky_cpu
sched_ext: Add rq parameter to dispatch_enqueue()
sched_ext: Fix ops.dequeue() semantics selftests/sched_ext: Add test to validate ops.dequeue() semantics
Documentation/scheduler/sched-ext.rst | 78 ++++- include/linux/sched/ext.h | 1 + kernel/sched/ext.c | 164 ++++++++-- kernel/sched/ext_internal.h | 7 + tools/sched_ext/include/scx/enum_defs.autogen.h | 1 + tools/sched_ext/include/scx/enums.autogen.bpf.h | 2 + tools/sched_ext/include/scx/enums.autogen.h | 1 + tools/testing/selftests/sched_ext/Makefile | 1 + tools/testing/selftests/sched_ext/dequeue.bpf.c | 394 ++++++++++++++++++++++++ tools/testing/selftests/sched_ext/dequeue.c | 274 ++++++++++++++++ 10 files changed, 890 insertions(+), 33 deletions(-) create mode 100644 tools/testing/selftests/sched_ext/dequeue.bpf.c create mode 100644 tools/testing/selftests/sched_ext/dequeue.c
Andrea Righi
reviewerPATCH 1/4 sched_ext: Properly mark SCX-internal migrations via sticky_cpu
Reposition the setting and clearing of sticky_cpu to better define the scope of SCX-internal migrations.
This ensures @sticky_cpu is set for the entire duration of an internal migration (from dequeue through enqueue), making it a reliable indicator that an SCX-internal migration is in progress. The dequeue and enqueue paths can then use @sticky_cpu to identify internal migrations and skip BPF scheduler notifications accordingly.
This prepares for a later commit fixing the ops.dequeue() semantics. No functional change intended.
--- kernel/sched/ext.c | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-)
kernel/sched/ext.c +8 -5
@@ -1476,9 +1476,6 @@ static void enqueue_task_scx(struct rq *rq, struct task_struct *p, int enq_flags enq_flags |= rq->scx.extra_enq_flags; -if (sticky_cpu >=0 ) - p->scx.sticky_cpu =-1 ; -/* * Restoring a running task will be immediately followed by * set_next_task_scx() which expects the task to not be on the BPF
@@ -1509,6 +1506,9 @@ static void enqueue_task_scx(struct rq *rq, struct task_struct *p, int enq_flagsdl_server_start (&rq->ext_server);do_enqueue_task (rq, p, enq_flags, sticky_cpu); + +if (sticky_cpu >=0 ) + p->scx.sticky_cpu =-1 ; out: rq->scx.flags &= ~SCX_RQ_IN_WAKEUP ;
@@ -1670,10 +1670,13 @@ static void move_remote_task_to_local_dsq(struct task_struct *p, u64 enq_flags, {lockdep_assert_rq_held (src_rq); -/* the following marks @p MIGRATING which excludes dequeue */ +/* +* Set sticky_cpu before deactivate_task() to properly mark the +* beginning of an SCX-internal migration. +*/ + p->scx.sticky_cpu =cpu_of (dst_rq);deactivate_task (src_rq, p,0 );set_task_cpu (p,cpu_of (dst_rq)); - p->scx.sticky_cpu =cpu_of (dst_rq);raw_spin_rq_unlock (src_rq);raw_spin_rq_lock (dst_rq);
Signature · 1 line
2.53.0
Andrea Righi
reviewerPATCH 2/4 sched_ext: Add rq parameter to dispatch_enqueue()
In reply to Andrea RighiThis prepares for a later commit fixing the ops.dequeue() semantics. No functional change intended.
--- kernel/sched/ext.c | 25 +++++++++++++------------ 1 file changed, 13 insertions(+), 12 deletions(-)
kernel/sched/ext.c +13 -12
@@ -1010,8 +1010,9 @@ static void local_dsq_post_enq(struct scx_dispatch_q *dsq, struct task_struct *presched_curr (rq); } -static void dispatch_enqueue (struct scx_sched *sch,struct scx_dispatch_q *dsq, -struct task_struct *p,u64 enq_flags) +static void dispatch_enqueue (struct scx_sched *sch,struct rq *rq, +struct scx_dispatch_q *dsq,struct task_struct *p, +u64 enq_flags) {bool is_local = dsq->id ==SCX_DSQ_LOCAL ;
@@ -1325,7 +1326,7 @@ static void direct_dispatch(struct scx_sched *sch, struct task_struct *p,return ; } -dispatch_enqueue (sch, dsq, p, +dispatch_enqueue (sch, rq, dsq, p, p->scx.ddsp_enq_flags |SCX_ENQ_CLEAR_OPSS ); }
@@ -1415,7 +1416,7 @@ static void do_enqueue_task(struct rq *rq, struct task_struct *p, u64 enq_flags,direct_dispatch (sch, p, enq_flags);return ; local_norefill: -dispatch_enqueue (sch, &rq->scx.local_dsq, p, enq_flags); +dispatch_enqueue (sch, rq, &rq->scx.local_dsq, p, enq_flags);return ; local: dsq = &rq->scx.local_dsq;
@@ -1435,7 +1436,7 @@ static void do_enqueue_task(struct rq *rq, struct task_struct *p, u64 enq_flags,*/ touch_core_sched (rq, p);refill_task_slice_dfl (sch, p); -dispatch_enqueue (sch, dsq, p, enq_flags); +dispatch_enqueue (sch, rq, dsq, p, enq_flags); }static bool task_runnable (const struct task_struct *p)
@@ -1888,7 +1889,7 @@ static struct rq *move_task_between_dsqs(struct scx_sched *sch,dispatch_dequeue_locked (p, src_dsq);raw_spin_unlock (&src_dsq->lock); -dispatch_enqueue (sch, dst_dsq, p, enq_flags); +dispatch_enqueue (sch, dst_rq, dst_dsq, p, enq_flags); }return dst_rq;
@@ -1978,14 +1979,14 @@ static void dispatch_to_local_dsq(struct scx_sched *sch, struct rq *rq,* If dispatching to @rq that @p is already on, no lock dancing needed. */ if (rq == src_rq && rq == dst_rq) { -dispatch_enqueue (sch, dst_dsq, p, +dispatch_enqueue (sch, rq, dst_dsq, p, enq_flags |SCX_ENQ_CLEAR_OPSS );return ; }if (src_rq != dst_rq &&unlikely (!task_can_run_on_remote_rq (sch, p, dst_rq, true))) { -dispatch_enqueue (sch,find_global_dsq (sch, p), p, +dispatch_enqueue (sch, rq,find_global_dsq (sch, p), p, enq_flags |SCX_ENQ_CLEAR_OPSS );return ; }
@@ -2023,7 +2024,7 @@ static void dispatch_to_local_dsq(struct scx_sched *sch, struct rq *rq,*/ if (src_rq == dst_rq) { p->scx.holding_cpu =-1 ; -dispatch_enqueue (sch, &dst_rq->scx.local_dsq, p, +dispatch_enqueue (sch, dst_rq, &dst_rq->scx.local_dsq, p, enq_flags); }else { move_remote_task_to_local_dsq(p, enq_flags,
@@ -2122,7 +2123,7 @@ static void finish_dispatch(struct scx_sched *sch, struct rq *rq,if (dsq->id ==SCX_DSQ_LOCAL )dispatch_to_local_dsq (sch, rq, dsq, p, enq_flags);else -dispatch_enqueue (sch, dsq, p, enq_flags |SCX_ENQ_CLEAR_OPSS ); +dispatch_enqueue (sch, rq, dsq, p, enq_flags |SCX_ENQ_CLEAR_OPSS ); }static void flush_dispatch_buf (struct scx_sched *sch,struct rq *rq)
@@ -2423,7 +2424,7 @@ static void put_prev_task_scx(struct rq *rq, struct task_struct *p,* DSQ. */ if (p->scx.slice && !scx_rq_bypassing (rq)) { -dispatch_enqueue (sch, &rq->scx.local_dsq, p, +dispatch_enqueue (sch, rq, &rq->scx.local_dsq, p,SCX_ENQ_HEAD ); goto switch_class; }
@@ -3954,7 +3955,7 @@ static u32 bypass_lb_cpu(struct scx_sched *sch, struct rq *rq,* between bypass DSQs. */ dispatch_dequeue_locked (p, donor_dsq); -dispatch_enqueue (sch, donee_dsq, p,SCX_ENQ_NESTED ); +dispatch_enqueue (sch, donee_rq, donee_dsq, p,SCX_ENQ_NESTED );/* * $donee might have been idle and need to be woken up. No need
Signature · 1 line
2.53.0
Andrea Righi
reviewerPATCH 3/4 sched_ext: Fix ops.dequeue() semantics
In reply to Andrea RighiCurrently, ops.dequeue() is only invoked when the sched_ext core knows that a task resides in BPF-managed data structures, which causes it to miss scheduling property change events. In addition, ops.dequeue() callbacks are completely skipped when tasks are dispatched to non-local DSQs from ops.select_cpu(). As a result, BPF schedulers cannot reliably track task state.
Fix this by guaranteeing that each task entering the BPF scheduler's custody triggers exactly one ops.dequeue() call when it leaves that custody, whether the exit is due to a dispatch (regular or via a core scheduling pick) or to a scheduling property change (e.g. sched_setaffinity(), sched_setscheduler(), set_user_nice(), NUMA balancing, etc.).
BPF scheduler custody concept: a task is considered to be in the BPF scheduler's custody when the scheduler is responsible for managing its lifecycle. This includes tasks dispatched to user-created DSQs or stored in the BPF scheduler's internal data structures from ops.enqueue(). Custody ends when the task is dispatched to a terminal DSQ (such as the local DSQ or %SCX_DSQ_GLOBAL), selected by core scheduling, or removed due to a property change.
Tasks directly dispatched to terminal DSQs bypass the BPF scheduler entirely and are never in its custody. Terminal DSQs include:
- Local DSQs (%SCX_DSQ_LOCAL or %SCX_DSQ_LOCAL_ON): per-CPU queues where tasks go directly to execution.
- Global DSQ (%SCX_DSQ_GLOBAL): the built-in fallback queue where the BPF scheduler is considered "done" with the task.
As a result, ops.dequeue() is not invoked for tasks directly dispatched to terminal DSQs.
To identify dequeues triggered by scheduling property changes, introduce the new ops.dequeue() flag %SCX_DEQ_SCHED_CHANGE: when this flag is set, the dequeue was caused by a scheduling property change.
New ops.dequeue() semantics:
- ops.dequeue() is invoked exactly once when the task leaves the BPF scheduler's custody, in one of the following cases:
a) regular dispatch: a task dispatched to a user DSQ or stored in internal BPF data structures is moved to a terminal DSQ (ops.dequeue() called without any special flags set),
b) core scheduling dispatch: core-sched picks task before dispatch (ops.dequeue() called with %SCX_DEQ_CORE_SCHED_EXEC flag set),
c) property change: task properties modified before dispatch, (ops.dequeue() called with %SCX_DEQ_SCHED_CHANGE flag set).
This allows BPF schedulers to:
- reliably track task ownership and lifecycle,
- maintain accurate accounting of managed tasks,
- update internal state when tasks change properties.
--- Documentation/scheduler/sched-ext.rst | 78 ++++++++++- include/linux/sched/ext.h | 1 + kernel/sched/ext.c | 126 ++++++++++++++++-- kernel/sched/ext_internal.h | 7 + .../sched_ext/include/scx/enum_defs.autogen.h | 1 + .../sched_ext/include/scx/enums.autogen.bpf.h | 2 + tools/sched_ext/include/scx/enums.autogen.h | 1 + 7 files changed, 200 insertions(+), 16 deletions(-)
Documentation/scheduler/sched-ext.rst +71 -7
@@ -228,16 +228,23 @@ The following briefly shows how a waking task is scheduled and executed. scheduler can wake up any cpu using the ``scx_bpf_kick_cpu()`` helper, using ``ops.select_cpu()`` judiciously can be simpler and more efficient. - A task can be immediately inserted into a DSQ from ``ops.select_cpu()`` - by calling ``scx_bpf_dsq_insert()``. If the task is inserted into - ``SCX_DSQ_LOCAL`` from ``ops.select_cpu()``, it will be inserted into the - local DSQ of whichever CPU is returned from ``ops.select_cpu()``. - Additionally, inserting directly from ``ops.select_cpu()`` will cause the - ``ops.enqueue()`` callback to be skipped. - Note that the scheduler core will ignore an invalid CPU selection, for example, if it's outside the allowed cpumask of the task. + A task can be immediately inserted into a DSQ from ``ops.select_cpu()`` + by calling ``scx_bpf_dsq_insert()`` or ``scx_bpf_dsq_insert_vtime()``. + + If the task is inserted into ``SCX_DSQ_LOCAL`` from + ``ops.select_cpu()``, it will be added to the local DSQ of whichever CPU + is returned from ``ops.select_cpu()``. Additionally, inserting directly + from ``ops.select_cpu()`` will cause the ``ops.enqueue()`` callback to + be skipped. + + Any other attempt to store a task in BPF-internal data structures from + ``ops.select_cpu()`` does not prevent ``ops.enqueue()`` from being + invoked. This is discouraged, as it can introduce racy behavior or + inconsistent state. + 2. Once the target CPU is selected, ``ops.enqueue()`` is invoked (unless the task was inserted directly from ``ops.select_cpu()``). ``ops.enqueue()`` can make one of the following decisions:
@@ -251,6 +258,61 @@ The following briefly shows how a waking task is scheduled and executed. * Queue the task on the BPF side. + **Task State Tracking and ops.dequeue() Semantics** + + A task is in the "BPF scheduler's custody" when the BPF scheduler is + responsible for managing its lifecycle. A task enters custody when it is + dispatched to a user DSQ or stored in the BPF scheduler's internal data + structures. Custody is entered only from ``ops.enqueue()`` for those + operations. The only exception is dispatching to a user DSQ from + ``ops.select_cpu()``: although the task is not yet technically in BPF + scheduler custody at that point, the dispatch has the same semantic + effect as dispatching from ``ops.enqueue()`` for custody-related + purposes. + + Once ``ops.enqueue()`` is called, the task may or may not enter custody + depending on what the scheduler does: + + * **Directly dispatched to terminal DSQs** (``SCX_DSQ_LOCAL``, + ``SCX_DSQ_LOCAL_ON | cpu``, or ``SCX_DSQ_GLOBAL``): the BPF scheduler + is done with the task - it either goes straight to a CPU's local run + queue or to the global DSQ as a fallback. The task never enters (or + exits) BPF custody, and ``ops.dequeue()`` will not be called. + + * **Dispatch to user-created DSQs** (custom DSQs): the task enters the + BPF scheduler's custody. When the task later leaves BPF custody + (dispatched to a terminal DSQ, picked by core-sched, or dequeued for + sleep/property changes), ``ops.dequeue()`` will be called exactly + once. + + * **Stored in BPF data structures** (e.g., internal BPF queues): the + task is in BPF custody. ``ops.dequeue()`` will be called when it + leaves (e.g., when ``ops.dispatch()`` moves it to a terminal DSQ, or + on property change / sleep). + + When a task leaves BPF scheduler custody, ``ops.dequeue()`` is invoked. + The dequeue can happen for different reasons, distinguished by flags: + + 1. **Regular dispatch**: when a task in BPF custody is dispatched to a + terminal DSQ from ``ops.dispatch()`` (leaving BPF custody for + execution), ``ops.dequeue()`` is triggered without any special flags. + + 2. **Core scheduling pick**: when ``CONFIG_SCHED_CORE`` is enabled and + core scheduling picks a task for execution while it's still in BPF + custody, ``ops.dequeue()`` is called with the + ``SCX_DEQ_CORE_SCHED_EXEC`` flag. + + 3. **Scheduling property change**: when a task property changes (via + operations like ``sched_setaffinity()``, ``sched_setscheduler()``, + priority changes, CPU migrations, etc.) while the task is still in + BPF custody, ``ops.dequeue()`` is called with the + ``SCX_DEQ_SCHED_CHANGE`` flag set in ``deq_flags``. + + **Important**: Once a task has left BPF custody (e.g., after being + dispatched to a terminal DSQ), property changes will not trigger + ``ops.dequeue()``, since the task is no longer managed by the BPF + scheduler. + 3. When a CPU is ready to schedule, it first looks at its local DSQ. If empty, it then looks at the global DSQ. If there still isn't a task to run, ``ops.dispatch()`` is invoked which can use the following two
@@ -318,6 +380,8 @@ by a sched_ext scheduler: /* Any usable CPU becomes available */ ops.dispatch(); /* Task is moved to a local DSQ */ + + ops.dequeue(); /* Exiting BPF scheduler */ } ops.running(); /* Task starts running on its assigned CPU */ while (task->scx.slice > 0 && task is runnable)
include/linux/sched/ext.h +1 -0
@@ -84,6 +84,7 @@ struct scx_dispatch_q {/* scx_entity.flags */ enum scx_ent_flags {SCX_TASK_QUEUED =1 <<0 ,/* on ext runqueue */ +SCX_TASK_IN_CUSTODY =1 <<1 ,/* in custody, needs ops.dequeue() when leaving */ SCX_TASK_RESET_RUNNABLE_AT =1 <<2 ,/* runnable_at should be reset */ SCX_TASK_DEQD_FOR_SLEEP =1 <<3 ,/* last dequeue was for SLEEP */
kernel/sched/ext.c +117 -9
192 lines not on this page · raw message
kernel/sched/ext_internal.h +7 -0
@@ -982,6 +982,13 @@ enum scx_deq_flags {* it hasn't been dispatched yet. Dequeue from the BPF side. */ SCX_DEQ_CORE_SCHED_EXEC =1LLU <<32 , + +/* +* The task is being dequeued due to a property change (e.g., +* sched_setaffinity(), sched_setscheduler(), set_user_nice(), +* etc.). +*/ + SCX_DEQ_SCHED_CHANGE = 1LLU << 33, }; enum scx_pick_idle_cpu_flags {
tools/sched_ext/include/scx/enum_defs.autogen.h +1 -0
@@ -21,6 +21,7 @@#define HAVE_SCX_CPU_PREEMPT_UNKNOWN #define HAVE_SCX_DEQ_SLEEP #define HAVE_SCX_DEQ_CORE_SCHED_EXEC +#define HAVE_SCX_DEQ_SCHED_CHANGE #define HAVE_SCX_DSQ_FLAG_BUILTIN #define HAVE_SCX_DSQ_FLAG_LOCAL_ON #define HAVE_SCX_DSQ_INVALID
tools/sched_ext/include/scx/enums.autogen.bpf.h +2 -0
@@ -127,3 +127,5 @@ const volatile u64 __SCX_ENQ_CLEAR_OPSS __weak;const volatile u64 __SCX_ENQ_DSQ_PRIQ__weak ;#define SCX_ENQ_DSQ_PRIQ __SCX_ENQ_DSQ_PRIQ +const volatile u64 __SCX_DEQ_SCHED_CHANGE__weak ; +#define SCX_DEQ_SCHED_CHANGE __SCX_DEQ_SCHED_CHANGE
tools/sched_ext/include/scx/enums.autogen.h +1 -0
@@ -46,4 +46,5 @@SCX_ENUM_SET (skel, scx_enq_flags,SCX_ENQ_LAST ); \SCX_ENUM_SET (skel, scx_enq_flags,SCX_ENQ_CLEAR_OPSS ); \SCX_ENUM_SET (skel, scx_enq_flags,SCX_ENQ_DSQ_PRIQ ); \ +SCX_ENUM_SET (skel, scx_deq_flags,SCX_DEQ_SCHED_CHANGE ); \ }while (0 )
Signature · 1 line
2.53.0
Andrea Righi
reviewerPATCH 4/4 selftests/sched_ext: Add test to validate ops.dequeue() semantics
In reply to Andrea RighiAdd a new kselftest to validate that the new ops.dequeue() semantics work correctly for all task lifecycle scenarios, including the distinction between terminal DSQs (where BPF scheduler is done with the task), user DSQs (where BPF scheduler manages the task lifecycle) and BPF data structures, regardless of which event performs the dispatch.
The test validates the following scenarios:
- From ops.select_cpu():
- scenario 0 (local DSQ): tasks dispatched to the local DSQ bypass the BPF scheduler entirely; they never enter BPF custody, so ops.dequeue() is not called,
- scenario 1 (global DSQ): tasks dispatched to SCX_DSQ_GLOBAL also bypass the BPF scheduler, like the local DSQ; ops.dequeue() is not called,
- scenario 2 (user DSQ): tasks dispatched to user DSQs from ops.select_cpu(): tasks enter BPF scheduler's custody with full enqueue/dequeue lifecycle tracking and state machine validation, expects 1:1 enqueue/dequeue pairing,
- From ops.enqueue():
- scenario 3 (local DSQ): same behavior as scenario 0,
- scenario 4 (global DSQ): same behavior as scenario 1,
- scenario 5 (user DSQ): same behavior as scenario 2,
- scenario 6 (BPF internal queue): tasks are stored in a BPF queue from ops.enqueue() and consumed from ops.dispatch(); similarly to scenario 5, tasks enter BPF scheduler's custody with full lifecycle tracking and 1:1 enqueue/dequeue validation.
This verifies that:
- terminal DSQ dispatch (local, global) don't trigger ops.dequeue(),
- tasks dispatched to user DSQs, either from ops.select_cpu() or ops.enqueue(), enter BPF scheduler's custody and have exact 1:1 enqueue/dequeue pairing,
- tasks stored to internal BPF data structures from ops.enqueue() enter BPF scheduler's custody and have exact 1:1 enqueue/dequeue pairing,
- dispatch dequeues have no flags (normal workflow),
- property change dequeues have the %SCX_DEQ_SCHED_CHANGE flag set,
- no duplicate enqueues or invalid state transitions are happening.
--- tools/testing/selftests/sched_ext/Makefile | 1 + .../testing/selftests/sched_ext/dequeue.bpf.c | 394 ++++++++++++++++++ tools/testing/selftests/sched_ext/dequeue.c | 274 ++++++++++++ 3 files changed, 669 insertions(+) create mode 100644 tools/testing/selftests/sched_ext/dequeue.bpf.c create mode 100644 tools/testing/selftests/sched_ext/dequeue.c
tools/testing/selftests/sched_ext/Makefile +1 -0
@@ -161,6 +161,7 @@ all_test_bpfprogs := $(foreach prog,$(wildcard *.bpf.c),$(INCLUDE_DIR)/$(patsubs auto-test-targets := \ create_dsq \ + dequeue \ enq_last_no_enq_fails \ ddsp_bogus_dsq_fail \ ddsp_vtimelocal_fail \
tools/testing/selftests/sched_ext/dequeue.bpf.c new file +394 -0
394 lines not on this page · raw message
tools/testing/selftests/sched_ext/dequeue.c new file +274 -0
274 lines not on this page · raw message
Signature · 1 line
2.53.0
Re: [PATCH 4/4] selftests/sched_ext: Add test to validate ops.dequeue() semantics
Hi Andrea,
Looks clean, runs fine.
All minor stuff, no need for a respin here in v10.
tools/testing/selftests/sched_ext/dequeue.bpf.c line 71
Quoted 8 lines
tools/testing/selftests/sched_ext/dequeue.bpf.c +7
-0+/* +* Per-task state to track lifecycle and validate workflow semantics. +* State transitions: +* NONE -> ENQUEUED (on enqueue) +* ENQUEUED -> DISPATCHED (on dispatch dequeue) +* DISPATCHED -> NONE (on property change dequeue or re-enqueue) +* ENQUEUED -> NONE (on property change dequeue before dispatch)
NONE -> DISPATCHED is a valid transition too.
Quoted 8 lines
tools/testing/selftests/sched_ext/dequeue.bpf.c +8
-0+void BPF_STRUCT_OPS (dequeue_enqueue ,struct task_struct *p,u64 enq_flags) +{ +struct task_ctx *tctx; +s32 pid = p->pid; + + tctx =try_lookup_task_ctx (p); +if (!tctx) +return ;
tools/testing/selftests/sched_ext/dequeue.bpf.c line 167
Quoted 10 lines
tools/testing/selftests/sched_ext/dequeue.bpf.c +10
-0+ +switch (test_scenario) { +case 3 : +/* +* Direct dispatch to the local DSQ. +* +* Task bypasses BPF scheduler entirely: no enqueue +* tracking, no ops.dequeue() callbacks. +*/ +scx_bpf_dsq_insert (p,SCX_DSQ_LOCAL ,SCX_SLICE_DFL , enq_flags);
Could do with
tctx->state = TASK_DISPATCHED;
tools/testing/selftests/sched_ext/dequeue.bpf.c line 176
Quoted 9 lines
tools/testing/selftests/sched_ext/dequeue.bpf.c +9
-0+break ; +case 4 : +/* +* Direct dispatch to the global DSQ. +* +* Task bypasses BPF scheduler entirely: no enqueue +* tracking, no ops.dequeue() callbacks. +*/ +scx_bpf_dsq_insert (p,SCX_DSQ_GLOBAL ,SCX_SLICE_DFL , enq_flags);
...and also here
tctx->state = TASK_DISPATCHED;
to be consistent with how select_cpu sets this state, but I don't think setting DISPATCHED has any effect in select_cpu or here.
Quoted 1 line
+void
BPF_STRUCT_OPS(dequeue_dequeue, struct task_struct *p, u64 deq_flags)
...
Quoted 1 line
tools/testing/selftests/sched_ext/dequeue.bpf.c +1
-0+ }else {
...
tools/testing/selftests/sched_ext/dequeue.bpf.c line 297
Quoted 7 lines
tools/testing/selftests/sched_ext/dequeue.bpf.c +7
-0+/* +* Dispatch dequeue should not have %SCX_DEQ_SCHED_CHANGE +* flag. +*/ +if (deq_flags &SCX_DEQ_SCHED_CHANGE ) +scx_bpf_error ("%d (%s): SCX_DEQ_SCHED_CHANGE in dispatch dequeue seq=%llu" , + p->pid, p->comm, tctx->enqueue_seq);
This 'if' can be deleted.
Andrea Righi
reviewerPATCH v2 4/4 selftests/sched_ext: Add test to validate ops.dequeue() semantics
Add a new kselftest to validate that the new ops.dequeue() semantics work correctly for all task lifecycle scenarios, including the distinction between terminal DSQs (where BPF scheduler is done with the task), user DSQs (where BPF scheduler manages the task lifecycle) and BPF data structures, regardless of which event performs the dispatch.
The test validates the following scenarios:
- From ops.select_cpu():
- scenario 0 (local DSQ): tasks dispatched to the local DSQ bypass the BPF scheduler entirely; they never enter BPF custody, so ops.dequeue() is not called,
- scenario 1 (global DSQ): tasks dispatched to SCX_DSQ_GLOBAL also bypass the BPF scheduler, like the local DSQ; ops.dequeue() is not called,
- scenario 2 (user DSQ): tasks dispatched to user DSQs from ops.select_cpu(): tasks enter BPF scheduler's custody with full enqueue/dequeue lifecycle tracking and state machine validation, expects 1:1 enqueue/dequeue pairing,
- From ops.enqueue():
- scenario 3 (local DSQ): same behavior as scenario 0,
- scenario 4 (global DSQ): same behavior as scenario 1,
- scenario 5 (user DSQ): same behavior as scenario 2,
- scenario 6 (BPF internal queue): tasks are stored in a BPF queue from ops.enqueue() and consumed from ops.dispatch(); similarly to scenario 5, tasks enter BPF scheduler's custody with full lifecycle tracking and 1:1 enqueue/dequeue validation.
This verifies that:
- terminal DSQ dispatch (local, global) don't trigger ops.dequeue(),
- tasks dispatched to user DSQs, either from ops.select_cpu() or ops.enqueue(), enter BPF scheduler's custody and have exact 1:1 enqueue/dequeue pairing,
- tasks stored to internal BPF data structures from ops.enqueue() enter BPF scheduler's custody and have exact 1:1 enqueue/dequeue pairing,
- dispatch dequeues have no flags (normal workflow),
- property change dequeues have the %SCX_DEQ_SCHED_CHANGE flag set,
- no duplicate enqueues or invalid state transitions are happening.
---
v2: - set DISPATCHED on direct dispatch and remove unnecessary condition
(Daniel Jordan)
tools/testing/selftests/sched_ext/Makefile | 1 + .../testing/selftests/sched_ext/dequeue.bpf.c | 389 ++++++++++++++++++ tools/testing/selftests/sched_ext/dequeue.c | 274 ++++++++++++ 3 files changed, 664 insertions(+) create mode 100644 tools/testing/selftests/sched_ext/dequeue.bpf.c create mode 100644 tools/testing/selftests/sched_ext/dequeue.c
tools/testing/selftests/sched_ext/Makefile +1 -0
@@ -161,6 +161,7 @@ all_test_bpfprogs := $(foreach prog,$(wildcard *.bpf.c),$(INCLUDE_DIR)/$(patsubs auto-test-targets := \ create_dsq \ + dequeue \ enq_last_no_enq_fails \ ddsp_bogus_dsq_fail \ ddsp_vtimelocal_fail \
tools/testing/selftests/sched_ext/dequeue.bpf.c new file +389 -0
389 lines not on this page · raw message
tools/testing/selftests/sched_ext/dequeue.c new file +274 -0
274 lines not on this page · raw message
Signature · 1 line
2.53.0
Tejun Heo
reviewerQuoted 5 lines
Andrea Righi (4):
sched_ext: Properly mark SCX-internal migrations viasticky_cpu
sched_ext: Add rq parameter todispatch_enqueue()
sched_ext: Fixops.dequeue()semantics selftests/sched_ext: Add test to validateops.dequeue()semantics
Applied 1-4 to sched_ext/for-7.1 with the following minor changes:
- Dropped unused is_terminal_dsq() helper from 3/4.
- Fixed typo in 4/4 comment ("shared a user DSQ" -> "a shared user DSQ").
Thanks.
--
tejun