Skip to contentSkip to conversation

sched_ext: Fix ops.dequeue() semantics

PATCH Superseded Andrea Righi 3 participants 15 messages Feb 1 – Feb 4

Subsystems (4)

v4 of 10 Superseded Latest is v10 →

lore
Share & tools
lore

Andrea Righi

author reviewer
#0 lore raw

PATCH v4 1/1 sched_ext: Fix ops.dequeue() semantics

The callback ops.dequeue() is provided to let BPF schedulers observe when a task leaves the scheduler, either because it is dispatched or due to a task property change. However, this callback is currently unreliable and not invoked systematically, which can result in missed ops.dequeue() events.

In particular, once a task is removed from the scheduler (whether for dispatch or due to a property change) the BPF scheduler loses visibility of the task and the sched_ext core may not always trigger ops.dequeue().

This breaks accurate accounting (i.e., per-DSQ queued runtime sums) and prevents reliable tracking of task lifecycle transitions.

This patch set fixes the semantics of ops.dequeue(), by guaranteeing that each task entering the BPF scheduler's custody triggers exactly one ops.dequeue() call when it leaves that custody, whether the exit is due to a dispatch (regular or via a core scheduling pick) or to a scheduling property change (e.g. sched_setaffinity(), sched_setscheduler(), set_user_nice(), NUMA balancing, etc.).

To identify property change dequeues a new ops.dequeue() flag is
introduced: %SCX_DEQ_SCHED_CHANGE.

Together, these changes allow BPF schedulers to reliably track task ownership and maintain accurate accounting.

= Open issues =

Even though a few refinements are still pending, I'm sending a new patchset, so we can comment more effectively based on the latest agreed-upon semantics.

Open issues that still need agreement:

- Should we trigger ops.dequeue() for tasks dispatched to SCX_DSQ_GLOBAL? (do we treat SCX_DSQ_GLOBAL as a local DSQ or a user DSQ?). In the current implementation, SCX_DSQ_GLOBAL is treated like a built-in user DSQ -> ops.dequeue() invoked for tasks dispatched to SCX_DSQ_GLOBAL.

Changes in v4:
- Introduce the concept of "BPF scheduler custody"
- Do not trigger ops.dequeue() for tasks directly dispatched to local DSQs
- Trigger ops.dequeue() only once; after the task leaves BPF scheduler custody, further dequeue events are not reported.
- Link to v3: https://lore.kernel.org/all/[email protected]

Changes in v3:
- Rename SCX_DEQ_ASYNC to SCX_DEQ_SCHED_CHANGE
- Handle core-sched dequeues (Kuba)
- Link to v2: https://lore.kernel.org/all/[email protected]

Changes in v2:
- Distinguish between "dispatch" dequeues and "property change" dequeues (flag SCX_DEQ_ASYNC)
- Link to v1: https://lore.kernel.org/all/[email protected]

Andrea Righi (2):
sched_ext: Fix ops.dequeue() semantics selftests/sched_ext: Add test to validate ops.dequeue() semantics

 Documentation/scheduler/sched-ext.rst           |  76 ++++++++
 include/linux/sched/ext.h                       |   1 +
 kernel/sched/ext.c                              | 168 ++++++++++++++++-
 kernel/sched/ext_internal.h                     |   7 +
 tools/sched_ext/include/scx/enum_defs.autogen.h |   1 +
 tools/sched_ext/include/scx/enums.autogen.bpf.h |   2 +
 tools/sched_ext/include/scx/enums.autogen.h     |   1 +
 tools/testing/selftests/sched_ext/Makefile      |   1 +
 tools/testing/selftests/sched_ext/dequeue.bpf.c | 234 ++++++++++++++++++++++++
 tools/testing/selftests/sched_ext/dequeue.c     | 198 ++++++++++++++++++++
 10 files changed, 686 insertions(+), 3 deletions(-)
 create mode 100644 tools/testing/selftests/sched_ext/dequeue.bpf.c
 create mode 100644 tools/testing/selftests/sched_ext/dequeue.c
author reviewer
#1 lore raw

PATCH 1/2 sched_ext: Fix ops.dequeue() semantics

Currently, ops.dequeue() is only invoked when the sched_ext core knows that a task resides in BPF-managed data structures, which causes it to miss scheduling property change events. In addition, ops.dequeue() callbacks are completely skipped when tasks are dispatched to non-local DSQs from ops.select_cpu(). As a result, BPF schedulers cannot reliably track task state.

Fix this by guaranteeing that each task entering the BPF scheduler's custody triggers exactly one ops.dequeue() call when it leaves that custody, whether the exit is due to a dispatch (regular or via a core scheduling pick) or to a scheduling property change (e.g. sched_setaffinity(), sched_setscheduler(), set_user_nice(), NUMA balancing, etc.).

BPF scheduler custody concept: a task is considered to be in "BPF scheduler's custody" when it has been queued in BPF-managed data structures and the BPF scheduler is responsible for its lifecycle. Custody ends when the task is dispatched to a local DSQ, selected by core scheduling, or removed due to a property change.

Tasks directly dispatched to local DSQs (via %SCX_DSQ_LOCAL or %SCX_DSQ_LOCAL_ON) bypass the BPF scheduler entirely and are not in its custody. As a result, ops.dequeue() is not invoked for these tasks.

To identify dequeues triggered by scheduling property changes, introduce the new ops.dequeue() flag %SCX_DEQ_SCHED_CHANGE: when this flag is set, the dequeue was caused by a scheduling property change.

New ops.dequeue() semantics:
- ops.dequeue() is invoked exactly once when the task leaves the BPF scheduler's custody, in one of the following cases:
a) regular dispatch: task was dispatched to a non-local DSQ (global or user DSQ), ops.dequeue() called without any special flags set
b) core scheduling dispatch: core-sched picks task before dispatch, dequeue called with %SCX_DEQ_CORE_SCHED_EXEC flag set
c) property change: task properties modified before dispatch, dequeue called with %SCX_DEQ_SCHED_CHANGE flag set

This allows BPF schedulers to:
- reliably track task ownership and lifecycle,
- maintain accurate accounting of managed tasks,
- update internal state when tasks change properties.

Cc Tejun Heo <[email protected]>
Cc Emil Tsalapatis <[email protected]>
Cc Kuba Piecuch <[email protected]>
---
 Documentation/scheduler/sched-ext.rst         |  76 ++++++++
 include/linux/sched/ext.h                     |   1 +
 kernel/sched/ext.c                            | 168 +++++++++++++++++-
 kernel/sched/ext_internal.h                   |   7 +
 .../sched_ext/include/scx/enum_defs.autogen.h |   1 +
 .../sched_ext/include/scx/enums.autogen.bpf.h |   2 +
 tools/sched_ext/include/scx/enums.autogen.h   |   1 +
 7 files changed, 253 insertions(+), 3 deletions(-)
Documentation/scheduler/sched-ext.rst +76 -0
index 404fe6126a769..6d9e82e6ca9d4 100644
@@ -252,6 +252,80 @@ The following briefly shows how a waking task is scheduled and executed.
 
    * Queue the task on the BPF side.
 
+   **Task State Tracking and ops.dequeue() Semantics**
+
+   Once ``ops.select_cpu()`` or ``ops.enqueue()`` is called, the task may
+   enter the "BPF scheduler's custody" depending on where it's dispatched:
+
+   * **Direct dispatch to local DSQs** (``SCX_DSQ_LOCAL`` or
+     ``SCX_DSQ_LOCAL_ON | cpu``): The task bypasses the BPF scheduler
+     entirely and goes straight to the CPU's local run queue. The task
+     never enters BPF custody, and ``ops.dequeue()`` will not be called.
+
+   * **Dispatch to non-local DSQs** (``SCX_DSQ_GLOBAL`` or custom DSQs):
+     the task enters the BPF scheduler's custody. When the task later
+     leaves BPF custody (dispatched to a local DSQ, picked by core-sched,
+     or dequeued for sleep/property changes), ``ops.dequeue()`` will be
+     called exactly once.
+
+   * **Queued on BPF side**: The task is in BPF data structures and in BPF
+     custody, ``ops.dequeue()`` will be called when it leaves.
+
+   The key principle: **ops.dequeue() is called when a task leaves the BPF
+   scheduler's custody**. A task is in BPF custody if it's on a non-local
+   DSQ or in BPF data structures. Once dispatched to a local DSQ or after
+   ops.dequeue() is called, the task is out of BPF custody and the BPF
+   scheduler no longer needs to track it.
+
+   This works correctly with the ``ops.select_cpu()`` direct dispatch
+   optimization: even though it skips ``ops.enqueue()`` invocation, if the
+   task is dispatched to a non-local DSQ, it enters BPF custody and will
+   get ``ops.dequeue()`` when it leaves. This provides the performance
+   benefit of avoiding the ``ops.enqueue()`` roundtrip while maintaining
+   correct state tracking.
+
+   The dequeue can happen for different reasons, distinguished by flags:
+
+   1. **Regular dispatch workflow**: when the task is dispatched from a
+      non-local DSQ to a local DSQ (leaving BPF custody for execution),
+      ``ops.dequeue()`` is triggered without any special flags.
+
+   2. **Core scheduling pick**: when ``CONFIG_SCHED_CORE`` is enabled and
+      core scheduling picks a task for execution while it's still in BPF
+      custody, ``ops.dequeue()`` is called with the
+      ``SCX_DEQ_CORE_SCHED_EXEC`` flag.
+
+   3. **Scheduling property change**: when a task property changes (via
+      operations like ``sched_setaffinity()``, ``sched_setscheduler()``,
+      priority changes, CPU migrations, etc.) while the task is still in
+      BPF custody, ``ops.dequeue()`` is called with the
+      ``SCX_DEQ_SCHED_CHANGE`` flag set in ``deq_flags``.
+
+   **Important**: Once a task has left BPF custody (dispatched to local
+   DSQ), property changes will not trigger ``ops.dequeue()``, since the
+   task is no longer being managed by the BPF scheduler.
+
+   **Property Change Notifications for Running Tasks**:
+
+   For tasks that have left BPF custody (running or on local DSQs),
+   property changes can be intercepted through the dedicated callbacks:
+
+   * ``ops.set_cpumask()``: Called when a task's CPU affinity changes
+     (e.g., via ``sched_setaffinity()``). This callback is invoked for
+     all tasks regardless of their state or BPF custody.
+
+   * ``ops.set_weight()``: Called when a task's scheduling weight/priority
+     changes (e.g., via ``sched_setscheduler()`` or ``set_user_nice()``).
+     This callback is also invoked for all tasks.
+
+   These callbacks provide complete coverage for property changes,
+   complementing ``ops.dequeue()`` which only applies to tasks in BPF
+   custody.
+
+   BPF schedulers can choose not to implement ``ops.dequeue()`` if they
+   don't need to track these transitions. The sched_ext core will safely
+   handle all dequeue operations regardless.
+
 3. When a CPU is ready to schedule, it first looks at its local DSQ. If
    empty, it then looks at the global DSQ. If there still isn't a task to
    run, ``ops.dispatch()`` is invoked which can use the following two
@@ -319,6 +393,8 @@ by a sched_ext scheduler:
                 /* Any usable CPU becomes available */
 
                 ops.dispatch(); /* Task is moved to a local DSQ */
+
+                ops.dequeue(); /* Exiting BPF scheduler */
             }
             ops.running();      /* Task starts running on its assigned CPU */
             while (task->scx.slice > 0 && task is runnable)
include/linux/sched/ext.h +1 -0
index bcb962d5ee7d8..0d003d2845393 100644
@@ -84,6 +84,7 @@ struct scx_dispatch_q {
 /* scx_entity.flags */
 enum scx_ent_flags {
 	SCX_TASK_QUEUED		= 1 << 0, /* on ext runqueue */
+	SCX_TASK_OPS_ENQUEUED	= 1 << 1, /* under ext scheduler's custody */
 	SCX_TASK_RESET_RUNNABLE_AT = 1 << 2, /* runnable_at should be reset */
 	SCX_TASK_DEQD_FOR_SLEEP	= 1 << 3, /* last dequeue was for SLEEP */
 
kernel/sched/ext.c +165 -3

234 lines not on this page · raw message

kernel/sched/ext_internal.h +7 -0
index 386c677e4c9a0..befa9a5d6e53f 100644
@@ -982,6 +982,13 @@ enum scx_deq_flags {
 	 * it hasn't been dispatched yet. Dequeue from the BPF side.
 	 */
 	SCX_DEQ_CORE_SCHED_EXEC	= 1LLU << 32,
+
+	/*
+	 * The task is being dequeued due to a property change (e.g.,
+	 * sched_setaffinity(), sched_setscheduler(), set_user_nice(),
+	 * etc.).
+	 */
+	SCX_DEQ_SCHED_CHANGE	= 1LLU << 33,
 };
 
 enum scx_pick_idle_cpu_flags {
tools/sched_ext/include/scx/enum_defs.autogen.h +1 -0
index c2c33df9292c2..dcc945304760f 100644
@@ -21,6 +21,7 @@
 #define HAVE_SCX_CPU_PREEMPT_UNKNOWN
 #define HAVE_SCX_DEQ_SLEEP
 #define HAVE_SCX_DEQ_CORE_SCHED_EXEC
+#define HAVE_SCX_DEQ_SCHED_CHANGE
 #define HAVE_SCX_DSQ_FLAG_BUILTIN
 #define HAVE_SCX_DSQ_FLAG_LOCAL_ON
 #define HAVE_SCX_DSQ_INVALID
tools/sched_ext/include/scx/enums.autogen.bpf.h +2 -0
index 2f8002bcc19ad..5da50f9376844 100644
@@ -127,3 +127,5 @@ const volatile u64 __SCX_ENQ_CLEAR_OPSS __weak;
 const volatile u64 __SCX_ENQ_DSQ_PRIQ __weak;
 #define SCX_ENQ_DSQ_PRIQ __SCX_ENQ_DSQ_PRIQ
 
+const volatile u64 __SCX_DEQ_SCHED_CHANGE __weak;
+#define SCX_DEQ_SCHED_CHANGE __SCX_DEQ_SCHED_CHANGE
tools/sched_ext/include/scx/enums.autogen.h +1 -0
index fedec938584be..fc9a7a4d9dea5 100644
@@ -46,4 +46,5 @@
 	SCX_ENUM_SET(skel, scx_enq_flags, SCX_ENQ_LAST); \
 	SCX_ENUM_SET(skel, scx_enq_flags, SCX_ENQ_CLEAR_OPSS); \
 	SCX_ENUM_SET(skel, scx_enq_flags, SCX_ENQ_DSQ_PRIQ); \
+	SCX_ENUM_SET(skel, scx_deq_flags, SCX_DEQ_SCHED_CHANGE); \
 } while (0)
Signature · 1 line
2.52.0
#2 lore raw
Quoted 38 lines · Andrea Righi

Currently, ops.dequeue() is only invoked when the sched_ext core knows that a task resides in BPF-managed data structures, which causes it to miss scheduling property change events. In addition, ops.dequeue() callbacks are completely skipped when tasks are dispatched to non-local DSQs from ops.select_cpu(). As a result, BPF schedulers cannot reliably track task state.

Fix this by guaranteeing that each task entering the BPF scheduler's custody triggers exactly one ops.dequeue() call when it leaves that custody, whether the exit is due to a dispatch (regular or via a core scheduling pick) or to a scheduling property change (e.g. sched_setaffinity(), sched_setscheduler(), set_user_nice(), NUMA balancing, etc.).

BPF scheduler custody concept: a task is considered to be in "BPF scheduler's custody" when it has been queued in BPF-managed data structures and the BPF scheduler is responsible for its lifecycle. Custody ends when the task is dispatched to a local DSQ, selected by core scheduling, or removed due to a property change.

Tasks directly dispatched to local DSQs (via %SCX_DSQ_LOCAL or %SCX_DSQ_LOCAL_ON) bypass the BPF scheduler entirely and are not in its custody. As a result, ops.dequeue() is not invoked for these tasks.

To identify dequeues triggered by scheduling property changes, introduce the new ops.dequeue() flag %SCX_DEQ_SCHED_CHANGE: when this flag is set, the dequeue was caused by a scheduling property change.

New ops.dequeue() semantics:
- ops.dequeue() is invoked exactly once when the task leaves the BPF scheduler's custody, in one of the following cases:
a) regular dispatch: task was dispatched to a non-local DSQ (global or user DSQ), ops.dequeue() called without any special flags set
b) core scheduling dispatch: core-sched picks task before dispatch, dequeue called with %SCX_DEQ_CORE_SCHED_EXEC flag set
c) property change: task properties modified before dispatch, dequeue called with %SCX_DEQ_SCHED_CHANGE flag set

Quoted 5 lines

This allows BPF schedulers to:
- reliably track task ownership and lifecycle,
- maintain accurate accounting of managed tasks,
- update internal state when tasks change properties.

So I have finally gotten around updating scx_storm to the new semantics, see:
https://github.com/cloehle/scx/tree/cloehle/scx-storm-qmap-insert-local-dequeue-semantics

I don't think the new ops.dequeue() are enough to make inserts to local-on from anywhere safe, because it's still racing with dequeue from another CPU?

Furthermore I can reproduce the following with this patch applied quite easily with something like

hackbench -l 1000 & timeout 10 ./build/scheds/c/scx_storm

[   44.356878] sched_ext: BPF scheduler "simple" enabled
[   59.315370] sched_ext: BPF scheduler "simple" disabled (unregistered from user space)
[   85.366747] sched_ext: BPF scheduler "storm" enabled
[   85.371324] ------------[ cut here ]------------
[   85.373370] WARNING: kernel/sched/sched.h:1571 at update_locked_rq+0x64/0x6c, CPU#5: gmain/1111
[   85.373392] Modules linked in: qrtr
[   85.380088] ------------[ cut here ]------------
[   85.380719] ------------[ cut here ]------------
[   85.380722] WARNING: kernel/sched/sched.h:1571 at update_locked_rq+0x64/0x6c, CPU#10: kworker/u48:1/82
[   85.380728] Modules linked in: qrtr 8021q garp mrp stp llc binfmt_misc sm3_ce r8169 cdns3_pci_wrap nf_tables nfnetlink fuse dm_mod ipv6
[   85.380745] CPU: 10 UID: 0 PID: 82 Comm: kworker/u48:1 Tainted: G S                  6.19.0-rc7-cix-build+ #256 PREEMPT 
[   85.380749] Tainted: [S]=CPU_OUT_OF_SPEC
[   85.380750] Hardware name: Radxa Computer (Shenzhen) Co., Ltd. Radxa Orion O6/Radxa Orion O6, BIOS 1.1.0-1 2025-12-25T02:55:53+00:00
[   85.380754] Workqueue:  0x0 (events_unbound)
[   85.380760] Sched_ext: storm (enabled+all), task: runnable_at=+0ms
[   85.380762] pstate: 634000c9 (nZCv daIF +PAN -UAO +TCO +DIT -SSBS BTYPE=--)
[   85.380764] pc : update_locked_rq+0x64/0x6c
[   85.380767] lr : update_locked_rq+0x60/0x6c
[   85.380769] sp : ffff8000803a3bd0
[   85.380770] x29: ffff8000803a3bd0 x28: fffffdffbf622dc0 x27: ffff0000911e5040
[   85.380773] x26: 0000000000000000 x25: ffffd204426cad80 x24: ffffd20442ba5bb8
[   85.380776] x23: c00000000000000a x22: 0000000000000000 x21: ffffd20442ba4830
[   85.380778] x20: ffff00009af0b000 x19: ffff0001fef2ed80 x18: 0000000000000000
[   85.380781] x17: 0000000000000000 x16: 0000000000000000 x15: 0000aaaadd996940
[   85.380783] x14: 0000000000000000 x13: 00000000000a0000 x12: 0000000000000000
[   85.380786] x11: 0000000000000040 x10: ffffd204402e7ca0 x9 : ffffd2044324b000
[   85.380788] x8 : ffff0000810e0000 x7 : 0000d00202cc2dc0 x6 : 0000000000000050
[   85.380790] x5 : ffffd204426b5648 x4 : fffffdffbf622dc0 x3 : ffff0000810e0000
[   85.380793] x2 : 0000000000000002 x1 : ffff2dfdbc960000 x0 : 0000000000000000
[   85.380795] Call trace:
[   85.380796]  update_locked_rq+0x64/0x6c (P)
[   85.380799]  flush_dispatch_buf+0x2a8/0x2dc
[   85.380801]  pick_task_scx+0x2b0/0x6d4
[   85.380804]  __schedule+0x62c/0x1060
[   85.380811]  schedule+0x48/0x15c
[   85.380813]  worker_thread+0xdc/0x358
[   85.380824]  kthread+0x134/0x1fc
[   85.380831]  ret_from_fork+0x10/0x20
[   85.380839] irq event stamp: 34386
[   85.380840] hardirqs last  enabled at (34385): [<ffffd20441511408>] _raw_spin_unlock_irq+0x30/0x6c
[   85.380850] hardirqs last disabled at (34386): [<ffffd20441507100>] __schedule+0x510/0x1060
[   85.380852] softirqs last  enabled at (34014): [<ffffd204400c7280>] handle_softirqs+0x514/0x52c
[   85.380865] softirqs last disabled at (34007): [<ffffd204400105c4>] __do_softirq+0x14/0x20
[   85.380867] ---[ end trace 0000000000000000 ]---
[   85.380969] ------------[ cut here ]------------
[   85.380970] WARNING: kernel/sched/sched.h:1571 at update_locked_rq+0x64/0x6c, CPU#10: kworker/u48:1/82
[   85.380974] Modules linked in: qrtr 8021q garp mrp stp llc binfmt_misc sm3_ce r8169 cdns3_pci_wrap nf_tables nfnetlink fuse dm_mod ipv6
[   85.380984] CPU: 10 UID: 0 PID: 82 Comm: kworker/u48:1 Tainted: G S      W           6.19.0-rc7-cix-build+ #256 PREEMPT 
[   85.380987] Tainted: [S]=CPU_OUT_OF_SPEC, [W]=WARN
[   85.380988] Hardware name: Radxa Computer (Shenzhen) Co., Ltd. Radxa Orion O6/Radxa Orion O6, BIOS 1.1.0-1 2025-12-25T02:55:53+00:00
[   85.380990] Workqueue:  0x0 (events_unbound)
[   85.380993] Sched_ext: storm (enabled+all), task: runnable_at=+0ms
[   85.380994] pstate: 634000c9 (nZCv daIF +PAN -UAO +TCO +DIT -SSBS BTYPE=--)
[   85.380996] pc : update_locked_rq+0x64/0x6c
[   85.380997] lr : update_locked_rq+0x60/0x6c
[   85.380999] sp : ffff8000803a3bd0
[   85.381000] x29: ffff8000803a3bd0 x28: fffffdffbf622dc0 x27: ffff00009151b580
[   85.381002] x26: 0000000000000000 x25: ffffd204426cad80 x24: ffffd20442ba5bb8
[   85.381005] x23: c00000000000000a x22: 0000000000000000 x21: ffffd20442ba4830
[   85.381007] x20: ffff00009af0b000 x19: ffff0001fef52d80 x18: 0000000000000000
[   85.381009] x17: 0000000000000000 x16: 0000000000000000 x15: 0000aaaae6917960
[   85.381012] x14: 0000000000000000 x13: 00000000000a0000 x12: 0000000000000000
[   85.381014] x11: 0000000000000040 x10: ffffd204402e7ca0 x9 : ffffd2044324b000
[   85.381016] x8 : ffff0000810e0000 x7 : 0000d00202cc2dc0 x6 : 0000000000000050
[   85.381019] x5 : ffffd204426b5648 x4 : fffffdffbf622dc0 x3 : ffff0000810e0000
[   85.381021] x2 : 0000000000000002 x1 : ffff2dfdbc960000 x0 : 0000000000000000
[   85.381023] Call trace:
[   85.381024]  update_locked_rq+0x64/0x6c (P)
[   85.381026]  flush_dispatch_buf+0x2a8/0x2dc
[   85.381028]  pick_task_scx+0x2b0/0x6d4
[   85.381030]  __schedule+0x62c/0x1060
[   85.381032]  schedule+0x48/0x15c
[   85.381034]  worker_thread+0xdc/0x358
[   85.381036]  kthread+0x134/0x1fc
[   85.381039]  ret_from_fork+0x10/0x20
[   85.381041] irq event stamp: 34394
[   85.381042] hardirqs last  enabled at (34393): [<ffffd20441511408>] _raw_spin_unlock_irq+0x30/0x6c
[   85.381044] hardirqs last disabled at (34394): [<ffffd20441507100>] __schedule+0x510/0x1060
[   85.381046] softirqs last  enabled at (34014): [<ffffd204400c7280>] handle_softirqs+0x514/0x52c
[   85.381049] softirqs last disabled at (34007): [<ffffd204400105c4>] __do_softirq+0x14/0x20
[   85.381050] ---[ end trace 0000000000000000 ]---
[   85.381199] ------------[ cut here ]------------
[   85.381201] WARNING: kernel/sched/sched.h:1571 at update_locked_rq+0x64/0x6c, CPU#10: kworker/u48:1/82
author reviewer
#3 lore raw

Hi Christian,

Quoted 45 lines · Christian Loehle
Quoted 38 lines · Andrea Righi

Currently, ops.dequeue() is only invoked when the sched_ext core knows that a task resides in BPF-managed data structures, which causes it to miss scheduling property change events. In addition, ops.dequeue() callbacks are completely skipped when tasks are dispatched to non-local DSQs from ops.select_cpu(). As a result, BPF schedulers cannot reliably track task state.

Fix this by guaranteeing that each task entering the BPF scheduler's custody triggers exactly one ops.dequeue() call when it leaves that custody, whether the exit is due to a dispatch (regular or via a core scheduling pick) or to a scheduling property change (e.g. sched_setaffinity(), sched_setscheduler(), set_user_nice(), NUMA balancing, etc.).

BPF scheduler custody concept: a task is considered to be in "BPF scheduler's custody" when it has been queued in BPF-managed data structures and the BPF scheduler is responsible for its lifecycle. Custody ends when the task is dispatched to a local DSQ, selected by core scheduling, or removed due to a property change.

Tasks directly dispatched to local DSQs (via %SCX_DSQ_LOCAL or %SCX_DSQ_LOCAL_ON) bypass the BPF scheduler entirely and are not in its custody. As a result, ops.dequeue() is not invoked for these tasks.

To identify dequeues triggered by scheduling property changes, introduce the new ops.dequeue() flag %SCX_DEQ_SCHED_CHANGE: when this flag is set, the dequeue was caused by a scheduling property change.

New ops.dequeue() semantics:
- ops.dequeue() is invoked exactly once when the task leaves the BPF scheduler's custody, in one of the following cases:
a) regular dispatch: task was dispatched to a non-local DSQ (global or user DSQ), ops.dequeue() called without any special flags set
b) core scheduling dispatch: core-sched picks task before dispatch, dequeue called with %SCX_DEQ_CORE_SCHED_EXEC flag set
c) property change: task properties modified before dispatch, dequeue called with %SCX_DEQ_SCHED_CHANGE flag set

Quoted 5 lines

This allows BPF schedulers to:
- reliably track task ownership and lifecycle,
- maintain accurate accounting of managed tasks,
- update internal state when tasks change properties.

Quoted 6 lines

So I have finally gotten around updating scx_storm to the new semantics, see:
https://github.com/cloehle/scx/tree/cloehle/scx-storm-qmap-insert-local-dequeue-semantics

I don't think the new ops.dequeue() are enough to make inserts to local-on from anywhere safe, because it's still racing with dequeue from another CPU?

Yeah, with this patch set BPF schedulers get proper ops.dequeue() callbacks, but we're not fixing the usage of SCX_DSQ_LOCAL_ON from ops.dispatch().

When task properties change between scx_bpf_dsq_insert() and the actual dispatch, task_can_run_on_remote_rq() can still trigger a fatal scx_error().

The ops.dequeue(SCX_DEQ_SCHED_CHANGE) notifications happens after the property change, so it can't prevent already-queued dispatches from failing. The race window is between ops.dispatch() returning and dispatch_to_local_dsq() executing.

We can address this in a separate patch set. One thing at a time. :)

Quoted 11 lines

Furthermore I can reproduce the following with this patch applied quite easily with something like

hackbench -l 1000 & timeout 10 ./build/scheds/c/scx_storm

[   44.356878] sched_ext: BPF scheduler "simple" enabled
[   59.315370] sched_ext: BPF scheduler "simple" disabled (unregistered from user space)
[   85.366747] sched_ext: BPF scheduler "storm" enabled
[   85.371324] ------------[ cut here ]------------
[   85.373370] WARNING: kernel/sched/sched.h:1571 at update_locked_rq+0x64/0x6c, CPU#5: gmain/1111

Ah yes! I think I see it, can you try this on top?

Thanks,
-Andrea

 kernel/sched/ext.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/sched/ext.c +1 -1
index 6d6f1253039d8..d8fed4a49195d 100644
@@ -2248,7 +2248,7 @@ static void finish_dispatch(struct scx_sched *sch, struct rq *rq,
 			p->scx.flags |= SCX_TASK_OPS_ENQUEUED;
 		} else {
 			if (p->scx.flags & SCX_TASK_OPS_ENQUEUED)
-				SCX_CALL_OP_TASK(sch, SCX_KF_REST, dequeue, task_rq(p), p, 0);
+				SCX_CALL_OP_TASK(sch, SCX_KF_REST, dequeue, rq, p, 0);
 
 			p->scx.flags &= ~SCX_TASK_OPS_ENQUEUED;
 		}
author reviewer
#4 lore raw

On Mon, Feb 02, 2026 at 08:45:18AM +0100, Andrea Righi wrote:
...

Quoted 15 lines
Quoted 6 lines

So I have finally gotten around updating scx_storm to the new semantics, see:
https://github.com/cloehle/scx/tree/cloehle/scx-storm-qmap-insert-local-dequeue-semantics

I don't think the new ops.dequeue() are enough to make inserts to local-on from anywhere safe, because it's still racing with dequeue from another CPU?

Yeah, with this patch set BPF schedulers get proper ops.dequeue() callbacks, but we're not fixing the usage of SCX_DSQ_LOCAL_ON from ops.dispatch().

When task properties change between scx_bpf_dsq_insert() and the actual dispatch, task_can_run_on_remote_rq() can still trigger a fatal scx_error().

Quoted 6 lines

The ops.dequeue(SCX_DEQ_SCHED_CHANGE) notifications happens after the property change, so it can't prevent already-queued dispatches from failing. The race window is between ops.dispatch() returning and dispatch_to_local_dsq() executing.

We can address this in a separate patch set. One thing at a time. :)

Thinking more on this, the problem is that we're passing enforce=true to task_can_run_on_remote_rq(), triggering a critical failure - scx_error(). There's a logic in task_can_run_on_remote_rq() to fallback to the global DSQ, that doesn't happen if we pass enforce=true, due to scx_error().

However, instead of the global DSQ fallback, I was wondering if it'd be better to simply re-enqueue the task - setting SCX_ENQ_REENQ - if the target local DSQ isn't valid anymore when the dispatch is finalized.

In this way using SCX_DSQ_LOCAL_ON | cpu from ops.dispatch() would simply trigger a re-enqueue when "cpu" isn't valid anymore (due to concurrent affinity / migration disabled changes) and the BPF scheduler can handle that in another ops.enqueue().

What do you think?

Thanks,
-Andrea

#5 lore raw
Quoted 33 lines · Andrea Righi

On Mon, Feb 02, 2026 at 08:45:18AM +0100, Andrea Righi wrote:
...

Quoted 15 lines
Quoted 6 lines

So I have finally gotten around updating scx_storm to the new semantics, see:
https://github.com/cloehle/scx/tree/cloehle/scx-storm-qmap-insert-local-dequeue-semantics

I don't think the new ops.dequeue() are enough to make inserts to local-on from anywhere safe, because it's still racing with dequeue from another CPU?

Yeah, with this patch set BPF schedulers get proper ops.dequeue() callbacks, but we're not fixing the usage of SCX_DSQ_LOCAL_ON from ops.dispatch().

When task properties change between scx_bpf_dsq_insert() and the actual dispatch, task_can_run_on_remote_rq() can still trigger a fatal scx_error().

Quoted 6 lines

The ops.dequeue(SCX_DEQ_SCHED_CHANGE) notifications happens after the property change, so it can't prevent already-queued dispatches from failing. The race window is between ops.dispatch() returning and dispatch_to_local_dsq() executing.

We can address this in a separate patch set. One thing at a time. :)

Thinking more on this, the problem is that we're passing enforce=true to task_can_run_on_remote_rq(), triggering a critical failure - scx_error(). There's a logic in task_can_run_on_remote_rq() to fallback to the global DSQ, that doesn't happen if we pass enforce=true, due to scx_error().

However, instead of the global DSQ fallback, I was wondering if it'd be better to simply re-enqueue the task - setting SCX_ENQ_REENQ - if the target local DSQ isn't valid anymore when the dispatch is finalized.

Quoted 6 lines

In this way using SCX_DSQ_LOCAL_ON | cpu from ops.dispatch() would simply trigger a re-enqueue when "cpu" isn't valid anymore (due to concurrent affinity / migration disabled changes) and the BPF scheduler can handle that in another ops.enqueue().

What do you think?

I think that's a lot more versatile for the BPF scheduler than using the global DSQ as fallback in that case, so yeah I'm all for it!

author reviewer
#6 lore raw
Quoted 39 lines · Christian Loehle
Quoted 38 lines · Andrea Righi

On Mon, Feb 02, 2026 at 08:45:18AM +0100, Andrea Righi wrote:
...

Quoted 15 lines
Quoted 6 lines

So I have finally gotten around updating scx_storm to the new semantics, see:
https://github.com/cloehle/scx/tree/cloehle/scx-storm-qmap-insert-local-dequeue-semantics

I don't think the new ops.dequeue() are enough to make inserts to local-on from anywhere safe, because it's still racing with dequeue from another CPU?

Yeah, with this patch set BPF schedulers get proper ops.dequeue() callbacks, but we're not fixing the usage of SCX_DSQ_LOCAL_ON from ops.dispatch().

When task properties change between scx_bpf_dsq_insert() and the actual dispatch, task_can_run_on_remote_rq() can still trigger a fatal scx_error().

Quoted 6 lines

The ops.dequeue(SCX_DEQ_SCHED_CHANGE) notifications happens after the property change, so it can't prevent already-queued dispatches from failing. The race window is between ops.dispatch() returning and dispatch_to_local_dsq() executing.

We can address this in a separate patch set. One thing at a time. :)

Thinking more on this, the problem is that we're passing enforce=true to task_can_run_on_remote_rq(), triggering a critical failure - scx_error(). There's a logic in task_can_run_on_remote_rq() to fallback to the global DSQ, that doesn't happen if we pass enforce=true, due to scx_error().

However, instead of the global DSQ fallback, I was wondering if it'd be better to simply re-enqueue the task - setting SCX_ENQ_REENQ - if the target local DSQ isn't valid anymore when the dispatch is finalized.

In this way using SCX_DSQ_LOCAL_ON | cpu from ops.dispatch() would simply trigger a re-enqueue when "cpu" isn't valid anymore (due to concurrent affinity / migration disabled changes) and the BPF scheduler can handle that in another ops.enqueue().

Quoted 5 lines
Quoted 1 line

What do you think?

I think that's a lot more versatile for the BPF scheduler than using the global DSQ as fallback in that case, so yeah I'm all for it!

Ack, I already have a working patch do to this, I'll post it as a separate patch set.

Thanks,
-Andrea

#7 lore raw
In reply to Andrea Righi
Quoted 95 lines · Andrea Righi

Hi Christian,

Quoted 45 lines · Christian Loehle
Quoted 38 lines · Andrea Righi

Currently, ops.dequeue() is only invoked when the sched_ext core knows that a task resides in BPF-managed data structures, which causes it to miss scheduling property change events. In addition, ops.dequeue() callbacks are completely skipped when tasks are dispatched to non-local DSQs from ops.select_cpu(). As a result, BPF schedulers cannot reliably track task state.

Fix this by guaranteeing that each task entering the BPF scheduler's custody triggers exactly one ops.dequeue() call when it leaves that custody, whether the exit is due to a dispatch (regular or via a core scheduling pick) or to a scheduling property change (e.g. sched_setaffinity(), sched_setscheduler(), set_user_nice(), NUMA balancing, etc.).

BPF scheduler custody concept: a task is considered to be in "BPF scheduler's custody" when it has been queued in BPF-managed data structures and the BPF scheduler is responsible for its lifecycle. Custody ends when the task is dispatched to a local DSQ, selected by core scheduling, or removed due to a property change.

Tasks directly dispatched to local DSQs (via %SCX_DSQ_LOCAL or %SCX_DSQ_LOCAL_ON) bypass the BPF scheduler entirely and are not in its custody. As a result, ops.dequeue() is not invoked for these tasks.

To identify dequeues triggered by scheduling property changes, introduce the new ops.dequeue() flag %SCX_DEQ_SCHED_CHANGE: when this flag is set, the dequeue was caused by a scheduling property change.

New ops.dequeue() semantics:
- ops.dequeue() is invoked exactly once when the task leaves the BPF scheduler's custody, in one of the following cases:
a) regular dispatch: task was dispatched to a non-local DSQ (global or user DSQ), ops.dequeue() called without any special flags set
b) core scheduling dispatch: core-sched picks task before dispatch, dequeue called with %SCX_DEQ_CORE_SCHED_EXEC flag set
c) property change: task properties modified before dispatch, dequeue called with %SCX_DEQ_SCHED_CHANGE flag set

Quoted 5 lines

This allows BPF schedulers to:
- reliably track task ownership and lifecycle,
- maintain accurate accounting of managed tasks,
- update internal state when tasks change properties.

Quoted 6 lines

So I have finally gotten around updating scx_storm to the new semantics, see:
https://github.com/cloehle/scx/tree/cloehle/scx-storm-qmap-insert-local-dequeue-semantics

I don't think the new ops.dequeue() are enough to make inserts to local-on from anywhere safe, because it's still racing with dequeue from another CPU?

Yeah, with this patch set BPF schedulers get proper ops.dequeue() callbacks, but we're not fixing the usage of SCX_DSQ_LOCAL_ON from ops.dispatch().

When task properties change between scx_bpf_dsq_insert() and the actual dispatch, task_can_run_on_remote_rq() can still trigger a fatal scx_error().

The ops.dequeue(SCX_DEQ_SCHED_CHANGE) notifications happens after the property change, so it can't prevent already-queued dispatches from failing. The race window is between ops.dispatch() returning and dispatch_to_local_dsq() executing.

We can address this in a separate patch set. One thing at a time. :)

Quoted 11 lines

Furthermore I can reproduce the following with this patch applied quite easily with something like

hackbench -l 1000 & timeout 10 ./build/scheds/c/scx_storm

[   44.356878] sched_ext: BPF scheduler "simple" enabled
[   59.315370] sched_ext: BPF scheduler "simple" disabled (unregistered from user space)
[   85.366747] sched_ext: BPF scheduler "storm" enabled
[   85.371324] ------------[ cut here ]------------
[   85.373370] WARNING: kernel/sched/sched.h:1571 at update_locked_rq+0x64/0x6c, CPU#5: gmain/1111

Ah yes! I think I see it, can you try this on top?

Thanks,

(quoted diff) +0 -1
-Andrea
 
 kernel/sched/ext.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/sched/ext.c +0 -0
index 6d6f1253039d8..d8fed4a49195d 100644

kernel/sched/ext.c static void finish_dispatch(struct scx_sched *sch, struct rq *rq, · line 2254

Quoted 8 lines
kernel/sched/ext.c +1 -1
@@ -2248,7 +2248,7 @@ static void finish_dispatch(struct scx_sched *sch, struct rq *rq,
 			p->scx.flags |= SCX_TASK_OPS_ENQUEUED;
 		} else {
 			if (p->scx.flags & SCX_TASK_OPS_ENQUEUED)
-				SCX_CALL_OP_TASK(sch, SCX_KF_REST, dequeue, task_rq(p), p, 0);
+				SCX_CALL_OP_TASK(sch, SCX_KF_REST, dequeue, rq, p, 0);
 
 			p->scx.flags &= ~SCX_TASK_OPS_ENQUEUED;
 		}

Yup, that fixes it!

#8 lore raw
In reply to Andrea Righi

Hi Andrea,

kernel/sched/ext.c static void finish_dispatch(struct scx_sched *sch, struct rq *rq, · line 2254

Quoted 13 lines · Andrea Righi
kernel/sched/ext.c +1 -1
index 6d6f1253039d8..d8fed4a49195d 100644
@@ -2248,7 +2248,7 @@ static void finish_dispatch(struct scx_sched *sch, struct rq *rq,
 			p->scx.flags |= SCX_TASK_OPS_ENQUEUED;
 		} else {
 			if (p->scx.flags & SCX_TASK_OPS_ENQUEUED)
-				SCX_CALL_OP_TASK(sch, SCX_KF_REST, dequeue, task_rq(p), p, 0);
+				SCX_CALL_OP_TASK(sch, SCX_KF_REST, dequeue, rq, p, 0);
 
 			p->scx.flags &= ~SCX_TASK_OPS_ENQUEUED;
 		}

This looks risky from a locking perspective. Are we relying on SCX_OPSS_DISPATCHING to protect against racing dequeues? If so, it might be worth calling out in a comment.

Thanks,
Kuba

author reviewer
#9 lore raw

Hi Kuba,

sorry for the late response.

Quoted 14 lines · Kuba Piecuch

Hi Andrea,

Quoted 11 lines · Andrea Righi
kernel/sched/ext.c +1 -1
index 6d6f1253039d8..d8fed4a49195d 100644
@@ -2248,7 +2248,7 @@ static void finish_dispatch(struct scx_sched *sch, struct rq *rq,
 			p->scx.flags |= SCX_TASK_OPS_ENQUEUED;
 		} else {
 			if (p->scx.flags & SCX_TASK_OPS_ENQUEUED)
-				SCX_CALL_OP_TASK(sch, SCX_KF_REST, dequeue, task_rq(p), p, 0);
+				SCX_CALL_OP_TASK(sch, SCX_KF_REST, dequeue, rq, p, 0);
 
Quoted 6 lines

kernel/sched/ext.c static void finish_dispatch(struct scx_sched *sch, struct rq *rq, · line 2254

Quoted 2 lines
kernel/sched/ext.c +0 -0
 			p->scx.flags &= ~SCX_TASK_OPS_ENQUEUED;
 		}

This looks risky from a locking perspective. Are we relying on SCX_OPSS_DISPATCHING to protect against racing dequeues? If so, it might be worth calling out in a comment.

You're right, we're relying on SCX_OPSS_DISPATCHING to protect against racing dequeues and this definitely deserves a comment. How about something like the following?

Thanks,
-Andrea

---

kernel/sched/ext.c +9 -0
index 292adf10fee1b..b189339e74101 100644
@@ -2260,6 +2260,15 @@ static void finish_dispatch(struct scx_sched *sch, struct rq *rq,
 		if (!is_terminal_dsq(dsq_id)) {
 			p->scx.flags |= SCX_TASK_OPS_ENQUEUED;
 		} else {
+			/*
+			 * Locking: we're holding the @rq lock (the
+			 * dispatch CPU's rq), but not necessarily
+			 * task_rq(p), since @p may be from a remote CPU.
+			 *
+			 * This is safe because SCX_OPSS_DISPATCHING state
+			 * prevents racing dequeues, any concurrent
+			 * ops_dequeue() will wait for this state to clear.
+			 */
 			if (p->scx.flags & SCX_TASK_OPS_ENQUEUED)
 				SCX_CALL_OP_TASK(sch, SCX_KF_REST, dequeue, rq, p, 0);
 
#10 lore raw
Quoted 9 lines · Andrea Righi
kernel/sched/ext.c +1 -0
index 292adf10fee1b..b189339e74101 100644
@@ -2260,6 +2260,15 @@ static void finish_dispatch(struct scx_sched *sch, struct rq *rq,
 		if (!is_terminal_dsq(dsq_id)) {
 			p->scx.flags |= SCX_TASK_OPS_ENQUEUED;
 		} else {
+			/*

kernel/sched/ext.c static void finish_dispatch(struct scx_sched *sch, struct rq *rq, · line 2273

Quoted 10 lines
kernel/sched/ext.c +8 -0
+			 * Locking: we're holding the @rq lock (the
+			 * dispatch CPU's rq), but not necessarily
+			 * task_rq(p), since @p may be from a remote CPU.
+			 *
+			 * This is safe because SCX_OPSS_DISPATCHING state
+			 * prevents racing dequeues, any concurrent
+			 * ops_dequeue() will wait for this state to clear.
+			 */
 			if (p->scx.flags & SCX_TASK_OPS_ENQUEUED)
 				SCX_CALL_OP_TASK(sch, SCX_KF_REST, dequeue, rq, p, 0);

Looks good, thanks :)

#11 lore raw
In reply to Andrea Righi

Hi Andrea,

Looks good overall, but we need to settle on the global DSQ semantics, plus some edge cases that need clearing up.

Quoted 35 lines · Andrea Righi
Documentation/scheduler/sched-ext.rst +27 -0
index 404fe6126a769..6d9e82e6ca9d4 100644
@@ -252,6 +252,80 @@ The following briefly shows how a waking task is scheduled and executed.
 
    * Queue the task on the BPF side.
 
+   **Task State Tracking and ops.dequeue() Semantics**
+
+   Once ``ops.select_cpu()`` or ``ops.enqueue()`` is called, the task may
+   enter the "BPF scheduler's custody" depending on where it's dispatched:
+
+   * **Direct dispatch to local DSQs** (``SCX_DSQ_LOCAL`` or
+     ``SCX_DSQ_LOCAL_ON | cpu``): The task bypasses the BPF scheduler
+     entirely and goes straight to the CPU's local run queue. The task
+     never enters BPF custody, and ``ops.dequeue()`` will not be called.
+
+   * **Dispatch to non-local DSQs** (``SCX_DSQ_GLOBAL`` or custom DSQs):
+     the task enters the BPF scheduler's custody. When the task later
+     leaves BPF custody (dispatched to a local DSQ, picked by core-sched,
+     or dequeued for sleep/property changes), ``ops.dequeue()`` will be
+     called exactly once.
+
+   * **Queued on BPF side**: The task is in BPF data structures and in BPF
+     custody, ``ops.dequeue()`` will be called when it leaves.
+
+   The key principle: **ops.dequeue() is called when a task leaves the BPF
+   scheduler's custody**. A task is in BPF custody if it's on a non-local
+   DSQ or in BPF data structures. Once dispatched to a local DSQ or after
+   ops.dequeue() is called, the task is out of BPF custody and the BPF
+   scheduler no longer needs to track it.
+
+   This works correctly with the ``ops.select_cpu()`` direct dispatch
+   optimization: even though it skips ``ops.enqueue()`` invocation, if the

Documentation/scheduler/sched-ext.rst The following briefly shows how a waking task is scheduled and executed. · line 291

Quoted 10 lines
Documentation/scheduler/sched-ext.rst +10 -0
+   task is dispatched to a non-local DSQ, it enters BPF custody and will
+   get ``ops.dequeue()`` when it leaves. This provides the performance
+   benefit of avoiding the ``ops.enqueue()`` roundtrip while maintaining
+   correct state tracking.
+
+   The dequeue can happen for different reasons, distinguished by flags:
+
+   1. **Regular dispatch workflow**: when the task is dispatched from a
+      non-local DSQ to a local DSQ (leaving BPF custody for execution),
+      ``ops.dequeue()`` is triggered without any special flags.

Maybe add a note that this can happen asynchronously, without the BPF scheduler explicitly dispatching the task to a local DSQ, when the task is on a global DSQ? Or maybe make that case into a separate dequeue reason with its own flag, e.g. SCX_DEQ_PICKED_FROM_GLOBAL_DSQ?

include/linux/sched/ext.h struct scx_dispatch_q { · line 87

Quoted 9 lines
include/linux/sched/ext.h +1 -0
index bcb962d5ee7d8..0d003d2845393 100644
@@ -84,6 +84,7 @@ struct scx_dispatch_q {
 /* scx_entity.flags */
 enum scx_ent_flags {
 	SCX_TASK_QUEUED		= 1 << 0, /* on ext runqueue */
+	SCX_TASK_OPS_ENQUEUED	= 1 << 1, /* under ext scheduler's custody */

Nit: I think "in BPF scheduler's custody" would be a bit clearer, as "ext scheduler" could potentially be interpreted to mean SCHED_CLASS_EXT as a whole.

Quoted 14 lines
include/linux/sched/ext.h +10 -0
@@ -1523,6 +1603,30 @@ static void ops_dequeue(struct rq *rq, struct task_struct *p, u64 deq_flags)
 
 	switch (opss & SCX_OPSS_STATE_MASK) {
 	case SCX_OPSS_NONE:
+		/*
+		 * Task is not in BPF data structures (either dispatched to
+		 * a DSQ or running). Only call ops.dequeue() if the task
+		 * is still in BPF scheduler's custody
+		 * (%SCX_TASK_OPS_ENQUEUED is set).
+		 *
+		 * If the task has already been dispatched to a local DSQ
+		 * (left BPF custody), the flag will be clear and we skip
+		 * ops.dequeue()
+		 *

include/linux/sched/ext.h static void ops_dequeue(struct rq *rq, struct task_struct *p, u64 deq_flags) · line 1625

Quoted 10 lines
include/linux/sched/ext.h +10 -0
+		 * If this is a property change (not sleep/core-sched) and
+		 * the task is still in BPF custody, set the
+		 * %SCX_DEQ_SCHED_CHANGE flag.
+		 */
+		if (SCX_HAS_OP(sch, dequeue) &&
+		    p->scx.flags & SCX_TASK_OPS_ENQUEUED) {
+			u64 flags = deq_flags;
+
+			if (!(deq_flags & (DEQUEUE_SLEEP | SCX_DEQ_CORE_SCHED_EXEC)))
+				flags |= SCX_DEQ_SCHED_CHANGE;

I think this logic will result in ops.dequeue(SCHED_CHANGE) being called for tasks being picked from a global DSQ being migrated from a remote rq to the local rq, which, while technically correct since the task is migrating rqs, may be confusing, since it fits two cases in the documentation:

* Since the task is leaving BPF custody for execution, ops.dequeue() should be called without any special flags.
* Since the task is being migrated between rqs, ops.dequeue() should be called with SCX_DEQ_SCHED_CHANGE.

include/linux/sched/ext.h static void ops_dequeue(struct rq *rq, struct task_struct *p, u64 deq_flags) · line 1632

Quoted 7 lines
include/linux/sched/ext.h +4 -0
+
+			SCX_CALL_OP_TASK(sch, SCX_KF_REST, dequeue, rq, p, flags);
+			p->scx.flags &= ~SCX_TASK_OPS_ENQUEUED;
+		}
 		break;
 	case SCX_OPSS_QUEUEING:
 		/*

Thanks,
Kuba

author reviewer
#12 lore raw
Quoted 4 lines · Kuba Piecuch

Hi Andrea,

Looks good overall, but we need to settle on the global DSQ semantics, plus some edge cases that need clearing up.

On this one I think we settled on the assumption that SCX_DSQ_GLOBAL can be considered a "terminal DSQ", so we won't trigger ops.dequeue().

Quoted 44 lines · Andrea Righi
Quoted 42 lines
Documentation/scheduler/sched-ext.rst +34 -0
index 404fe6126a769..6d9e82e6ca9d4 100644
@@ -252,6 +252,80 @@ The following briefly shows how a waking task is scheduled and executed.
 
    * Queue the task on the BPF side.
 
+   **Task State Tracking and ops.dequeue() Semantics**
+
+   Once ``ops.select_cpu()`` or ``ops.enqueue()`` is called, the task may
+   enter the "BPF scheduler's custody" depending on where it's dispatched:
+
+   * **Direct dispatch to local DSQs** (``SCX_DSQ_LOCAL`` or
+     ``SCX_DSQ_LOCAL_ON | cpu``): The task bypasses the BPF scheduler
+     entirely and goes straight to the CPU's local run queue. The task
+     never enters BPF custody, and ``ops.dequeue()`` will not be called.
+
+   * **Dispatch to non-local DSQs** (``SCX_DSQ_GLOBAL`` or custom DSQs):
+     the task enters the BPF scheduler's custody. When the task later
+     leaves BPF custody (dispatched to a local DSQ, picked by core-sched,
+     or dequeued for sleep/property changes), ``ops.dequeue()`` will be
+     called exactly once.
+
+   * **Queued on BPF side**: The task is in BPF data structures and in BPF
+     custody, ``ops.dequeue()`` will be called when it leaves.
+
+   The key principle: **ops.dequeue() is called when a task leaves the BPF
+   scheduler's custody**. A task is in BPF custody if it's on a non-local
+   DSQ or in BPF data structures. Once dispatched to a local DSQ or after
+   ops.dequeue() is called, the task is out of BPF custody and the BPF
+   scheduler no longer needs to track it.
+
+   This works correctly with the ``ops.select_cpu()`` direct dispatch
+   optimization: even though it skips ``ops.enqueue()`` invocation, if the
+   task is dispatched to a non-local DSQ, it enters BPF custody and will
+   get ``ops.dequeue()`` when it leaves. This provides the performance
+   benefit of avoiding the ``ops.enqueue()`` roundtrip while maintaining
+   correct state tracking.
+
+   The dequeue can happen for different reasons, distinguished by flags:
+
Quoted 8 lines

Documentation/scheduler/sched-ext.rst The following briefly shows how a waking task is scheduled and executed. · line 291

Quoted 3 lines
Documentation/scheduler/sched-ext.rst +3 -0
+   1. **Regular dispatch workflow**: when the task is dispatched from a
+      non-local DSQ to a local DSQ (leaving BPF custody for execution),
+      ``ops.dequeue()`` is triggered without any special flags.

Maybe add a note that this can happen asynchronously, without the BPF scheduler explicitly dispatching the task to a local DSQ, when the task is on a global DSQ? Or maybe make that case into a separate dequeue reason with its own flag, e.g. SCX_DEQ_PICKED_FROM_GLOBAL_DSQ?

And I guess we don't need this if we consider SCX_DSQ_GLOBAL as a terminal DSQ, because we won't trigger ops.dequeue().

Quoted 14 lines

include/linux/sched/ext.h struct scx_dispatch_q { · line 87

Quoted 10 lines
include/linux/sched/ext.h +1 -0
index bcb962d5ee7d8..0d003d2845393 100644
@@ -84,6 +84,7 @@ struct scx_dispatch_q {
 /* scx_entity.flags */
 enum scx_ent_flags {
 	SCX_TASK_QUEUED		= 1 << 0, /* on ext runqueue */
+	SCX_TASK_OPS_ENQUEUED	= 1 << 1, /* under ext scheduler's custody */

Nit: I think "in BPF scheduler's custody" would be a bit clearer, as "ext scheduler" could potentially be interpreted to mean SCHED_CLASS_EXT as a whole.

Ack. Will change that.

Quoted 31 lines
Quoted 15 lines
include/linux/sched/ext.h +10 -0
@@ -1523,6 +1603,30 @@ static void ops_dequeue(struct rq *rq, struct task_struct *p, u64 deq_flags)
 
 	switch (opss & SCX_OPSS_STATE_MASK) {
 	case SCX_OPSS_NONE:
+		/*
+		 * Task is not in BPF data structures (either dispatched to
+		 * a DSQ or running). Only call ops.dequeue() if the task
+		 * is still in BPF scheduler's custody
+		 * (%SCX_TASK_OPS_ENQUEUED is set).
+		 *
+		 * If the task has already been dispatched to a local DSQ
+		 * (left BPF custody), the flag will be clear and we skip
+		 * ops.dequeue()
+		 *

include/linux/sched/ext.h static void ops_dequeue(struct rq *rq, struct task_struct *p, u64 deq_flags) · line 1625

Quoted 10 lines
include/linux/sched/ext.h +10 -0
+		 * If this is a property change (not sleep/core-sched) and
+		 * the task is still in BPF custody, set the
+		 * %SCX_DEQ_SCHED_CHANGE flag.
+		 */
+		if (SCX_HAS_OP(sch, dequeue) &&
+		    p->scx.flags & SCX_TASK_OPS_ENQUEUED) {
+			u64 flags = deq_flags;
+
+			if (!(deq_flags & (DEQUEUE_SLEEP | SCX_DEQ_CORE_SCHED_EXEC)))
+				flags |= SCX_DEQ_SCHED_CHANGE;

I think this logic will result in ops.dequeue(SCHED_CHANGE) being called for tasks being picked from a global DSQ being migrated from a remote rq to the local rq, which, while technically correct since the task is migrating rqs, may be confusing, since it fits two cases in the documentation:

Quoted 4 lines

* Since the task is leaving BPF custody for execution, ops.dequeue() should be called without any special flags.
* Since the task is being migrated between rqs, ops.dequeue() should be called with SCX_DEQ_SCHED_CHANGE.

This also should be fixed with the new logic, because a task disptched to a global DSQ is considered outside of the BPF scheduler's custody, so ops.dequeue() is not invoked at all.

I'll post a new patch set later today, so we can better discuss if all these assumptions have been addressed properly. :)

Quoted 11 lines

include/linux/sched/ext.h static void ops_dequeue(struct rq *rq, struct task_struct *p, u64 deq_flags) · line 1632

Quoted 8 lines
include/linux/sched/ext.h +4 -0
+
+			SCX_CALL_OP_TASK(sch, SCX_KF_REST, dequeue, rq, p, flags);
+			p->scx.flags &= ~SCX_TASK_OPS_ENQUEUED;
+		}
 		break;
 	case SCX_OPSS_QUEUEING:
 		/*

Thanks,
Kuba

Thanks,
-Andrea

#13 lore raw
Quoted 8 lines · Andrea Righi
Quoted 4 lines · Kuba Piecuch

Hi Andrea,

Looks good overall, but we need to settle on the global DSQ semantics, plus some edge cases that need clearing up.

On this one I think we settled on the assumption that SCX_DSQ_GLOBAL can be considered a "terminal DSQ", so we won't trigger ops.dequeue().

Correct, I made this comment before we settled it.

Thanks,
Kuba

author reviewer
#14 lore raw

PATCH 2/2 selftests/sched_ext: Add test to validate ops.dequeue() semantics

In reply to Andrea Righi

Add a new kselftest to validate the following scenarios:
- scenario 0 (Local DSQ): tasks don't increment enqueue counters since they never enter BPF scheduler custody (no tracking or validation, expects 0 enqueues and 0 dequeues),
- scenario 1 (User DSQ): full enqueue/dequeue lifecycle tracking with state machine validation (expects 1:1 enqueue/dequeue pairing).

The test validates that:
- local DSQ dispatch don't trigger ops.dequeue(),
- non-local DSQ dispatch has exact 1:1 ops.enqueue/dequeue() pairing,
- dispatch dequeues have no flags (normal workflow),
- property change dequeues have %SCX_DEQ_SCHED_CHANGE flag,
- no duplicate enqueues or invalid state transitions are happening.

This validates that the new ops.dequeue() semantics work correctly for all task lifecycle scenarios.

Cc Tejun Heo <[email protected]>
Cc Emil Tsalapatis <[email protected]>
Cc Kuba Piecuch <[email protected]>
---
 tools/testing/selftests/sched_ext/Makefile    |   1 +
 .../testing/selftests/sched_ext/dequeue.bpf.c | 238 ++++++++++++++++++
 tools/testing/selftests/sched_ext/dequeue.c   | 201 +++++++++++++++
 3 files changed, 440 insertions(+)
 create mode 100644 tools/testing/selftests/sched_ext/dequeue.bpf.c
 create mode 100644 tools/testing/selftests/sched_ext/dequeue.c
tools/testing/selftests/sched_ext/Makefile +1 -0
index 5fe45f9c5f8fd..764e91edabf93 100644
@@ -161,6 +161,7 @@ all_test_bpfprogs := $(foreach prog,$(wildcard *.bpf.c),$(INCLUDE_DIR)/$(patsubs
 
 auto-test-targets :=			\
 	create_dsq			\
+	dequeue				\
 	enq_last_no_enq_fails		\
 	ddsp_bogus_dsq_fail		\
 	ddsp_vtimelocal_fail		\
tools/testing/selftests/sched_ext/dequeue.bpf.c new file +238 -0

238 lines not on this page · raw message

tools/testing/selftests/sched_ext/dequeue.c new file +201 -0

201 lines not on this page · raw message

Signature · 1 line
2.52.0