sched: Make proxy execution compatible with sched_ext
Subsystems (3)
Andrea Righi
reviewerPATCH v2 1/1 sched: Make proxy execution compatible with sched_ext
This series enables using proxy execution with sched_ext and is based on early work by John Stultz [1].
Background
==========
Proxy execution (proxy-exec) lets a waiting task ("donor") donate its execution context to a mutex owner, so the owner can run while the donor stays eligible on the runqueue.
Currently, proxy execution and sched_ext are mutually exclusive at build time:
we can't enable CONFIG_SCHED_PROXY_EXEC=y and CONFIG_SCHED_CLASS_EXT=y in the same kernel.
This restriction can be problematic for Linux distributions and for anyone who wants to ship one kernel and choose features at runtime.
Why they are mutually exclusive?
================================
sched_ext schedulers drive dispatch through their own interfaces. A proxy-exec handoff can run a task that the BPF scheduler never dispatched through that path. sched_ext callbacks then observe a "current" task that does not match what the BPF side considers running, so kfuncs and helper state can see an inconsistent view of the executing task.
sched_ext also tracks runnable work through Dispatch Queues (DSQs) and BPF chosen dispatch rules, while the core scheduler still maintains classic per-CPU runqueues and pick paths. A proxy handoff can therefore switch the CPU to a task that the BPF scheduler never inserted or ordered through its DSQ interface.
DSQ state, vtime, and "who is running" bookkeeping inside the BPF program can then disagree with what the core actually executes, so helpers and kfuncs that assume their dispatched task is current may observe stale or inconsistent state.
Design: supporting proxy execution with sched_ext
=================================================
Proxy execution support is an optional per-scheduler capability: a BPF scheduler can set SCX_OPS_ENQ_BLOCKED to receive mutex blocked tasks, without this flag mutex waiters block normally and proxy execution is automaticaly disabled. When this flag is set, blocked donors are passed through ops.enqueue(), where scx_bpf_task_is_blocked() lets BPF recognize them and apply its own admission and ordering policy.
Knowing the mutex owner's location is optional, BPF may enqueue a donor on its current CPU's local DSQ and let the core resolve the owner and perform the proxy exec handoff. Schedulers that want to reduce handoff latency can use scx_bpf_task_proxy_cpu() or scx_bpf_task_proxy_cid() to obtain the owner location and steer the donation there when affinity constraints and policy allow it. The core still revalidates the owner relationship to perform the actual migration.
The donor-to-owner handoff is modeled like a "function call" from the scheduler's perspective. The donor remains the running scheduling entity selected by BPF: its scheduling context, runtime and slice are consumed while the core temporarily invokes the mutex owner's code to make the critical section progress. It is not a scheduler-visible switch to the owner.
Accordingly, the donor remains the scheduling context presented to sched_ext, while the mutex owner is treated solely as the execution context selected internally by the core scheduler. Scheduling state is accounted against rq->donor where appropriate, while rq->curr identifies the execution context. The internal owner substitution does not generate synthetic sched_ext callbacks for a task that BPF did not dispatch.
The sched_ext callback bookkeeping is adjusted accordingly. Blocked proxy donors do not generate spurious ops.running() callbacks and ops.stopping() is only called when ops.running() was emitted. The normal sched_ext migration path also leaves blocked donors to the proxy machinery, and cross-CPU migration is avoided for migration-disabled and single-CPU tasks.
scx_qmap is modified to demonstrate the opt-in policy: the scheduler immediately admits blocked donors, prefers the owner's cid when allowed by the donor's affinity and inserts the donor at the head of the selected local DSQ.
A new kselftest (enq_blocked) is also introduced to validate the proxy execution support with sched_ext. The test creates a three-task priority inversion on one
CPU: a low-priority owner (nice +19) holds a kernel mutex, a high-priority donor (nice -20) blocks on it and a nice 0 contender competes for the CPU. It runs the same workload with SCX_OPS_ENQ_BLOCKED disabled and enabled, validates blocked donor admission, the reported proxy CPU and prints average mutex hold/wait times with their deltas for manual comparison without enforcing performance thresholds. Access to the mutex is provided by a loadable kernel module built via TEST_GEN_MODS_DIR, with the test responsible for loading, unloading, and managing the module's lifecycle.
Example kselftest run:
$ sudo tools/testing/selftests/sched_ext/runner -t enq_blocked ===== START ===== TEST: enq_blocked DESCRIPTION: Verify BPF-driven proxy donor admission OUTPUT:
[SCX_OPS_ENQ_BLOCKED=disabled]
proxy_exec=enabled
owner_nice=19
donor_nice=-20
contender_nice=0
mutex_hold_avg_ns=254084719 (254.084 ms, samples=10)
mutex_wait_avg_ns=254095120 (254.095 ms, samples=10)
nr_blocked_enqueues=0[SCX_OPS_ENQ_BLOCKED=enabled]
proxy_exec=enabled
owner_nice=19
donor_nice=-20
contender_nice=0
mutex_hold_avg_ns=228884734 (228.884 ms, samples=10)
mutex_wait_avg_ns=207903720 (207.903 ms, samples=10)
nr_blocked_enqueues=51[delta: enabled - disabled]
mutex_hold_delta_ns=-25199985 (-9.92%)
mutex_wait_delta_ns=-46191400 (-18.18%) ok 1 enq_blocked #
===== END =====
=============================
RESULTS:
PASSED: 1
SKIPPED: 0
FAILED: 0
References
==========
[1] https://lore.kernel.org/all/[email protected]
Git tree: git://git.kernel.org/pub/scm/linux/kernel/git/arighi/linux.git scx-proxy-exec
Changes in v2:
- Rebased onto sched_ext/for-7.3 and adapted the series to the split sched_ext implementation and cid-form scheduler interfaces.
- Replaced the global sched_proxy_exec_scx boot-time opt-in with the per-scheduler SCX_OPS_ENQ_BLOCKED capability, allowing BPF to control donor admission and ordering through ops.enqueue().
- Added scx_bpf_task_is_blocked(), scx_bpf_task_proxy_cpu(), and scx_bpf_task_proxy_cid(); enforce CPU/cid API separation for cid-form schedulers.
- Added proxy exec support to scx_qmap, including optional owner-cid steering, affinity validation, and fallback to the donor's current cid.
- Added a kselftest with a kernel mutex test and a three-task priority inversion workload, test is executed with blocked task admission disabled and enabled, validate the behavior, and report hold/wait-time deltas.
- Link to v1: https://lore.kernel.org/all/[email protected]/
Andrea Righi (10):
sched/core: Skip migration disabled tasks in proxy execution sched/core: Skip put_prev_task/set_next_task re-entry for sched_ext donors
sched_ext: Fix TOCTOU race in consume_remote_task()
sched_ext: Fix ops.running/stopping() pairing for proxy-exec donors
sched_ext: Save/restore kf_tasks[] when task ops nest
sched_ext: Skip ops.runnable() when nested in SCX_CALL_OP_TASK
sched_ext: Delegate proxy donor admission to BPF schedulers
sched_ext: Add selftest for blocked donor admission
sched_ext: scx_qmap: Add proxy execution support
sched: Allow enabling proxy exec with sched_ext
John Stultz (2):
sched/ext: Split curr|donor references properly sched/ext: Avoid migrating blocked tasks with proxy execution
include/linux/sched/ext.h | 9 + init/Kconfig | 2 - kernel/sched/core.c | 59 +- kernel/sched/ext/ext.c | 209 +++++- kernel/sched/ext/ext.h | 8 + kernel/sched/ext/internal.h | 58 +- kernel/sched/sched.h | 6 + tools/sched_ext/include/scx/common.bpf.h | 3 + tools/sched_ext/include/scx/compat.h | 1 + tools/sched_ext/scx_qmap.bpf.c | 18 +- tools/testing/selftests/sched_ext/.gitignore | 4 + tools/testing/selftests/sched_ext/Makefile | 2 + tools/testing/selftests/sched_ext/config | 2 + .../selftests/sched_ext/enq_blocked.bpf.c | 116 +++ .../testing/selftests/sched_ext/enq_blocked.c | 682 ++++++++++++++++++ .../testing/selftests/sched_ext/enq_blocked.h | 21 + .../selftests/sched_ext/test_modules/Makefile | 13 + .../test_modules/scx_enq_blocked_test.c | 134 ++++ 18 files changed, 1299 insertions(+), 48 deletions(-) create mode 100644 tools/testing/selftests/sched_ext/enq_blocked.bpf.c create mode 100644 tools/testing/selftests/sched_ext/enq_blocked.c create mode 100644 tools/testing/selftests/sched_ext/enq_blocked.h create mode 100644 tools/testing/selftests/sched_ext/test_modules/Makefile create mode 100644 tools/testing/selftests/sched_ext/test_modules/scx_enq_blocked_test.c
Andrea Righi
reviewerPATCH 1/12 sched/core: Skip migration disabled tasks in proxy execution
Never attempt to migrate migration-disabled tasks or tasks that can only run on a single CPU when switching donor's execution context, preventing task pinning violations.
--- kernel/sched/core.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+)
kernel/sched/core.c +14 -0
@@ -6936,6 +6936,20 @@ find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf)*/ if (curr_in_chain)return proxy_resched_idle (rq); +/* +* Tasks pinned to a single CPU (per-CPU kthreads via +* kthread_bind(), tasks under migrate_disable()) cannot +* be moved to @owner_cpu. proxy_migrate_task() uses +* __set_task_cpu() which would silently violate the +* pinning and leave the task to run on a CPU outside +* its cpus_ptr once it is unblocked. Deactivate it on +* this CPU; the owner running elsewhere will wake @p +* back up when the mutex becomes available. +*/ +if (p->nr_cpus_allowed ==1 ||is_migration_disabled (p)) { +__clear_task_blocked_on (p,NULL ); + goto deactivate; + } goto migrate_task; }
Signature · 1 line
2.55.0
K Prateek Nayak
reviewerRe: [PATCH 01/12] sched/core: Skip migration disabled tasks in proxy execution
Hello Andrea,
Quoted 14 lines · Andrea Righi
Never attempt to migrate migration-disabled tasks or tasks that can only run on a single CPU when switching donor's execution context, preventing task pinning violations.
Signed-off-by Andrea Righi <[email protected]>--- kernel/sched/core.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+)kernel/sched/core.c +0
-0
kernel/sched/core.c find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf) · line 6942
Quoted 7 lines
kernel/sched/core.c +4
-0@@ -6936,6 +6936,20 @@ find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf)*/ if (curr_in_chain)return proxy_resched_idle (rq); +/* +* Tasks pinned to a single CPU (per-CPU kthreads via +* kthread_bind(), tasks under migrate_disable()) cannot +* be moved to @owner_cpu. proxy_migrate_task() uses
We only move it to donate the vruntime context. It is never actually run there.
kernel/sched/core.c find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf) · line 6945
Quoted 3 lines
kernel/sched/core.c +3
-0+* __set_task_cpu() which would silently violate the +* pinning and leave the task to run on a CPU outside +* its cpus_ptr once it is unblocked. Deactivate it on
For the task to run as normal, p->is_blocked needs to be cleared. It is only done in the wakeup path (and sometimes in find_proxy_task() if the task is rq->curr) which ensures all the affinity / migrate disable bits get fixed when the task gets to actually run.
Where / how is this being violated?
kernel/sched/core.c find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf) · line 6952
Quoted 7 lines
kernel/sched/core.c +7
-0+* this CPU; the owner running elsewhere will wake @p +* back up when the mutex becomes available. +*/ +if (p->nr_cpus_allowed ==1 ||is_migration_disabled (p)) { +__clear_task_blocked_on (p,NULL ); + goto deactivate; + }
Proxy depends on the ability to migrate the donor context to owner's CPU without actually running the task. The task is superficially on the CPU only to give it's runtime share to the lock owner. Feels like you are tripping a shortcoming in ext core if this is a problem.
Quoted 3 lines
kernel/sched/core.c +0
-0goto migrate_task; }
Signature · 2 lines
Thanks and Regards, Prateek
Andrea Righi
reviewerRe: [PATCH 01/12] sched/core: Skip migration disabled tasks in proxy execution
Hi Prateek,
Quoted 39 lines · K Prateek Nayak
Hello Andrea,
Quoted 14 lines · Andrea Righi
Never attempt to migrate migration-disabled tasks or tasks that can only run on a single CPU when switching donor's execution context, preventing task pinning violations.
Signed-off-by Andrea Righi <[email protected]>--- kernel/sched/core.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+)kernel/sched/core.c +0
-0kernel/sched/core.c find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf) · line 6942
Quoted 7 lines
kernel/sched/core.c +4
-0@@ -6936,6 +6936,20 @@ find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf)*/ if (curr_in_chain)return proxy_resched_idle (rq); +/* +* Tasks pinned to a single CPU (per-CPU kthreads via +* kthread_bind(), tasks under migrate_disable()) cannot +* be moved to @owner_cpu. proxy_migrate_task() uses We only move it to donate the vruntime context. It is never actually run there.
kernel/sched/core.c find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf) · line 6945
Quoted 3 lines
kernel/sched/core.c +3
-0+* __set_task_cpu() which would silently violate the +* pinning and leave the task to run on a CPU outside +* its cpus_ptr once it is unblocked. Deactivate it on For the task to run as normal,
p->is_blockedneeds to be cleared. It is only done in the wakeup path (and sometimes infind_proxy_task()if the task isrq->curr) which ensures all the affinity / migrate disable bits get fixed when the task gets to actually run.Where / how is this being violated?
Quoted 12 lines
kernel/sched/core.c find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf) · line 6952
Quoted 7 lines
kernel/sched/core.c +7
-0+* this CPU; the owner running elsewhere will wake @p +* back up when the mutex becomes available. +*/ +if (p->nr_cpus_allowed ==1 ||is_migration_disabled (p)) { +__clear_task_blocked_on (p,NULL ); + goto deactivate; + }Proxy depends on the ability to migrate the donor context to owner's CPU without actually running the task. The task is superficially on the CPU only to give it's runtime share to the lock owner. Feels like you are tripping a shortcoming in ext core if this is a problem.
Agree with all of the above. As I mention in the other email, I had this in the previous version, because I was hitting migration-disabled errors. Maybe they were related to an issue in the sched_ext core, I'll remove this and repeat my tests.
Thanks,
-Andrea
Peter Zijlstra
maintainerRe: [PATCH 01/12] sched/core: Skip migration disabled tasks in proxy execution
In reply to Andrea RighiQuoted 17 lines · Andrea Righi
Never attempt to migrate migration-disabled tasks or tasks that can only run on a single CPU when switching donor's execution context, preventing task pinning violations.
Signed-off-by Andrea Righi <[email protected]>--- kernel/sched/core.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+)kernel/sched/core.c +0
-0@@ -6936,6 +6936,20 @@ find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf)*/ if (curr_in_chain)return proxy_resched_idle (rq);
kernel/sched/core.c find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf) · line 6948
Quoted 10 lines
kernel/sched/core.c +10
-0+/* +* Tasks pinned to a single CPU (per-CPU kthreads via +* kthread_bind(), tasks under migrate_disable()) cannot +* be moved to @owner_cpu. proxy_migrate_task() uses +* __set_task_cpu() which would silently violate the +* pinning and leave the task to run on a CPU outside +* its cpus_ptr once it is unblocked. Deactivate it on +* this CPU; the owner running elsewhere will wake @p +* back up when the mutex becomes available. +*/
No, this is actually OK. Remember, we only migrate the scheduling context, but the task as such won't ever execute on the remote CPU.
Andrea Righi
reviewerRe: [PATCH 01/12] sched/core: Skip migration disabled tasks in proxy execution
Hi Peter,
Quoted 23 lines · Peter Zijlstra
Quoted 22 lines · Andrea Righi
Never attempt to migrate migration-disabled tasks or tasks that can only run on a single CPU when switching donor's execution context, preventing task pinning violations.
Signed-off-by Andrea Righi <[email protected]>--- kernel/sched/core.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+)kernel/sched/core.c +5
-0@@ -6936,6 +6936,20 @@ find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf)*/ if (curr_in_chain)return proxy_resched_idle (rq); +/* +* Tasks pinned to a single CPU (per-CPU kthreads via +* kthread_bind(), tasks under migrate_disable()) cannot +* be moved to @owner_cpu. proxy_migrate_task() uses +* __set_task_cpu() which would silently violate the
Quoted 8 lines
kernel/sched/core.c find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf) · line 6948
Quoted 5 lines
kernel/sched/core.c +5
-0+* pinning and leave the task to run on a CPU outside +* its cpus_ptr once it is unblocked. Deactivate it on +* this CPU; the owner running elsewhere will wake @p +* back up when the mutex becomes available. +*/ No, this is actually OK. Remember, we only migrate the scheduling context, but the task as such won't ever execute on the remote CPU.
Yeah... makes sense. This patch is actually coming from the previous series, maybe I was hitting a different issue with the sched_ext core. I'll remove this one and repeat my tests.
Thanks,
-Andrea
Andrea Righi
reviewerPATCH 2/12 sched/core: Skip put_prev_task/set_next_task re-entry for sched_ext donors
In reply to Andrea RighiIn __schedule(), the proxy-exec donor-stabilization block calls put_prev_task() and set_next_task() when rq->donor == prev_donor and prev != next.
For sched_ext tasks, re-entering set_next_task_scx() for a donor that has already been seen by BPF ops.running via the normal pick path causes issues. It fires SCX_CALL_OP_TASK(sch, running, rq, donor) a second time, and sch->ops dispatch can land on a vtable slot in a state that yields a NULL function pointer or corrupts the stack.
Fix this by skipping the put_prev_task/set_next_task re-entry when the donor is in the ext_sched_class, since sched_ext tracks curr/donor itself.
--- kernel/sched/core.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)
kernel/sched/core.c +7 -2
@@ -7185,9 +7185,14 @@ static void __sched notrace __schedule(int sched_mode)* anything, since B == B. However, A might have * missed a RT/DL balance opportunity due to being * on_cpu. +* +* sched_ext tracks curr/donor itself; re-entering set_next_task_scx +* here dispatches through a stale/NULL BPF ops vtable. */ - donor->sched_class->put_prev_task (rq, donor, donor); - donor->sched_class->set_next_task (rq, donor, true); +if (donor->sched_class != &ext_sched_class) { + donor->sched_class->put_prev_task (rq, donor, donor); + donor->sched_class->set_next_task (rq, donor, true); + } } }else {rq_set_donor (rq, next);
Signature · 1 line
2.55.0
Peter Zijlstra
maintainerRe: [PATCH 02/12] sched/core: Skip put_prev_task/set_next_task re-entry for sched_ext donors
Quoted 10 lines · Andrea Righi
In
__schedule(), the proxy-exec donor-stabilization block callsput_prev_task()andset_next_task()whenrq->donor==prev_donorand prev != next.For
sched_exttasks, re-enteringset_next_task_scx()for a donor that has already been seen by BPF ops.running via the normal pick path causes issues. It firesSCX_CALL_OP_TASK(sch, running, rq, donor)a second time, andsch->opsdispatch can land on a vtable slot in a state that yields a NULL function pointer or corrupts the stack.
Quoted 3 lines
Fix this by skipping the put_prev_task/set_next_task re-entry when the donor is in the
ext_sched_class, sincesched_exttracks curr/donor itself.
This really sounds like a bug in ext; how is this different from the sched_change pattern doing a put/set cycle?
Andrea Righi
reviewerRe: [PATCH 02/12] sched/core: Skip put_prev_task/set_next_task re-entry for sched_ext donors
Quoted 11 lines · Peter Zijlstra
Quoted 10 lines · Andrea Righi
In
__schedule(), the proxy-exec donor-stabilization block callsput_prev_task()andset_next_task()whenrq->donor==prev_donorand prev != next.For
sched_exttasks, re-enteringset_next_task_scx()for a donor that has already been seen by BPF ops.running via the normal pick path causes issues. It firesSCX_CALL_OP_TASK(sch, running, rq, donor)a second time, andsch->opsdispatch can land on a vtable slot in a state that yields a NULL function pointer or corrupts the stack.
Quoted 6 lines
Quoted 3 lines
Fix this by skipping the put_prev_task/set_next_task re-entry when the donor is in the
ext_sched_class, sincesched_exttracks curr/donor itself.This really sounds like a bug in ext; how is this different from the
sched_changepattern doing a put/set cycle?
I think you're right, the patch differs from sched_change because it doesn't bracket the put/set cycle with a dequeue/enqueue, but sched_ext still needs to support the put/set contract.
The underlying problem is that set_next_task_scx() triggers ops.running() for a blocked proxy donor even though the donor never becomes the execution context, so we need to prevent triggering ops.running() in this case. But this should be handled in sched_ext by pairing ops.running()/ops.stopping() only when the donor actually runs. This is addressed later in the series with explicit running-state tracking. I'll drop this core special case and rework that sched_ext fix.
Thanks,
-Andrea
Andrea Righi
reviewerPATCH 3/12 sched_ext: Split curr|donor references properly
In reply to Andrea RighiFrom: John Stultz <[email protected]>
With proxy-exec, we want to do the accounting against the donor most of the time. Without proxy-exec, there should be no difference as the rq->donor and rq->curr are the same.
So rework the logic to reference the rq->donor where appropriate.
Also add donor info to scx_dump_state().
Since CONFIG_SCHED_PROXY_EXEC currently depends on !CONFIG_SCHED_CLASS_EXT, this should have no effect (other than the extra donor output in scx_dump_state), but this is one step needed to eventually remove that constraint for proxy-exec.
--- kernel/sched/ext/ext.c | 28 ++++++++++++++++------------ 1 file changed, 16 insertions(+), 12 deletions(-)
kernel/sched/ext/ext.c +16 -12
@@ -1145,17 +1145,17 @@ static void touch_core_sched_dispatch(struct rq *rq, struct task_struct *p)static void update_curr_scx (struct rq *rq) { -struct task_struct *curr = rq->curr; +struct task_struct *donor = rq->donor;s64 delta_exec; delta_exec =update_curr_common (rq);if (unlikely (delta_exec <=0 ))return ; -if (curr->scx.slice !=SCX_SLICE_INF ) { - curr->scx.slice -=min_t (u64, curr->scx.slice, delta_exec); -if (!curr->scx.slice) -touch_core_sched (rq, curr); +if (donor->scx.slice !=SCX_SLICE_INF ) { + donor->scx.slice -=min_t (u64, donor->scx.slice, delta_exec); +if (!donor->scx.slice) +touch_core_sched (rq, donor); }dl_server_update (&rq->ext_server, delta_exec);
@@ -1316,8 +1316,8 @@ static void local_dsq_post_enq(struct scx_sched *sch, struct scx_dispatch_q *dsqif (rq->scx.flags &SCX_RQ_IN_BALANCE )return ; -if ((enq_flags &SCX_ENQ_PREEMPT ) && p != rq->curr && - rq->curr->sched_class == &ext_sched_class) { +if ((enq_flags &SCX_ENQ_PREEMPT ) && p != rq->donor && + rq->donor->sched_class == &ext_sched_class) { rq->curr->scx.slice =0 ;resched_curr (rq); }
@@ -2464,7 +2464,8 @@ static void dispatch_to_local_dsq(struct scx_sched *sch, struct rq *rq, }/* if the destination CPU is idle, wake it up */ -if (!fallback &&sched_class_above (p->sched_class, dst_rq->curr->sched_class)) +if (!fallback &&sched_class_above (p->sched_class, + dst_rq->donor->sched_class))resched_curr (dst_rq); }
@@ -2876,7 +2877,7 @@ static struct task_struct *first_local_task(struct rq *rq)static struct task_struct *do_pick_task_scx (struct rq *rq,struct rq_flags *rf,bool force_scx) { -struct task_struct *prev = rq->curr; +struct task_struct *prev = rq->donor;bool keep_prev;struct task_struct *p;
@@ -4029,7 +4030,7 @@ static void run_deferred(struct rq *rq)#ifdef CONFIG_NO_HZ_FULL bool scx_can_stop_tick (struct rq *rq) { -struct task_struct *p = rq->curr; +struct task_struct *p = rq->donor;struct scx_sched *sch =scx_task_sched (p);if (p->sched_class != &ext_sched_class)
@@ -6007,6 +6008,9 @@ static void scx_dump_cpu(struct scx_sched *sch, struct seq_buf *s,dump_line (&ns," curr=%s[%d] class=%ps" , rq->curr->comm, rq->curr->pid, rq->curr->sched_class); +dump_line (&ns," donor=%s[%d] class=%ps" , + rq->donor->comm, rq->donor->pid, + rq->donor->sched_class);if (!cpumask_empty (rq->scx.cpus_to_kick))dump_line (&ns," cpus_to_kick : %*pb" ,cpumask_pr_args (rq->scx.cpus_to_kick));
@@ -7452,7 +7456,7 @@ static bool kick_one_cpu(s32 cpu, struct rq *this_rq, unsigned long *ksyncs)unsigned long flags;raw_spin_rq_lock_irqsave (rq, flags); - cur_class = rq->curr->sched_class; + cur_class = rq->donor->sched_class;/* * During CPU hotplug, a CPU may depend on kicking itself to make
@@ -7464,7 +7468,7 @@ static bool kick_one_cpu(s32 cpu, struct rq *this_rq, unsigned long *ksyncs) !sched_class_above(cur_class, &ext_sched_class)) {if (cpumask_test_cpu (cpu, this_scx->cpus_to_preempt)) {if (cur_class == &ext_sched_class) - rq->curr->scx.slice =0 ; + rq->donor->scx.slice =0 ;cpumask_clear_cpu (cpu, this_scx->cpus_to_preempt); }
Signature · 1 line
2.55.0
Re: [PATCH 03/12] sched_ext: Split curr|donor references properly
Quoted 5 lines · Andrea Righi
From: John Stultz <[email protected]>
With proxy-exec, we want to do the accounting against the donor most of the time. Without proxy-exec, there should be no difference as the
rq->donorandrq->currare the same.
Trying to understand more of the situation when donor and curr in different sched_class? one is in scx and the other is not.
could you pls help to explain more with this information in commit message?
Quoted 43 lines
So rework the logic to reference the
rq->donorwhere appropriate.Also add donor info to
scx_dump_state().Since
CONFIG_SCHED_PROXY_EXECcurrently depends on !CONFIG_SCHED_CLASS_EXT, this should have no effect (other than the extra donor output inscx_dump_state), but this is one step needed to eventually remove that constraint for proxy-exec.Signed-off-by John Stultz <[email protected]>--- kernel/sched/ext/ext.c | 28 ++++++++++++++++------------ 1 file changed, 16 insertions(+), 12 deletions(-)kernel/sched/ext/ext.c +5
-5@@ -1145,17 +1145,17 @@ static void touch_core_sched_dispatch(struct rq *rq, struct task_struct *p)static void update_curr_scx (struct rq *rq) { -struct task_struct *curr = rq->curr; +struct task_struct *donor = rq->donor;s64 delta_exec; delta_exec =update_curr_common (rq);if (unlikely (delta_exec <=0 ))return ; -if (curr->scx.slice !=SCX_SLICE_INF ) { - curr->scx.slice -=min_t (u64, curr->scx.slice, delta_exec); -if (!curr->scx.slice) -touch_core_sched (rq, curr); +if (donor->scx.slice !=SCX_SLICE_INF ) { + donor->scx.slice -=min_t (u64, donor->scx.slice, delta_exec); +if (!donor->scx.slice) +touch_core_sched (rq, donor); }dl_server_update (&rq->ext_server, delta_exec);
kernel/sched/ext/ext.c static void local_dsq_post_enq(struct scx_sched *sch, struct scx_dispatch_q *dsq · line 1321
Quoted 8 lines
kernel/sched/ext/ext.c +2
-2@@ -1316,8 +1316,8 @@ static void local_dsq_post_enq(struct scx_sched *sch, struct scx_dispatch_q *dsqif (rq->scx.flags &SCX_RQ_IN_BALANCE )return ; -if ((enq_flags &SCX_ENQ_PREEMPT ) && p != rq->curr && - rq->curr->sched_class == &ext_sched_class) { +if ((enq_flags &SCX_ENQ_PREEMPT ) && p != rq->donor && + rq->donor->sched_class == &ext_sched_class) { rq->curr->scx.slice =0 ;
Do you forget to update rq->curr with rq->donor here?
Quoted 58 lines
kernel/sched/ext/ext.c +9
-5resched_curr (rq); }@@ -2464,7 +2464,8 @@ static void dispatch_to_local_dsq(struct scx_sched *sch, struct rq *rq, }/* if the destination CPU is idle, wake it up */ -if (!fallback &&sched_class_above (p->sched_class, dst_rq->curr->sched_class)) +if (!fallback &&sched_class_above (p->sched_class, + dst_rq->donor->sched_class))resched_curr (dst_rq); }@@ -2876,7 +2877,7 @@ static struct task_struct *first_local_task(struct rq *rq)static struct task_struct *do_pick_task_scx (struct rq *rq,struct rq_flags *rf,bool force_scx) { -struct task_struct *prev = rq->curr; +struct task_struct *prev = rq->donor;bool keep_prev;struct task_struct *p;@@ -4029,7 +4030,7 @@ static void run_deferred(struct rq *rq)#ifdef CONFIG_NO_HZ_FULL bool scx_can_stop_tick (struct rq *rq) { -struct task_struct *p = rq->curr; +struct task_struct *p = rq->donor;struct scx_sched *sch =scx_task_sched (p);if (p->sched_class != &ext_sched_class)@@ -6007,6 +6008,9 @@ static void scx_dump_cpu(struct scx_sched *sch, struct seq_buf *s,dump_line (&ns," curr=%s[%d] class=%ps" , rq->curr->comm, rq->curr->pid, rq->curr->sched_class); +dump_line (&ns," donor=%s[%d] class=%ps" , + rq->donor->comm, rq->donor->pid, + rq->donor->sched_class);if (!cpumask_empty (rq->scx.cpus_to_kick))dump_line (&ns," cpus_to_kick : %*pb" ,cpumask_pr_args (rq->scx.cpus_to_kick));@@ -7452,7 +7456,7 @@ static bool kick_one_cpu(s32 cpu, struct rq *this_rq, unsigned long *ksyncs)unsigned long flags;raw_spin_rq_lock_irqsave (rq, flags); - cur_class = rq->curr->sched_class; + cur_class = rq->donor->sched_class;/* * During CPU hotplug, a CPU may depend on kicking itself to make @@ -7464,7 +7468,7 @@ static bool kick_one_cpu(s32 cpu, struct rq *this_rq, unsigned long *ksyncs) !sched_class_above(cur_class, &ext_sched_class)) {if (cpumask_test_cpu (cpu, this_scx->cpus_to_preempt)) {if (cur_class == &ext_sched_class) - rq->curr->scx.slice =0 ; + rq->donor->scx.slice =0 ;cpumask_clear_cpu (cpu, this_scx->cpus_to_preempt); }
Signature · 2 lines
Thx and BRs, Aiqun(Maria) Yu
Andrea Righi
reviewerRe: [PATCH 03/12] sched_ext: Split curr|donor references properly
Hi Maria,
Quoted 10 lines · Aiqun(Maria) Yu
Quoted 5 lines · Andrea Righi
From: John Stultz <[email protected]>
With proxy-exec, we want to do the accounting against the donor most of the time. Without proxy-exec, there should be no difference as the
rq->donorandrq->currare the same.Trying to understand more of the situation when donor and curr in different
sched_class? one is in scx and the other is not.
could you pls help to explain more with this information in commit message?
Sure, I'll add more details. Do you think something like the following would help (maybe in a shorter form)?
Let use FAIR and EXT and assume we run the BPF scheduler in partial mode. We can replace FAIR with RT/deadline, the result is the same.
Terminology:
D = blocked donor
M = mutex
O = mutex owner
T = competing runnable task
D -----------------> M -------------> O ----------------> T [donor] blocked on [mutex] owned by [owner] preempted by [task]
\_________________________________^
donates scheduling context
During a proxy exec switch:
- D supplies scheduling class, priority and runtime budget
- O is the task whose code is physically executing
- T is a competing task that preempts O
Scenarios:
1) D is EXT, O is EXT, T is EXT
Result:
- D can interrupt T depending on the BPF scheduling policy
- O is executed with D's EXT priority and runtime budget
- When D runs, T waits in EXT
2) D is EXT, O is EXT, T is FAIR
Result:
- D is visible to the BPF scheduler
- D cannot preempt T (EXT < FAIR)
- Once T stops, BPF dispatches D
- D executes O using D's EXT priority and runtime budget
- if T becomes runnable again, it preempts the D/O proxy execution
3) D is EXT, O is FAIR, T is EXT
Result:
- Not possible, T can't preempt O (EXT < FAIR)
4) D is EXT, O is FAIR, T is FAIR
Result:
- D cannot boost O because EXT < FAIR
- O and T continue competing under FAIR
- O eventually runs and releases M
- D then wakes and resumes normal EXT scheduling
5) D is FAIR, O is EXT, T is EXT
Result:
- D preempts T immediately (higher sched class)
- O is executed with D's FAIR priority and runtime budget
- When D runs, T waits in EXT
- D is not visible to the BPF scheduler
6) D is FAIR, O is EXT, T is FAIR
Result:
- D runs based on its FAIR deadline (competing with T)
- O is executed with D's FAIR priority and runtime budget
- When D runs, T waits in FAIR
- D is not visible to the BPF scheduler
7) D is FAIR, O is FAIR, T is EXT
Result:
- Not possible, T can't preempt O (EXT < FAIR)
8) D is FAIR, O is FAIR, T is FAIR
Result:
- O, T and D all have FAIR scheduling contexts
- D remains runnable as a blocked proxy donor
- When CFS selects D, O executes using D's FAIR scheduling context
- When CFS selects O, O executes using its own FAIR context
- When CFS selects T, T executes normally
- D is not visible to the BPF scheduler
Quoted 47 lines
Quoted 47 lines
So rework the logic to reference the
rq->donorwhere appropriate.Also add donor info to
scx_dump_state().Since
CONFIG_SCHED_PROXY_EXECcurrently depends on !CONFIG_SCHED_CLASS_EXT, this should have no effect (other than the extra donor output inscx_dump_state), but this is one step needed to eventually remove that constraint for proxy-exec.Signed-off-by John Stultz <[email protected]>--- kernel/sched/ext/ext.c | 28 ++++++++++++++++------------ 1 file changed, 16 insertions(+), 12 deletions(-)kernel/sched/ext/ext.c +5
-5@@ -1145,17 +1145,17 @@ static void touch_core_sched_dispatch(struct rq *rq, struct task_struct *p)static void update_curr_scx (struct rq *rq) { -struct task_struct *curr = rq->curr; +struct task_struct *donor = rq->donor;s64 delta_exec; delta_exec =update_curr_common (rq);if (unlikely (delta_exec <=0 ))return ; -if (curr->scx.slice !=SCX_SLICE_INF ) { - curr->scx.slice -=min_t (u64, curr->scx.slice, delta_exec); -if (!curr->scx.slice) -touch_core_sched (rq, curr); +if (donor->scx.slice !=SCX_SLICE_INF ) { + donor->scx.slice -=min_t (u64, donor->scx.slice, delta_exec); +if (!donor->scx.slice) +touch_core_sched (rq, donor); }dl_server_update (&rq->ext_server, delta_exec);@@ -1316,8 +1316,8 @@ static void local_dsq_post_enq(struct scx_sched *sch, struct scx_dispatch_q *dsqif (rq->scx.flags &SCX_RQ_IN_BALANCE )return ;
Quoted 7 lines
kernel/sched/ext/ext.c static void local_dsq_post_enq(struct scx_sched *sch, struct scx_dispatch_q *dsq · line 1321
Quoted 5 lines
kernel/sched/ext/ext.c +2
-2-if ((enq_flags &SCX_ENQ_PREEMPT ) && p != rq->curr && - rq->curr->sched_class == &ext_sched_class) { +if ((enq_flags &SCX_ENQ_PREEMPT ) && p != rq->donor && + rq->donor->sched_class == &ext_sched_class) { rq->curr->scx.slice =0 ;Do you forget to update
rq->currwithrq->donorhere?
Yes, good catch. This should be rq->donor->scx.slice = 0. I'll fix it in the next version.
Thanks,
-Andrea
Andrea Righi
reviewerPATCH 4/12 sched_ext: Avoid migrating blocked tasks with proxy execution
In reply to Andrea RighiFrom: John Stultz <[email protected]>
With proxy execution enabled, mutex blocked tasks stay on the runqueue. Later with donor migration they will be migrated when necessary by the core scheduler to boost lock owners.
Don't try to migrate mutex blocked tasks, the proxy logic will handle that.
--- kernel/sched/ext/ext.c | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+)
kernel/sched/ext/ext.c +25 -0
@@ -2150,6 +2150,14 @@ static bool task_can_run_on_remote_rq(struct scx_sched *sch,WARN_ON_ONCE (task_cpu (p) == cpu); +/* Make sure tasks aren't on a cpu */ +if (task_on_cpu (task_rq (p), p)) +return false; + +/* Don't migrate blocked tasks, proxy-exec will handle this */ +if (task_is_blocked (p)) +return false; +/* * If @p has migration disabled, @p->cpus_ptr is updated to contain only * the pinned CPU in migrate_disable_switch() while @p is being switched
@@ -2784,6 +2792,23 @@ static void put_prev_task_scx(struct rq *rq, struct task_struct *p,if (p->scx.flags &SCX_TASK_QUEUED ) {set_task_runnable (rq, p); +/* +* Mutex-blocked donors stay queued on the runqueue under proxy +* execution, but the donor never runs as itself, proxy-exec +* walks the blocked_on chain on the next __schedule() and runs +* the lock owner in its place. +* +* Put the donor on the local DSQ directly, so pick_next_task() +* can still see it, find_proxy_task() will be invoked on +* next->blocked_on and either run the chain owner here, or call +* proxy_force_return() and let BPF make a new dispatch decision +* once the task is no longer blocked. +*/ +if (task_is_blocked (p)) { +dispatch_enqueue (sch, rq, &rq->scx.local_dsq, p,0 ); + goto switch_class; + } +/* * If @p has slice left and is being put, @p is getting * preempted by a higher priority scheduler class or core-sched
Signature · 1 line
2.55.0
Re: [PATCH 04/12] sched_ext: Avoid migrating blocked tasks with proxy execution
Quoted 6 lines · Andrea Righi
From: John Stultz <[email protected]>
With proxy execution enabled, mutex blocked tasks stay on the runqueue. Later with donor migration they will be migrated when necessary by the core scheduler to boost lock owners.
Quoted 6 lines
Don't try to migrate mutex blocked tasks, the proxy logic will handle that.
Co-developed-by Andrea Righi <[email protected]>Signed-off-by Andrea Righi <[email protected]>Signed-off-by John Stultz <[email protected]>
SOB was suggested to have the current committer at the last line. Not sure if it is the same rule for the subsystem here.
My understanding would be:
Quoted 9 lines
--- kernel/sched/ext/ext.c | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+)kernel/sched/ext/ext.c +0
-0
kernel/sched/ext/ext.c static bool task_can_run_on_remote_rq(struct scx_sched *sch, · line 2159
Quoted 10 lines
kernel/sched/ext/ext.c +7
-0@@ -2150,6 +2150,14 @@ static bool task_can_run_on_remote_rq(struct scx_sched *sch,WARN_ON_ONCE (task_cpu (p) == cpu); +/* Make sure tasks aren't on a cpu */ +if (task_on_cpu (task_rq (p), p)) +return false; + +/* Don't migrate blocked tasks, proxy-exec will handle this */ +if (task_is_blocked (p)) +return false;
what about return true here for owner_cpu?
if the owner_cpu is in the move_task_between_dsqs stage for example.
Quoted 28 lines
kernel/sched/ext/ext.c +18
-0+/* * If @p has migration disabled, @p->cpus_ptr is updated to contain only * the pinned CPU in migrate_disable_switch() while @p is being switched @@ -2784,6 +2792,23 @@ static void put_prev_task_scx(struct rq *rq, struct task_struct *p,if (p->scx.flags &SCX_TASK_QUEUED ) {set_task_runnable (rq, p); +/* +* Mutex-blocked donors stay queued on the runqueue under proxy +* execution, but the donor never runs as itself, proxy-exec +* walks the blocked_on chain on the next __schedule() and runs +* the lock owner in its place. +* +* Put the donor on the local DSQ directly, so pick_next_task() +* can still see it, find_proxy_task() will be invoked on +* next->blocked_on and either run the chain owner here, or call +* proxy_force_return() and let BPF make a new dispatch decision +* once the task is no longer blocked. +*/ +if (task_is_blocked (p)) { +dispatch_enqueue (sch, rq, &rq->scx.local_dsq, p,0 ); + goto switch_class; + } +/* * If @p has slice left and is being put, @p is getting * preempted by a higher priority scheduler class or core-sched
Signature · 2 lines
Thx and BRs, Aiqun(Maria) Yu
Andrea Righi
reviewerRe: [PATCH 04/12] sched_ext: Avoid migrating blocked tasks with proxy execution
Hi Maria,
Quoted 14 lines · Aiqun(Maria) Yu
Quoted 6 lines · Andrea Righi
From: John Stultz <[email protected]>
With proxy execution enabled, mutex blocked tasks stay on the runqueue. Later with donor migration they will be migrated when necessary by the core scheduler to boost lock owners.
Quoted 6 lines
Don't try to migrate mutex blocked tasks, the proxy logic will handle that.
Co-developed-by Andrea Righi <[email protected]>Signed-off-by Andrea Righi <[email protected]>Signed-off-by John Stultz <[email protected]>
Quoted 7 lines
SOB was suggested to have the current committer at the last line. Not sure if it is the same rule for the subsystem here.
My understanding would be:
Signed-off-by John Stultz <[email protected]>Co-developed-by Andrea Righi <[email protected]>Signed-off-by Andrea Righi <[email protected]>
You're right, I'll fix it in the next version.
Quoted 14 lines
Quoted 14 lines
--- kernel/sched/ext/ext.c | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+)kernel/sched/ext/ext.c +0
-0@@ -2150,6 +2150,14 @@ static bool task_can_run_on_remote_rq(struct scx_sched *sch,WARN_ON_ONCE (task_cpu (p) == cpu);
Quoted 10 lines
kernel/sched/ext/ext.c static bool task_can_run_on_remote_rq(struct scx_sched *sch, · line 2159
Quoted 7 lines
kernel/sched/ext/ext.c +7
-0+/* Make sure tasks aren't on a cpu */ +if (task_on_cpu (task_rq (p), p)) +return false; + +/* Don't migrate blocked tasks, proxy-exec will handle this */ +if (task_is_blocked (p)) +return false;what about return true here for
owner_cpu?
if theowner_cpuis in themove_task_between_dsqsstage for example.
Good point. Returning false for every blocked task is too restrictive, because it'd reject potential remote placements requested by the BPF scheduler from ops.enqueue().
But we can't simply return true either, otherwise we would bypass all the checks below, allowing sched_ext to move the donor even if the destination is outside its affinity mask or the destination rq is offline.
I think we should do something like this:
if (task_is_blocked(p) && task_current_donor(task_rq(p), p))
return false;In practice, normal migration can happen if a blocker donor is not already the rq's current scheduling context.
With that, the BPF scheduler should be able to decide the "call back home" CPU for the donor, if it performs a direct dispatch to a local DSQ different than its current CPU's local DSQ.
Thanks,
-Andrea
Andrea Righi
reviewerPATCH 5/12 sched_ext: Fix TOCTOU race in consume_remote_task()
In reply to Andrea RighiWhen pulling a task from a non-local DSQ, consume_dispatch_q() checks if the task can run on the destination rq via task_can_run_on_remote_rq(). However, it then drops the destination rq lock and locks the source rq in consume_remote_task() -> unlink_dsq_and_lock_src_rq(). During this window, the task might have become migration disabled, making it invalid to migrate it to the destination rq.
Fix this by re-evaluating task_can_run_on_remote_rq() in consume_remote_task() after the source rq is locked. If the task can no longer be migrated, we clear its DSQ association, reset the holding CPU, and enqueue it to the source rq's local DSQ instead.
While the destination rq lock is dropped, clear the tracked rq state and restore it after reacquiring the lock. Otherwise, a nested ops.dequeue() callback can attempt to restore an rq which is no longer locked.
--- kernel/sched/ext/ext.c | 43 +++++++++++++++++++++++++++++++++++------- 1 file changed, 36 insertions(+), 7 deletions(-)
kernel/sched/ext/ext.c +36 -7
@@ -2248,20 +2248,49 @@ static bool unlink_dsq_and_lock_src_rq(struct task_struct *p, !WARN_ON_ONCE (src_rq !=task_rq (p)); } -static bool consume_remote_task (struct rq *this_rq, +static bool consume_remote_task (struct scx_sched *sch,struct rq *this_rq,struct task_struct *p,u64 enq_flags,struct scx_dispatch_q *dsq,struct rq *src_rq) { +struct rq *tracked_rq =scx_locked_rq (); +bool consumed = false; + +/* +* consume_remote_task() may be called from an SCX op with @this_rq +* recorded as the currently locked rq. Clear the tracking while the rq +* lock is dropped so nested callbacks don't save and later try to restore +* an rq which isn't locked anymore. +*/ +if (tracked_rq) { +WARN_ON_ONCE (tracked_rq != this_rq); +update_locked_rq (NULL ); + }raw_spin_rq_unlock (this_rq);if (unlink_dsq_and_lock_src_rq (p, dsq, src_rq)) { +if (unlikely (!task_can_run_on_remote_rq (sch, p, this_rq, true))) { + p->scx.dsq =NULL ; + p->scx.holding_cpu =-1 ; +scx_dispatch_enqueue (sch, src_rq, &src_rq->scx.local_dsq, p, + enq_flags |SCX_ENQ_CLEAR_OPSS ); +if (sched_class_above (p->sched_class, src_rq->donor->sched_class)) +resched_curr (src_rq); +raw_spin_rq_unlock (src_rq); + goto relock; + }move_remote_task_to_local_dsq (p, enq_flags, src_rq, this_rq); -return true; - }else { -raw_spin_rq_unlock (src_rq); -raw_spin_rq_lock (this_rq); -return false; + consumed = true; + goto restore; } +raw_spin_rq_unlock (src_rq); + +relock: +raw_spin_rq_lock (this_rq); +restore: +if (tracked_rq) +update_locked_rq (tracked_rq); + +return consumed; }/**
@@ -2371,7 +2400,7 @@ bool scx_consume_dispatch_q(struct scx_sched *sch, struct rq *rq, }if (task_can_run_on_remote_rq (sch, p, rq, false)) { -if (likely (consume_remote_task (rq, p, enq_flags, dsq, task_rq))) +if (likely (consume_remote_task (sch, rq, p, enq_flags, dsq, task_rq)))return true; goto retry; }
Signature · 1 line
2.55.0
Andrea Righi
reviewerPATCH 6/12 sched_ext: Fix ops.running/stopping() pairing for proxy-exec donors
In reply to Andrea RighiWith proxy-exec, pick_next_task() can return a task with blocked_on set (a proxy donor); put_prev_set_next_task() then calls set_next_task_scx() on this "ghost" task, which fires ops.running(). However, the task never actually runs.
If we simply short-circuit set_next_task_scx() for blocked tasks, we break DSQ bookkeeping. If we only skip ops.running(), we create an ops.enqueue() -> ops.stopping() pair without running, because ops.stopping() is still called in put_prev_task_scx().
Fix this by introducing a new flag SCX_TASK_IS_RUNNING to track whether ops.running() was actually called. Skip ops.running() for blocked tasks, and only call ops.stopping() if SCX_TASK_IS_RUNNING is set. This ensures that running and stopping callbacks are perfectly paired even when a blocked task is picked as a proxy donor.
--- include/linux/sched/ext.h | 2 ++ kernel/sched/core.c | 2 +- kernel/sched/ext/ext.c | 14 +++++++++++--- kernel/sched/ext/ext.h | 6 ++++++ 4 files changed, 20 insertions(+), 4 deletions(-)
include/linux/sched/ext.h +2 -0
@@ -102,6 +102,8 @@ enum scx_ent_flags {SCX_TASK_SUB_INIT =1 <<4 ,/* task being initialized for a sub sched */ SCX_TASK_IMMED =1 <<5 ,/* task is on local DSQ with %SCX_ENQ_IMMED */ + SCX_TASK_IS_RUNNING = 1 << 6, /* ops.running() has been called */ +/* * Bits 8 to 10 are used to carry task state: *
kernel/sched/core.c +1 -1
@@ -7189,7 +7189,7 @@ static void __sched notrace __schedule(int sched_mode)* sched_ext tracks curr/donor itself; re-entering set_next_task_scx * here dispatches through a stale/NULL BPF ops vtable. */ -if (donor->sched_class != &ext_sched_class) { +if (!is_ext_class (donor)) { donor->sched_class->put_prev_task (rq, donor, donor); donor->sched_class->set_next_task (rq, donor, true); }
kernel/sched/ext/ext.c +11 -3
@@ -1985,9 +1985,11 @@ static bool dequeue_task_scx(struct rq *rq, struct task_struct *p, int core_deq_* information meaningful to the BPF scheduler and can be suppressed by * skipping the callbacks if the task is !QUEUED. */ -if (SCX_HAS_OP (sch, stopping) &&task_current (rq, p)) { +if (SCX_HAS_OP (sch, stopping) &&task_current (rq, p) && + (p->scx.flags &SCX_TASK_IS_RUNNING )) {update_curr_scx (rq);SCX_CALL_OP_TASK (sch, stopping, rq, p, false); + p->scx.flags &= ~SCX_TASK_IS_RUNNING ; }if (SCX_HAS_OP (sch, quiescent) && !task_on_rq_migrating (p))
@@ -2725,8 +2727,11 @@ static void set_next_task_scx(struct rq *rq, struct task_struct *p, bool first) p->se.exec_start =rq_clock_task (rq);/* see dequeue_task_scx() on why we skip when !QUEUED */ -if (SCX_HAS_OP (sch, running) && (p->scx.flags &SCX_TASK_QUEUED )) +if (SCX_HAS_OP (sch, running) && (p->scx.flags &SCX_TASK_QUEUED ) && + !task_is_blocked (p)) {SCX_CALL_OP_TASK (sch, running, rq, p); + p->scx.flags |=SCX_TASK_IS_RUNNING ; + }clr_task_runnable (p, true);
@@ -2815,8 +2820,11 @@ static void put_prev_task_scx(struct rq *rq, struct task_struct *p,update_curr_scx (rq);/* see dequeue_task_scx() on why we skip when !QUEUED */ -if (SCX_HAS_OP (sch, stopping) && (p->scx.flags &SCX_TASK_QUEUED )) +if (SCX_HAS_OP (sch, stopping) && (p->scx.flags &SCX_TASK_QUEUED ) && + (p->scx.flags &SCX_TASK_IS_RUNNING )) {SCX_CALL_OP_TASK (sch, stopping, rq, p, true); + p->scx.flags &= ~SCX_TASK_IS_RUNNING ; + }if (p->scx.flags &SCX_TASK_QUEUED ) {set_task_runnable (rq, p);
kernel/sched/ext/ext.h +6 -0
@@ -35,6 +35,11 @@ static inline bool task_on_scx(const struct task_struct *p)return scx_enabled () && p->sched_class == &ext_sched_class; } +static inline bool is_ext_class (const struct task_struct *p) +{ +return p->sched_class == &ext_sched_class; +} +#ifdef CONFIG_SCHED_CORE bool scx_prio_less (const struct task_struct *a,const struct task_struct *b,bool in_fi);
@@ -53,6 +58,7 @@ static inline void scx_rq_activate(struct rq *rq) {}static inline void scx_rq_deactivate (struct rq *rq) {}static inline int scx_check_setscheduler (struct task_struct *p,int policy) {return 0 ; }static inline bool task_on_scx (const struct task_struct *p) {return false; } +static inline bool is_ext_class (const struct task_struct *p) {return false; }static inline bool scx_allow_ttwu_queue (const struct task_struct *p) {return true; }static inline void init_sched_ext_class (void ) {}
Signature · 1 line
2.55.0
Andrea Righi
reviewerPATCH 7/12 sched_ext: Save/restore kf_tasks[] when task ops nest
In reply to Andrea RighiSCX_CALL_OP_TASK*() stored the subject task in current->scx.kf_tasks[] and assumed ops would not nest. A BPF ops.running() callback can call kfuncs (e.g. scx_bpf_dsq_insert) that enqueue work and trigger enqueue_task_scx() -> ops.runnable(), which used SCX_CALL_OP_TASK again and overwrote kf_tasks[0] then cleared it, leaving the running context wrong and leading to NULL function dispatches from BPF helpers.
Save and restore kf_tasks[] (both slots for the two-task variant) around each invocation so nested task-based ops preserve the outer context.
--- kernel/sched/ext/ext.c | 10 ++++++++-- kernel/sched/ext/internal.h | 33 +++++++++++++++++++++++---------- 2 files changed, 31 insertions(+), 12 deletions(-)
kernel/sched/ext/ext.c +8 -2
@@ -410,8 +410,12 @@ static inline void scx_call_op_set_cpumask(struct scx_sched *sch, struct rq *rq,struct task_struct *task, const struct cpumask *cpumask) { - WARN_ON_ONCE(current->scx.kf_tasks[0]); + struct task_struct *__scx_kf0_sv = current->scx.kf_tasks[0]; +struct task_struct *__scx_kf1_sv = current->scx.kf_tasks[1 ]; + + current->scx.kf_nest++; current->scx.kf_tasks[0 ] = task; + current->scx.kf_tasks[1 ] =NULL ;if (rq)update_locked_rq (rq);
@@ -430,7 +434,9 @@ static inline void scx_call_op_set_cpumask(struct scx_sched *sch, struct rq *rq,if (rq)update_locked_rq (NULL ); - current->scx.kf_tasks[0 ] =NULL ; + current->scx.kf_tasks[0 ] = __scx_kf0_sv; + current->scx.kf_tasks[1 ] = __scx_kf1_sv; + current->scx.kf_nest--; }enum scx_dsq_iter_flags {
kernel/sched/ext/internal.h +23 -10
@@ -1792,37 +1792,50 @@ do { \* pi_lock held by try_to_wake_up() with rq tracking via scx_rq.in_select_cpu. * So if kf_tasks[] is set, @p's scheduler-protected fields are stable. * -* kf_tasks[] can not stack, so task-based SCX ops must not nest. The -* WARN_ON_ONCE() in each macro catches a re-entry of any of the three variants -* while a previous one is still in progress. +* Task-based SCX ops may nest (e.g. ops.running() calling a kfunc that ends up +* in enqueue_task_scx() -> ops.runnable()). Save and restore kf_tasks[] around +* each invocation so the outer op's context is restored for kfuncs and for +* further nested calls. Single-task ops save/restore both slots and clear +* kf_tasks[1] while active so a nested call under SCX_CALL_OP_2TASKS_RET does +* not leave the outer pair's second task authenticated for kfuncs. */ #define SCX_CALL_OP_TASK(sch, op, locked_rq, task, args...) \ do { \ - WARN_ON_ONCE(current->scx.kf_tasks[0]); \ + struct task_struct *__scx_kf0_sv = current->scx.kf_tasks[0]; \ + struct task_struct *__scx_kf1_sv = current->scx.kf_tasks[1]; \ + \ current->scx.kf_tasks[0] = task; \ + current->scx.kf_tasks[1] = NULL; \ SCX_CALL_OP((sch), op, locked_rq, task, ##args); \ - current->scx.kf_tasks[0] = NULL; \ + current->scx.kf_tasks[0] = __scx_kf0_sv; \ + current->scx.kf_tasks[1] = __scx_kf1_sv; \ } while (0) #define SCX_CALL_OP_TASK_RET(sch, op, locked_rq, task, args...) \ ({ \ __typeof__((sch)->ops.op(task, ##args)) __ret; \ - WARN_ON_ONCE(current->scx.kf_tasks[0]); \ + struct task_struct *__scx_kf0_sv = current->scx.kf_tasks[0]; \ + struct task_struct *__scx_kf1_sv = current->scx.kf_tasks[1]; \ + \ current->scx.kf_tasks[0] = task; \ + current->scx.kf_tasks[1] = NULL; \ __ret = SCX_CALL_OP_RET((sch), op, locked_rq, task, ##args); \ - current->scx.kf_tasks[0] = NULL; \ + current->scx.kf_tasks[0] = __scx_kf0_sv; \ + current->scx.kf_tasks[1] = __scx_kf1_sv; \ __ret; \ }) #define SCX_CALL_OP_2TASKS_RET(sch, op, locked_rq, task0, task1, args...) \ ({ \ __typeof__((sch)->ops.op(task0, task1, ##args)) __ret; \ - WARN_ON_ONCE(current->scx.kf_tasks[0]); \ + struct task_struct *__scx_kf0_sv = current->scx.kf_tasks[0]; \ + struct task_struct *__scx_kf1_sv = current->scx.kf_tasks[1]; \ + \ current->scx.kf_tasks[0] = task0; \ current->scx.kf_tasks[1] = task1; \ __ret = SCX_CALL_OP_RET((sch), op, locked_rq, task0, task1, ##args); \ - current->scx.kf_tasks[0] = NULL; \ - current->scx.kf_tasks[1] = NULL; \ + current->scx.kf_tasks[0] = __scx_kf0_sv; \ + current->scx.kf_tasks[1] = __scx_kf1_sv; \ __ret; \ })
Signature · 1 line
2.55.0
Andrea Righi
reviewerPATCH 8/12 sched_ext: Skip ops.runnable() when nested in SCX_CALL_OP_TASK
In reply to Andrea Righiops.running() can pull in enqueue_task_scx() -> ops.runnable() on the same current task while kf_tasks[] save/restore is still insufficient for every BPF/kfunc combination, leading to NULL dispatches and stack corruption.
Track SCX_CALL_OP_TASK nesting in current->scx.kf_nest (incremented by all SCX_CALL_OP_TASK* macros) and omit the ops.runnable() callback when non-zero. The full enqueue path including ops.enqueue() still runs, only the runnable hook is skipped in this case.
--- include/linux/sched/ext.h | 7 +++++++ kernel/sched/ext/ext.c | 3 ++- kernel/sched/ext/internal.h | 6 ++++++ 3 files changed, 15 insertions(+), 1 deletion(-)
include/linux/sched/ext.h +7 -0
@@ -201,6 +201,13 @@ struct sched_ext_entity {s32 holding_cpu;s32 selected_cpu;struct task_struct *kf_tasks[2 ];/* see SCX_CALL_OP_TASK() */ +/* +* Nesting depth of SCX_CALL_OP_TASK() on this task as %current (e.g. +* during schedule() %current is still the previous task). Used to skip +* ops.runnable() when invoked from inside another task op such as +* ops.running() to avoid breaking BPF re-entrance guarantees. +*/ +u32 kf_nest;struct list_head runnable_node;/* rq->scx.runnable_list */ unsigned long runnable_at;
kernel/sched/ext/ext.c +2 -1
@@ -1862,7 +1862,8 @@ static void enqueue_task_scx(struct rq *rq, struct task_struct *p, int core_enq_ rq->scx.nr_running++;add_nr_running (rq,1 ); -if (SCX_HAS_OP (sch, runnable) && !task_on_rq_migrating (p)) +if (SCX_HAS_OP (sch, runnable) && !task_on_rq_migrating (p) && + !READ_ONCE (current->scx.kf_nest))SCX_CALL_OP_TASK (sch, runnable, rq, p, enq_flags);if (enq_flags &SCX_ENQ_WAKEUP )
kernel/sched/ext/internal.h +6 -0
@@ -1804,11 +1804,13 @@ do { \struct task_struct *__scx_kf0_sv = current->scx.kf_tasks[0 ]; \struct task_struct *__scx_kf1_sv = current->scx.kf_tasks[1 ]; \ \ + current->scx.kf_nest++; \ current->scx.kf_tasks[0 ] = task; \ current->scx.kf_tasks[1 ] =NULL ; \ SCX_CALL_OP((sch), op, locked_rq, task, ##args); \ current->scx.kf_tasks[0 ] = __scx_kf0_sv; \ current->scx.kf_tasks[1 ] = __scx_kf1_sv; \ + current->scx.kf_nest--; \ }while (0 ) #define SCX_CALL_OP_TASK_RET(sch, op, locked_rq, task, args...) \
@@ -1817,11 +1819,13 @@ do { \struct task_struct *__scx_kf0_sv = current->scx.kf_tasks[0 ]; \struct task_struct *__scx_kf1_sv = current->scx.kf_tasks[1 ]; \ \ + current->scx.kf_nest++; \ current->scx.kf_tasks[0 ] = task; \ current->scx.kf_tasks[1 ] =NULL ; \ __ret = SCX_CALL_OP_RET((sch), op, locked_rq, task, ##args); \ current->scx.kf_tasks[0] = __scx_kf0_sv; \ current->scx.kf_tasks[1] = __scx_kf1_sv; \ + current->scx.kf_nest--; \ __ret; \ })
@@ -1831,11 +1835,13 @@ do { \struct task_struct *__scx_kf0_sv = current->scx.kf_tasks[0 ]; \struct task_struct *__scx_kf1_sv = current->scx.kf_tasks[1 ]; \ \ + current->scx.kf_nest++; \ current->scx.kf_tasks[0 ] = task0; \ current->scx.kf_tasks[1 ] = task1; \ __ret = SCX_CALL_OP_RET((sch), op, locked_rq, task0, task1, ##args); \ current->scx.kf_tasks[0] = __scx_kf0_sv; \ current->scx.kf_tasks[1] = __scx_kf1_sv; \ + current->scx.kf_nest--; \ __ret; \ })
Signature · 1 line
2.55.0
Andrea Righi
reviewerPATCH 9/12 sched_ext: Delegate proxy donor admission to BPF schedulers
In reply to Andrea RighiProxy execution keeps a blocked donor runnable so its scheduling context can execute the mutex owner. Dispatching sched_ext donors on a local DSQ bypasses the BPF scheduler's ordering policy and can give donors more CPU priority than intended to perform the proxy execution handoff to the mutex owner.
Add SCX_OPS_ENQ_BLOCKED as an explicit proxy execution capability. Tasks owned by schedulers without the flag block normally. Schedulers with the flag receive blocked donors through ops.enqueue() and can use scx_bpf_task_is_blocked() to apply their own admission policy.
The donor starts associated with its original CPU. A BPF scheduler may dispatch it directly into that CPU's local DSQ to let the core resolve the mutex owner and execute it with the donor's scheduling context.
Knowing the mutex owner's location in BPF is not strictly required, but steering the donation there can reduce handoff latency when migration fits the donor's affinity constraints and the scheduler's policy. Add scx_bpf_task_proxy_cpu() and scx_bpf_task_proxy_cid() to report that destination in the corresponding address space, and let the BPF scheduler decide to migrate or not the donor to the mutex owner's location.
Reschedule a retained donor when its mutex wakes it so ops.dispatch() can reconsider the now-unblocked task. Make SCX_OPS_ENQ_BLOCKED override the exiting and migration-disabled enqueue fallbacks so opted-in schedulers receive all eligible donor requests.
--- kernel/sched/core.c | 36 +++++++- kernel/sched/ext/ext.c | 108 +++++++++++++++++++---- kernel/sched/ext/ext.h | 2 + kernel/sched/ext/internal.h | 19 +++- kernel/sched/sched.h | 6 ++ tools/sched_ext/include/scx/common.bpf.h | 3 + tools/sched_ext/include/scx/compat.h | 1 + 7 files changed, 156 insertions(+), 19 deletions(-)
kernel/sched/core.c +35 -1
@@ -7018,6 +7018,39 @@ find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf)proxy_migrate_task (rq, rf, p, owner_cpu);return NULL ; } + +int task_proxy_cpu (struct task_struct *p) +{ +struct task_struct *owner; +struct mutex *mutex; +int owner_cpu; + +if (!task_is_blocked (p)) +return -ENOENT ; + + mutex =READ_ONCE (p->blocked_on); +if (!mutex) +return -ENOENT ; + +guard (raw_spinlock)(&mutex->wait_lock); +guard (raw_spinlock)(&p->blocked_lock); + +if (mutex !=__get_task_blocked_on (p)) +return -EAGAIN ; + + owner =__mutex_owner (mutex); +if (!owner) +return -ENOENT ; +if (!READ_ONCE (owner->on_rq) || owner->se.sched_delayed) +return -EAGAIN ; + + owner_cpu =task_cpu (owner); +if (owner_cpu !=task_cpu (p) && + (p->nr_cpus_allowed ==1 ||is_migration_disabled (p))) +return -EOPNOTSUPP ; + +return owner_cpu; +}#else /* SCHED_PROXY_EXEC */ static struct task_struct *find_proxy_task (struct rq *rq,struct task_struct *donor,struct rq_flags *rf)
@@ -7148,7 +7181,8 @@ static void __sched notrace __schedule(int sched_mode)* task_is_blocked() will always be false). */ try_to_block_task (rq, prev, &prev_state, - !task_is_blocked (prev)); + !task_is_blocked (prev) || + !scx_allow_proxy_exec (prev)); switch_count = &prev->nvcsw; }
kernel/sched/ext/ext.c +91 -17
172 lines not on this page · raw message
kernel/sched/ext/ext.h +2 -0
@@ -20,6 +20,7 @@ void scx_rq_deactivate(struct rq *rq);int scx_check_setscheduler (struct task_struct *p,int policy);bool task_should_scx (int policy);bool scx_allow_ttwu_queue (const struct task_struct *p); +bool scx_allow_proxy_exec (const struct task_struct *p);void init_sched_ext_class (void );static inline u32 scx_cpuperf_target (s32 cpu)
@@ -60,6 +61,7 @@ static inline int scx_check_setscheduler(struct task_struct *p, int policy) { restatic inline bool task_on_scx (const struct task_struct *p) {return false; }static inline bool is_ext_class (const struct task_struct *p) {return false; }static inline bool scx_allow_ttwu_queue (const struct task_struct *p) {return true; } +static inline bool scx_allow_proxy_exec (const struct task_struct *p) {return true; }static inline void init_sched_ext_class (void ) {}#endif /* CONFIG_SCHED_CLASS_EXT */
kernel/sched/ext/internal.h +18 -1
@@ -212,6 +212,22 @@ enum scx_ops_flags {*/ SCX_OPS_TID_TO_TASK =1LLU <<8 , +/* +* If set, mutex-blocked tasks remain runnable as proxy donors and are +* passed to ops.enqueue(). The BPF scheduler can identify them with +* scx_bpf_task_is_blocked() and query the next mutex owner's CPU or cid +* with scx_bpf_task_proxy_cpu() or scx_bpf_task_proxy_cid(). It controls +* when donors are dispatched and whether they should preempt work on the +* owner's CPU. +* +* If clear, mutex-blocked tasks are removed from the runqueue normally +* and cannot donate their scheduling context through proxy execution. +* +* For blocked donors, this flag takes precedence over +* %SCX_OPS_ENQ_EXITING and %SCX_OPS_ENQ_MIGRATION_DISABLED. +*/ +SCX_OPS_ENQ_BLOCKED =1LLU <<9 , +SCX_OPS_ALL_FLAGS =SCX_OPS_KEEP_BUILTIN_IDLE |SCX_OPS_ENQ_LAST | SCX_OPS_ENQ_EXITING |
@@ -220,7 +236,8 @@ enum scx_ops_flags {SCX_OPS_SWITCH_PARTIAL |SCX_OPS_BUILTIN_IDLE_PER_NODE |SCX_OPS_ALWAYS_ENQ_IMMED | -SCX_OPS_TID_TO_TASK , +SCX_OPS_TID_TO_TASK | +SCX_OPS_ENQ_BLOCKED ,/* high 8 bits are internal, don't include in SCX_OPS_ALL_FLAGS */ __SCX_OPS_INTERNAL_MASK = 0xffLLU << 56,
kernel/sched/sched.h +6 -0
@@ -2470,6 +2470,12 @@ static inline bool task_is_blocked(struct task_struct *p)return !!p->blocked_on; } +#ifdef CONFIG_SCHED_PROXY_EXEC +int task_proxy_cpu (struct task_struct *p); +#else +static inline int task_proxy_cpu (struct task_struct *p) {return -EOPNOTSUPP ; } +#endif +static inline int task_on_cpu (struct rq *rq,struct task_struct *p) {return p->on_cpu;
tools/sched_ext/include/scx/common.bpf.h +3 -0
@@ -95,6 +95,9 @@ s32 scx_bpf_pick_idle_cpu(const cpumask_t *cpus_allowed, u64 flags) __ksym; s32 scx_bpf_pick_any_cpu_node(const cpumask_t *cpus_allowed, int node, u64 flags) __ksym __weak; s32 scx_bpf_pick_any_cpu(const cpumask_t *cpus_allowed, u64 flags) __ksym; bool scx_bpf_task_running(const struct task_struct *p) __ksym; +bool scx_bpf_task_is_blocked(struct task_struct *p) __ksym __weak; +s32 scx_bpf_task_proxy_cpu(struct task_struct *p) __ksym __weak; +s32 scx_bpf_task_proxy_cid(struct task_struct *p) __ksym __weak; s32 scx_bpf_task_cpu(const struct task_struct *p) __ksym;struct rq *scx_bpf_locked_rq (void ) __ksym;struct task_struct *scx_bpf_cpu_curr (s32 cpu) __ksym__weak ;
tools/sched_ext/include/scx/compat.h +1 -0
@@ -117,6 +117,7 @@ static inline bool __COMPAT_struct_has_field(const char *type, const char *field#define SCX_OPS_ALLOW_QUEUED_WAKEUP SCX_OPS_FLAG(SCX_OPS_ALLOW_QUEUED_WAKEUP)#define SCX_OPS_BUILTIN_IDLE_PER_NODE SCX_OPS_FLAG(SCX_OPS_BUILTIN_IDLE_PER_NODE)#define SCX_OPS_ALWAYS_ENQ_IMMED SCX_OPS_FLAG(SCX_OPS_ALWAYS_ENQ_IMMED) +#define SCX_OPS_ENQ_BLOCKED SCX_OPS_FLAG(SCX_OPS_ENQ_BLOCKED)#define SCX_PICK_IDLE_FLAG (name) __COMPAT_ENUM_OR_ZERO("scx_pick_idle_cpu_flags", #name)
Signature · 1 line
2.55.0
K Prateek Nayak
reviewerRe: [PATCH 09/12] sched_ext: Delegate proxy donor admission to BPF schedulers
Hello Andrea,
Quoted 16 lines · Andrea Righi
kernel/sched/core.c +2
-1@@ -7148,7 +7181,8 @@ static void __sched notrace __schedule(int sched_mode)* task_is_blocked() will always be false). */ try_to_block_task (rq, prev, &prev_state, - !task_is_blocked (prev)); + !task_is_blocked (prev) || + !scx_allow_proxy_exec (prev)); switch_count = &prev->nvcsw; }kernel/sched/ext/ext.c +0
-0@@ -23,6 +23,14 @@
kernel/sched/ext/ext.c line 33
Quoted 10 lines
kernel/sched/ext/ext.c +8
-0DEFINE_RAW_SPINLOCK (scx_sched_lock); +bool scx_allow_proxy_exec (const struct task_struct *p) +{ +if (!task_on_scx (p)) +return true; + +return scx_task_sched (p)->ops.flags &SCX_OPS_ENQ_BLOCKED ; +} +
What happens when we switch in a a sched-ext scheduler that doesn't support proxy?
There would be a bunch proxy donors already preset at the time of the scheduler loading but I don't see anything in the enable path that would take care of those tasks by blocking them completely.
Is it handled elsewhere?
Signature · 2 lines
Thanks and Regards, Prateek
Andrea Righi
reviewerRe: [PATCH 09/12] sched_ext: Delegate proxy donor admission to BPF schedulers
Hi Prateek,
Quoted 33 lines · K Prateek Nayak
Hello Andrea,
Quoted 16 lines · Andrea Righi
kernel/sched/core.c +2
-1@@ -7148,7 +7181,8 @@ static void __sched notrace __schedule(int sched_mode)* task_is_blocked() will always be false). */ try_to_block_task (rq, prev, &prev_state, - !task_is_blocked (prev)); + !task_is_blocked (prev) || + !scx_allow_proxy_exec (prev)); switch_count = &prev->nvcsw; }kernel/sched/ext/ext.c +0
-0@@ -23,6 +23,14 @@kernel/sched/ext/ext.c line 33
Quoted 10 lines
kernel/sched/ext/ext.c +8
-0DEFINE_RAW_SPINLOCK (scx_sched_lock); +bool scx_allow_proxy_exec (const struct task_struct *p) +{ +if (!task_on_scx (p)) +return true; + +return scx_task_sched (p)->ops.flags &SCX_OPS_ENQ_BLOCKED ; +} +What happens when we switch in a a sched-ext scheduler that doesn't support proxy?
Quoted 5 lines
There would be a bunch proxy donors already preset at the time of the scheduler loading but I don't see anything in the enable path that would take care of those tasks by blocking them completely.
Is it handled elsewhere?
Good catch! It's not currently handled, a proxy donor that blocked before sched_ext was enabled can remain queued after being transferred to a scheduler that doesn't have SCX_OPS_ENQ_BLOCKED.
We should probably deactivate existing blocked donors while switching them to a scheduler without proxy support.
I'll fix this in the next version and add this case to the kselftest.
Thanks,
-Andrea
Andrea Righi
reviewerPATCH 10/12 sched_ext: Add selftest for blocked donor admission
In reply to Andrea RighiSCX_OPS_ENQ_BLOCKED allows BPF schedulers to receive blocked proxy donors through ops.enqueue(), while scx_bpf_task_is_blocked() identifies donor admission requests and scx_bpf_task_proxy_cpu() returns the CPU where the mutex owner is blocked. Add test coverage for this interface, including validation of the owner's CPU.
Exercise a three-task priority inversion on a single CPU using a weighted vruntime BPF scheduler. A nice +19 owner holds a shared mutex, a nice -20 donor blocks on it and a nice 0 CPU contender delays the owner.
Run the workload with SCX_OPS_ENQ_BLOCKED first disabled and then enabled. In the enabled run, count blocked donor enqueues, validate the proxy CPU, and dispatch the donor there to facilitate proxy execution. Report average mutex hold and wait times and their deltas without enforcing performance results.
Proxy execution coverage requires CONFIG_SCHED_PROXY_EXEC=y, which the selftest config selects. Expect blocked enqueue events when generic proxy execution is enabled and none when proxy execution is disabled.
The mutex is provided via a loadable kernel module, built via TEST_GEN_MODS_DIR, with the test responsible for loading, unloading, and managing its lifecycle.
--- tools/testing/selftests/sched_ext/.gitignore | 4 + tools/testing/selftests/sched_ext/Makefile | 2 + tools/testing/selftests/sched_ext/config | 2 + .../selftests/sched_ext/enq_blocked.bpf.c | 116 +++ .../testing/selftests/sched_ext/enq_blocked.c | 682 ++++++++++++++++++ .../testing/selftests/sched_ext/enq_blocked.h | 21 + .../selftests/sched_ext/test_modules/Makefile | 13 + .../test_modules/scx_enq_blocked_test.c | 134 ++++ 8 files changed, 974 insertions(+) create mode 100644 tools/testing/selftests/sched_ext/enq_blocked.bpf.c create mode 100644 tools/testing/selftests/sched_ext/enq_blocked.c create mode 100644 tools/testing/selftests/sched_ext/enq_blocked.h create mode 100644 tools/testing/selftests/sched_ext/test_modules/Makefile create mode 100644 tools/testing/selftests/sched_ext/test_modules/scx_enq_blocked_test.c
tools/testing/selftests/sched_ext/.gitignore +4 -0
@@ -4,3 +4,7 @@ !Makefile !.gitignore !config +!test_modules/ +!test_modules/scx_enq_blocked_test.c +!test_modules/Makefile +test_modules/*.mod.c
tools/testing/selftests/sched_ext/Makefile +2 -0
@@ -5,6 +5,7 @@ include ../../../scripts/Makefile.arch include ../../../scripts/Makefile.include TEST_GEN_PROGS := runner +TEST_GEN_MODS_DIR := test_modules # override lib.mk's default rules OVERRIDE_TARGETS := 1
@@ -164,6 +165,7 @@ all_test_bpfprogs := $(foreach prog,$(wildcard *.bpf.c),$(INCLUDE_DIR)/$(patsubs auto-test-targets := \ create_dsq \ dequeue \ + enq_blocked \ enq_last_no_enq_fails \ ddsp_bogus_dsq_fail \ ddsp_vtimelocal_fail \
tools/testing/selftests/sched_ext/config +2 -0
@@ -6,3 +6,5 @@ CONFIG_BPF=y CONFIG_BPF_SYSCALL=y CONFIG_DEBUG_INFO=y CONFIG_DEBUG_INFO_BTF=y +CONFIG_EXPERT=y +CONFIG_SCHED_PROXY_EXEC=y
tools/testing/selftests/sched_ext/enq_blocked.bpf.c new file +116 -0
116 lines not on this page · raw message
tools/testing/selftests/sched_ext/enq_blocked.c new file +682 -0
682 lines not on this page · raw message
tools/testing/selftests/sched_ext/enq_blocked.h new file +21 -0
@@ -0,0 +1,21 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES */ +#ifndef __ENQ_BLOCKED_H +#define __ENQ_BLOCKED_H + +#include <linux/ioctl.h> +#include <linux/types.h> + +struct enq_blocked_stats { +__u64 hold_time_ns; +__u64 wait_time_ns; +__u64 nr_holds; +__u64 nr_waits; +}; + +#define ENQ_BLOCKED_IOCTL_OWNER _IO('s', 1) +#define ENQ_BLOCKED_IOCTL_DONOR _IO('s', 2) +#define ENQ_BLOCKED_IOCTL_RESET_STATS _IO('s', 3) +#define ENQ_BLOCKED_IOCTL_GET_STATS _IOR('s', 4, struct enq_blocked_stats) + +#endif /* __ENQ_BLOCKED_H */
tools/testing/selftests/sched_ext/test_modules/Makefile new file +13 -0
@@ -0,0 +1,13 @@ +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES + +TESTMODS_DIR := $(realpath $(dir $(abspath $(lastword $(MAKEFILE_LIST))))) +KDIR ?= $(if $(O),$(O),$(realpath ../../../../..)) + +obj-m += scx_enq_blocked_test.o + +all: + +$(Q)$(MAKE) -C $(KDIR) M=$(TESTMODS_DIR) modules + +clean: + +$(Q)$(MAKE) -C $(KDIR) M=$(TESTMODS_DIR) clean
tools/testing/selftests/sched_ext/test_modules/scx_enq_blocked_test.c new file +134 -0
134 lines not on this page · raw message
Signature · 1 line
2.55.0
Andrea Righi
reviewerPATCH 11/12 sched_ext: scx_qmap: Add proxy execution support
In reply to Andrea RighiBlocked sched_ext tasks must remain available to the scheduler before the core can use their scheduling contexts to execute mutex owners. BPF schedulers opt into receiving these donor requests through SCX_OPS_ENQ_BLOCKED and are responsible for deciding when to admit them.
Enable SCX_OPS_ENQ_BLOCKED in scx_qmap and immediately admit every blocked donor. Query the mutex owner's cid and use it when allowed by the donor's affinity, falling back to the donor's current cid otherwise. Insert the donor at the head of the selected local DSQ.
Applying the same policy after a proxy migration expedites the donor on the owner's CPU so proxy execution can begin without waiting behind unrelated local work.
--- tools/sched_ext/scx_qmap.bpf.c | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-)
tools/sched_ext/scx_qmap.bpf.c +17 -1
@@ -396,6 +396,21 @@ void BPF_STRUCT_OPS(qmap_enqueue, struct task_struct *p, u64 enq_flags)*/ taskc->core_sched_seq = qa.core_sched_tail_seqs[idx]++; +/* +* Admit blocked proxy donors immediately. Dispatch directly on the mutex +* owner's cid when available to facilitate the proxy execution switch. +* SCX_ENQ_HEAD also puts the donor at the head of the local DSQ and +* expedites the destination enqueue after a proxy migration. +*/ +if (scx_bpf_task_is_blocked (p)) { + cid =scx_bpf_task_proxy_cid (p); +if (cid <0 || !cmask_test (cid, &taskc->cpus_allowed)) + cid =scx_bpf_task_cid (p); +scx_bpf_dsq_insert (p,SCX_DSQ_LOCAL_ON | cid, + slice_ns, enq_flags |SCX_ENQ_HEAD ); +return ; + } +/* * IMMED stress testing: Every immed_stress_nth'th enqueue, dispatch * directly to prev_cpu's local DSQ even when busy to force dsq->nr > 1
@@ -1216,7 +1231,8 @@ void BPF_STRUCT_OPS(qmap_sub_detach, struct scx_sub_detach_args *args) } SCX_OPS_CID_DEFINE(qmap_ops, - .flags = SCX_OPS_ENQ_EXITING | SCX_OPS_TID_TO_TASK, + .flags = SCX_OPS_ENQ_EXITING | SCX_OPS_TID_TO_TASK | + SCX_OPS_ENQ_BLOCKED, .select_cid = (void *)qmap_select_cid, .enqueue = (void *)qmap_enqueue, .dequeue = (void *)qmap_dequeue,
Signature · 1 line
2.55.0
Andrea Righi
reviewerPATCH 12/12 sched: Allow enabling proxy exec with sched_ext
In reply to Andrea RighiNow that sched_ext supports proxy execution, allow enabling both options together.
--- init/Kconfig | 2 -- 1 file changed, 2 deletions(-)
init/Kconfig +0 -2
@@ -934,8 +934,6 @@ config SCHED_PROXY_EXEC bool "Proxy Execution" # Avoid some build failures w/ PREEMPT_RT until it can be fixed depends on !PREEMPT_RT - # Need to investigate how to inform sched_ext of split contexts - depends on !SCHED_CLASS_EXT # Not particularly useful until we get to multi-rq proxying depends on EXPERT help
Signature · 1 line
2.55.0