Skip to contentSkip to conversation

sched: Make proxy execution compatible with sched_ext

PATCH Superseded Andrea Righi 4 participants 25 messages Jul 2 – Jul 3

Subsystems (3)

v2 of 14 Superseded Latest is v14 →

lore
Share & tools
lore

Andrea Righi

author reviewer
#0 lore raw

PATCH v2 1/1 sched: Make proxy execution compatible with sched_ext

This series enables using proxy execution with sched_ext and is based on early work by John Stultz [1].

Background
==========

Proxy execution (proxy-exec) lets a waiting task ("donor") donate its execution context to a mutex owner, so the owner can run while the donor stays eligible on the runqueue.

Currently, proxy execution and sched_ext are mutually exclusive at build time:
we can't enable CONFIG_SCHED_PROXY_EXEC=y and CONFIG_SCHED_CLASS_EXT=y in the same kernel.

This restriction can be problematic for Linux distributions and for anyone who wants to ship one kernel and choose features at runtime.

Why they are mutually exclusive?
================================

sched_ext schedulers drive dispatch through their own interfaces. A proxy-exec handoff can run a task that the BPF scheduler never dispatched through that path. sched_ext callbacks then observe a "current" task that does not match what the BPF side considers running, so kfuncs and helper state can see an inconsistent view of the executing task.

sched_ext also tracks runnable work through Dispatch Queues (DSQs) and BPF chosen dispatch rules, while the core scheduler still maintains classic per-CPU runqueues and pick paths. A proxy handoff can therefore switch the CPU to a task that the BPF scheduler never inserted or ordered through its DSQ interface.

DSQ state, vtime, and "who is running" bookkeeping inside the BPF program can then disagree with what the core actually executes, so helpers and kfuncs that assume their dispatched task is current may observe stale or inconsistent state.

Design: supporting proxy execution with sched_ext
=================================================

Proxy execution support is an optional per-scheduler capability: a BPF scheduler can set SCX_OPS_ENQ_BLOCKED to receive mutex blocked tasks, without this flag mutex waiters block normally and proxy execution is automaticaly disabled. When this flag is set, blocked donors are passed through ops.enqueue(), where scx_bpf_task_is_blocked() lets BPF recognize them and apply its own admission and ordering policy.

Knowing the mutex owner's location is optional, BPF may enqueue a donor on its current CPU's local DSQ and let the core resolve the owner and perform the proxy exec handoff. Schedulers that want to reduce handoff latency can use scx_bpf_task_proxy_cpu() or scx_bpf_task_proxy_cid() to obtain the owner location and steer the donation there when affinity constraints and policy allow it. The core still revalidates the owner relationship to perform the actual migration.

The donor-to-owner handoff is modeled like a "function call" from the scheduler's perspective. The donor remains the running scheduling entity selected by BPF: its scheduling context, runtime and slice are consumed while the core temporarily invokes the mutex owner's code to make the critical section progress. It is not a scheduler-visible switch to the owner.

Accordingly, the donor remains the scheduling context presented to sched_ext, while the mutex owner is treated solely as the execution context selected internally by the core scheduler. Scheduling state is accounted against rq->donor where appropriate, while rq->curr identifies the execution context. The internal owner substitution does not generate synthetic sched_ext callbacks for a task that BPF did not dispatch.

The sched_ext callback bookkeeping is adjusted accordingly. Blocked proxy donors do not generate spurious ops.running() callbacks and ops.stopping() is only called when ops.running() was emitted. The normal sched_ext migration path also leaves blocked donors to the proxy machinery, and cross-CPU migration is avoided for migration-disabled and single-CPU tasks.

scx_qmap is modified to demonstrate the opt-in policy: the scheduler immediately admits blocked donors, prefers the owner's cid when allowed by the donor's affinity and inserts the donor at the head of the selected local DSQ.

A new kselftest (enq_blocked) is also introduced to validate the proxy execution support with sched_ext. The test creates a three-task priority inversion on one
CPU: a low-priority owner (nice +19) holds a kernel mutex, a high-priority donor (nice -20) blocks on it and a nice 0 contender competes for the CPU. It runs the same workload with SCX_OPS_ENQ_BLOCKED disabled and enabled, validates blocked donor admission, the reported proxy CPU and prints average mutex hold/wait times with their deltas for manual comparison without enforcing performance thresholds. Access to the mutex is provided by a loadable kernel module built via TEST_GEN_MODS_DIR, with the test responsible for loading, unloading, and managing the module's lifecycle.

Example kselftest run:

  $ sudo tools/testing/selftests/sched_ext/runner -t enq_blocked
  ===== START =====
  TEST: enq_blocked
  DESCRIPTION: Verify BPF-driven proxy donor admission
  OUTPUT:

[SCX_OPS_ENQ_BLOCKED=disabled]

    proxy_exec=enabled
    owner_nice=19
    donor_nice=-20
    contender_nice=0
    mutex_hold_avg_ns=254084719 (254.084 ms, samples=10)
    mutex_wait_avg_ns=254095120 (254.095 ms, samples=10)
    nr_blocked_enqueues=0

[SCX_OPS_ENQ_BLOCKED=enabled]

    proxy_exec=enabled
    owner_nice=19
    donor_nice=-20
    contender_nice=0
    mutex_hold_avg_ns=228884734 (228.884 ms, samples=10)
    mutex_wait_avg_ns=207903720 (207.903 ms, samples=10)
    nr_blocked_enqueues=51

[delta: enabled - disabled]

    mutex_hold_delta_ns=-25199985 (-9.92%)
    mutex_wait_delta_ns=-46191400 (-18.18%)

ok 1 enq_blocked #
===== END =====

=============================

RESULTS:

PASSED: 1
SKIPPED: 0
FAILED: 0

References
==========

[1] https://lore.kernel.org/all/[email protected]

Git tree: git://git.kernel.org/pub/scm/linux/kernel/git/arighi/linux.git scx-proxy-exec

Changes in v2:
- Rebased onto sched_ext/for-7.3 and adapted the series to the split sched_ext implementation and cid-form scheduler interfaces.
- Replaced the global sched_proxy_exec_scx boot-time opt-in with the per-scheduler SCX_OPS_ENQ_BLOCKED capability, allowing BPF to control donor admission and ordering through ops.enqueue().
- Added scx_bpf_task_is_blocked(), scx_bpf_task_proxy_cpu(), and scx_bpf_task_proxy_cid(); enforce CPU/cid API separation for cid-form schedulers.
- Added proxy exec support to scx_qmap, including optional owner-cid steering, affinity validation, and fallback to the donor's current cid.
- Added a kselftest with a kernel mutex test and a three-task priority inversion workload, test is executed with blocked task admission disabled and enabled, validate the behavior, and report hold/wait-time deltas.
- Link to v1: https://lore.kernel.org/all/[email protected]/

Andrea Righi (10):
sched/core: Skip migration disabled tasks in proxy execution sched/core: Skip put_prev_task/set_next_task re-entry for sched_ext donors
sched_ext: Fix TOCTOU race in consume_remote_task()
sched_ext: Fix ops.running/stopping() pairing for proxy-exec donors
sched_ext: Save/restore kf_tasks[] when task ops nest
sched_ext: Skip ops.runnable() when nested in SCX_CALL_OP_TASK
sched_ext: Delegate proxy donor admission to BPF schedulers
sched_ext: Add selftest for blocked donor admission
sched_ext: scx_qmap: Add proxy execution support
sched: Allow enabling proxy exec with sched_ext

John Stultz (2):
sched/ext: Split curr|donor references properly sched/ext: Avoid migrating blocked tasks with proxy execution

 include/linux/sched/ext.h                     |   9 +
 init/Kconfig                                  |   2 -
 kernel/sched/core.c                           |  59 +-
 kernel/sched/ext/ext.c                        | 209 +++++-
 kernel/sched/ext/ext.h                        |   8 +
 kernel/sched/ext/internal.h                   |  58 +-
 kernel/sched/sched.h                          |   6 +
 tools/sched_ext/include/scx/common.bpf.h      |   3 +
 tools/sched_ext/include/scx/compat.h          |   1 +
 tools/sched_ext/scx_qmap.bpf.c                |  18 +-
 tools/testing/selftests/sched_ext/.gitignore  |   4 +
 tools/testing/selftests/sched_ext/Makefile    |   2 +
 tools/testing/selftests/sched_ext/config      |   2 +
 .../selftests/sched_ext/enq_blocked.bpf.c     | 116 +++
 .../testing/selftests/sched_ext/enq_blocked.c | 682 ++++++++++++++++++
 .../testing/selftests/sched_ext/enq_blocked.h |  21 +
 .../selftests/sched_ext/test_modules/Makefile |  13 +
 .../test_modules/scx_enq_blocked_test.c       | 134 ++++
 18 files changed, 1299 insertions(+), 48 deletions(-)
 create mode 100644 tools/testing/selftests/sched_ext/enq_blocked.bpf.c
 create mode 100644 tools/testing/selftests/sched_ext/enq_blocked.c
 create mode 100644 tools/testing/selftests/sched_ext/enq_blocked.h
 create mode 100644 tools/testing/selftests/sched_ext/test_modules/Makefile
 create mode 100644 tools/testing/selftests/sched_ext/test_modules/scx_enq_blocked_test.c
author reviewer
#1 lore raw

PATCH 1/12 sched/core: Skip migration disabled tasks in proxy execution

Never attempt to migrate migration-disabled tasks or tasks that can only run on a single CPU when switching donor's execution context, preventing task pinning violations.

---
 kernel/sched/core.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
kernel/sched/core.c +14 -0
index 3cc6fb1d20547..8a3eecc7caf5d 100644
@@ -6936,6 +6936,20 @@ find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf)
 			 */
 			if (curr_in_chain)
 				return proxy_resched_idle(rq);
+			/*
+			 * Tasks pinned to a single CPU (per-CPU kthreads via
+			 * kthread_bind(), tasks under migrate_disable()) cannot
+			 * be moved to @owner_cpu. proxy_migrate_task() uses
+			 * __set_task_cpu() which would silently violate the
+			 * pinning and leave the task to run on a CPU outside
+			 * its cpus_ptr once it is unblocked. Deactivate it on
+			 * this CPU; the owner running elsewhere will wake @p
+			 * back up when the mutex becomes available.
+			 */
+			if (p->nr_cpus_allowed == 1 || is_migration_disabled(p)) {
+				__clear_task_blocked_on(p, NULL);
+				goto deactivate;
+			}
 			goto migrate_task;
 		}
 
Signature · 1 line
2.55.0
#2 lore raw

Re: [PATCH 01/12] sched/core: Skip migration disabled tasks in proxy execution

Hello Andrea,

Quoted 14 lines · Andrea Righi

Never attempt to migrate migration-disabled tasks or tasks that can only run on a single CPU when switching donor's execution context, preventing task pinning violations.

---
 kernel/sched/core.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
kernel/sched/core.c +0 -0
index 3cc6fb1d20547..8a3eecc7caf5d 100644

kernel/sched/core.c find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf) · line 6942

Quoted 7 lines
kernel/sched/core.c +4 -0
@@ -6936,6 +6936,20 @@ find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf)
 			 */
 			if (curr_in_chain)
 				return proxy_resched_idle(rq);
+			/*
+			 * Tasks pinned to a single CPU (per-CPU kthreads via
+			 * kthread_bind(), tasks under migrate_disable()) cannot
+			 * be moved to @owner_cpu. proxy_migrate_task() uses

We only move it to donate the vruntime context. It is never actually run there.

kernel/sched/core.c find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf) · line 6945

Quoted 3 lines
kernel/sched/core.c +3 -0
+			 * __set_task_cpu() which would silently violate the
+			 * pinning and leave the task to run on a CPU outside
+			 * its cpus_ptr once it is unblocked. Deactivate it on

For the task to run as normal, p->is_blocked needs to be cleared. It is only done in the wakeup path (and sometimes in find_proxy_task() if the task is rq->curr) which ensures all the affinity / migrate disable bits get fixed when the task gets to actually run.

Where / how is this being violated?

kernel/sched/core.c find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf) · line 6952

Quoted 7 lines
kernel/sched/core.c +7 -0
+			 * this CPU; the owner running elsewhere will wake @p
+			 * back up when the mutex becomes available.
+			 */
+			if (p->nr_cpus_allowed == 1 || is_migration_disabled(p)) {
+				__clear_task_blocked_on(p, NULL);
+				goto deactivate;
+			}

Proxy depends on the ability to migrate the donor context to owner's CPU without actually running the task. The task is superficially on the CPU only to give it's runtime share to the lock owner. Feels like you are tripping a shortcoming in ext core if this is a problem.

Quoted 3 lines
kernel/sched/core.c +0 -0
 			goto migrate_task;
 		}
 
Signature · 2 lines
Thanks and Regards,
Prateek
author reviewer
#3 lore raw

Re: [PATCH 01/12] sched/core: Skip migration disabled tasks in proxy execution

Hi Prateek,

Quoted 39 lines · K Prateek Nayak

Hello Andrea,

Quoted 14 lines · Andrea Righi

Never attempt to migrate migration-disabled tasks or tasks that can only run on a single CPU when switching donor's execution context, preventing task pinning violations.

---
 kernel/sched/core.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
kernel/sched/core.c +0 -0
index 3cc6fb1d20547..8a3eecc7caf5d 100644

kernel/sched/core.c find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf) · line 6942

Quoted 7 lines
kernel/sched/core.c +4 -0
@@ -6936,6 +6936,20 @@ find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf)
 			 */
 			if (curr_in_chain)
 				return proxy_resched_idle(rq);
+			/*
+			 * Tasks pinned to a single CPU (per-CPU kthreads via
+			 * kthread_bind(), tasks under migrate_disable()) cannot
+			 * be moved to @owner_cpu. proxy_migrate_task() uses

We only move it to donate the vruntime context. It is never actually run there.

kernel/sched/core.c find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf) · line 6945

Quoted 3 lines
kernel/sched/core.c +3 -0
+			 * __set_task_cpu() which would silently violate the
+			 * pinning and leave the task to run on a CPU outside
+			 * its cpus_ptr once it is unblocked. Deactivate it on

For the task to run as normal, p->is_blocked needs to be cleared. It is only done in the wakeup path (and sometimes in find_proxy_task() if the task is rq->curr) which ensures all the affinity / migrate disable bits get fixed when the task gets to actually run.

Where / how is this being violated?

Quoted 12 lines

kernel/sched/core.c find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf) · line 6952

Quoted 7 lines
kernel/sched/core.c +7 -0
+			 * this CPU; the owner running elsewhere will wake @p
+			 * back up when the mutex becomes available.
+			 */
+			if (p->nr_cpus_allowed == 1 || is_migration_disabled(p)) {
+				__clear_task_blocked_on(p, NULL);
+				goto deactivate;
+			}

Proxy depends on the ability to migrate the donor context to owner's CPU without actually running the task. The task is superficially on the CPU only to give it's runtime share to the lock owner. Feels like you are tripping a shortcoming in ext core if this is a problem.

Agree with all of the above. As I mention in the other email, I had this in the previous version, because I was hitting migration-disabled errors. Maybe they were related to an issue in the sched_ext core, I'll remove this and repeat my tests.

Thanks,
-Andrea

maintainer
#4 lore raw

Re: [PATCH 01/12] sched/core: Skip migration disabled tasks in proxy execution

In reply to Andrea Righi
Quoted 17 lines · Andrea Righi

Never attempt to migrate migration-disabled tasks or tasks that can only run on a single CPU when switching donor's execution context, preventing task pinning violations.

---
 kernel/sched/core.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
kernel/sched/core.c +0 -0
index 3cc6fb1d20547..8a3eecc7caf5d 100644
@@ -6936,6 +6936,20 @@ find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf)
 			 */
 			if (curr_in_chain)
 				return proxy_resched_idle(rq);

kernel/sched/core.c find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf) · line 6948

Quoted 10 lines
kernel/sched/core.c +10 -0
+			/*
+			 * Tasks pinned to a single CPU (per-CPU kthreads via
+			 * kthread_bind(), tasks under migrate_disable()) cannot
+			 * be moved to @owner_cpu. proxy_migrate_task() uses
+			 * __set_task_cpu() which would silently violate the
+			 * pinning and leave the task to run on a CPU outside
+			 * its cpus_ptr once it is unblocked. Deactivate it on
+			 * this CPU; the owner running elsewhere will wake @p
+			 * back up when the mutex becomes available.
+			 */

No, this is actually OK. Remember, we only migrate the scheduling context, but the task as such won't ever execute on the remote CPU.

author reviewer
#5 lore raw

Re: [PATCH 01/12] sched/core: Skip migration disabled tasks in proxy execution

Hi Peter,

Quoted 23 lines · Peter Zijlstra
Quoted 22 lines · Andrea Righi

Never attempt to migrate migration-disabled tasks or tasks that can only run on a single CPU when switching donor's execution context, preventing task pinning violations.

---
 kernel/sched/core.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
kernel/sched/core.c +5 -0
index 3cc6fb1d20547..8a3eecc7caf5d 100644
@@ -6936,6 +6936,20 @@ find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf)
 			 */
 			if (curr_in_chain)
 				return proxy_resched_idle(rq);
+			/*
+			 * Tasks pinned to a single CPU (per-CPU kthreads via
+			 * kthread_bind(), tasks under migrate_disable()) cannot
+			 * be moved to @owner_cpu. proxy_migrate_task() uses
+			 * __set_task_cpu() which would silently violate the
Quoted 8 lines

kernel/sched/core.c find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf) · line 6948

Quoted 5 lines
kernel/sched/core.c +5 -0
+			 * pinning and leave the task to run on a CPU outside
+			 * its cpus_ptr once it is unblocked. Deactivate it on
+			 * this CPU; the owner running elsewhere will wake @p
+			 * back up when the mutex becomes available.
+			 */

No, this is actually OK. Remember, we only migrate the scheduling context, but the task as such won't ever execute on the remote CPU.

Yeah... makes sense. This patch is actually coming from the previous series, maybe I was hitting a different issue with the sched_ext core. I'll remove this one and repeat my tests.

Thanks,
-Andrea

author reviewer
#6 lore raw

PATCH 2/12 sched/core: Skip put_prev_task/set_next_task re-entry for sched_ext donors

In reply to Andrea Righi

In __schedule(), the proxy-exec donor-stabilization block calls put_prev_task() and set_next_task() when rq->donor == prev_donor and prev != next.

For sched_ext tasks, re-entering set_next_task_scx() for a donor that has already been seen by BPF ops.running via the normal pick path causes issues. It fires SCX_CALL_OP_TASK(sch, running, rq, donor) a second time, and sch->ops dispatch can land on a vtable slot in a state that yields a NULL function pointer or corrupts the stack.

Fix this by skipping the put_prev_task/set_next_task re-entry when the donor is in the ext_sched_class, since sched_ext tracks curr/donor itself.

---
 kernel/sched/core.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)
kernel/sched/core.c +7 -2
index 8a3eecc7caf5d..2e4c98b4ea6b0 100644
@@ -7185,9 +7185,14 @@ static void __sched notrace __schedule(int sched_mode)
 			 * anything, since B == B. However, A might have
 			 * missed a RT/DL balance opportunity due to being
 			 * on_cpu.
+			 *
+			 * sched_ext tracks curr/donor itself; re-entering set_next_task_scx
+			 * here dispatches through a stale/NULL BPF ops vtable.
 			 */
-			donor->sched_class->put_prev_task(rq, donor, donor);
-			donor->sched_class->set_next_task(rq, donor, true);
+			if (donor->sched_class != &ext_sched_class) {
+				donor->sched_class->put_prev_task(rq, donor, donor);
+				donor->sched_class->set_next_task(rq, donor, true);
+			}
 		}
 	} else {
 		rq_set_donor(rq, next);
Signature · 1 line
2.55.0
maintainer
#7 lore raw

Re: [PATCH 02/12] sched/core: Skip put_prev_task/set_next_task re-entry for sched_ext donors

Quoted 10 lines · Andrea Righi

In __schedule(), the proxy-exec donor-stabilization block calls put_prev_task() and set_next_task() when rq->donor == prev_donor and prev != next.

For sched_ext tasks, re-entering set_next_task_scx() for a donor that has already been seen by BPF ops.running via the normal pick path causes issues. It fires SCX_CALL_OP_TASK(sch, running, rq, donor) a second time, and sch->ops dispatch can land on a vtable slot in a state that yields a NULL function pointer or corrupts the stack.

Quoted 3 lines

Fix this by skipping the put_prev_task/set_next_task re-entry when the donor is in the ext_sched_class, since sched_ext tracks curr/donor itself.

This really sounds like a bug in ext; how is this different from the sched_change pattern doing a put/set cycle?

author reviewer
#8 lore raw

Re: [PATCH 02/12] sched/core: Skip put_prev_task/set_next_task re-entry for sched_ext donors

Quoted 11 lines · Peter Zijlstra
Quoted 10 lines · Andrea Righi

In __schedule(), the proxy-exec donor-stabilization block calls put_prev_task() and set_next_task() when rq->donor == prev_donor and prev != next.

For sched_ext tasks, re-entering set_next_task_scx() for a donor that has already been seen by BPF ops.running via the normal pick path causes issues. It fires SCX_CALL_OP_TASK(sch, running, rq, donor) a second time, and sch->ops dispatch can land on a vtable slot in a state that yields a NULL function pointer or corrupts the stack.

Quoted 6 lines
Quoted 3 lines

Fix this by skipping the put_prev_task/set_next_task re-entry when the donor is in the ext_sched_class, since sched_ext tracks curr/donor itself.

This really sounds like a bug in ext; how is this different from the sched_change pattern doing a put/set cycle?

I think you're right, the patch differs from sched_change because it doesn't bracket the put/set cycle with a dequeue/enqueue, but sched_ext still needs to support the put/set contract.

The underlying problem is that set_next_task_scx() triggers ops.running() for a blocked proxy donor even though the donor never becomes the execution context, so we need to prevent triggering ops.running() in this case. But this should be handled in sched_ext by pairing ops.running()/ops.stopping() only when the donor actually runs. This is addressed later in the series with explicit running-state tracking. I'll drop this core special case and rework that sched_ext fix.

Thanks,
-Andrea

author reviewer
#9 lore raw

PATCH 3/12 sched_ext: Split curr|donor references properly

In reply to Andrea Righi

From: John Stultz <[email protected]>

With proxy-exec, we want to do the accounting against the donor most of the time. Without proxy-exec, there should be no difference as the rq->donor and rq->curr are the same.

So rework the logic to reference the rq->donor where appropriate.

Also add donor info to scx_dump_state().

Since CONFIG_SCHED_PROXY_EXEC currently depends on !CONFIG_SCHED_CLASS_EXT, this should have no effect (other than the extra donor output in scx_dump_state), but this is one step needed to eventually remove that constraint for proxy-exec.

---
 kernel/sched/ext/ext.c | 28 ++++++++++++++++------------
 1 file changed, 16 insertions(+), 12 deletions(-)
kernel/sched/ext/ext.c +16 -12
index 1a0ec985da77d..1588565050679 100644
@@ -1145,17 +1145,17 @@ static void touch_core_sched_dispatch(struct rq *rq, struct task_struct *p)
 
 static void update_curr_scx(struct rq *rq)
 {
-	struct task_struct *curr = rq->curr;
+	struct task_struct *donor = rq->donor;
 	s64 delta_exec;
 
 	delta_exec = update_curr_common(rq);
 	if (unlikely(delta_exec <= 0))
 		return;
 
-	if (curr->scx.slice != SCX_SLICE_INF) {
-		curr->scx.slice -= min_t(u64, curr->scx.slice, delta_exec);
-		if (!curr->scx.slice)
-			touch_core_sched(rq, curr);
+	if (donor->scx.slice != SCX_SLICE_INF) {
+		donor->scx.slice -= min_t(u64, donor->scx.slice, delta_exec);
+		if (!donor->scx.slice)
+			touch_core_sched(rq, donor);
 	}
 
 	dl_server_update(&rq->ext_server, delta_exec);
@@ -1316,8 +1316,8 @@ static void local_dsq_post_enq(struct scx_sched *sch, struct scx_dispatch_q *dsq
 	if (rq->scx.flags & SCX_RQ_IN_BALANCE)
 		return;
 
-	if ((enq_flags & SCX_ENQ_PREEMPT) && p != rq->curr &&
-	    rq->curr->sched_class == &ext_sched_class) {
+	if ((enq_flags & SCX_ENQ_PREEMPT) && p != rq->donor &&
+	    rq->donor->sched_class == &ext_sched_class) {
 		rq->curr->scx.slice = 0;
 		resched_curr(rq);
 	}
@@ -2464,7 +2464,8 @@ static void dispatch_to_local_dsq(struct scx_sched *sch, struct rq *rq,
 		}
 
 		/* if the destination CPU is idle, wake it up */
-		if (!fallback && sched_class_above(p->sched_class, dst_rq->curr->sched_class))
+		if (!fallback && sched_class_above(p->sched_class,
+						      dst_rq->donor->sched_class))
 			resched_curr(dst_rq);
 	}
 
@@ -2876,7 +2877,7 @@ static struct task_struct *first_local_task(struct rq *rq)
 static struct task_struct *
 do_pick_task_scx(struct rq *rq, struct rq_flags *rf, bool force_scx)
 {
-	struct task_struct *prev = rq->curr;
+	struct task_struct *prev = rq->donor;
 	bool keep_prev;
 	struct task_struct *p;
 
@@ -4029,7 +4030,7 @@ static void run_deferred(struct rq *rq)
 #ifdef CONFIG_NO_HZ_FULL
 bool scx_can_stop_tick(struct rq *rq)
 {
-	struct task_struct *p = rq->curr;
+	struct task_struct *p = rq->donor;
 	struct scx_sched *sch = scx_task_sched(p);
 
 	if (p->sched_class != &ext_sched_class)
@@ -6007,6 +6008,9 @@ static void scx_dump_cpu(struct scx_sched *sch, struct seq_buf *s,
 	dump_line(&ns, "          curr=%s[%d] class=%ps",
 		  rq->curr->comm, rq->curr->pid,
 		  rq->curr->sched_class);
+	dump_line(&ns, "          donor=%s[%d] class=%ps",
+		  rq->donor->comm, rq->donor->pid,
+		  rq->donor->sched_class);
 	if (!cpumask_empty(rq->scx.cpus_to_kick))
 		dump_line(&ns, "  cpus_to_kick   : %*pb",
 			  cpumask_pr_args(rq->scx.cpus_to_kick));
@@ -7452,7 +7456,7 @@ static bool kick_one_cpu(s32 cpu, struct rq *this_rq, unsigned long *ksyncs)
 	unsigned long flags;
 
 	raw_spin_rq_lock_irqsave(rq, flags);
-	cur_class = rq->curr->sched_class;
+	cur_class = rq->donor->sched_class;
 
 	/*
 	 * During CPU hotplug, a CPU may depend on kicking itself to make
@@ -7464,7 +7468,7 @@ static bool kick_one_cpu(s32 cpu, struct rq *this_rq, unsigned long *ksyncs)
 	    !sched_class_above(cur_class, &ext_sched_class)) {
 		if (cpumask_test_cpu(cpu, this_scx->cpus_to_preempt)) {
 			if (cur_class == &ext_sched_class)
-				rq->curr->scx.slice = 0;
+				rq->donor->scx.slice = 0;
 			cpumask_clear_cpu(cpu, this_scx->cpus_to_preempt);
 		}
 
Signature · 1 line
2.55.0
#10 lore raw

Re: [PATCH 03/12] sched_ext: Split curr|donor references properly

Quoted 5 lines · Andrea Righi

From: John Stultz <[email protected]>

With proxy-exec, we want to do the accounting against the donor most of the time. Without proxy-exec, there should be no difference as the rq->donor and rq->curr are the same.

Trying to understand more of the situation when donor and curr in different sched_class? one is in scx and the other is not.
could you pls help to explain more with this information in commit message?

Quoted 43 lines

So rework the logic to reference the rq->donor where appropriate.

Also add donor info to scx_dump_state().

Since CONFIG_SCHED_PROXY_EXEC currently depends on !CONFIG_SCHED_CLASS_EXT, this should have no effect (other than the extra donor output in scx_dump_state), but this is one step needed to eventually remove that constraint for proxy-exec.

---
 kernel/sched/ext/ext.c | 28 ++++++++++++++++------------
 1 file changed, 16 insertions(+), 12 deletions(-)
kernel/sched/ext/ext.c +5 -5
index 1a0ec985da77d..1588565050679 100644
@@ -1145,17 +1145,17 @@ static void touch_core_sched_dispatch(struct rq *rq, struct task_struct *p)
 
 static void update_curr_scx(struct rq *rq)
 {
-	struct task_struct *curr = rq->curr;
+	struct task_struct *donor = rq->donor;
 	s64 delta_exec;
 
 	delta_exec = update_curr_common(rq);
 	if (unlikely(delta_exec <= 0))
 		return;
 
-	if (curr->scx.slice != SCX_SLICE_INF) {
-		curr->scx.slice -= min_t(u64, curr->scx.slice, delta_exec);
-		if (!curr->scx.slice)
-			touch_core_sched(rq, curr);
+	if (donor->scx.slice != SCX_SLICE_INF) {
+		donor->scx.slice -= min_t(u64, donor->scx.slice, delta_exec);
+		if (!donor->scx.slice)
+			touch_core_sched(rq, donor);
 	}
 
 	dl_server_update(&rq->ext_server, delta_exec);

kernel/sched/ext/ext.c static void local_dsq_post_enq(struct scx_sched *sch, struct scx_dispatch_q *dsq · line 1321

Quoted 8 lines
kernel/sched/ext/ext.c +2 -2
@@ -1316,8 +1316,8 @@ static void local_dsq_post_enq(struct scx_sched *sch, struct scx_dispatch_q *dsq
 	if (rq->scx.flags & SCX_RQ_IN_BALANCE)
 		return;
 
-	if ((enq_flags & SCX_ENQ_PREEMPT) && p != rq->curr &&
-	    rq->curr->sched_class == &ext_sched_class) {
+	if ((enq_flags & SCX_ENQ_PREEMPT) && p != rq->donor &&
+	    rq->donor->sched_class == &ext_sched_class) {
 		rq->curr->scx.slice = 0;

Do you forget to update rq->curr with rq->donor here?

Quoted 58 lines
kernel/sched/ext/ext.c +9 -5
 		resched_curr(rq);
 	}
@@ -2464,7 +2464,8 @@ static void dispatch_to_local_dsq(struct scx_sched *sch, struct rq *rq,
 		}
 
 		/* if the destination CPU is idle, wake it up */
-		if (!fallback && sched_class_above(p->sched_class, dst_rq->curr->sched_class))
+		if (!fallback && sched_class_above(p->sched_class,
+						      dst_rq->donor->sched_class))
 			resched_curr(dst_rq);
 	}
 
@@ -2876,7 +2877,7 @@ static struct task_struct *first_local_task(struct rq *rq)
 static struct task_struct *
 do_pick_task_scx(struct rq *rq, struct rq_flags *rf, bool force_scx)
 {
-	struct task_struct *prev = rq->curr;
+	struct task_struct *prev = rq->donor;
 	bool keep_prev;
 	struct task_struct *p;
 
@@ -4029,7 +4030,7 @@ static void run_deferred(struct rq *rq)
 #ifdef CONFIG_NO_HZ_FULL
 bool scx_can_stop_tick(struct rq *rq)
 {
-	struct task_struct *p = rq->curr;
+	struct task_struct *p = rq->donor;
 	struct scx_sched *sch = scx_task_sched(p);
 
 	if (p->sched_class != &ext_sched_class)
@@ -6007,6 +6008,9 @@ static void scx_dump_cpu(struct scx_sched *sch, struct seq_buf *s,
 	dump_line(&ns, "          curr=%s[%d] class=%ps",
 		  rq->curr->comm, rq->curr->pid,
 		  rq->curr->sched_class);
+	dump_line(&ns, "          donor=%s[%d] class=%ps",
+		  rq->donor->comm, rq->donor->pid,
+		  rq->donor->sched_class);
 	if (!cpumask_empty(rq->scx.cpus_to_kick))
 		dump_line(&ns, "  cpus_to_kick   : %*pb",
 			  cpumask_pr_args(rq->scx.cpus_to_kick));
@@ -7452,7 +7456,7 @@ static bool kick_one_cpu(s32 cpu, struct rq *this_rq, unsigned long *ksyncs)
 	unsigned long flags;
 
 	raw_spin_rq_lock_irqsave(rq, flags);
-	cur_class = rq->curr->sched_class;
+	cur_class = rq->donor->sched_class;
 
 	/*
 	 * During CPU hotplug, a CPU may depend on kicking itself to make
@@ -7464,7 +7468,7 @@ static bool kick_one_cpu(s32 cpu, struct rq *this_rq, unsigned long *ksyncs)
 	    !sched_class_above(cur_class, &ext_sched_class)) {
 		if (cpumask_test_cpu(cpu, this_scx->cpus_to_preempt)) {
 			if (cur_class == &ext_sched_class)
-				rq->curr->scx.slice = 0;
+				rq->donor->scx.slice = 0;
 			cpumask_clear_cpu(cpu, this_scx->cpus_to_preempt);
 		}
 
Signature · 2 lines
Thx and BRs,
Aiqun(Maria) Yu
author reviewer
#11 lore raw

Re: [PATCH 03/12] sched_ext: Split curr|donor references properly

Hi Maria,

Quoted 10 lines · Aiqun(Maria) Yu
Quoted 5 lines · Andrea Righi

From: John Stultz <[email protected]>

With proxy-exec, we want to do the accounting against the donor most of the time. Without proxy-exec, there should be no difference as the rq->donor and rq->curr are the same.

Trying to understand more of the situation when donor and curr in different sched_class? one is in scx and the other is not.
could you pls help to explain more with this information in commit message?

Sure, I'll add more details. Do you think something like the following would help (maybe in a shorter form)?

Let use FAIR and EXT and assume we run the BPF scheduler in partial mode. We can replace FAIR with RT/deadline, the result is the same.

Terminology:

D = blocked donor
M = mutex
O = mutex owner
T = competing runnable task

D -----------------> M -------------> O ----------------> T [donor] blocked on [mutex] owned by [owner] preempted by [task]
\_________________________________^
donates scheduling context

During a proxy exec switch:
- D supplies scheduling class, priority and runtime budget
- O is the task whose code is physically executing
- T is a competing task that preempts O

Scenarios:

1) D is EXT, O is EXT, T is EXT

Result:
- D can interrupt T depending on the BPF scheduling policy
- O is executed with D's EXT priority and runtime budget
- When D runs, T waits in EXT

2) D is EXT, O is EXT, T is FAIR

Result:
- D is visible to the BPF scheduler
- D cannot preempt T (EXT < FAIR)
- Once T stops, BPF dispatches D
- D executes O using D's EXT priority and runtime budget
- if T becomes runnable again, it preempts the D/O proxy execution

3) D is EXT, O is FAIR, T is EXT

Result:
- Not possible, T can't preempt O (EXT < FAIR)

4) D is EXT, O is FAIR, T is FAIR

Result:
- D cannot boost O because EXT < FAIR
- O and T continue competing under FAIR
- O eventually runs and releases M
- D then wakes and resumes normal EXT scheduling

5) D is FAIR, O is EXT, T is EXT

Result:
- D preempts T immediately (higher sched class)
- O is executed with D's FAIR priority and runtime budget
- When D runs, T waits in EXT
- D is not visible to the BPF scheduler

6) D is FAIR, O is EXT, T is FAIR

Result:
- D runs based on its FAIR deadline (competing with T)
- O is executed with D's FAIR priority and runtime budget
- When D runs, T waits in FAIR
- D is not visible to the BPF scheduler

7) D is FAIR, O is FAIR, T is EXT

Result:
- Not possible, T can't preempt O (EXT < FAIR)

8) D is FAIR, O is FAIR, T is FAIR

Result:
- O, T and D all have FAIR scheduling contexts
- D remains runnable as a blocked proxy donor
- When CFS selects D, O executes using D's FAIR scheduling context
- When CFS selects O, O executes using its own FAIR context
- When CFS selects T, T executes normally
- D is not visible to the BPF scheduler

Quoted 47 lines
Quoted 47 lines

So rework the logic to reference the rq->donor where appropriate.

Also add donor info to scx_dump_state().

Since CONFIG_SCHED_PROXY_EXEC currently depends on !CONFIG_SCHED_CLASS_EXT, this should have no effect (other than the extra donor output in scx_dump_state), but this is one step needed to eventually remove that constraint for proxy-exec.

---
 kernel/sched/ext/ext.c | 28 ++++++++++++++++------------
 1 file changed, 16 insertions(+), 12 deletions(-)
kernel/sched/ext/ext.c +5 -5
index 1a0ec985da77d..1588565050679 100644
@@ -1145,17 +1145,17 @@ static void touch_core_sched_dispatch(struct rq *rq, struct task_struct *p)
 
 static void update_curr_scx(struct rq *rq)
 {
-	struct task_struct *curr = rq->curr;
+	struct task_struct *donor = rq->donor;
 	s64 delta_exec;
 
 	delta_exec = update_curr_common(rq);
 	if (unlikely(delta_exec <= 0))
 		return;
 
-	if (curr->scx.slice != SCX_SLICE_INF) {
-		curr->scx.slice -= min_t(u64, curr->scx.slice, delta_exec);
-		if (!curr->scx.slice)
-			touch_core_sched(rq, curr);
+	if (donor->scx.slice != SCX_SLICE_INF) {
+		donor->scx.slice -= min_t(u64, donor->scx.slice, delta_exec);
+		if (!donor->scx.slice)
+			touch_core_sched(rq, donor);
 	}
 
 	dl_server_update(&rq->ext_server, delta_exec);
@@ -1316,8 +1316,8 @@ static void local_dsq_post_enq(struct scx_sched *sch, struct scx_dispatch_q *dsq
 	if (rq->scx.flags & SCX_RQ_IN_BALANCE)
 		return;
 
Quoted 7 lines

kernel/sched/ext/ext.c static void local_dsq_post_enq(struct scx_sched *sch, struct scx_dispatch_q *dsq · line 1321

Quoted 5 lines
kernel/sched/ext/ext.c +2 -2
-	if ((enq_flags & SCX_ENQ_PREEMPT) && p != rq->curr &&
-	    rq->curr->sched_class == &ext_sched_class) {
+	if ((enq_flags & SCX_ENQ_PREEMPT) && p != rq->donor &&
+	    rq->donor->sched_class == &ext_sched_class) {
 		rq->curr->scx.slice = 0;

Do you forget to update rq->curr with rq->donor here?

Yes, good catch. This should be rq->donor->scx.slice = 0. I'll fix it in the next version.

Thanks,
-Andrea

author reviewer
#12 lore raw

PATCH 4/12 sched_ext: Avoid migrating blocked tasks with proxy execution

In reply to Andrea Righi

From: John Stultz <[email protected]>

With proxy execution enabled, mutex blocked tasks stay on the runqueue. Later with donor migration they will be migrated when necessary by the core scheduler to boost lock owners.

Don't try to migrate mutex blocked tasks, the proxy logic will handle that.

---
 kernel/sched/ext/ext.c | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)
kernel/sched/ext/ext.c +25 -0
index 1588565050679..9a672b9a55f6e 100644
@@ -2150,6 +2150,14 @@ static bool task_can_run_on_remote_rq(struct scx_sched *sch,
 
 	WARN_ON_ONCE(task_cpu(p) == cpu);
 
+	/* Make sure tasks aren't on a cpu */
+	if (task_on_cpu(task_rq(p), p))
+		return false;
+
+	/* Don't migrate blocked tasks, proxy-exec will handle this */
+	if (task_is_blocked(p))
+		return false;
+
 	/*
 	 * If @p has migration disabled, @p->cpus_ptr is updated to contain only
 	 * the pinned CPU in migrate_disable_switch() while @p is being switched
@@ -2784,6 +2792,23 @@ static void put_prev_task_scx(struct rq *rq, struct task_struct *p,
 	if (p->scx.flags & SCX_TASK_QUEUED) {
 		set_task_runnable(rq, p);
 
+		/*
+		 * Mutex-blocked donors stay queued on the runqueue under proxy
+		 * execution, but the donor never runs as itself, proxy-exec
+		 * walks the blocked_on chain on the next __schedule() and runs
+		 * the lock owner in its place.
+		 *
+		 * Put the donor on the local DSQ directly, so pick_next_task()
+		 * can still see it, find_proxy_task() will be invoked on
+		 * next->blocked_on and either run the chain owner here, or call
+		 * proxy_force_return() and let BPF make a new dispatch decision
+		 * once the task is no longer blocked.
+		 */
+		if (task_is_blocked(p)) {
+			dispatch_enqueue(sch, rq, &rq->scx.local_dsq, p, 0);
+			goto switch_class;
+		}
+
 		/*
 		 * If @p has slice left and is being put, @p is getting
 		 * preempted by a higher priority scheduler class or core-sched
Signature · 1 line
2.55.0
#13 lore raw

Re: [PATCH 04/12] sched_ext: Avoid migrating blocked tasks with proxy execution

Quoted 6 lines · Andrea Righi

From: John Stultz <[email protected]>

With proxy execution enabled, mutex blocked tasks stay on the runqueue. Later with donor migration they will be migrated when necessary by the core scheduler to boost lock owners.

Quoted 6 lines

Don't try to migrate mutex blocked tasks, the proxy logic will handle that.

SOB was suggested to have the current committer at the last line. Not sure if it is the same rule for the subsystem here.

My understanding would be:

Quoted 9 lines
---
 kernel/sched/ext/ext.c | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)
kernel/sched/ext/ext.c +0 -0
index 1588565050679..9a672b9a55f6e 100644

kernel/sched/ext/ext.c static bool task_can_run_on_remote_rq(struct scx_sched *sch, · line 2159

Quoted 10 lines
kernel/sched/ext/ext.c +7 -0
@@ -2150,6 +2150,14 @@ static bool task_can_run_on_remote_rq(struct scx_sched *sch,
 
 	WARN_ON_ONCE(task_cpu(p) == cpu);
 
+	/* Make sure tasks aren't on a cpu */
+	if (task_on_cpu(task_rq(p), p))
+		return false;
+
+	/* Don't migrate blocked tasks, proxy-exec will handle this */
+	if (task_is_blocked(p))
+		return false;

what about return true here for owner_cpu?
if the owner_cpu is in the move_task_between_dsqs stage for example.

Quoted 28 lines
kernel/sched/ext/ext.c +18 -0
+
 	/*
 	 * If @p has migration disabled, @p->cpus_ptr is updated to contain only
 	 * the pinned CPU in migrate_disable_switch() while @p is being switched
@@ -2784,6 +2792,23 @@ static void put_prev_task_scx(struct rq *rq, struct task_struct *p,
 	if (p->scx.flags & SCX_TASK_QUEUED) {
 		set_task_runnable(rq, p);
 
+		/*
+		 * Mutex-blocked donors stay queued on the runqueue under proxy
+		 * execution, but the donor never runs as itself, proxy-exec
+		 * walks the blocked_on chain on the next __schedule() and runs
+		 * the lock owner in its place.
+		 *
+		 * Put the donor on the local DSQ directly, so pick_next_task()
+		 * can still see it, find_proxy_task() will be invoked on
+		 * next->blocked_on and either run the chain owner here, or call
+		 * proxy_force_return() and let BPF make a new dispatch decision
+		 * once the task is no longer blocked.
+		 */
+		if (task_is_blocked(p)) {
+			dispatch_enqueue(sch, rq, &rq->scx.local_dsq, p, 0);
+			goto switch_class;
+		}
+
 		/*
 		 * If @p has slice left and is being put, @p is getting
 		 * preempted by a higher priority scheduler class or core-sched
Signature · 2 lines
Thx and BRs,
Aiqun(Maria) Yu
author reviewer
#14 lore raw

Re: [PATCH 04/12] sched_ext: Avoid migrating blocked tasks with proxy execution

Hi Maria,

Quoted 14 lines · Aiqun(Maria) Yu
Quoted 6 lines · Andrea Righi

From: John Stultz <[email protected]>

With proxy execution enabled, mutex blocked tasks stay on the runqueue. Later with donor migration they will be migrated when necessary by the core scheduler to boost lock owners.

Quoted 6 lines

Don't try to migrate mutex blocked tasks, the proxy logic will handle that.

Quoted 7 lines

SOB was suggested to have the current committer at the last line. Not sure if it is the same rule for the subsystem here.

My understanding would be:

You're right, I'll fix it in the next version.

Quoted 14 lines
Quoted 14 lines

---
 kernel/sched/ext/ext.c | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)
kernel/sched/ext/ext.c +0 -0
index 1588565050679..9a672b9a55f6e 100644
@@ -2150,6 +2150,14 @@ static bool task_can_run_on_remote_rq(struct scx_sched *sch,
 
 	WARN_ON_ONCE(task_cpu(p) == cpu);
 
Quoted 10 lines

kernel/sched/ext/ext.c static bool task_can_run_on_remote_rq(struct scx_sched *sch, · line 2159

Quoted 7 lines
kernel/sched/ext/ext.c +7 -0
+	/* Make sure tasks aren't on a cpu */
+	if (task_on_cpu(task_rq(p), p))
+		return false;
+
+	/* Don't migrate blocked tasks, proxy-exec will handle this */
+	if (task_is_blocked(p))
+		return false;

what about return true here for owner_cpu?
if the owner_cpu is in the move_task_between_dsqs stage for example.

Good point. Returning false for every blocked task is too restrictive, because it'd reject potential remote placements requested by the BPF scheduler from ops.enqueue().

But we can't simply return true either, otherwise we would bypass all the checks below, allowing sched_ext to move the donor even if the destination is outside its affinity mask or the destination rq is offline.

I think we should do something like this:

  if (task_is_blocked(p) && task_current_donor(task_rq(p), p))
          return false;

In practice, normal migration can happen if a blocker donor is not already the rq's current scheduling context.

With that, the BPF scheduler should be able to decide the "call back home" CPU for the donor, if it performs a direct dispatch to a local DSQ different than its current CPU's local DSQ.

Thanks,
-Andrea

author reviewer
#15 lore raw

PATCH 5/12 sched_ext: Fix TOCTOU race in consume_remote_task()

In reply to Andrea Righi

When pulling a task from a non-local DSQ, consume_dispatch_q() checks if the task can run on the destination rq via task_can_run_on_remote_rq(). However, it then drops the destination rq lock and locks the source rq in consume_remote_task() -> unlink_dsq_and_lock_src_rq(). During this window, the task might have become migration disabled, making it invalid to migrate it to the destination rq.

Fix this by re-evaluating task_can_run_on_remote_rq() in consume_remote_task() after the source rq is locked. If the task can no longer be migrated, we clear its DSQ association, reset the holding CPU, and enqueue it to the source rq's local DSQ instead.

While the destination rq lock is dropped, clear the tracked rq state and restore it after reacquiring the lock. Otherwise, a nested ops.dequeue() callback can attempt to restore an rq which is no longer locked.

---
 kernel/sched/ext/ext.c | 43 +++++++++++++++++++++++++++++++++++-------
 1 file changed, 36 insertions(+), 7 deletions(-)
kernel/sched/ext/ext.c +36 -7
index 9a672b9a55f6e..189ba9c42043a 100644
@@ -2248,20 +2248,49 @@ static bool unlink_dsq_and_lock_src_rq(struct task_struct *p,
 		!WARN_ON_ONCE(src_rq != task_rq(p));
 }
 
-static bool consume_remote_task(struct rq *this_rq,
+static bool consume_remote_task(struct scx_sched *sch, struct rq *this_rq,
 				struct task_struct *p, u64 enq_flags,
 				struct scx_dispatch_q *dsq, struct rq *src_rq)
 {
+	struct rq *tracked_rq = scx_locked_rq();
+	bool consumed = false;
+
+	/*
+	 * consume_remote_task() may be called from an SCX op with @this_rq
+	 * recorded as the currently locked rq. Clear the tracking while the rq
+	 * lock is dropped so nested callbacks don't save and later try to restore
+	 * an rq which isn't locked anymore.
+	 */
+	if (tracked_rq) {
+		WARN_ON_ONCE(tracked_rq != this_rq);
+		update_locked_rq(NULL);
+	}
 	raw_spin_rq_unlock(this_rq);
 
 	if (unlink_dsq_and_lock_src_rq(p, dsq, src_rq)) {
+		if (unlikely(!task_can_run_on_remote_rq(sch, p, this_rq, true))) {
+			p->scx.dsq = NULL;
+			p->scx.holding_cpu = -1;
+			scx_dispatch_enqueue(sch, src_rq, &src_rq->scx.local_dsq, p,
+					     enq_flags | SCX_ENQ_CLEAR_OPSS);
+			if (sched_class_above(p->sched_class, src_rq->donor->sched_class))
+				resched_curr(src_rq);
+			raw_spin_rq_unlock(src_rq);
+			goto relock;
+		}
 		move_remote_task_to_local_dsq(p, enq_flags, src_rq, this_rq);
-		return true;
-	} else {
-		raw_spin_rq_unlock(src_rq);
-		raw_spin_rq_lock(this_rq);
-		return false;
+		consumed = true;
+		goto restore;
 	}
+	raw_spin_rq_unlock(src_rq);
+
+relock:
+	raw_spin_rq_lock(this_rq);
+restore:
+	if (tracked_rq)
+		update_locked_rq(tracked_rq);
+
+	return consumed;
 }
 
 /**
@@ -2371,7 +2400,7 @@ bool scx_consume_dispatch_q(struct scx_sched *sch, struct rq *rq,
 		}
 
 		if (task_can_run_on_remote_rq(sch, p, rq, false)) {
-			if (likely(consume_remote_task(rq, p, enq_flags, dsq, task_rq)))
+			if (likely(consume_remote_task(sch, rq, p, enq_flags, dsq, task_rq)))
 				return true;
 			goto retry;
 		}
Signature · 1 line
2.55.0
author reviewer
#16 lore raw

PATCH 6/12 sched_ext: Fix ops.running/stopping() pairing for proxy-exec donors

In reply to Andrea Righi

With proxy-exec, pick_next_task() can return a task with blocked_on set (a proxy donor); put_prev_set_next_task() then calls set_next_task_scx() on this "ghost" task, which fires ops.running(). However, the task never actually runs.

If we simply short-circuit set_next_task_scx() for blocked tasks, we break DSQ bookkeeping. If we only skip ops.running(), we create an ops.enqueue() -> ops.stopping() pair without running, because ops.stopping() is still called in put_prev_task_scx().

Fix this by introducing a new flag SCX_TASK_IS_RUNNING to track whether ops.running() was actually called. Skip ops.running() for blocked tasks, and only call ops.stopping() if SCX_TASK_IS_RUNNING is set. This ensures that running and stopping callbacks are perfectly paired even when a blocked task is picked as a proxy donor.

---
 include/linux/sched/ext.h |  2 ++
 kernel/sched/core.c       |  2 +-
 kernel/sched/ext/ext.c    | 14 +++++++++++---
 kernel/sched/ext/ext.h    |  6 ++++++
 4 files changed, 20 insertions(+), 4 deletions(-)
include/linux/sched/ext.h +2 -0
index 75cb8b119fb79..e599bb86f8acd 100644
@@ -102,6 +102,8 @@ enum scx_ent_flags {
 	SCX_TASK_SUB_INIT	= 1 << 4, /* task being initialized for a sub sched */
 	SCX_TASK_IMMED		= 1 << 5, /* task is on local DSQ with %SCX_ENQ_IMMED */
 
+	SCX_TASK_IS_RUNNING	= 1 << 6, /* ops.running() has been called */
+
 	/*
 	 * Bits 8 to 10 are used to carry task state:
 	 *
kernel/sched/core.c +1 -1
index 2e4c98b4ea6b0..6aedb26c08ee7 100644
@@ -7189,7 +7189,7 @@ static void __sched notrace __schedule(int sched_mode)
 			 * sched_ext tracks curr/donor itself; re-entering set_next_task_scx
 			 * here dispatches through a stale/NULL BPF ops vtable.
 			 */
-			if (donor->sched_class != &ext_sched_class) {
+			if (!is_ext_class(donor)) {
 				donor->sched_class->put_prev_task(rq, donor, donor);
 				donor->sched_class->set_next_task(rq, donor, true);
 			}
kernel/sched/ext/ext.c +11 -3
index 189ba9c42043a..b0ec579e3a3ef 100644
@@ -1985,9 +1985,11 @@ static bool dequeue_task_scx(struct rq *rq, struct task_struct *p, int core_deq_
 	 * information meaningful to the BPF scheduler and can be suppressed by
 	 * skipping the callbacks if the task is !QUEUED.
 	 */
-	if (SCX_HAS_OP(sch, stopping) && task_current(rq, p)) {
+	if (SCX_HAS_OP(sch, stopping) && task_current(rq, p) &&
+	    (p->scx.flags & SCX_TASK_IS_RUNNING)) {
 		update_curr_scx(rq);
 		SCX_CALL_OP_TASK(sch, stopping, rq, p, false);
+		p->scx.flags &= ~SCX_TASK_IS_RUNNING;
 	}
 
 	if (SCX_HAS_OP(sch, quiescent) && !task_on_rq_migrating(p))
@@ -2725,8 +2727,11 @@ static void set_next_task_scx(struct rq *rq, struct task_struct *p, bool first)
 	p->se.exec_start = rq_clock_task(rq);
 
 	/* see dequeue_task_scx() on why we skip when !QUEUED */
-	if (SCX_HAS_OP(sch, running) && (p->scx.flags & SCX_TASK_QUEUED))
+	if (SCX_HAS_OP(sch, running) && (p->scx.flags & SCX_TASK_QUEUED) &&
+	    !task_is_blocked(p)) {
 		SCX_CALL_OP_TASK(sch, running, rq, p);
+		p->scx.flags |= SCX_TASK_IS_RUNNING;
+	}
 
 	clr_task_runnable(p, true);
 
@@ -2815,8 +2820,11 @@ static void put_prev_task_scx(struct rq *rq, struct task_struct *p,
 	update_curr_scx(rq);
 
 	/* see dequeue_task_scx() on why we skip when !QUEUED */
-	if (SCX_HAS_OP(sch, stopping) && (p->scx.flags & SCX_TASK_QUEUED))
+	if (SCX_HAS_OP(sch, stopping) && (p->scx.flags & SCX_TASK_QUEUED) &&
+	    (p->scx.flags & SCX_TASK_IS_RUNNING)) {
 		SCX_CALL_OP_TASK(sch, stopping, rq, p, true);
+		p->scx.flags &= ~SCX_TASK_IS_RUNNING;
+	}
 
 	if (p->scx.flags & SCX_TASK_QUEUED) {
 		set_task_runnable(rq, p);
kernel/sched/ext/ext.h +6 -0
index 0b7fc46aee08c..c7fa4d06ac7d3 100644
@@ -35,6 +35,11 @@ static inline bool task_on_scx(const struct task_struct *p)
 	return scx_enabled() && p->sched_class == &ext_sched_class;
 }
 
+static inline bool is_ext_class(const struct task_struct *p)
+{
+	return p->sched_class == &ext_sched_class;
+}
+
 #ifdef CONFIG_SCHED_CORE
 bool scx_prio_less(const struct task_struct *a, const struct task_struct *b,
 		   bool in_fi);
@@ -53,6 +58,7 @@ static inline void scx_rq_activate(struct rq *rq) {}
 static inline void scx_rq_deactivate(struct rq *rq) {}
 static inline int scx_check_setscheduler(struct task_struct *p, int policy) { return 0; }
 static inline bool task_on_scx(const struct task_struct *p) { return false; }
+static inline bool is_ext_class(const struct task_struct *p) { return false; }
 static inline bool scx_allow_ttwu_queue(const struct task_struct *p) { return true; }
 static inline void init_sched_ext_class(void) {}
 
Signature · 1 line
2.55.0
author reviewer
#17 lore raw

PATCH 7/12 sched_ext: Save/restore kf_tasks[] when task ops nest

In reply to Andrea Righi

SCX_CALL_OP_TASK*() stored the subject task in current->scx.kf_tasks[] and assumed ops would not nest. A BPF ops.running() callback can call kfuncs (e.g. scx_bpf_dsq_insert) that enqueue work and trigger enqueue_task_scx() -> ops.runnable(), which used SCX_CALL_OP_TASK again and overwrote kf_tasks[0] then cleared it, leaving the running context wrong and leading to NULL function dispatches from BPF helpers.

Save and restore kf_tasks[] (both slots for the two-task variant) around each invocation so nested task-based ops preserve the outer context.

---
 kernel/sched/ext/ext.c      | 10 ++++++++--
 kernel/sched/ext/internal.h | 33 +++++++++++++++++++++++----------
 2 files changed, 31 insertions(+), 12 deletions(-)
kernel/sched/ext/ext.c +8 -2
index b0ec579e3a3ef..4cefe5acb36ff 100644
@@ -410,8 +410,12 @@ static inline void scx_call_op_set_cpumask(struct scx_sched *sch, struct rq *rq,
 					   struct task_struct *task,
 					   const struct cpumask *cpumask)
 {
-	WARN_ON_ONCE(current->scx.kf_tasks[0]);
+	struct task_struct *__scx_kf0_sv = current->scx.kf_tasks[0];
+	struct task_struct *__scx_kf1_sv = current->scx.kf_tasks[1];
+
+	current->scx.kf_nest++;
 	current->scx.kf_tasks[0] = task;
+	current->scx.kf_tasks[1] = NULL;
 	if (rq)
 		update_locked_rq(rq);
 
@@ -430,7 +434,9 @@ static inline void scx_call_op_set_cpumask(struct scx_sched *sch, struct rq *rq,
 
 	if (rq)
 		update_locked_rq(NULL);
-	current->scx.kf_tasks[0] = NULL;
+	current->scx.kf_tasks[0] = __scx_kf0_sv;
+	current->scx.kf_tasks[1] = __scx_kf1_sv;
+	current->scx.kf_nest--;
 }
 
 enum scx_dsq_iter_flags {
kernel/sched/ext/internal.h +23 -10
index f9fe7c6ebc4be..7d92826fc3a5f 100644
@@ -1792,37 +1792,50 @@ do {										\
  * pi_lock held by try_to_wake_up() with rq tracking via scx_rq.in_select_cpu.
  * So if kf_tasks[] is set, @p's scheduler-protected fields are stable.
  *
- * kf_tasks[] can not stack, so task-based SCX ops must not nest. The
- * WARN_ON_ONCE() in each macro catches a re-entry of any of the three variants
- * while a previous one is still in progress.
+ * Task-based SCX ops may nest (e.g. ops.running() calling a kfunc that ends up
+ * in enqueue_task_scx() -> ops.runnable()). Save and restore kf_tasks[] around
+ * each invocation so the outer op's context is restored for kfuncs and for
+ * further nested calls. Single-task ops save/restore both slots and clear
+ * kf_tasks[1] while active so a nested call under SCX_CALL_OP_2TASKS_RET does
+ * not leave the outer pair's second task authenticated for kfuncs.
  */
 #define SCX_CALL_OP_TASK(sch, op, locked_rq, task, args...)			\
 do {										\
-	WARN_ON_ONCE(current->scx.kf_tasks[0]);					\
+	struct task_struct *__scx_kf0_sv = current->scx.kf_tasks[0];		\
+	struct task_struct *__scx_kf1_sv = current->scx.kf_tasks[1];		\
+										\
 	current->scx.kf_tasks[0] = task;					\
+	current->scx.kf_tasks[1] = NULL;					\
 	SCX_CALL_OP((sch), op, locked_rq, task, ##args);			\
-	current->scx.kf_tasks[0] = NULL;					\
+	current->scx.kf_tasks[0] = __scx_kf0_sv;				\
+	current->scx.kf_tasks[1] = __scx_kf1_sv;				\
 } while (0)
 
 #define SCX_CALL_OP_TASK_RET(sch, op, locked_rq, task, args...)			\
 ({										\
 	__typeof__((sch)->ops.op(task, ##args)) __ret;				\
-	WARN_ON_ONCE(current->scx.kf_tasks[0]);					\
+	struct task_struct *__scx_kf0_sv = current->scx.kf_tasks[0];		\
+	struct task_struct *__scx_kf1_sv = current->scx.kf_tasks[1];		\
+										\
 	current->scx.kf_tasks[0] = task;					\
+	current->scx.kf_tasks[1] = NULL;					\
 	__ret = SCX_CALL_OP_RET((sch), op, locked_rq, task, ##args);		\
-	current->scx.kf_tasks[0] = NULL;					\
+	current->scx.kf_tasks[0] = __scx_kf0_sv;				\
+	current->scx.kf_tasks[1] = __scx_kf1_sv;				\
 	__ret;									\
 })
 
 #define SCX_CALL_OP_2TASKS_RET(sch, op, locked_rq, task0, task1, args...)	\
 ({										\
 	__typeof__((sch)->ops.op(task0, task1, ##args)) __ret;			\
-	WARN_ON_ONCE(current->scx.kf_tasks[0]);					\
+	struct task_struct *__scx_kf0_sv = current->scx.kf_tasks[0];		\
+	struct task_struct *__scx_kf1_sv = current->scx.kf_tasks[1];		\
+										\
 	current->scx.kf_tasks[0] = task0;					\
 	current->scx.kf_tasks[1] = task1;					\
 	__ret = SCX_CALL_OP_RET((sch), op, locked_rq, task0, task1, ##args);	\
-	current->scx.kf_tasks[0] = NULL;					\
-	current->scx.kf_tasks[1] = NULL;					\
+	current->scx.kf_tasks[0] = __scx_kf0_sv;				\
+	current->scx.kf_tasks[1] = __scx_kf1_sv;				\
 	__ret;									\
 })
 
Signature · 1 line
2.55.0
author reviewer
#18 lore raw

PATCH 8/12 sched_ext: Skip ops.runnable() when nested in SCX_CALL_OP_TASK

In reply to Andrea Righi

ops.running() can pull in enqueue_task_scx() -> ops.runnable() on the same current task while kf_tasks[] save/restore is still insufficient for every BPF/kfunc combination, leading to NULL dispatches and stack corruption.

Track SCX_CALL_OP_TASK nesting in current->scx.kf_nest (incremented by all SCX_CALL_OP_TASK* macros) and omit the ops.runnable() callback when non-zero. The full enqueue path including ops.enqueue() still runs, only the runnable hook is skipped in this case.

---
 include/linux/sched/ext.h   | 7 +++++++
 kernel/sched/ext/ext.c      | 3 ++-
 kernel/sched/ext/internal.h | 6 ++++++
 3 files changed, 15 insertions(+), 1 deletion(-)
include/linux/sched/ext.h +7 -0
index e599bb86f8acd..a0c2077216094 100644
@@ -201,6 +201,13 @@ struct sched_ext_entity {
 	s32			holding_cpu;
 	s32			selected_cpu;
 	struct task_struct	*kf_tasks[2];	/* see SCX_CALL_OP_TASK() */
+	/*
+	 * Nesting depth of SCX_CALL_OP_TASK() on this task as %current (e.g.
+	 * during schedule() %current is still the previous task). Used to skip
+	 * ops.runnable() when invoked from inside another task op such as
+	 * ops.running() to avoid breaking BPF re-entrance guarantees.
+	 */
+	u32			kf_nest;
 
 	struct list_head	runnable_node;	/* rq->scx.runnable_list */
 	unsigned long		runnable_at;
kernel/sched/ext/ext.c +2 -1
index 4cefe5acb36ff..c48d043dbe58f 100644
@@ -1862,7 +1862,8 @@ static void enqueue_task_scx(struct rq *rq, struct task_struct *p, int core_enq_
 	rq->scx.nr_running++;
 	add_nr_running(rq, 1);
 
-	if (SCX_HAS_OP(sch, runnable) && !task_on_rq_migrating(p))
+	if (SCX_HAS_OP(sch, runnable) && !task_on_rq_migrating(p) &&
+	    !READ_ONCE(current->scx.kf_nest))
 		SCX_CALL_OP_TASK(sch, runnable, rq, p, enq_flags);
 
 	if (enq_flags & SCX_ENQ_WAKEUP)
kernel/sched/ext/internal.h +6 -0
index 7d92826fc3a5f..1774e44aebcf7 100644
@@ -1804,11 +1804,13 @@ do {										\
 	struct task_struct *__scx_kf0_sv = current->scx.kf_tasks[0];		\
 	struct task_struct *__scx_kf1_sv = current->scx.kf_tasks[1];		\
 										\
+	current->scx.kf_nest++;							\
 	current->scx.kf_tasks[0] = task;					\
 	current->scx.kf_tasks[1] = NULL;					\
 	SCX_CALL_OP((sch), op, locked_rq, task, ##args);			\
 	current->scx.kf_tasks[0] = __scx_kf0_sv;				\
 	current->scx.kf_tasks[1] = __scx_kf1_sv;				\
+	current->scx.kf_nest--;							\
 } while (0)
 
 #define SCX_CALL_OP_TASK_RET(sch, op, locked_rq, task, args...)			\
@@ -1817,11 +1819,13 @@ do {										\
 	struct task_struct *__scx_kf0_sv = current->scx.kf_tasks[0];		\
 	struct task_struct *__scx_kf1_sv = current->scx.kf_tasks[1];		\
 										\
+	current->scx.kf_nest++;							\
 	current->scx.kf_tasks[0] = task;					\
 	current->scx.kf_tasks[1] = NULL;					\
 	__ret = SCX_CALL_OP_RET((sch), op, locked_rq, task, ##args);		\
 	current->scx.kf_tasks[0] = __scx_kf0_sv;				\
 	current->scx.kf_tasks[1] = __scx_kf1_sv;				\
+	current->scx.kf_nest--;							\
 	__ret;									\
 })
 
@@ -1831,11 +1835,13 @@ do {										\
 	struct task_struct *__scx_kf0_sv = current->scx.kf_tasks[0];		\
 	struct task_struct *__scx_kf1_sv = current->scx.kf_tasks[1];		\
 										\
+	current->scx.kf_nest++;							\
 	current->scx.kf_tasks[0] = task0;					\
 	current->scx.kf_tasks[1] = task1;					\
 	__ret = SCX_CALL_OP_RET((sch), op, locked_rq, task0, task1, ##args);	\
 	current->scx.kf_tasks[0] = __scx_kf0_sv;				\
 	current->scx.kf_tasks[1] = __scx_kf1_sv;				\
+	current->scx.kf_nest--;							\
 	__ret;									\
 })
 
Signature · 1 line
2.55.0
author reviewer
#19 lore raw

PATCH 9/12 sched_ext: Delegate proxy donor admission to BPF schedulers

In reply to Andrea Righi

Proxy execution keeps a blocked donor runnable so its scheduling context can execute the mutex owner. Dispatching sched_ext donors on a local DSQ bypasses the BPF scheduler's ordering policy and can give donors more CPU priority than intended to perform the proxy execution handoff to the mutex owner.

Add SCX_OPS_ENQ_BLOCKED as an explicit proxy execution capability. Tasks owned by schedulers without the flag block normally. Schedulers with the flag receive blocked donors through ops.enqueue() and can use scx_bpf_task_is_blocked() to apply their own admission policy.

The donor starts associated with its original CPU. A BPF scheduler may dispatch it directly into that CPU's local DSQ to let the core resolve the mutex owner and execute it with the donor's scheduling context.

Knowing the mutex owner's location in BPF is not strictly required, but steering the donation there can reduce handoff latency when migration fits the donor's affinity constraints and the scheduler's policy. Add scx_bpf_task_proxy_cpu() and scx_bpf_task_proxy_cid() to report that destination in the corresponding address space, and let the BPF scheduler decide to migrate or not the donor to the mutex owner's location.

Reschedule a retained donor when its mutex wakes it so ops.dispatch() can reconsider the now-unblocked task. Make SCX_OPS_ENQ_BLOCKED override the exiting and migration-disabled enqueue fallbacks so opted-in schedulers receive all eligible donor requests.

---
 kernel/sched/core.c                      |  36 +++++++-
 kernel/sched/ext/ext.c                   | 108 +++++++++++++++++++----
 kernel/sched/ext/ext.h                   |   2 +
 kernel/sched/ext/internal.h              |  19 +++-
 kernel/sched/sched.h                     |   6 ++
 tools/sched_ext/include/scx/common.bpf.h |   3 +
 tools/sched_ext/include/scx/compat.h     |   1 +
 7 files changed, 156 insertions(+), 19 deletions(-)
kernel/sched/core.c +35 -1
index 6aedb26c08ee7..f0edfbe8ce232 100644
@@ -7018,6 +7018,39 @@ find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf)
 	proxy_migrate_task(rq, rf, p, owner_cpu);
 	return NULL;
 }
+
+int task_proxy_cpu(struct task_struct *p)
+{
+	struct task_struct *owner;
+	struct mutex *mutex;
+	int owner_cpu;
+
+	if (!task_is_blocked(p))
+		return -ENOENT;
+
+	mutex = READ_ONCE(p->blocked_on);
+	if (!mutex)
+		return -ENOENT;
+
+	guard(raw_spinlock)(&mutex->wait_lock);
+	guard(raw_spinlock)(&p->blocked_lock);
+
+	if (mutex != __get_task_blocked_on(p))
+		return -EAGAIN;
+
+	owner = __mutex_owner(mutex);
+	if (!owner)
+		return -ENOENT;
+	if (!READ_ONCE(owner->on_rq) || owner->se.sched_delayed)
+		return -EAGAIN;
+
+	owner_cpu = task_cpu(owner);
+	if (owner_cpu != task_cpu(p) &&
+	    (p->nr_cpus_allowed == 1 || is_migration_disabled(p)))
+		return -EOPNOTSUPP;
+
+	return owner_cpu;
+}
 #else /* SCHED_PROXY_EXEC */
 static struct task_struct *
 find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf)
@@ -7148,7 +7181,8 @@ static void __sched notrace __schedule(int sched_mode)
 		 * task_is_blocked() will always be false).
 		 */
 		try_to_block_task(rq, prev, &prev_state,
-				  !task_is_blocked(prev));
+				  !task_is_blocked(prev) ||
+				  !scx_allow_proxy_exec(prev));
 		switch_count = &prev->nvcsw;
 	}
 
kernel/sched/ext/ext.c +91 -17

172 lines not on this page · raw message

kernel/sched/ext/ext.h +2 -0
index c7fa4d06ac7d3..08c64547b0143 100644
@@ -20,6 +20,7 @@ void scx_rq_deactivate(struct rq *rq);
 int scx_check_setscheduler(struct task_struct *p, int policy);
 bool task_should_scx(int policy);
 bool scx_allow_ttwu_queue(const struct task_struct *p);
+bool scx_allow_proxy_exec(const struct task_struct *p);
 void init_sched_ext_class(void);
 
 static inline u32 scx_cpuperf_target(s32 cpu)
@@ -60,6 +61,7 @@ static inline int scx_check_setscheduler(struct task_struct *p, int policy) { re
 static inline bool task_on_scx(const struct task_struct *p) { return false; }
 static inline bool is_ext_class(const struct task_struct *p) { return false; }
 static inline bool scx_allow_ttwu_queue(const struct task_struct *p) { return true; }
+static inline bool scx_allow_proxy_exec(const struct task_struct *p) { return true; }
 static inline void init_sched_ext_class(void) {}
 
 #endif	/* CONFIG_SCHED_CLASS_EXT */
kernel/sched/ext/internal.h +18 -1
index 1774e44aebcf7..a958c9d3144b3 100644
@@ -212,6 +212,22 @@ enum scx_ops_flags {
 	 */
 	SCX_OPS_TID_TO_TASK		= 1LLU << 8,
 
+	/*
+	 * If set, mutex-blocked tasks remain runnable as proxy donors and are
+	 * passed to ops.enqueue(). The BPF scheduler can identify them with
+	 * scx_bpf_task_is_blocked() and query the next mutex owner's CPU or cid
+	 * with scx_bpf_task_proxy_cpu() or scx_bpf_task_proxy_cid(). It controls
+	 * when donors are dispatched and whether they should preempt work on the
+	 * owner's CPU.
+	 *
+	 * If clear, mutex-blocked tasks are removed from the runqueue normally
+	 * and cannot donate their scheduling context through proxy execution.
+	 *
+	 * For blocked donors, this flag takes precedence over
+	 * %SCX_OPS_ENQ_EXITING and %SCX_OPS_ENQ_MIGRATION_DISABLED.
+	 */
+	SCX_OPS_ENQ_BLOCKED		= 1LLU << 9,
+
 	SCX_OPS_ALL_FLAGS		= SCX_OPS_KEEP_BUILTIN_IDLE |
 					  SCX_OPS_ENQ_LAST |
 					  SCX_OPS_ENQ_EXITING |
@@ -220,7 +236,8 @@ enum scx_ops_flags {
 					  SCX_OPS_SWITCH_PARTIAL |
 					  SCX_OPS_BUILTIN_IDLE_PER_NODE |
 					  SCX_OPS_ALWAYS_ENQ_IMMED |
-					  SCX_OPS_TID_TO_TASK,
+					  SCX_OPS_TID_TO_TASK |
+					  SCX_OPS_ENQ_BLOCKED,
 
 	/* high 8 bits are internal, don't include in SCX_OPS_ALL_FLAGS */
 	__SCX_OPS_INTERNAL_MASK		= 0xffLLU << 56,
kernel/sched/sched.h +6 -0
index 56acf502ba260..c4eb70371f7af 100644
@@ -2470,6 +2470,12 @@ static inline bool task_is_blocked(struct task_struct *p)
 	return !!p->blocked_on;
 }
 
+#ifdef CONFIG_SCHED_PROXY_EXEC
+int task_proxy_cpu(struct task_struct *p);
+#else
+static inline int task_proxy_cpu(struct task_struct *p) { return -EOPNOTSUPP; }
+#endif
+
 static inline int task_on_cpu(struct rq *rq, struct task_struct *p)
 {
 	return p->on_cpu;
tools/sched_ext/include/scx/common.bpf.h +3 -0
index bd51986c4c42e..4b82af377a15d 100644
@@ -95,6 +95,9 @@ s32 scx_bpf_pick_idle_cpu(const cpumask_t *cpus_allowed, u64 flags) __ksym;
 s32 scx_bpf_pick_any_cpu_node(const cpumask_t *cpus_allowed, int node, u64 flags) __ksym __weak;
 s32 scx_bpf_pick_any_cpu(const cpumask_t *cpus_allowed, u64 flags) __ksym;
 bool scx_bpf_task_running(const struct task_struct *p) __ksym;
+bool scx_bpf_task_is_blocked(struct task_struct *p) __ksym __weak;
+s32 scx_bpf_task_proxy_cpu(struct task_struct *p) __ksym __weak;
+s32 scx_bpf_task_proxy_cid(struct task_struct *p) __ksym __weak;
 s32 scx_bpf_task_cpu(const struct task_struct *p) __ksym;
 struct rq *scx_bpf_locked_rq(void) __ksym;
 struct task_struct *scx_bpf_cpu_curr(s32 cpu) __ksym __weak;
tools/sched_ext/include/scx/compat.h +1 -0
index 23d9ef3e4c9d2..8d5606e465080 100644
@@ -117,6 +117,7 @@ static inline bool __COMPAT_struct_has_field(const char *type, const char *field
 #define SCX_OPS_ALLOW_QUEUED_WAKEUP SCX_OPS_FLAG(SCX_OPS_ALLOW_QUEUED_WAKEUP)
 #define SCX_OPS_BUILTIN_IDLE_PER_NODE SCX_OPS_FLAG(SCX_OPS_BUILTIN_IDLE_PER_NODE)
 #define SCX_OPS_ALWAYS_ENQ_IMMED SCX_OPS_FLAG(SCX_OPS_ALWAYS_ENQ_IMMED)
+#define SCX_OPS_ENQ_BLOCKED SCX_OPS_FLAG(SCX_OPS_ENQ_BLOCKED)
 
 #define SCX_PICK_IDLE_FLAG(name) __COMPAT_ENUM_OR_ZERO("scx_pick_idle_cpu_flags", #name)
 
Signature · 1 line
2.55.0
#20 lore raw

Re: [PATCH 09/12] sched_ext: Delegate proxy donor admission to BPF schedulers

Hello Andrea,

Quoted 16 lines · Andrea Righi
kernel/sched/core.c +2 -1
@@ -7148,7 +7181,8 @@ static void __sched notrace __schedule(int sched_mode)
 		 * task_is_blocked() will always be false).
 		 */
 		try_to_block_task(rq, prev, &prev_state,
-				  !task_is_blocked(prev));
+				  !task_is_blocked(prev) ||
+				  !scx_allow_proxy_exec(prev));
 		switch_count = &prev->nvcsw;
 	}
 
kernel/sched/ext/ext.c +0 -0
index c48d043dbe58f..8ffbf857acd51 100644
@@ -23,6 +23,14 @@
 

kernel/sched/ext/ext.c line 33

Quoted 10 lines
kernel/sched/ext/ext.c +8 -0
 DEFINE_RAW_SPINLOCK(scx_sched_lock);
 
+bool scx_allow_proxy_exec(const struct task_struct *p)
+{
+	if (!task_on_scx(p))
+		return true;
+
+	return scx_task_sched(p)->ops.flags & SCX_OPS_ENQ_BLOCKED;
+}
+

What happens when we switch in a a sched-ext scheduler that doesn't support proxy?

There would be a bunch proxy donors already preset at the time of the scheduler loading but I don't see anything in the enable path that would take care of those tasks by blocking them completely.

Is it handled elsewhere?

Signature · 2 lines
Thanks and Regards,
Prateek
author reviewer
#21 lore raw

Re: [PATCH 09/12] sched_ext: Delegate proxy donor admission to BPF schedulers

Hi Prateek,

Quoted 33 lines · K Prateek Nayak

Hello Andrea,

Quoted 16 lines · Andrea Righi
kernel/sched/core.c +2 -1
@@ -7148,7 +7181,8 @@ static void __sched notrace __schedule(int sched_mode)
 		 * task_is_blocked() will always be false).
 		 */
 		try_to_block_task(rq, prev, &prev_state,
-				  !task_is_blocked(prev));
+				  !task_is_blocked(prev) ||
+				  !scx_allow_proxy_exec(prev));
 		switch_count = &prev->nvcsw;
 	}
 
kernel/sched/ext/ext.c +0 -0
index c48d043dbe58f..8ffbf857acd51 100644
@@ -23,6 +23,14 @@
 

kernel/sched/ext/ext.c line 33

Quoted 10 lines
kernel/sched/ext/ext.c +8 -0
 DEFINE_RAW_SPINLOCK(scx_sched_lock);
 
+bool scx_allow_proxy_exec(const struct task_struct *p)
+{
+	if (!task_on_scx(p))
+		return true;
+
+	return scx_task_sched(p)->ops.flags & SCX_OPS_ENQ_BLOCKED;
+}
+

What happens when we switch in a a sched-ext scheduler that doesn't support proxy?

Quoted 5 lines

There would be a bunch proxy donors already preset at the time of the scheduler loading but I don't see anything in the enable path that would take care of those tasks by blocking them completely.

Is it handled elsewhere?

Good catch! It's not currently handled, a proxy donor that blocked before sched_ext was enabled can remain queued after being transferred to a scheduler that doesn't have SCX_OPS_ENQ_BLOCKED.

We should probably deactivate existing blocked donors while switching them to a scheduler without proxy support.

I'll fix this in the next version and add this case to the kselftest.

Thanks,
-Andrea

author reviewer
#22 lore raw

PATCH 10/12 sched_ext: Add selftest for blocked donor admission

In reply to Andrea Righi

SCX_OPS_ENQ_BLOCKED allows BPF schedulers to receive blocked proxy donors through ops.enqueue(), while scx_bpf_task_is_blocked() identifies donor admission requests and scx_bpf_task_proxy_cpu() returns the CPU where the mutex owner is blocked. Add test coverage for this interface, including validation of the owner's CPU.

Exercise a three-task priority inversion on a single CPU using a weighted vruntime BPF scheduler. A nice +19 owner holds a shared mutex, a nice -20 donor blocks on it and a nice 0 CPU contender delays the owner.

Run the workload with SCX_OPS_ENQ_BLOCKED first disabled and then enabled. In the enabled run, count blocked donor enqueues, validate the proxy CPU, and dispatch the donor there to facilitate proxy execution. Report average mutex hold and wait times and their deltas without enforcing performance results.

Proxy execution coverage requires CONFIG_SCHED_PROXY_EXEC=y, which the selftest config selects. Expect blocked enqueue events when generic proxy execution is enabled and none when proxy execution is disabled.

The mutex is provided via a loadable kernel module, built via TEST_GEN_MODS_DIR, with the test responsible for loading, unloading, and managing its lifecycle.

---
 tools/testing/selftests/sched_ext/.gitignore  |   4 +
 tools/testing/selftests/sched_ext/Makefile    |   2 +
 tools/testing/selftests/sched_ext/config      |   2 +
 .../selftests/sched_ext/enq_blocked.bpf.c     | 116 +++
 .../testing/selftests/sched_ext/enq_blocked.c | 682 ++++++++++++++++++
 .../testing/selftests/sched_ext/enq_blocked.h |  21 +
 .../selftests/sched_ext/test_modules/Makefile |  13 +
 .../test_modules/scx_enq_blocked_test.c       | 134 ++++
 8 files changed, 974 insertions(+)
 create mode 100644 tools/testing/selftests/sched_ext/enq_blocked.bpf.c
 create mode 100644 tools/testing/selftests/sched_ext/enq_blocked.c
 create mode 100644 tools/testing/selftests/sched_ext/enq_blocked.h
 create mode 100644 tools/testing/selftests/sched_ext/test_modules/Makefile
 create mode 100644 tools/testing/selftests/sched_ext/test_modules/scx_enq_blocked_test.c
tools/testing/selftests/sched_ext/.gitignore +4 -0
index ae5491a114c09..54a1fd2af713d 100644
@@ -4,3 +4,7 @@
 !Makefile
 !.gitignore
 !config
+!test_modules/
+!test_modules/scx_enq_blocked_test.c
+!test_modules/Makefile
+test_modules/*.mod.c
tools/testing/selftests/sched_ext/Makefile +2 -0
index 5d2dffca0e918..8eeece67d1de1 100644
@@ -5,6 +5,7 @@ include ../../../scripts/Makefile.arch
 include ../../../scripts/Makefile.include
 
 TEST_GEN_PROGS := runner
+TEST_GEN_MODS_DIR := test_modules
 
 # override lib.mk's default rules
 OVERRIDE_TARGETS := 1
@@ -164,6 +165,7 @@ all_test_bpfprogs := $(foreach prog,$(wildcard *.bpf.c),$(INCLUDE_DIR)/$(patsubs
 auto-test-targets :=			\
 	create_dsq			\
 	dequeue				\
+	enq_blocked			\
 	enq_last_no_enq_fails		\
 	ddsp_bogus_dsq_fail		\
 	ddsp_vtimelocal_fail		\
tools/testing/selftests/sched_ext/config +2 -0
index aa901b05c8ad6..affa3cf33470a 100644
@@ -6,3 +6,5 @@ CONFIG_BPF=y
 CONFIG_BPF_SYSCALL=y
 CONFIG_DEBUG_INFO=y
 CONFIG_DEBUG_INFO_BTF=y
+CONFIG_EXPERT=y
+CONFIG_SCHED_PROXY_EXEC=y
tools/testing/selftests/sched_ext/enq_blocked.bpf.c new file +116 -0

116 lines not on this page · raw message

tools/testing/selftests/sched_ext/enq_blocked.c new file +682 -0

682 lines not on this page · raw message

tools/testing/selftests/sched_ext/enq_blocked.h new file +21 -0
new file mode 100644
index 0000000000000..e2f9be0f57c9d
@@ -0,0 +1,21 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES */
+#ifndef __ENQ_BLOCKED_H
+#define __ENQ_BLOCKED_H
+
+#include <linux/ioctl.h>
+#include <linux/types.h>
+
+struct enq_blocked_stats {
+	__u64 hold_time_ns;
+	__u64 wait_time_ns;
+	__u64 nr_holds;
+	__u64 nr_waits;
+};
+
+#define ENQ_BLOCKED_IOCTL_OWNER	_IO('s', 1)
+#define ENQ_BLOCKED_IOCTL_DONOR	_IO('s', 2)
+#define ENQ_BLOCKED_IOCTL_RESET_STATS	_IO('s', 3)
+#define ENQ_BLOCKED_IOCTL_GET_STATS	_IOR('s', 4, struct enq_blocked_stats)
+
+#endif /* __ENQ_BLOCKED_H */
tools/testing/selftests/sched_ext/test_modules/Makefile new file +13 -0
new file mode 100644
index 0000000000000..a0e9e9401ead6
@@ -0,0 +1,13 @@
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES
+
+TESTMODS_DIR := $(realpath $(dir $(abspath $(lastword $(MAKEFILE_LIST)))))
+KDIR ?= $(if $(O),$(O),$(realpath ../../../../..))
+
+obj-m += scx_enq_blocked_test.o
+
+all:
+	+$(Q)$(MAKE) -C $(KDIR) M=$(TESTMODS_DIR) modules
+
+clean:
+	+$(Q)$(MAKE) -C $(KDIR) M=$(TESTMODS_DIR) clean
tools/testing/selftests/sched_ext/test_modules/scx_enq_blocked_test.c new file +134 -0

134 lines not on this page · raw message

Signature · 1 line
2.55.0
author reviewer
#23 lore raw

PATCH 11/12 sched_ext: scx_qmap: Add proxy execution support

In reply to Andrea Righi

Blocked sched_ext tasks must remain available to the scheduler before the core can use their scheduling contexts to execute mutex owners. BPF schedulers opt into receiving these donor requests through SCX_OPS_ENQ_BLOCKED and are responsible for deciding when to admit them.

Enable SCX_OPS_ENQ_BLOCKED in scx_qmap and immediately admit every blocked donor. Query the mutex owner's cid and use it when allowed by the donor's affinity, falling back to the donor's current cid otherwise. Insert the donor at the head of the selected local DSQ.

Applying the same policy after a proxy migration expedites the donor on the owner's CPU so proxy execution can begin without waiting behind unrelated local work.

---
 tools/sched_ext/scx_qmap.bpf.c | 18 +++++++++++++++++-
 1 file changed, 17 insertions(+), 1 deletion(-)
tools/sched_ext/scx_qmap.bpf.c +17 -1
index fd9a82a676278..5ecd2ffe05270 100644
@@ -396,6 +396,21 @@ void BPF_STRUCT_OPS(qmap_enqueue, struct task_struct *p, u64 enq_flags)
 	 */
 	taskc->core_sched_seq = qa.core_sched_tail_seqs[idx]++;
 
+	/*
+	 * Admit blocked proxy donors immediately. Dispatch directly on the mutex
+	 * owner's cid when available to facilitate the proxy execution switch.
+	 * SCX_ENQ_HEAD also puts the donor at the head of the local DSQ and
+	 * expedites the destination enqueue after a proxy migration.
+	 */
+	if (scx_bpf_task_is_blocked(p)) {
+		cid = scx_bpf_task_proxy_cid(p);
+		if (cid < 0 || !cmask_test(cid, &taskc->cpus_allowed))
+			cid = scx_bpf_task_cid(p);
+		scx_bpf_dsq_insert(p, SCX_DSQ_LOCAL_ON | cid,
+				   slice_ns, enq_flags | SCX_ENQ_HEAD);
+		return;
+	}
+
 	/*
 	 * IMMED stress testing: Every immed_stress_nth'th enqueue, dispatch
 	 * directly to prev_cpu's local DSQ even when busy to force dsq->nr > 1
@@ -1216,7 +1231,8 @@ void BPF_STRUCT_OPS(qmap_sub_detach, struct scx_sub_detach_args *args)
 }
 
 SCX_OPS_CID_DEFINE(qmap_ops,
-	       .flags			= SCX_OPS_ENQ_EXITING | SCX_OPS_TID_TO_TASK,
+	       .flags			= SCX_OPS_ENQ_EXITING | SCX_OPS_TID_TO_TASK |
+					  SCX_OPS_ENQ_BLOCKED,
 	       .select_cid		= (void *)qmap_select_cid,
 	       .enqueue			= (void *)qmap_enqueue,
 	       .dequeue			= (void *)qmap_dequeue,
Signature · 1 line
2.55.0
author reviewer
#24 lore raw

PATCH 12/12 sched: Allow enabling proxy exec with sched_ext

In reply to Andrea Righi

Now that sched_ext supports proxy execution, allow enabling both options together.

---
 init/Kconfig | 2 --
 1 file changed, 2 deletions(-)
init/Kconfig +0 -2
index 5230d4879b1c8..0817e62266e03 100644
@@ -934,8 +934,6 @@ config SCHED_PROXY_EXEC
 	bool "Proxy Execution"
 	# Avoid some build failures w/ PREEMPT_RT until it can be fixed
 	depends on !PREEMPT_RT
-	# Need to investigate how to inform sched_ext of split contexts
-	depends on !SCHED_CLASS_EXT
 	# Not particularly useful until we get to multi-rq proxying
 	depends on EXPERT
 	help
Signature · 1 line
2.55.0