Skip to contentSkip to conversation

sched: Make proxy execution compatible with sched_ext

PATCH Superseded Andrea Righi 3 participants 16 messages Jul 6

Subsystems (3)

v3 of 14 Superseded Latest is v14 →

lore
Share & tools
lore

Andrea Righi

author reviewer
#0 lore raw

PATCH v3 1/1 sched: Make proxy execution compatible with sched_ext

This series enables using proxy execution with sched_ext and is based on early work by John Stultz [1].

Background
==========

Proxy execution (proxy-exec) lets a waiting task ("donor") donate its scheduling context to a mutex owner, so the owner can run while the donor stays eligible on the runqueue.

Currently, proxy execution and sched_ext are mutually exclusive at build time:
we can't enable CONFIG_SCHED_PROXY_EXEC=y and CONFIG_SCHED_CLASS_EXT=y in the same kernel.

This restriction can be problematic for Linux distributions and for anyone who wants to ship one kernel and choose features at runtime.

Why are they mutually exclusive?
================================

sched_ext schedulers drive dispatch through their own interfaces. A proxy-exec handoff can run a task that the BPF scheduler never dispatched through that path. sched_ext callbacks then observe a "current" task that does not match what the BPF side considers running, so kfuncs and helper state can see an inconsistent view of the executing task.

sched_ext also tracks runnable work through Dispatch Queues (DSQs) and BPF chosen dispatch rules, while the core scheduler still maintains classic per-CPU runqueues and pick paths. A proxy handoff can therefore switch the CPU to a task that the BPF scheduler never inserted or ordered through its DSQ interface.

DSQ state, vtime, and "who is running" bookkeeping inside the BPF program can then disagree with what the core actually executes, so helpers and kfuncs that assume their dispatched task is current may observe stale or inconsistent state.

Design: supporting proxy execution with sched_ext
=================================================

Provide proxy execution in sched_ext as optional per-scheduler capability: a BPF scheduler can set the ops flag SCX_OPS_ENQ_BLOCKED to keep mutex-blocked donors runnable and receive them through ops.enqueue() with SCX_ENQ_BLOCKED set.

This flag requires an ops.enqueue() callback and gives BPF control over whether, where, and in what order to dispatch each donor. The core then walks the mutex-owner chain and, when needed, migrates the donor's scheduling context to the owner's rq before executing the owner.

Without the SCX_OPS_ENQ_BLOCKED flag, mutex waiters block normally and do not participate in proxy execution while owned by that scheduler.

The donor-to-owner handoff is modeled like a "function call" from the scheduler's perspective. The donor remains the running scheduling entity selected by BPF: its scheduling context, runtime and slice are consumed while the core temporarily invokes the mutex owner's code to make the critical section progress. It is not a scheduler-visible switch to the owner.

Accordingly, the donor remains the scheduling context presented to sched_ext, while the mutex owner is treated solely as the execution context selected internally by the core scheduler. Scheduling state is accounted against rq->donor where appropriate, while rq->curr identifies the execution context. The internal owner substitution does not generate synthetic sched_ext callbacks for a task that BPF did not dispatch.

Scheduler ownership changes need special handling because a donor may already be blocked when a root scheduler is enabled or when a task moves between root and sub-schedulers. If the incoming scheduler does not set SCX_OPS_ENQ_BLOCKED, the retained donor is removed from the runqueue and stays on the normal mutex wait path. Otherwise it remains eligible for BPF admission.

The mutex owner's CPU can be queried from BPF using scx_bpf_task_proxy_cpu(p) or scx_bpf_task_proxy_cid(p), with p being the mutex-blocked task. It's up to the BPF scheduler to decide if and how to use this information. For example, a BPF scheduler may inspect the task currently running at that location and request preemption to expedite the proxy-execution handoff. Schedulers that decide to not use this information can keep the donor on its current CPU/cid and let the core perform the handoff.

scx_qmap has been modified with a -B option to enable queueing mutex-blocked tasks for proxy execution.

A new kselftest (enq_blocked) is also introduced to validate proxy execution with sched_ext. The test creates a three-task priority inversion on one CPU: a low-priority owner (nice +19) holds a kernel mutex, a high-priority donor (nice -20) blocks on it and a nice 0 contender competes for the CPU. The first donor blocks before the BPF scheduler is attached, allowing the test to verify that switching to a scheduler without SCX_OPS_ENQ_BLOCKED removes the retained donor from the runqueue, while switching to one with the flag preserves it. Access to the mutex is provided by a loadable kernel module built via TEST_GEN_MODS_DIR, with the test responsible for its lifecycle.

Example kselftest run:

  $ sudo tools/testing/selftests/sched_ext/runner -t enq_blocked
  ===== START =====
  TEST: enq_blocked
  DESCRIPTION: Verify BPF-driven proxy donor admission
  OUTPUT:

[SCX_OPS_ENQ_BLOCKED=disabled]

    proxy_exec=enabled
    owner_nice=19
    donor_nice=-20
    contender_nice=0
    mutex_hold_avg_ns=254084719 (254.084 ms, samples=10)
    mutex_wait_avg_ns=254095120 (254.095 ms, samples=10)
    nr_blocked_enqueues=0

[SCX_OPS_ENQ_BLOCKED=enabled]

    proxy_exec=enabled
    owner_nice=19
    donor_nice=-20
    contender_nice=0
    mutex_hold_avg_ns=228884734 (228.884 ms, samples=10)
    mutex_wait_avg_ns=207903720 (207.903 ms, samples=10)
    nr_blocked_enqueues=51

[delta: enabled - disabled]

    mutex_hold_delta_ns=-25199985 (-9.92%)
    mutex_wait_delta_ns=-46191400 (-18.18%)

ok 1 enq_blocked #
===== END =====

=============================

RESULTS:

PASSED: 1
SKIPPED: 0
FAILED: 0

References
==========

[1] https://lore.kernel.org/all/[email protected]

Git tree: git://git.kernel.org/pub/scm/linux/kernel/git/arighi/linux.git scx-proxy-exec

Changes in v3:
- Dropped the core restrictions on proxy-migrating migration-disabled and single-CPU donors: proxy execution moves the scheduling context, not the task's execution context (Peter Zijlstra, K Prateek Nayak)
- Dropped the sched_ext-specific put_prev_task()/set_next_task() exception and fixed ops.running()/ops.stopping() pairing inside sched_ext instead; track a real running transition even when either callback is absent (Peter Zijlstra)
- Dropped the kf_tasks[] nesting and nested ops.runnable() patches from v2; extensive proxy-exec testing did not reproduce task-op re-entry, so retain the existing non-nesting invariant
- Replaced scx_bpf_task_is_blocked() with SCX_ENQ_BLOCKED in ops.enqueue() flags
- Expanded the curr/donor description to cover mixed scheduling classes and fixed local preemption to expire rq->donor's slice (Aiqun Maria Yu)
- Allow inactive blocked donors to be placed on a remote local DSQ while preventing normal migration of an active rq donor (Aiqun Maria Yu)
- Deactivate retained donors when ownership changes to a root or sub-scheduler without SCX_OPS_ENQ_BLOCKED, and extend the selftest to cover attaching a scheduler after the donor blocks (K Prateek Nayak)
- Hardened remote DSQ consumption by rejecting active tasks and re-checking migration eligibility after locking the source rq; fall back to the global DSQ without treating an eligibility change during the lock handoff as a BPF scheduler error
- scx_qmap has a command line option (-B) to enable blocked-donor queueing
- Link to v2: https://lore.kernel.org/all/[email protected]/

Changes in v2:
- Rebased onto sched_ext/for-7.3 and adapted the series to the split sched_ext implementation and cid-form scheduler interfaces.
- Replaced the global sched_proxy_exec_scx boot-time opt-in with the per-scheduler SCX_OPS_ENQ_BLOCKED capability, allowing BPF to control donor admission and ordering through ops.enqueue().
- Added scx_bpf_task_is_blocked(), scx_bpf_task_proxy_cpu(), and scx_bpf_task_proxy_cid(); enforce CPU/cid API separation for cid-form schedulers.
- Added proxy exec support to scx_qmap, including optional owner-cid steering, affinity validation, and fallback to the donor's current cid.
- Added a kselftest with a kernel mutex test and a three-task priority inversion workload, test is executed with blocked task admission disabled and enabled, validates the behavior, and reports hold/wait-time deltas.
- Link to v1: https://lore.kernel.org/all/[email protected]/

Andrea Righi (7):
sched/core: Drop mutex locks before proxy rescheduling
sched_ext: Fix TOCTOU race in consume_remote_task()
sched_ext: Fix ops.running/stopping() pairing for proxy-exec donors
sched_ext: Delegate proxy donor admission to BPF schedulers
sched_ext: Add selftest for blocked donor admission
sched_ext: scx_qmap: Add proxy execution support
sched: Allow enabling proxy exec with sched_ext

John Stultz (2):
sched_ext: Split curr|donor references properly
sched_ext: Handle blocked donor migration with proxy execution

 include/linux/sched/ext.h                          |   2 +
 init/Kconfig                                       |   2 -
 kernel/sched/core.c                                |  68 +-
 kernel/sched/ext/ext.c                             | 245 ++++++-
 kernel/sched/ext/ext.h                             |   2 +
 kernel/sched/ext/internal.h                        |  25 +-
 kernel/sched/ext/sub.c                             |  12 +-
 kernel/sched/sched.h                               |   8 +
 tools/sched_ext/include/scx/common.bpf.h           |   2 +
 tools/sched_ext/include/scx/compat.bpf.h           |  18 +
 tools/sched_ext/include/scx/compat.h               |   1 +
 tools/sched_ext/include/scx/enum_defs.autogen.h    |   1 +
 tools/sched_ext/include/scx/enums.autogen.bpf.h    |   3 +
 tools/sched_ext/include/scx/enums.autogen.h        |   1 +
 tools/sched_ext/scx_qmap.bpf.c                     |  13 +
 tools/sched_ext/scx_qmap.c                         |   9 +-
 tools/testing/selftests/sched_ext/.gitignore       |   4 +
 tools/testing/selftests/sched_ext/Makefile         |   2 +
 tools/testing/selftests/sched_ext/config           |   2 +
 .../testing/selftests/sched_ext/enq_blocked.bpf.c  | 128 ++++
 tools/testing/selftests/sched_ext/enq_blocked.c    | 732 +++++++++++++++++++++
 tools/testing/selftests/sched_ext/enq_blocked.h    |  27 +
 .../selftests/sched_ext/test_modules/Makefile      |  13 +
 .../sched_ext/test_modules/scx_enq_blocked_test.c  | 193 ++++++
 24 files changed, 1464 insertions(+), 49 deletions(-)
 create mode 100644 tools/testing/selftests/sched_ext/enq_blocked.bpf.c
 create mode 100644 tools/testing/selftests/sched_ext/enq_blocked.c
 create mode 100644 tools/testing/selftests/sched_ext/enq_blocked.h
 create mode 100644 tools/testing/selftests/sched_ext/test_modules/Makefile
 create mode 100644 tools/testing/selftests/sched_ext/test_modules/scx_enq_blocked_test.c
author reviewer
#1 lore raw

PATCH 1/9 sched/core: Drop mutex locks before proxy rescheduling

find_proxy_task() can call proxy_resched_idle() while holding a mutex wait_lock and the blocked task lock. Guard cleanup does not run until after the return expression is evaluated.

proxy_resched_idle() invokes scheduling-class callbacks through put_prev_set_next_task(). Calling those callbacks with the mutex locks held prevents them from safely inspecting the proxy chain, as doing so may require acquiring the same locks.

This is a preparatory change to support proxy execution in sched_ext. It leaves the guard scope before calling proxy_resched_idle() so sched_ext callbacks can safely inspect the proxy chain.

---
 kernel/sched/core.c | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)
kernel/sched/core.c +6 -4
index 3cc6fb1d20547..0139cd4a8be7e 100644
@@ -6923,7 +6923,7 @@ find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf)
 		if (!READ_ONCE(owner->on_rq) || owner->se.sched_delayed) {
 			/* XXX Don't handle blocked owners/delayed dequeue yet */
 			if (curr_in_chain)
-				return proxy_resched_idle(rq);
+				goto resched_idle;
 			__clear_task_blocked_on(p, NULL);
 			goto deactivate;
 		}
@@ -6935,7 +6935,7 @@ find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf)
 			 * and leave that CPU to sort things out.
 			 */
 			if (curr_in_chain)
-				return proxy_resched_idle(rq);
+				goto resched_idle;
 			goto migrate_task;
 		}
 
@@ -6948,7 +6948,7 @@ find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf)
 			 * case we should end up back in find_proxy_task(), this time
 			 * hopefully with all relevant tasks already enqueued.
 			 */
-			return proxy_resched_idle(rq);
+			goto resched_idle;
 		}
 
 		/*
@@ -6985,7 +6985,7 @@ find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf)
 			 * So schedule rq->idle so that ttwu_runnable() can get the rq
 			 * lock and mark owner as running.
 			 */
-			return proxy_resched_idle(rq);
+			goto resched_idle;
 		}
 		/*
 		 * OK, now we're absolutely sure @owner is on this
@@ -6997,6 +6997,8 @@ find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf)
 	WARN_ON_ONCE(owner && !owner->on_rq);
 	return owner;
 
+resched_idle:
+	return proxy_resched_idle(rq);
 deactivate:
 	proxy_deactivate(rq, p);
 	return NULL;
Signature · 1 line
2.55.0
reviewer Reviewed-by on v1 1/9
#2 lore raw

Re: [PATCH 1/9] sched/core: Drop mutex locks before proxy rescheduling

Hello Andrea,

Quoted 9 lines · Andrea Righi

find_proxy_task() can call proxy_resched_idle() while holding a mutex wait_lock and the blocked task lock. Guard cleanup does not run until after the return expression is evaluated.

proxy_resched_idle() invokes scheduling-class callbacks through put_prev_set_next_task(). Calling those callbacks with the mutex locks held prevents them from safely inspecting the proxy chain, as doing so may require acquiring the same locks.

Quoted 5 lines

This is a preparatory change to support proxy execution in sched_ext. It leaves the guard scope before calling proxy_resched_idle() so sched_ext callbacks can safely inspect the proxy chain.

I don't see the need to hold anything more than the rq_lock for proxy_resched_idle(). Feel free to include:

Signature · 2 lines
Thanks and Regards,
Prateek
author reviewer
#3 lore raw

PATCH 2/9 sched_ext: Split curr|donor references properly

In reply to Andrea Righi

From: John Stultz <[email protected]>

With proxy execution, the task selected by the scheduler and the task physically executing can differ. A blocked mutex waiter donates its scheduling context to the lock owner:

D -----------------> M -------------> O ----------------> T [donor] blocked on [mutex] owned by [owner] preempted by [task]
\_________________________________^
donates scheduling context

where:

D = blocked donor
M = mutex
O = mutex owner
T = competing runnable task

During a proxy execution switch, D supplies the scheduling class, priority, and runtime budget, while O supplies the execution context: O is the task whose code physically executes. T is a competing runnable task which may preempt the D/O proxy execution.

Consider FAIR and EXT tasks with sched_ext running in partial mode. FAIR can be replaced with a higher scheduling class such as RT or deadline without changing the class interaction described here. The possible combinations are:

1. D is EXT, O is EXT, T is EXT

D can interrupt T according to BPF scheduling policy. O executes with D's EXT priority and runtime budget, while T waits in EXT.

2. D is EXT, O is EXT, T is FAIR

D is visible to the BPF scheduler, but cannot preempt T because EXT is below FAIR. Once T stops, BPF can dispatch D and O executes with D's EXT priority and runtime budget. If T becomes runnable again, it preempts the D/O proxy execution.

3. D is EXT, O is FAIR, T is EXT

This cannot represent T preempting O because EXT is below FAIR.

4. D is EXT, O is FAIR, T is FAIR

D cannot boost O above T because EXT is below FAIR. O and T continue competing under FAIR. Once O releases M, D wakes and resumes normal EXT scheduling.

5. D is FAIR, O is EXT, T is EXT

D preempts T as the higher-class scheduling context. O executes with D's FAIR priority and runtime budget, while T waits in EXT. D is not visible to the BPF scheduler.

6. D is FAIR, O is EXT, T is FAIR

D competes with T according to its FAIR deadline. When D is selected, O executes with D's FAIR priority and runtime budget. D is not visible to the BPF scheduler.

7. D is FAIR, O is FAIR, T is EXT

This cannot represent T preempting O because EXT is below FAIR.

8. D is FAIR, O is FAIR, T is FAIR

O, T, and D all have FAIR scheduling contexts. D remains runnable as a blocked proxy donor. When CFS selects D, O executes using D's FAIR scheduling context. When CFS selects O, O executes using its own FAIR context, and when CFS selects T, T executes normally. D is not visible to the BPF scheduler.

Thus, sched_ext policy and accounting must generally use rq->donor, the scheduler-selected task which supplies the scheduling context, rather than rq->curr, the task whose code physically executes. Without proxy execution they are the same task.

Use rq->donor for sched_ext slice accounting, preemption, task picking, tick handling, and CPU kicks. Also include donor information in scx_dump_state() so both contexts are visible when they differ.

CONFIG_SCHED_PROXY_EXEC currently depends on !CONFIG_SCHED_CLASS_EXT, so this has no effect other than the additional scx_dump_state() output. It prepares sched_ext for removing that restriction.

---
 kernel/sched/ext/ext.c | 30 +++++++++++++++++-------------
 1 file changed, 17 insertions(+), 13 deletions(-)
kernel/sched/ext/ext.c +17 -13
index 1a0ec985da77d..242f6ffbba350 100644
@@ -1145,17 +1145,17 @@ static void touch_core_sched_dispatch(struct rq *rq, struct task_struct *p)
 
 static void update_curr_scx(struct rq *rq)
 {
-	struct task_struct *curr = rq->curr;
+	struct task_struct *donor = rq->donor;
 	s64 delta_exec;
 
 	delta_exec = update_curr_common(rq);
 	if (unlikely(delta_exec <= 0))
 		return;
 
-	if (curr->scx.slice != SCX_SLICE_INF) {
-		curr->scx.slice -= min_t(u64, curr->scx.slice, delta_exec);
-		if (!curr->scx.slice)
-			touch_core_sched(rq, curr);
+	if (donor->scx.slice != SCX_SLICE_INF) {
+		donor->scx.slice -= min_t(u64, donor->scx.slice, delta_exec);
+		if (!donor->scx.slice)
+			touch_core_sched(rq, donor);
 	}
 
 	dl_server_update(&rq->ext_server, delta_exec);
@@ -1316,9 +1316,9 @@ static void local_dsq_post_enq(struct scx_sched *sch, struct scx_dispatch_q *dsq
 	if (rq->scx.flags & SCX_RQ_IN_BALANCE)
 		return;
 
-	if ((enq_flags & SCX_ENQ_PREEMPT) && p != rq->curr &&
-	    rq->curr->sched_class == &ext_sched_class) {
-		rq->curr->scx.slice = 0;
+	if ((enq_flags & SCX_ENQ_PREEMPT) && p != rq->donor &&
+	    rq->donor->sched_class == &ext_sched_class) {
+		rq->donor->scx.slice = 0;
 		resched_curr(rq);
 	}
 }
@@ -2464,7 +2464,8 @@ static void dispatch_to_local_dsq(struct scx_sched *sch, struct rq *rq,
 		}
 
 		/* if the destination CPU is idle, wake it up */
-		if (!fallback && sched_class_above(p->sched_class, dst_rq->curr->sched_class))
+		if (!fallback && sched_class_above(p->sched_class,
+						      dst_rq->donor->sched_class))
 			resched_curr(dst_rq);
 	}
 
@@ -2876,7 +2877,7 @@ static struct task_struct *first_local_task(struct rq *rq)
 static struct task_struct *
 do_pick_task_scx(struct rq *rq, struct rq_flags *rf, bool force_scx)
 {
-	struct task_struct *prev = rq->curr;
+	struct task_struct *prev = rq->donor;
 	bool keep_prev;
 	struct task_struct *p;
 
@@ -4029,7 +4030,7 @@ static void run_deferred(struct rq *rq)
 #ifdef CONFIG_NO_HZ_FULL
 bool scx_can_stop_tick(struct rq *rq)
 {
-	struct task_struct *p = rq->curr;
+	struct task_struct *p = rq->donor;
 	struct scx_sched *sch = scx_task_sched(p);
 
 	if (p->sched_class != &ext_sched_class)
@@ -6007,6 +6008,9 @@ static void scx_dump_cpu(struct scx_sched *sch, struct seq_buf *s,
 	dump_line(&ns, "          curr=%s[%d] class=%ps",
 		  rq->curr->comm, rq->curr->pid,
 		  rq->curr->sched_class);
+	dump_line(&ns, "          donor=%s[%d] class=%ps",
+		  rq->donor->comm, rq->donor->pid,
+		  rq->donor->sched_class);
 	if (!cpumask_empty(rq->scx.cpus_to_kick))
 		dump_line(&ns, "  cpus_to_kick   : %*pb",
 			  cpumask_pr_args(rq->scx.cpus_to_kick));
@@ -7452,7 +7456,7 @@ static bool kick_one_cpu(s32 cpu, struct rq *this_rq, unsigned long *ksyncs)
 	unsigned long flags;
 
 	raw_spin_rq_lock_irqsave(rq, flags);
-	cur_class = rq->curr->sched_class;
+	cur_class = rq->donor->sched_class;
 
 	/*
 	 * During CPU hotplug, a CPU may depend on kicking itself to make
@@ -7464,7 +7468,7 @@ static bool kick_one_cpu(s32 cpu, struct rq *this_rq, unsigned long *ksyncs)
 	    !sched_class_above(cur_class, &ext_sched_class)) {
 		if (cpumask_test_cpu(cpu, this_scx->cpus_to_preempt)) {
 			if (cur_class == &ext_sched_class)
-				rq->curr->scx.slice = 0;
+				rq->donor->scx.slice = 0;
 			cpumask_clear_cpu(cpu, this_scx->cpus_to_preempt);
 		}
 
Signature · 1 line
2.55.0
#4 lore raw

Re: [PATCH 2/9] sched_ext: Split curr|donor references properly

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:

New issues:
- [High] update_curr_scx() unconditionally consumes delta_exec for non-EXT donor tasks when an EXT owner task goes to sleep, improperly charging the EXT DL server and breaking CPU time accounting for the donor task.
- [Medium] The detailed task state dump in scx_dump_cpu() incorrectly assumes rq->curr is the only active task to dump, causing the actual BPF-scheduled EXT task (rq->donor) to be omitted when proxy execution is active and the contexts differ.

Pre-existing issues:
- [High] scx_can_stop_tick() evaluates the tick dependency of the outgoing task rather than the incoming task during set_next_task_scx(), leaving the tick improperly stopped on NOHZ_FULL CPUs. --

commit 176c10999ef611ede6c4d2387d4c2485699560c2
Author: John Stultz <[email protected]>

sched_ext: Split curr|donor references properly

This patch modifies sched_ext to use rq->donor instead of rq->curr for slice accounting, preemption, task picking, tick handling, and CPU kicks. This separation prepares the scheduler for proxy execution, where the task supplying the scheduling context (donor) can differ from the physically executing task.

Quoted 12 lines
kernel/sched/ext/ext.c +1 -1
index 1a0ec985da77d..242f6ffbba350 100644
@@ -1145,17 +1145,17 @@ static void touch_core_sched_dispatch(struct rq *rq, struct task_struct *p)
 
 static void update_curr_scx(struct rq *rq)
 {
-	struct task_struct *curr = rq->curr;
+	struct task_struct *donor = rq->donor;
 	s64 delta_exec;
 

kernel/sched/ext/ext.c static void touch_core_sched_dispatch(struct rq *rq, struct task_struct *p) · line 1156

Quoted 10 lines
kernel/sched/ext/ext.c +2 -4
 	delta_exec = update_curr_common(rq);
 	if (unlikely(delta_exec <= 0))
 		return;
 
-	if (curr->scx.slice != SCX_SLICE_INF) {
-		curr->scx.slice -= min_t(u64, curr->scx.slice, delta_exec);
-		if (!curr->scx.slice)
-			touch_core_sched(rq, curr);
+	if (donor->scx.slice != SCX_SLICE_INF) {
+		donor->scx.slice -= min_t(u64, donor->scx.slice, delta_exec);

[Severity: High]
Does this code apply EXT-specific accounting to a non-EXT donor task when the owner task goes to sleep?

If proxy execution is active, where the donor is a FAIR task and the owner (currently executing) is an EXT task, update_curr_scx() will be called because the executing task belongs to the EXT class.

Without a check ensuring the donor is actually in the EXT class (e.g., if (donor->sched_class != &ext_sched_class) return;), could the EXT DL server be erroneously charged for the FAIR task's time?

[ ... ]

kernel/sched/ext/ext.c static void run_deferred(struct rq *rq) · line 4036

Quoted 9 lines
kernel/sched/ext/ext.c +1 -1
@@ -4029,7 +4030,7 @@ static void run_deferred(struct rq *rq)
 #ifdef CONFIG_NO_HZ_FULL
 bool scx_can_stop_tick(struct rq *rq)
 {
-	struct task_struct *p = rq->curr;
+	struct task_struct *p = rq->donor;
 	struct scx_sched *sch = scx_task_sched(p);
 
 	if (p->sched_class != &ext_sched_class)

[Severity: High]
This is a pre-existing issue, but does scx_can_stop_tick() evaluate the tick dependency of the outgoing task rather than the incoming task?

When switching to an EXT task on a NOHZ_FULL CPU, set_next_task_scx() triggers a check:

set_next_task_scx()
sched_update_tick_dependency()

    scx_can_stop_tick()

At this point, rq->donor (and previously rq->curr) has not yet been updated to the newly selected task, which happens later in __schedule().

If the previous task was not an EXT task (e.g., idle), won't scx_can_stop_tick() return true and wrongly bypass the BPF scheduler's request for ticks?

[ ... ]

kernel/sched/ext/ext.c static void scx_dump_cpu(struct scx_sched *sch, struct seq_buf *s, · line 6016

Quoted 10 lines
kernel/sched/ext/ext.c +3 -0
@@ -6007,6 +6008,9 @@ static void scx_dump_cpu(struct scx_sched *sch, struct seq_buf *s,
 	dump_line(&ns, "          curr=%s[%d] class=%ps",
 		  rq->curr->comm, rq->curr->pid,
 		  rq->curr->sched_class);
+	dump_line(&ns, "          donor=%s[%d] class=%ps",
+		  rq->donor->comm, rq->donor->pid,
+		  rq->donor->sched_class);
 	if (!cpumask_empty(rq->scx.cpus_to_kick))
 		dump_line(&ns, "  cpus_to_kick   : %*pb",
 			  cpumask_pr_args(rq->scx.cpus_to_kick));

[Severity: Medium]
Will this miss dumping the detailed state of the EXT donor task if rq->donor and rq->curr differ?

Later in scx_dump_cpu(), the detailed task dump only checks rq->curr:

	if (rq->curr->sched_class == &ext_sched_class &&
	    (dump_all_tasks || scx_task_on_sched(sch, rq->curr)))
		scx_dump_task(sch, s, dctx, rq, rq->curr, '*');

If proxy execution is active and rq->donor is an EXT task but rq->curr is a FAIR task, the actual BPF-scheduled EXT task (the donor) would be completely omitted from the detailed dump during a crash or sysrq.

[ ... ]

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
author reviewer
#5 lore raw

PATCH 3/9 sched_ext: Fix TOCTOU race in consume_remote_task()

In reply to Andrea Righi

When pulling a task from a non-local DSQ, scx_consume_dispatch_q() checks whether the task can run on the destination rq via task_can_run_on_remote_rq(). However, it then drops the destination rq lock and locks the source rq in consume_remote_task() -> unlink_dsq_and_lock_src_rq(). During this window, the task might become migration disabled, making it invalid to migrate to the destination rq.

With proxy execution enabled, a mutex-intensive workload such as stress-ng --pipeherd 0 can trigger this race when a donor becomes active on its source rq during the unlocked window. Migrating the active execution context can make it resume with IRQ and preemption state from the wrong scheduling path, triggering sleeping-while-atomic warnings and subsequent lockdep corruption.

Re-evaluate task_can_run_on_remote_rq() after locking the source rq. Use a non-enforcing check because eligibility may have changed during the lock handoff independently of BPF policy; reporting that transient race through scx_error() would unnecessarily abort the scheduler. If the task can no longer migrate, clear its DSQ association, reset the holding CPU, and enqueue it to the global DSQ instead. Normal consumption filtering then skips it on ineligible CPUs while allowing an eligible CPU to pick it without forcing it onto the source local DSQ.

Also reject tasks which are currently executing. Although an on-CPU task should normally not be available for remote DSQ consumption, checking it explicitly prevents a stale or racy candidate from reaching deactivate_task() and set_task_cpu().

While the destination rq lock is dropped, clear the tracked rq state and restore it after reacquiring the lock. Otherwise, a nested ops.dequeue() callback can attempt to restore an rq which is no longer locked.

Closing the race and preserving rq tracking across the lock handoff are prerequisites for correctly supporting proxy execution with sched_ext.

---
 kernel/sched/ext/ext.c | 62 ++++++++++++++++++++++++++++++++++++------
 1 file changed, 53 insertions(+), 9 deletions(-)
kernel/sched/ext/ext.c +53 -9
index 242f6ffbba350..4c6cd694c86db 100644
@@ -2131,8 +2131,10 @@ static void move_remote_task_to_local_dsq(struct task_struct *p, u64 enq_flags,
  * - The BPF scheduler is bypassed while the rq is offline and we can always say
  *   no to the BPF scheduler initiated migrations while offline.
  *
- * The caller must ensure that @p and @rq are on different CPUs.
- * If enforce == true, caller must hold @p's rq lock.
+ * The caller must ensure that @p and @rq are on different CPUs. If @enforce is
+ * true, report violations attributable to BPF-directed migrations. The caller
+ * must hold @p's rq lock to avoid reporting a transient race as a scheduler
+ * error.
  */
 static bool task_can_run_on_remote_rq(struct scx_sched *sch,
 				      struct task_struct *p, struct rq *rq,
@@ -2150,6 +2152,10 @@ static bool task_can_run_on_remote_rq(struct scx_sched *sch,
 
 	WARN_ON_ONCE(task_cpu(p) == cpu);
 
+	/* Don't migrate a task which is running on a CPU. */
+	if (task_on_cpu(task_rq(p), p))
+		return false;
+
 	/*
 	 * If @p has migration disabled, @p->cpus_ptr is updated to contain only
 	 * the pinned CPU in migrate_disable_switch() while @p is being switched
@@ -2240,20 +2246,58 @@ static bool unlink_dsq_and_lock_src_rq(struct task_struct *p,
 		!WARN_ON_ONCE(src_rq != task_rq(p));
 }
 
-static bool consume_remote_task(struct rq *this_rq,
+static bool consume_remote_task(struct scx_sched *sch, struct rq *this_rq,
 				struct task_struct *p, u64 enq_flags,
 				struct scx_dispatch_q *dsq, struct rq *src_rq)
 {
+	struct rq *tracked_rq = scx_locked_rq();
+	bool consumed = false;
+
+	/*
+	 * consume_remote_task() may be called from an SCX op with @this_rq
+	 * recorded as the currently locked rq. Clear the tracking while the rq
+	 * lock is dropped so nested callbacks don't save and later try to restore
+	 * an rq which isn't locked anymore.
+	 */
+	if (tracked_rq) {
+		WARN_ON_ONCE(tracked_rq != this_rq);
+		update_locked_rq(NULL);
+	}
 	raw_spin_rq_unlock(this_rq);
 
 	if (unlink_dsq_and_lock_src_rq(p, dsq, src_rq)) {
+		/*
+		 * Eligibility may have changed while switching rq locks. This is a
+		 * kernel-side race, not an invalid BPF placement request, so don't
+		 * abort the scheduler on failure. Fall back to the global DSQ, where
+		 * normal consumption filters can select an eligible CPU without
+		 * forcing the task onto the source local DSQ.
+		 */
+		if (unlikely(!task_can_run_on_remote_rq(sch, p, this_rq, false))) {
+			p->scx.dsq = NULL;
+			p->scx.holding_cpu = -1;
+			scx_dispatch_enqueue(sch, src_rq,
+					     find_global_dsq(sch, task_cpu(p)), p,
+					     enq_flags | SCX_ENQ_CLEAR_OPSS |
+					     SCX_ENQ_GDSQ_FALLBACK);
+			if (sched_class_above(p->sched_class, src_rq->donor->sched_class))
+				resched_curr(src_rq);
+			raw_spin_rq_unlock(src_rq);
+			goto relock;
+		}
 		move_remote_task_to_local_dsq(p, enq_flags, src_rq, this_rq);
-		return true;
-	} else {
-		raw_spin_rq_unlock(src_rq);
-		raw_spin_rq_lock(this_rq);
-		return false;
+		consumed = true;
+		goto restore;
 	}
+	raw_spin_rq_unlock(src_rq);
+
+relock:
+	raw_spin_rq_lock(this_rq);
+restore:
+	if (tracked_rq)
+		update_locked_rq(tracked_rq);
+
+	return consumed;
 }
 
 /**
@@ -2363,7 +2407,7 @@ bool scx_consume_dispatch_q(struct scx_sched *sch, struct rq *rq,
 		}
 
 		if (task_can_run_on_remote_rq(sch, p, rq, false)) {
-			if (likely(consume_remote_task(rq, p, enq_flags, dsq, task_rq)))
+			if (likely(consume_remote_task(sch, rq, p, enq_flags, dsq, task_rq)))
 				return true;
 			goto retry;
 		}
Signature · 1 line
2.55.0
#6 lore raw

Re: [PATCH 3/9] sched_ext: Fix TOCTOU race in consume_remote_task()

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Pre-existing issues:
- [High] Pre-existing tracking bug in dispatch_to_local_dsq(): it drops the locked_rq lock without clearing the scx_locked_rq_state tracking, which triggers a lockdep crash if a nested ops.dequeue() callback executes in the fallback path. --

commit aff47a8e37704807f1fefe8e15adcb99c8836308
Author: Andrea Righi <[email protected]>

sched_ext: Fix TOCTOU race in consume_remote_task()

This commit addresses a TOCTOU race condition in sched_ext during task migration between runqueues. It ensures task eligibility is re-evaluated after locking the source runqueue and properly tracks runqueue lock state while dropping the lock to avoid lockdep warnings.

Quoted 4 lines
kernel/sched/ext/ext.c +0 -0
index 242f6ffbba350..4c6cd694c86db 100644

[ ... ]

Quoted 13 lines
kernel/sched/ext/ext.c +5 -1
@@ -2240,20 +2246,58 @@ static bool unlink_dsq_and_lock_src_rq(struct task_struct *p,
 		!WARN_ON_ONCE(src_rq != task_rq(p));
 }
 
-static bool consume_remote_task(struct rq *this_rq,
+static bool consume_remote_task(struct scx_sched *sch, struct rq *this_rq,
 				struct task_struct *p, u64 enq_flags,
 				struct scx_dispatch_q *dsq, struct rq *src_rq)
 {
+	struct rq *tracked_rq = scx_locked_rq();
+	bool consumed = false;
+
+	/*

kernel/sched/ext/ext.c static bool unlink_dsq_and_lock_src_rq(struct task_struct *p, · line 2266

Quoted 10 lines
kernel/sched/ext/ext.c +9 -0
+	 * consume_remote_task() may be called from an SCX op with @this_rq
+	 * recorded as the currently locked rq. Clear the tracking while the rq
+	 * lock is dropped so nested callbacks don't save and later try to restore
+	 * an rq which isn't locked anymore.
+	 */
+	if (tracked_rq) {
+		WARN_ON_ONCE(tracked_rq != this_rq);
+		update_locked_rq(NULL);
+	}
 	raw_spin_rq_unlock(this_rq);

[Severity: High]
This is a pre-existing issue, but does dispatch_to_local_dsq() have a similar problem with stale rq lock tracking?

Similar to consume_remote_task(), dispatch_to_local_dsq() drops the locked_rq without temporarily clearing scx_locked_rq_state:

kernel/sched/ext/ext.c:dispatch_to_local_dsq() {
    ...
	/* switch to @src_rq lock */
	if (locked_rq != src_rq) {
		raw_spin_rq_unlock(locked_rq);
		locked_rq = src_rq;
		raw_spin_rq_lock(src_rq);
	}
    ...
}

If task_can_run_on_remote_rq() fails, it falls back to enqueueing to the global DSQ:

		} else if (unlikely(!task_can_run_on_remote_rq(sch, p, dst_rq, true))) {
			p->scx.holding_cpu = -1;
			fallback = true;
			scx_dispatch_enqueue(sch, src_rq, find_global_dsq(sch, task_cpu(p)),
					     p, enq_flags | SCX_ENQ_GDSQ_FALLBACK);

When scx_dispatch_enqueue() triggers the nested ops.dequeue() callback via SCX_CALL_OP_TASK, the stale, unlocked rq state is saved and later restored in SCX_CALL_OP, which triggers a lockdep assertion failure.

Could this lead to a lockdep crash if a nested ops.dequeue() executes in the fallback path?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
author reviewer
#7 lore raw

PATCH 4/9 sched_ext: Handle blocked donor migration with proxy execution

In reply to Andrea Righi

From: John Stultz <[email protected]>

With proxy execution enabled, mutex-blocked donors stay runnable so their scheduling context can execute the lock owner.

sched_ext may normally relocate a queued blocked donor. set_task_cpu() updates both task_cpu() and wake_cpu, making the destination rq the donor's new callback home. This remains valid if the donor was previously proxy-migrated: the BPF scheduler is intentionally selecting a new return destination, which the next proxy migration will preserve. The usual migration-disabled, affinity, and rq-online checks apply.

An active donor cannot be moved normally because the source rq still references it through rq->donor for scheduling-class operations and runtime accounting. Moving it would associate the task with a different rq while leaving those source-rq references in place. The core proxy migration path avoids this by switching the rq donor to idle before moving the scheduling context.

Allow normal sched_ext migration of blocked donors unless the donor is active on its rq. In that case, defer migration until it is switched out or leave it to the proxy machinery.

Keep blocked donors on the local DSQ when they are put so they remain visible to the proxy pick path.

---
 kernel/sched/ext/ext.c | 28 ++++++++++++++++++++++++++++
 1 file changed, 28 insertions(+)
kernel/sched/ext/ext.c +28 -0
index 4c6cd694c86db..e7cf88b2be7f6 100644
@@ -2156,6 +2156,18 @@ static bool task_can_run_on_remote_rq(struct scx_sched *sch,
 	if (task_on_cpu(task_rq(p), p))
 		return false;
 
+	/*
+	 * A blocked donor may be moved normally to select a new callback rq.
+	 * set_task_cpu() updates wake_cpu and makes the destination rq its new
+	 * callback home, even if the donor was previously proxy-migrated.
+	 *
+	 * Don't move an active donor while the source rq still references it for
+	 * scheduling and accounting. The migration can be retried after the donor
+	 * is switched out.
+	 */
+	if (task_is_blocked(p) && task_current_donor(task_rq(p), p))
+		return false;
+
 	/*
 	 * If @p has migration disabled, @p->cpus_ptr is updated to contain only
 	 * the pinned CPU in migrate_disable_switch() while @p is being switched
@@ -2828,6 +2840,22 @@ static void put_prev_task_scx(struct rq *rq, struct task_struct *p,
 	if (p->scx.flags & SCX_TASK_QUEUED) {
 		set_task_runnable(rq, p);
 
+		/*
+		 * Mutex-blocked donors stay queued on the runqueue under proxy
+		 * execution, but the donor never runs as itself, proxy-exec
+		 * walks the blocked_on chain on the next __schedule() and runs
+		 * the lock owner in its place.
+		 *
+		 * Put the donor on the local DSQ directly so pick_next_task()
+		 * can still see it. find_proxy_task() will either run the chain
+		 * owner or deactivate the donor so the wakeup path can return it
+		 * and let BPF make a new dispatch decision once it is unblocked.
+		 */
+		if (task_is_blocked(p)) {
+			scx_dispatch_enqueue(sch, rq, &rq->scx.local_dsq, p, 0);
+			goto switch_class;
+		}
+
 		/*
 		 * If @p has slice left and is being put, @p is getting
 		 * preempted by a higher priority scheduler class or core-sched
Signature · 1 line
2.55.0
#8 lore raw

Re: [PATCH 4/9] sched_ext: Handle blocked donor migration with proxy execution

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] Plain data race on `rq->donor` in `task_current_donor()` during lockless evaluation triggers…

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] Plain data race on rq->donor in task_current_donor() during lockless evaluation triggers KCSAN warnings. --

commit 6a880849b19162c80e0a10e017da8716e5ef3566
Author: John Stultz <[email protected]>

sched_ext: Handle blocked donor migration with proxy execution

With proxy execution enabled, mutex-blocked donors stay runnable so their scheduling context can execute the lock owner. This commit allows normal sched_ext migration of blocked donors unless the donor is active on its rq.

Quoted 4 lines
kernel/sched/ext/ext.c +0 -0
index 4c6cd694c86db..e7cf88b2be7f6 100644

[ ... ]

Quoted 4 lines
kernel/sched/ext/ext.c +0 -0
@@ -2156,6 +2156,18 @@ static bool task_can_run_on_remote_rq(struct scx_sched *sch,
 	if (task_on_cpu(task_rq(p), p))
 		return false;
 

kernel/sched/ext/ext.c static bool task_can_run_on_remote_rq(struct scx_sched *sch, · line 2168

Quoted 10 lines
kernel/sched/ext/ext.c +10 -0
+	/*
+	 * A blocked donor may be moved normally to select a new callback rq.
+	 * set_task_cpu() updates wake_cpu and makes the destination rq its new
+	 * callback home, even if the donor was previously proxy-migrated.
+	 *
+	 * Don't move an active donor while the source rq still references it for
+	 * scheduling and accounting. The migration can be retried after the donor
+	 * is switched out.
+	 */
+	if (task_is_blocked(p) && task_current_donor(task_rq(p), p))

[Severity: Medium]
Will this trigger KCSAN warnings due to a data race on rq->donor?

When consume_dispatch_q() calls task_can_run_on_remote_rq() with enforce=false as an opportunistic lockless check, it only holds dsq->lock, not the runqueue lock for task_rq(p).

This causes task_current_donor() to perform a plain read of rq->donor:

kernel/sched/sched.h:task_current_donor() {
    return rq->donor == p;
}

Since rq->donor can be concurrently modified by the runqueue's owner CPU, the plain read violates the memory model. Even if it is logically safe because the condition is re-evaluated later with the lock held, should this be annotated with READ_ONCE() or data_race() to avoid KCSAN warnings?

Quoted 2 lines
kernel/sched/ext/ext.c +2 -0
+		return false;
+
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=4
author reviewer
#9 lore raw

PATCH 5/9 sched_ext: Fix ops.running/stopping() pairing for proxy-exec donors

In reply to Andrea Righi

With proxy-exec, pick_next_task() can return a task with blocked_on set (a proxy donor). put_prev_set_next_task() then calls set_next_task_scx() on this "ghost" task even though the task only provides scheduling context and never actually runs.

Calling ops.running() for such a donor produces a spurious running event. Simply suppressing ops.running() is not sufficient because the following put_prev_task_scx() would still invoke ops.stopping(), resulting in an unpaired stopping event.

Introduce SCX_TASK_IS_RUNNING to track whether a task entered a real running transition. Set and clear the flag independently of ops.running() and ops.stopping(), as the callbacks are independently optional. Invoke ops.running() only for non-blocked tasks and invoke ops.stopping() only after a real running transition. This keeps the callbacks paired for proxy donors while preserving stopping notifications for schedulers which only implement ops.stopping().

---
 include/linux/sched/ext.h |  2 ++
 kernel/sched/ext/ext.c    | 33 +++++++++++++++++++++++++--------
 2 files changed, 27 insertions(+), 8 deletions(-)
include/linux/sched/ext.h +2 -0
index 75cb8b119fb79..c6f58e1a66cdb 100644
@@ -102,6 +102,8 @@ enum scx_ent_flags {
 	SCX_TASK_SUB_INIT	= 1 << 4, /* task being initialized for a sub sched */
 	SCX_TASK_IMMED		= 1 << 5, /* task is on local DSQ with %SCX_ENQ_IMMED */
 
+	SCX_TASK_IS_RUNNING	= 1 << 6, /* entered a real running transition */
+
 	/*
 	 * Bits 8 to 10 are used to carry task state:
 	 *
kernel/sched/ext/ext.c +25 -8
index e7cf88b2be7f6..85e07cbe626f6 100644
@@ -1985,9 +1985,13 @@ static bool dequeue_task_scx(struct rq *rq, struct task_struct *p, int core_deq_
 	 * information meaningful to the BPF scheduler and can be suppressed by
 	 * skipping the callbacks if the task is !QUEUED.
 	 */
-	if (SCX_HAS_OP(sch, stopping) && task_current(rq, p)) {
-		update_curr_scx(rq);
-		SCX_CALL_OP_TASK(sch, stopping, rq, p, false);
+	if (task_current(rq, p) &&
+	    (p->scx.flags & SCX_TASK_IS_RUNNING)) {
+		if (SCX_HAS_OP(sch, stopping)) {
+			update_curr_scx(rq);
+			SCX_CALL_OP_TASK(sch, stopping, rq, p, false);
+		}
+		p->scx.flags &= ~SCX_TASK_IS_RUNNING;
 	}
 
 	if (SCX_HAS_OP(sch, quiescent) && !task_on_rq_migrating(p))
@@ -2743,9 +2747,17 @@ static void set_next_task_scx(struct rq *rq, struct task_struct *p, bool first)
 
 	p->se.exec_start = rq_clock_task(rq);
 
-	/* see dequeue_task_scx() on why we skip when !QUEUED */
-	if (SCX_HAS_OP(sch, running) && (p->scx.flags & SCX_TASK_QUEUED))
-		SCX_CALL_OP_TASK(sch, running, rq, p);
+	/*
+	 * See dequeue_task_scx() for why we skip when !QUEUED. A blocked proxy
+	 * donor is also skipped because it provides scheduling context but never
+	 * runs itself.
+	 */
+	if ((p->scx.flags & SCX_TASK_QUEUED) && !task_is_blocked(p)) {
+		if (SCX_HAS_OP(sch, running))
+			SCX_CALL_OP_TASK(sch, running, rq, p);
+
+		p->scx.flags |= SCX_TASK_IS_RUNNING;
+	}
 
 	clr_task_runnable(p, true);
 
@@ -2834,8 +2846,13 @@ static void put_prev_task_scx(struct rq *rq, struct task_struct *p,
 	update_curr_scx(rq);
 
 	/* see dequeue_task_scx() on why we skip when !QUEUED */
-	if (SCX_HAS_OP(sch, stopping) && (p->scx.flags & SCX_TASK_QUEUED))
-		SCX_CALL_OP_TASK(sch, stopping, rq, p, true);
+	if ((p->scx.flags & SCX_TASK_QUEUED) &&
+	    (p->scx.flags & SCX_TASK_IS_RUNNING)) {
+		if (SCX_HAS_OP(sch, stopping))
+			SCX_CALL_OP_TASK(sch, stopping, rq, p, true);
+
+		p->scx.flags &= ~SCX_TASK_IS_RUNNING;
+	}
 
 	if (p->scx.flags & SCX_TASK_QUEUED) {
 		set_task_runnable(rq, p);
Signature · 1 line
2.55.0
author reviewer
#10 lore raw

PATCH 6/9 sched_ext: Delegate proxy donor admission to BPF schedulers

In reply to Andrea Righi

Proxy execution keeps a blocked donor runnable so its scheduling context can execute the mutex owner. Dispatching sched_ext donors on a local DSQ bypasses the BPF scheduler ordering policy and can give donors more CPU priority than intended to perform the proxy execution handoff.

Add SCX_OPS_ENQ_BLOCKED as an explicit proxy execution capability. Tasks owned by schedulers without the flag block normally. Schedulers with the flag receive blocked donors through ops.enqueue() with SCX_ENQ_BLOCKED set in enq_flags and can apply their own admission policy.

From a high-level perspective, the resulting flow for a BPF scheduler with SCX_OPS_ENQ_BLOCKED is:

    D ------ blocked on -----> M ------ owned by -----> O
 [donor]                    [mutex]                [owner]
    |
    | ops.enqueue(D, SCX_ENQ_BLOCKED)
    | BPF dispatches D to CPUi
    v
 +-----------------+
 | CPUi local DSQ  |
 +-------+---------+
         |
         | pick_next_task() selects D
         v
 +-----------------+
 |   proxy exec    | move D to O's rq
 +-------+---------+
         |
         | run O using D's scheduling context
         v
    rq->curr  = O
    rq->donor = D
         |
         | O releases M
         v
 +-----------------+
 |   proxy exec    | return D to CPUi via wakeup
 +-----------------+

Scheduler ownership can change after a donor has already blocked. Since sched_change preserves queued state, deactivate retained donors before moving them to a root or sub-scheduler without the flag. They remain on the mutex wait path and wake normally when the mutex becomes available.

The donor starts associated with its original CPU. A BPF scheduler may dispatch it directly into that CPU local DSQ to let the core resolve the mutex owner and execute it with the donor scheduling context.

Although proxy execution does not require BPF to know the mutex owner's location, scx_bpf_task_proxy_cpu() and scx_bpf_task_proxy_cid() are provided to allow BPF schedulers to determine the owner's CPU or cid and make placement decisions, such as steering the donor toward the owner, according to their scheduling policy. Provide __COMPAT wrappers which return -EOPNOTSUPP when either kfunc is unavailable so schedulers remain loadable on older kernels.

Reschedule a retained donor when its mutex wakes it so ops.dispatch() can reconsider the now-unblocked task. Make SCX_OPS_ENQ_BLOCKED override the exiting and migration-disabled enqueue fallbacks so opted-in schedulers receive all eligible donor requests.

---
 kernel/sched/core.c                           |  58 ++++++++-
 kernel/sched/ext/ext.c                        | 112 +++++++++++++++---
 kernel/sched/ext/ext.h                        |   2 +
 kernel/sched/ext/internal.h                   |  25 +++-
 kernel/sched/ext/sub.c                        |  12 +-
 kernel/sched/sched.h                          |   8 ++
 tools/sched_ext/include/scx/common.bpf.h      |   2 +
 tools/sched_ext/include/scx/compat.bpf.h      |  18 +++
 tools/sched_ext/include/scx/compat.h          |   1 +
 .../sched_ext/include/scx/enum_defs.autogen.h |   1 +
 .../sched_ext/include/scx/enums.autogen.bpf.h |   3 +
 tools/sched_ext/include/scx/enums.autogen.h   |   1 +
 12 files changed, 222 insertions(+), 21 deletions(-)
kernel/sched/core.c +57 -1
index 0139cd4a8be7e..f3b6c3c952e69 100644
@@ -6764,6 +6764,28 @@ static void proxy_deactivate(struct rq *rq, struct task_struct *donor)
 	block_task(rq, donor, state);
 }
 
+/*
+ * Remove a retained proxy donor before changing its scheduler ownership.
+ * The caller holds p->pi_lock, so p cannot wake and migrate after block_task()
+ * drops it from the runqueue.
+ */
+void sched_proxy_block_task(struct rq *rq, struct task_struct *p)
+{
+	unsigned long state = READ_ONCE(p->__state);
+
+	lockdep_assert_held(&p->pi_lock);
+	lockdep_assert_rq_held(rq);
+
+	if (!task_is_blocked(p) || !task_on_rq_queued(p))
+		return;
+	if (WARN_ON_ONCE(state == TASK_RUNNING))
+		return;
+
+	if (task_current_donor(rq, p))
+		proxy_resched_idle(rq);
+	block_task(rq, p, state);
+}
+
 static inline void proxy_release_rq_lock(struct rq *rq, struct rq_flags *rf)
 	__releases(__rq_lockp(rq))
 {
@@ -7006,6 +7028,39 @@ find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf)
 	proxy_migrate_task(rq, rf, p, owner_cpu);
 	return NULL;
 }
+
+int task_proxy_cpu(struct task_struct *p)
+{
+	struct task_struct *owner;
+	struct mutex *mutex;
+	int owner_cpu;
+
+	if (!task_is_blocked(p))
+		return -ENOENT;
+
+	mutex = READ_ONCE(p->blocked_on);
+	if (!mutex)
+		return -ENOENT;
+
+	guard(raw_spinlock)(&mutex->wait_lock);
+	guard(raw_spinlock)(&p->blocked_lock);
+
+	if (mutex != __get_task_blocked_on(p))
+		return -EAGAIN;
+
+	owner = __mutex_owner(mutex);
+	if (!owner)
+		return -ENOENT;
+	if (!READ_ONCE(owner->on_rq) || owner->se.sched_delayed)
+		return -EAGAIN;
+
+	owner_cpu = task_cpu(owner);
+	if (owner_cpu != task_cpu(p) &&
+	    (p->nr_cpus_allowed == 1 || is_migration_disabled(p)))
+		return -EOPNOTSUPP;
+
+	return owner_cpu;
+}
 #else /* SCHED_PROXY_EXEC */
 static struct task_struct *
 find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf)
@@ -7136,7 +7191,8 @@ static void __sched notrace __schedule(int sched_mode)
 		 * task_is_blocked() will always be false).
 		 */
 		try_to_block_task(rq, prev, &prev_state,
-				  !task_is_blocked(prev));
+				  !task_is_blocked(prev) ||
+				  !scx_allow_proxy_exec(prev));
 		switch_count = &prev->nvcsw;
 	}
 
kernel/sched/ext/ext.c +96 -16

182 lines not on this page · raw message

kernel/sched/ext/ext.h +2 -0
index 0b7fc46aee08c..95aa0025c88e9 100644
@@ -20,6 +20,7 @@ void scx_rq_deactivate(struct rq *rq);
 int scx_check_setscheduler(struct task_struct *p, int policy);
 bool task_should_scx(int policy);
 bool scx_allow_ttwu_queue(const struct task_struct *p);
+bool scx_allow_proxy_exec(const struct task_struct *p);
 void init_sched_ext_class(void);
 
 static inline u32 scx_cpuperf_target(s32 cpu)
@@ -54,6 +55,7 @@ static inline void scx_rq_deactivate(struct rq *rq) {}
 static inline int scx_check_setscheduler(struct task_struct *p, int policy) { return 0; }
 static inline bool task_on_scx(const struct task_struct *p) { return false; }
 static inline bool scx_allow_ttwu_queue(const struct task_struct *p) { return true; }
+static inline bool scx_allow_proxy_exec(const struct task_struct *p) { return true; }
 static inline void init_sched_ext_class(void) {}
 
 #endif	/* CONFIG_SCHED_CLASS_EXT */
kernel/sched/ext/internal.h +24 -1
index f9fe7c6ebc4be..e966860a3d2fc 100644
@@ -212,6 +212,21 @@ enum scx_ops_flags {
 	 */
 	SCX_OPS_TID_TO_TASK		= 1LLU << 8,
 
+	/*
+	 * If set, mutex-blocked tasks remain runnable as proxy donors and are
+	 * passed to ops.enqueue() with %SCX_ENQ_BLOCKED. The BPF scheduler can
+	 * query the next mutex owner's CPU or cid with scx_bpf_task_proxy_cpu()
+	 * or scx_bpf_task_proxy_cid(). It controls when donors are dispatched and
+	 * whether they should preempt work on the owner's CPU.
+	 *
+	 * If clear, mutex-blocked tasks are removed from the runqueue normally
+	 * and cannot donate their scheduling context through proxy execution.
+	 *
+	 * For blocked donors, this flag takes precedence over
+	 * %SCX_OPS_ENQ_EXITING and %SCX_OPS_ENQ_MIGRATION_DISABLED.
+	 */
+	SCX_OPS_ENQ_BLOCKED		= 1LLU << 9,
+
 	SCX_OPS_ALL_FLAGS		= SCX_OPS_KEEP_BUILTIN_IDLE |
 					  SCX_OPS_ENQ_LAST |
 					  SCX_OPS_ENQ_EXITING |
@@ -220,7 +235,8 @@ enum scx_ops_flags {
 					  SCX_OPS_SWITCH_PARTIAL |
 					  SCX_OPS_BUILTIN_IDLE_PER_NODE |
 					  SCX_OPS_ALWAYS_ENQ_IMMED |
-					  SCX_OPS_TID_TO_TASK,
+					  SCX_OPS_TID_TO_TASK |
+					  SCX_OPS_ENQ_BLOCKED,
 
 	/* high 8 bits are internal, don't include in SCX_OPS_ALL_FLAGS */
 	__SCX_OPS_INTERNAL_MASK		= 0xffLLU << 56,
@@ -1339,6 +1355,12 @@ enum scx_enq_flags {
 	 */
 	SCX_ENQ_LAST		= 1LLU << 41,
 
+	/*
+	 * The task is blocked on a mutex and is being kept runnable as a proxy
+	 * donor. Only passed to ops.enqueue() when %SCX_OPS_ENQ_BLOCKED is set.
+	 */
+	SCX_ENQ_BLOCKED		= 1LLU << 42,
+
 	/* high 8 bits are internal */
 	__SCX_ENQ_INTERNAL_MASK	= 0xffLLU << 56,
 
@@ -1624,6 +1646,7 @@ void scx_task_iter_start(struct scx_task_iter *iter, struct cgroup *cgrp);
 void scx_task_iter_unlock(struct scx_task_iter *iter);
 void scx_task_iter_stop(struct scx_task_iter *iter);
 struct task_struct *scx_task_iter_next_locked(struct scx_task_iter *iter);
+void scx_prepare_task_sched_change(struct task_struct *p, struct scx_sched *sch);
 bool scx_consume_dispatch_q(struct scx_sched *sch, struct rq *rq,
 			    struct scx_dispatch_q *dsq, u64 enq_flags);
 bool scx_consume_global_dsq(struct scx_sched *sch, struct rq *rq);
kernel/sched/ext/sub.c +9 -3
index 0504204272732..0a5b84575ba5a 100644
@@ -117,7 +117,9 @@ static void scx_fail_parent(struct scx_sched *sch,
 		if (scx_task_on_sched(parent, p))
 			continue;
 
-		scoped_guard (sched_change, p, DEQUEUE_SAVE | DEQUEUE_MOVE) {
+		scx_prepare_task_sched_change(p, parent);
+		scoped_guard (sched_change, p, DEQUEUE_SAVE | DEQUEUE_MOVE |
+			      DEQUEUE_NOCLOCK) {
 			scx_disable_and_exit_task(sch, p);
 			scx_set_task_sched(p, parent);
 		}
@@ -209,7 +211,9 @@ void scx_sub_disable(struct scx_sched *sch)
 			continue;
 		}
 
-		scoped_guard (sched_change, p, DEQUEUE_SAVE | DEQUEUE_MOVE) {
+		scx_prepare_task_sched_change(p, parent);
+		scoped_guard (sched_change, p, DEQUEUE_SAVE | DEQUEUE_MOVE |
+			      DEQUEUE_NOCLOCK) {
 			/*
 			 * $p is initialized for $parent and still attached to
 			 * @sch. Disable and exit for @sch, switch over to
@@ -503,7 +507,9 @@ void scx_sub_enable_workfn(struct kthread_work *work)
 		if (!(p->scx.flags & SCX_TASK_SUB_INIT))
 			continue;
 
-		scoped_guard (sched_change, p, DEQUEUE_SAVE | DEQUEUE_MOVE) {
+		scx_prepare_task_sched_change(p, sch);
+		scoped_guard (sched_change, p, DEQUEUE_SAVE | DEQUEUE_MOVE |
+			      DEQUEUE_NOCLOCK) {
 			/*
 			 * $p must be either READY or ENABLED. If ENABLED,
 			 * __scx_disabled_and_exit_task() first disables and
kernel/sched/sched.h +8 -0
index 56acf502ba260..1e4872ab46848 100644
@@ -2470,6 +2470,14 @@ static inline bool task_is_blocked(struct task_struct *p)
 	return !!p->blocked_on;
 }
 
+#ifdef CONFIG_SCHED_PROXY_EXEC
+int task_proxy_cpu(struct task_struct *p);
+void sched_proxy_block_task(struct rq *rq, struct task_struct *p);
+#else
+static inline int task_proxy_cpu(struct task_struct *p) { return -EOPNOTSUPP; }
+static inline void sched_proxy_block_task(struct rq *rq, struct task_struct *p) {}
+#endif
+
 static inline int task_on_cpu(struct rq *rq, struct task_struct *p)
 {
 	return p->on_cpu;
tools/sched_ext/include/scx/common.bpf.h +2 -0
index bd51986c4c42e..74d57b0aec39e 100644
@@ -95,6 +95,8 @@ s32 scx_bpf_pick_idle_cpu(const cpumask_t *cpus_allowed, u64 flags) __ksym;
 s32 scx_bpf_pick_any_cpu_node(const cpumask_t *cpus_allowed, int node, u64 flags) __ksym __weak;
 s32 scx_bpf_pick_any_cpu(const cpumask_t *cpus_allowed, u64 flags) __ksym;
 bool scx_bpf_task_running(const struct task_struct *p) __ksym;
+s32 scx_bpf_task_proxy_cpu(struct task_struct *p) __ksym __weak;
+s32 scx_bpf_task_proxy_cid(struct task_struct *p) __ksym __weak;
 s32 scx_bpf_task_cpu(const struct task_struct *p) __ksym;
 struct rq *scx_bpf_locked_rq(void) __ksym;
 struct task_struct *scx_bpf_cpu_curr(s32 cpu) __ksym __weak;
tools/sched_ext/include/scx/compat.bpf.h +18 -0
index 133058578668d..d25996163206a 100644
@@ -133,6 +133,24 @@ static inline void scx_bpf_cid_override(const s32 *cpu_to_cid, u32 cpu_to_cid__s
 		return scx_bpf_cid_override___compat(cpu_to_cid, cpu_to_cid__sz);
 }
 
+/*
+ * v7.3: scx_bpf_task_proxy_cpu() and scx_bpf_task_proxy_cid() for querying
+ * the next proxy execution destination. Return -EOPNOTSUPP if unavailable.
+ */
+static inline s32 __COMPAT_scx_bpf_task_proxy_cpu(struct task_struct *p)
+{
+	if (bpf_ksym_exists(scx_bpf_task_proxy_cpu))
+		return scx_bpf_task_proxy_cpu(p);
+	return -EOPNOTSUPP;
+}
+
+static inline s32 __COMPAT_scx_bpf_task_proxy_cid(struct task_struct *p)
+{
+	if (bpf_ksym_exists(scx_bpf_task_proxy_cid))
+		return scx_bpf_task_proxy_cid(p);
+	return -EOPNOTSUPP;
+}
+
 /**
  * __COMPAT_is_enq_cpu_selected - Test if SCX_ENQ_CPU_SELECTED is on
  * in a compatible way. We will preserve this __COMPAT helper until v6.16.
tools/sched_ext/include/scx/compat.h +1 -0
index 23d9ef3e4c9d2..8d5606e465080 100644
@@ -117,6 +117,7 @@ static inline bool __COMPAT_struct_has_field(const char *type, const char *field
 #define SCX_OPS_ALLOW_QUEUED_WAKEUP SCX_OPS_FLAG(SCX_OPS_ALLOW_QUEUED_WAKEUP)
 #define SCX_OPS_BUILTIN_IDLE_PER_NODE SCX_OPS_FLAG(SCX_OPS_BUILTIN_IDLE_PER_NODE)
 #define SCX_OPS_ALWAYS_ENQ_IMMED SCX_OPS_FLAG(SCX_OPS_ALWAYS_ENQ_IMMED)
+#define SCX_OPS_ENQ_BLOCKED SCX_OPS_FLAG(SCX_OPS_ENQ_BLOCKED)
 
 #define SCX_PICK_IDLE_FLAG(name) __COMPAT_ENUM_OR_ZERO("scx_pick_idle_cpu_flags", #name)
 
tools/sched_ext/include/scx/enum_defs.autogen.h +1 -0
index da4b459820fdd..79b31eb7db7cb 100644
@@ -55,6 +55,7 @@
 #define HAVE_SCX_ENQ_IMMED
 #define HAVE_SCX_ENQ_REENQ
 #define HAVE_SCX_ENQ_LAST
+#define HAVE_SCX_ENQ_BLOCKED
 #define HAVE___SCX_ENQ_INTERNAL_MASK
 #define HAVE_SCX_ENQ_CLEAR_OPSS
 #define HAVE_SCX_ENQ_DSQ_PRIQ
tools/sched_ext/include/scx/enums.autogen.bpf.h +3 -0
index dafccbb6b69d2..7efe7b9346b49 100644
@@ -130,6 +130,9 @@ const volatile u64 __SCX_ENQ_REENQ __weak;
 const volatile u64 __SCX_ENQ_LAST __weak;
 #define SCX_ENQ_LAST __SCX_ENQ_LAST
 
+const volatile u64 __SCX_ENQ_BLOCKED __weak;
+#define SCX_ENQ_BLOCKED __SCX_ENQ_BLOCKED
+
 const volatile u64 __SCX_ENQ_CLEAR_OPSS __weak;
 #define SCX_ENQ_CLEAR_OPSS __SCX_ENQ_CLEAR_OPSS
 
tools/sched_ext/include/scx/enums.autogen.h +1 -0
index bbd4901f4fce3..f8fbeb7fbf95b 100644
@@ -47,6 +47,7 @@
 	SCX_ENUM_SET(skel, scx_enq_flags, SCX_ENQ_IMMED); \
 	SCX_ENUM_SET(skel, scx_enq_flags, SCX_ENQ_REENQ); \
 	SCX_ENUM_SET(skel, scx_enq_flags, SCX_ENQ_LAST); \
+	SCX_ENUM_SET(skel, scx_enq_flags, SCX_ENQ_BLOCKED); \
 	SCX_ENUM_SET(skel, scx_enq_flags, SCX_ENQ_CLEAR_OPSS); \
 	SCX_ENUM_SET(skel, scx_enq_flags, SCX_ENQ_DSQ_PRIQ); \
 	SCX_ENUM_SET(skel, scx_deq_flags, SCX_DEQ_SCHED_CHANGE); \
Signature · 1 line
2.55.0
#11 lore raw

Re: [PATCH 6/9] sched_ext: Delegate proxy donor admission to BPF schedulers

Thank you for your contribution! Sashiko AI review found 5 potential issue(s) to consider: - [Critical] task_proxy_cpu() locklessly accesses p->blocked_on and dereferences the mutex to acquire its…

Thank you for your contribution! Sashiko AI review found 5 potential issue(s) to consider:
- [Critical] task_proxy_cpu() locklessly accesses p->blocked_on and dereferences the mutex to acquire its wait_lock, leading to a Use-After-Free if the target task is awakened and the mutex is concurrently destroyed.
- [High] task_proxy_cpu() acquires p->blocked_lock using raw_spinlock without disabling interrupts, risking a hard deadlock against hardware IRQ handlers.
- [High] sched_proxy_block_task() asynchronously swaps the proxy donor to idle without forcing a reschedule on the target CPU, leaving the target CPU to run the mutex owner indefinitely under idle accounting.
- [High] sched_setscheduler() fails to block proxy donors when transitioning them to a BPF scheduler without SCX_OPS_ENQ_BLOCKED, violating scheduler admission policies.
- [High] scx_prepare_task_sched_change() is called locklessly, modifying runqueue state and updating the runqueue clock without holding rq->lock or pi_lock.
--

commit fbf923089be5e0555003149fc8823b7f5e5723f7
Author: Andrea Righi <[email protected]>
Subject: sched_ext: Delegate proxy donor admission to BPF schedulers

Proxy execution keeps a blocked donor runnable so its scheduling context can execute the mutex owner. This commit adds SCX_OPS_ENQ_BLOCKED to delegate proxy execution admission to BPF schedulers, allowing them to handle blocked donors according to their own policies. Schedulers without this capability have their donors blocked normally via sched_proxy_block_task().

Quoted 4 lines
kernel/sched/core.c +0 -0
index 3673b17f6098c..19a6a78ef8472 100644

[ ... ]

Quoted 15 lines
kernel/sched/core.c +11 -0
@@ -6767,6 +6767,28 @@ static void proxy_deactivate(struct rq *rq, struct task_struct *donor)
 	block_task(rq, donor, state);
 }
 
+/*
+ * Remove a retained proxy donor before changing its scheduler ownership.
+ * The caller holds p->pi_lock, so p cannot wake and migrate after block_task()
+ * drops it from the runqueue.
+ */
+void sched_proxy_block_task(struct rq *rq, struct task_struct *p)
+{
+	unsigned long state = READ_ONCE(p->__state);
+
+	lockdep_assert_held(&p->pi_lock);
+	lockdep_assert_rq_held(rq);

kernel/sched/core.c static void proxy_deactivate(struct rq *rq, struct task_struct *donor) · line 6790

Quoted 10 lines
kernel/sched/core.c +10 -0
+
+	if (!task_is_blocked(p) || !task_on_rq_queued(p))
+		return;
+	if (WARN_ON_ONCE(state == TASK_RUNNING))
+		return;
+
+	if (task_current_donor(rq, p))
+		proxy_resched_idle(rq);
+	block_task(rq, p, state);
+}

[Severity: High]
Is this missing a resched_curr(rq) call?

When scx_prepare_task_sched_change() calls sched_proxy_block_task() remotely, it replaces rq->donor with rq->idle via proxy_resched_idle(rq). If no IPI is sent to notify the remote CPU (since resched_curr() is omitted), won't the remote CPU continue running the mutex owner indefinitely under idle accounting, completely bypassing scheduler preemption?

[ ... ]

Quoted 9 lines
kernel/sched/core.c +5 -0
@@ -7009,6 +7031,39 @@ find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf)
 	proxy_migrate_task(rq, rf, p, owner_cpu);
 	return NULL;
 }
+
+int task_proxy_cpu(struct task_struct *p)
+{
+	struct task_struct *owner;
+	struct mutex *mutex;

kernel/sched/core.c find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf) · line 7048

Quoted 10 lines
kernel/sched/core.c +10 -0
+	int owner_cpu;
+
+	if (!task_is_blocked(p))
+		return -ENOENT;
+
+	mutex = READ_ONCE(p->blocked_on);
+	if (!mutex)
+		return -ENOENT;
+
+	guard(raw_spinlock)(&mutex->wait_lock);

[Severity: Critical]
Can this result in a use-after-free?

The scx_bpf_task_proxy_cpu() kfunc can invoke this on any RCU-protected task. The lockless read of p->blocked_on above occurs without holding p->pi_lock or rq->lock. If the target task concurrently wakes up, finishes its execution, and frees the mutex, the wait_lock acquisition here would be dereferencing freed memory.

kernel/sched/core.c find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf) · line 7046

Quoted 1 line
kernel/sched/core.c +1 -0
+	guard(raw_spinlock)(&p->blocked_lock);

[Severity: High]
Can this cause a spinlock deadlock against hardware IRQs?

The scx_bpf_task_proxy_cpu() kfunc can be called from IRQ-enabled contexts like BPF_PROG_TYPE_SYSCALL programs. Because guard(raw_spinlock) does not disable interrupts (unlike raw_spinlock_irqsave), if an interrupt fires while the lock is held and tries to wake the same task (calling clear_task_blocked_on() from hardirq context), it will deadlock trying to acquire p->blocked_lock.

[ ... ]

Quoted 4 lines
kernel/sched/ext/ext.c +0 -0
index 85e07cbe626f6..a47ddbf741ea3 100644

[ ... ]

Quoted 14 lines
kernel/sched/ext/ext.c +10 -0
@@ -23,6 +23,27 @@
 
 DEFINE_RAW_SPINLOCK(scx_sched_lock);
 
+bool scx_allow_proxy_exec(const struct task_struct *p)
+{
+	if (!task_on_scx(p))
+		return true;
+
+	return scx_task_sched(p)->ops.flags & SCX_OPS_ENQ_BLOCKED;
+}
+
+/*
+ * Called immediately before sched_change_begin(). The caller must pass

kernel/sched/ext/ext.c line 45

Quoted 10 lines
kernel/sched/ext/ext.c +10 -0
+ * DEQUEUE_NOCLOCK so the rq clock is updated only once.
+ */
+void scx_prepare_task_sched_change(struct task_struct *p, struct scx_sched *sch)
+{
+	update_rq_clock(task_rq(p));
+
+	/* Block retained donors that the incoming scheduler cannot manage. */
+	if (!(sch->ops.flags & SCX_OPS_ENQ_BLOCKED))
+		sched_proxy_block_task(task_rq(p), p);
+}

[Severity: High]
How are blocked donors handled when a task policy is changed via the sched_setscheduler() syscall?

It appears __sched_setscheduler() changes the task's class but does not invoke scx_prepare_task_sched_change(), nor does scx_check_setscheduler() validate this transition. If a blocked proxy donor transitions to a BPF scheduler that lacks SCX_OPS_ENQ_BLOCKED, won't it bypass the admission check and get enqueued as a normal runnable task instead of a blocked donor?

[ ... ]

kernel/sched/ext/ext.c static void scx_root_enable_workfn(struct kthread_work *work) · line 6993

Quoted 10 lines
kernel/sched/ext/ext.c +4 -0
@@ -6950,6 +6985,10 @@ static void scx_root_enable_workfn(struct kthread_work *work)
 
 		if (old_class != new_class)
 			queue_flags |= DEQUEUE_CLASS;
+		if (new_class == &ext_sched_class) {
+			scx_prepare_task_sched_change(p, sch);
+			queue_flags |= DEQUEUE_NOCLOCK;
+		}
 
 		scoped_guard (sched_change, p, queue_flags) {

[Severity: High]
Is it safe to call scx_prepare_task_sched_change() outside the sched_change guard?

scx_prepare_task_sched_change() calls update_rq_clock() and sched_proxy_block_task(), both of which strictly require rq->lock. However, here they are called locklessly before scoped_guard() acquires p->pi_lock and rq->lock. Could this cause runqueue RB-tree corruption and lockdep panics?

[ ... ]

Quoted 4 lines
kernel/sched/ext/sub.c +0 -0
index 0504204272732..0a5b84575ba5a 100644

[ ... ]

kernel/sched/ext/sub.c static void scx_fail_parent(struct scx_sched *sch, · line 122

Quoted 8 lines
kernel/sched/ext/sub.c +3 -1
@@ -117,7 +117,9 @@ static void scx_fail_parent(struct scx_sched *sch,
 		if (scx_task_on_sched(parent, p))
 			continue;
 
-		scoped_guard (sched_change, p, DEQUEUE_SAVE | DEQUEUE_MOVE) {
+		scx_prepare_task_sched_change(p, parent);
+		scoped_guard (sched_change, p, DEQUEUE_SAVE | DEQUEUE_MOVE |
+			      DEQUEUE_NOCLOCK) {

[Severity: High]
Same issue here. scx_prepare_task_sched_change() is called locklessly before the rq->lock is acquired by the sched_change guard.

[ ... ]

kernel/sched/ext/sub.c void scx_sub_disable(struct scx_sched *sch) · line 216

Quoted 8 lines
kernel/sched/ext/sub.c +3 -1
@@ -209,7 +211,9 @@ void scx_sub_disable(struct scx_sched *sch)
 			continue;
 		}
 
-		scoped_guard (sched_change, p, DEQUEUE_SAVE | DEQUEUE_MOVE) {
+		scx_prepare_task_sched_change(p, parent);
+		scoped_guard (sched_change, p, DEQUEUE_SAVE | DEQUEUE_MOVE |
+			      DEQUEUE_NOCLOCK) {

[Severity: High]
Same issue here. scx_prepare_task_sched_change() is called locklessly.

[ ... ]

kernel/sched/ext/sub.c void scx_sub_enable_workfn(struct kthread_work *work) · line 512

Quoted 8 lines
kernel/sched/ext/sub.c +3 -1
@@ -503,7 +507,9 @@ void scx_sub_enable_workfn(struct kthread_work *work)
 		if (!(p->scx.flags & SCX_TASK_SUB_INIT))
 			continue;
 
-		scoped_guard (sched_change, p, DEQUEUE_SAVE | DEQUEUE_MOVE) {
+		scx_prepare_task_sched_change(p, sch);
+		scoped_guard (sched_change, p, DEQUEUE_SAVE | DEQUEUE_MOVE |
+			      DEQUEUE_NOCLOCK) {

[Severity: High]
Same issue here. scx_prepare_task_sched_change() is called locklessly.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=6
#12 lore raw

Re: [PATCH 6/9] sched_ext: Delegate proxy donor admission to BPF schedulers

In reply to Andrea Righi

Hello Andrea,

Quoted 11 lines · Andrea Righi
kernel/sched/core.c +3 -0
index 0139cd4a8be7e..f3b6c3c952e69 100644
@@ -6764,6 +6764,28 @@ static void proxy_deactivate(struct rq *rq, struct task_struct *donor)
 	block_task(rq, donor, state);
 }
 
+/*
+ * Remove a retained proxy donor before changing its scheduler ownership.
+ * The caller holds p->pi_lock, so p cannot wake and migrate after block_task()

kernel/sched/core.c static void proxy_deactivate(struct rq *rq, struct task_struct *donor) · line 6779

Quoted 10 lines
kernel/sched/core.c +10 -0
+ * drops it from the runqueue.
+ */
+void sched_proxy_block_task(struct rq *rq, struct task_struct *p)
+{
+	unsigned long state = READ_ONCE(p->__state);
+
+	lockdep_assert_held(&p->pi_lock);
+	lockdep_assert_rq_held(rq);
+
+	if (!task_is_blocked(p) || !task_on_rq_queued(p))

nit.

You may want to use p->is_blocked indicator for this which marks if the task has gone beyond the blocking point in __schedule(). I'll ensure we doesn't prematurely block task before here before it hits the correct blocking point in __schedule().

Quoted 16 lines
kernel/sched/core.c +9 -0
+		return;
+	if (WARN_ON_ONCE(state == TASK_RUNNING))
+		return;
+
+	if (task_current_donor(rq, p))
+		proxy_resched_idle(rq);
+	block_task(rq, p, state);
+}
+
 static inline void proxy_release_rq_lock(struct rq *rq, struct rq_flags *rf)
 	__releases(__rq_lockp(rq))
 {
@@ -7006,6 +7028,39 @@ find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf)
 	proxy_migrate_task(rq, rf, p, owner_cpu);
 	return NULL;
 }

kernel/sched/core.c find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf) · line 7040

Quoted 10 lines
kernel/sched/core.c +10 -0
+
+int task_proxy_cpu(struct task_struct *p)
+{
+	struct task_struct *owner;
+	struct mutex *mutex;
+	int owner_cpu;
+
+	if (!task_is_blocked(p))
+		return -ENOENT;
+

This entry check should also use p->is_blocked now. Before that, proxy doesn't start for core.

Quoted 5 lines
kernel/sched/core.c +5 -0
+	mutex = READ_ONCE(p->blocked_on);
+	if (!mutex)
+		return -ENOENT;
+
+	guard(raw_spinlock)(&mutex->wait_lock);

kernel/sched/core.c find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf) · line 7055

Quoted 10 lines
kernel/sched/core.c +10 -0
+	guard(raw_spinlock)(&p->blocked_lock);
+
+	if (mutex != __get_task_blocked_on(p))
+		return -EAGAIN;
+
+	owner = __mutex_owner(mutex);
+	if (!owner)
+		return -ENOENT;
+	if (!READ_ONCE(owner->on_rq) || owner->se.sched_delayed)
+		return -EAGAIN;

This probabaly warrants a different return code since you don't want to keep retrying until the owner wakes up.

kernel/sched/core.c find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf) · line 7060

Quoted 5 lines
kernel/sched/core.c +5 -0
+
+	owner_cpu = task_cpu(owner);
+	if (owner_cpu != task_cpu(p) &&
+	    (p->nr_cpus_allowed == 1 || is_migration_disabled(p)))
+		return -EOPNOTSUPP;

Do we need this? Proxy can work even if the owner is on a CPU where the donor cannot run.

Quoted 6 lines
kernel/sched/core.c +3 -0
+
+	return owner_cpu;
+}
 #else /* SCHED_PROXY_EXEC */
 static struct task_struct *
 find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf)
Signature · 2 lines
Thanks and Regards,
Prateek
author reviewer
#13 lore raw

PATCH 7/9 sched_ext: Add selftest for blocked donor admission

In reply to Andrea Righi

SCX_OPS_ENQ_BLOCKED allows BPF schedulers to receive blocked proxy donors through ops.enqueue(). SCX_ENQ_BLOCKED identifies donor admission requests in enq_flags, while __COMPAT_scx_bpf_task_proxy_cpu() returns the CPU of the mutex owner or a fallback error when the kfunc is unavailable. Add test coverage for this interface and owner CPU.

Exercise a three-task priority inversion on a single CPU using a weighted vruntime BPF scheduler. A nice +19 owner holds a shared mutex, a nice -20 donor blocks on it and a nice 0 CPU contender delays the owner.

Block the first donor before attaching the BPF scheduler and hold the owner until attachment completes. Verify that the donor starts sleeping and on-rq, is removed from the runqueue when the incoming scheduler does not support blocked donors, and remains on-rq when it does.

Run the workload with SCX_OPS_ENQ_BLOCKED first disabled and then enabled. In the enabled run, count blocked donor enqueues, validate the proxy CPU, and dispatch the donor there to facilitate proxy execution. Report average mutex hold and wait times without enforcing performance results.

Proxy execution coverage requires CONFIG_SCHED_PROXY_EXEC=y, which the selftest config selects. The mutex fixture is a loadable kernel module built through TEST_GEN_MODS_DIR and managed by the test.

---
 tools/testing/selftests/sched_ext/.gitignore  |   4 +
 tools/testing/selftests/sched_ext/Makefile    |   2 +
 tools/testing/selftests/sched_ext/config      |   2 +
 .../selftests/sched_ext/enq_blocked.bpf.c     | 128 +++
 .../testing/selftests/sched_ext/enq_blocked.c | 732 ++++++++++++++++++
 .../testing/selftests/sched_ext/enq_blocked.h |  27 +
 .../selftests/sched_ext/test_modules/Makefile |  13 +
 .../test_modules/scx_enq_blocked_test.c       | 193 +++++
 8 files changed, 1101 insertions(+)
 create mode 100644 tools/testing/selftests/sched_ext/enq_blocked.bpf.c
 create mode 100644 tools/testing/selftests/sched_ext/enq_blocked.c
 create mode 100644 tools/testing/selftests/sched_ext/enq_blocked.h
 create mode 100644 tools/testing/selftests/sched_ext/test_modules/Makefile
 create mode 100644 tools/testing/selftests/sched_ext/test_modules/scx_enq_blocked_test.c
tools/testing/selftests/sched_ext/.gitignore +4 -0
index ae5491a114c09..54a1fd2af713d 100644
@@ -4,3 +4,7 @@
 !Makefile
 !.gitignore
 !config
+!test_modules/
+!test_modules/scx_enq_blocked_test.c
+!test_modules/Makefile
+test_modules/*.mod.c
tools/testing/selftests/sched_ext/Makefile +2 -0
index 5d2dffca0e918..8eeece67d1de1 100644
@@ -5,6 +5,7 @@ include ../../../scripts/Makefile.arch
 include ../../../scripts/Makefile.include
 
 TEST_GEN_PROGS := runner
+TEST_GEN_MODS_DIR := test_modules
 
 # override lib.mk's default rules
 OVERRIDE_TARGETS := 1
@@ -164,6 +165,7 @@ all_test_bpfprogs := $(foreach prog,$(wildcard *.bpf.c),$(INCLUDE_DIR)/$(patsubs
 auto-test-targets :=			\
 	create_dsq			\
 	dequeue				\
+	enq_blocked			\
 	enq_last_no_enq_fails		\
 	ddsp_bogus_dsq_fail		\
 	ddsp_vtimelocal_fail		\
tools/testing/selftests/sched_ext/config +2 -0
index aa901b05c8ad6..affa3cf33470a 100644
@@ -6,3 +6,5 @@ CONFIG_BPF=y
 CONFIG_BPF_SYSCALL=y
 CONFIG_DEBUG_INFO=y
 CONFIG_DEBUG_INFO_BTF=y
+CONFIG_EXPERT=y
+CONFIG_SCHED_PROXY_EXEC=y
tools/testing/selftests/sched_ext/enq_blocked.bpf.c new file +128 -0

128 lines not on this page · raw message

tools/testing/selftests/sched_ext/enq_blocked.c new file +732 -0

732 lines not on this page · raw message

tools/testing/selftests/sched_ext/enq_blocked.h new file +27 -0
new file mode 100644
index 0000000000000..68094fb06c875
@@ -0,0 +1,27 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES */
+#ifndef __ENQ_BLOCKED_H
+#define __ENQ_BLOCKED_H
+
+#include <linux/ioctl.h>
+#include <linux/types.h>
+
+struct enq_blocked_stats {
+	__u64 hold_time_ns;
+	__u64 wait_time_ns;
+	__u64 nr_holds;
+	__u64 nr_waits;
+};
+
+#define ENQ_BLOCKED_IOCTL_OWNER	_IO('s', 1)
+#define ENQ_BLOCKED_IOCTL_DONOR	_IO('s', 2)
+#define ENQ_BLOCKED_IOCTL_RESET_STATS	_IO('s', 3)
+#define ENQ_BLOCKED_IOCTL_GET_STATS	_IOR('s', 4, struct enq_blocked_stats)
+#define ENQ_BLOCKED_IOCTL_PREP_ATTACH	_IO('s', 5)
+#define ENQ_BLOCKED_IOCTL_ATTACH_DONE	_IO('s', 6)
+#define ENQ_BLOCKED_IOCTL_DONOR_STATE	_IO('s', 7)
+
+#define ENQ_BLOCKED_DONOR_SLEEPING	(1U << 0)
+#define ENQ_BLOCKED_DONOR_ON_RQ		(1U << 1)
+
+#endif /* __ENQ_BLOCKED_H */
tools/testing/selftests/sched_ext/test_modules/Makefile new file +13 -0
new file mode 100644
index 0000000000000..a0e9e9401ead6
@@ -0,0 +1,13 @@
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES
+
+TESTMODS_DIR := $(realpath $(dir $(abspath $(lastword $(MAKEFILE_LIST)))))
+KDIR ?= $(if $(O),$(O),$(realpath ../../../../..))
+
+obj-m += scx_enq_blocked_test.o
+
+all:
+	+$(Q)$(MAKE) -C $(KDIR) M=$(TESTMODS_DIR) modules
+
+clean:
+	+$(Q)$(MAKE) -C $(KDIR) M=$(TESTMODS_DIR) clean
tools/testing/selftests/sched_ext/test_modules/scx_enq_blocked_test.c new file +193 -0

193 lines not on this page · raw message

Signature · 1 line
2.55.0
author reviewer
#14 lore raw

PATCH 8/9 sched_ext: scx_qmap: Add proxy execution support

In reply to Andrea Righi

Add a -B option to opt scx_qmap into queueing mutex-blocked tasks for proxy execution. Without the option, SCX_OPS_ENQ_BLOCKED remains clear and mutex waiters block normally. With -B, set SCX_OPS_ENQ_BLOCKED and enable the corresponding BPF-side blocked-donor admission policy.

When qmap_enqueue() receives SCX_ENQ_BLOCKED for a donor with runtime budget remaining, insert it directly into the local DSQ of its current cid without refilling its slice. This lets proxy execution continue using the donor's existing budget.

Once the donor exhausts its slice, let it proceed through qmap's normal enqueue path. This applies the same placement and slice refill decisions as other enqueues instead of imposing a proxy-specific policy.

---
 tools/sched_ext/scx_qmap.bpf.c | 13 +++++++++++++
 tools/sched_ext/scx_qmap.c     |  9 +++++++--
 2 files changed, 20 insertions(+), 2 deletions(-)
tools/sched_ext/scx_qmap.bpf.c +13 -0
index fd9a82a676278..2ba3e2a4f625f 100644
@@ -49,6 +49,7 @@ const volatile u64 sub_cgroup_id;
 const volatile s32 disallow_tgid;
 const volatile bool suppress_dump;
 const volatile bool always_enq_immed;
+const volatile bool enq_blocked;
 const volatile u32 immed_stress_nth;
 const volatile u32 max_tasks;
 
@@ -396,6 +397,18 @@ void BPF_STRUCT_OPS(qmap_enqueue, struct task_struct *p, u64 enq_flags)
 	 */
 	taskc->core_sched_seq = qa.core_sched_tail_seqs[idx]++;
 
+	/*
+	 * Admit a blocked donor with runtime budget left by inserting it into
+	 * its current cid's local DSQ, reusing its remaining slice. If the
+	 * slice is exhausted, fall through the normal enqueue path.
+	 */
+	if (enq_blocked &&
+	    (enq_flags & SCX_ENQ_BLOCKED) && p->scx.slice) {
+		scx_bpf_dsq_insert(p, SCX_DSQ_LOCAL_ON | scx_bpf_task_cid(p),
+				   0, enq_flags);
+		return;
+	}
+
 	/*
 	 * IMMED stress testing: Every immed_stress_nth'th enqueue, dispatch
 	 * directly to prev_cpu's local DSQ even when busy to force dsq->nr > 1
tools/sched_ext/scx_qmap.c +7 -2
index 67ddd483a4c75..236a9c9bb13f6 100644
@@ -23,7 +23,7 @@ const char help_fmt[] =
 "See the top-level comment in .bpf.c for more details.\n"
 "\n"
 "Usage: %s [-s SLICE_US] [-e COUNT] [-t COUNT] [-T COUNT] [-l COUNT] [-b COUNT]\n"
-"       [-N COUNT] [-P] [-M] [-H] [-c CG_PATH] [-d PID] [-D LEN] [-S] [-p] [-I]\n"
+"       [-N COUNT] [-P] [-M] [-H] [-c CG_PATH] [-d PID] [-D LEN] [-S] [-p] [-I] [-B]\n"
 "       [-F COUNT] [-v]\n"
 "\n"
 "  -s SLICE_US   Override slice duration\n"
@@ -42,6 +42,7 @@ const char help_fmt[] =
 "  -S            Suppress qmap-specific debug dump\n"
 "  -p            Switch only tasks on SCHED_EXT policy instead of all\n"
 "  -I            Turn on SCX_OPS_ALWAYS_ENQ_IMMED\n"
+"  -B            Turn on SCX_OPS_ENQ_BLOCKED\n"
 "  -F COUNT      IMMED stress: force every COUNT'th enqueue to a busy local DSQ (use with -I)\n"
 "  -C MODE       cid-override test (shuffle|bad-dup|bad-range)\n"
 "  -v            Print libbpf debug messages\n"
@@ -89,7 +90,7 @@ int main(int argc, char **argv)
 	skel->rodata->slice_ns = __COMPAT_ENUM_OR_ZERO("scx_public_consts", "SCX_SLICE_DFL");
 	skel->rodata->max_tasks = 16384;
 
-	while ((opt = getopt(argc, argv, "s:e:t:T:l:b:N:PMHc:d:D:SpIF:C:vh")) != -1) {
+	while ((opt = getopt(argc, argv, "s:e:t:T:l:b:N:PMHc:d:D:SpIBF:C:vh")) != -1) {
 		switch (opt) {
 		case 's':
 			skel->rodata->slice_ns = strtoull(optarg, NULL, 0) * 1000;
@@ -149,6 +150,10 @@ int main(int argc, char **argv)
 			skel->rodata->always_enq_immed = true;
 			skel->struct_ops.qmap_ops->flags |= SCX_OPS_ALWAYS_ENQ_IMMED;
 			break;
+		case 'B':
+			skel->rodata->enq_blocked = true;
+			skel->struct_ops.qmap_ops->flags |= SCX_OPS_ENQ_BLOCKED;
+			break;
 		case 'F':
 			skel->rodata->immed_stress_nth = strtoul(optarg, NULL, 0);
 			break;
Signature · 1 line
2.55.0
author reviewer
#15 lore raw

PATCH 9/9 sched: Allow enabling proxy exec with sched_ext

In reply to Andrea Righi

Now that sched_ext can handle proxy donors and BPF schedulers can opt in to blocked-donor enqueueing with SCX_OPS_ENQ_BLOCKED, remove the !SCHED_CLASS_EXT dependency from SCHED_PROXY_EXEC and allow both options to be enabled together.

---
 init/Kconfig | 2 --
 1 file changed, 2 deletions(-)
init/Kconfig +0 -2
index 5230d4879b1c8..0817e62266e03 100644
@@ -934,8 +934,6 @@ config SCHED_PROXY_EXEC
 	bool "Proxy Execution"
 	# Avoid some build failures w/ PREEMPT_RT until it can be fixed
 	depends on !PREEMPT_RT
-	# Need to investigate how to inform sched_ext of split contexts
-	depends on !SCHED_CLASS_EXT
 	# Not particularly useful until we get to multi-rq proxying
 	depends on EXPERT
 	help
Signature · 1 line
2.55.0