Skip to contentSkip to conversation

sched: Make proxy execution compatible with sched_ext

PATCH Superseded Andrea Righi 7 participants 61 messages Aug 31 – Sep 17

Subsystems (4)

v13 of 14 Superseded Latest is v14 →

lore
Share & tools
lore

Andrea Righi

author reviewer
#0 lore raw

PATCH v13 1/1 sched: Make proxy execution compatible with sched_ext

This series enables proxy execution with sched_ext and is based on early work by John Stultz [1].

Background
==========

Proxy execution (proxy-exec) lets a waiting task ("donor") donate its scheduling context to a mutex owner, so the owner can run while the donor stays eligible on the runqueue.

Currently, proxy-exec and sched_ext are mutually exclusive at build time: we can't enable CONFIG_SCHED_PROXY_EXEC=y and CONFIG_SCHED_CLASS_EXT=y in the same kernel.

This restriction can be problematic for Linux distributions and for anyone who wants to ship one kernel and choose features at runtime.

Why are they mutually exclusive?
================================

sched_ext schedulers drive dispatch through their own interfaces. A proxy-exec handoff can run a task that the BPF scheduler never dispatched through that path. sched_ext callbacks then observe a "current" task that does not match what the BPF side considers running, so kfuncs and helper state can see an inconsistent view of the executing task.

sched_ext also tracks runnable work through Dispatch Queues (DSQs) and BPF chosen dispatch rules, while the core scheduler still maintains classic per-CPU runqueues and pick paths. A proxy handoff can therefore switch the CPU to a task that the BPF scheduler never inserted or ordered through its DSQ interface.

DSQ state, vtime, and "who is running" bookkeeping inside the BPF program can then disagree with what the core actually executes, so helpers and kfuncs that assume their dispatched task is current may observe stale or inconsistent state.

Design: supporting proxy-exec with sched_ext
============================================

Provide proxy-exec in sched_ext as optional per-scheduler capability: a BPF scheduler can set the ops flag SCX_OPS_ENQ_BLOCKED to keep mutex-blocked donors runnable and receive them through ops.enqueue() with SCX_ENQ_BLOCKED set.

This flag requires an ops.enqueue() callback and gives BPF control over whether, where, and in what order to dispatch each donor. The core then walks the mutex-owner chain and, when needed, migrates the donor's scheduling context to the owner's rq before executing the owner.

Without the SCX_OPS_ENQ_BLOCKED flag, mutex waiters block normally and do not participate in proxy-exec while owned by that scheduler.

Once BPF dispatches a donor, proxy execution may move its scheduling context to the mutex owner's rq. This move is distinct from BPF placement: wake_cpu continues to identify the donor's pre-proxy CPU, while task_cpu() identifies the proxy CPU. When the donor wakes, the normal wakeup path uses wake_cpu as prev_cpu when returning the task to the BPF scheduler, which can then choose its next placement normally.

The donor-to-owner handoff is modeled like a "function call" from the scheduler's perspective. The donor remains the running scheduling entity selected by BPF: its scheduling context, runtime and slice are consumed while the core temporarily invokes the mutex owner's code to make the critical section progress. It is not a scheduler-visible switch to the owner.

Accordingly, the donor remains the scheduling context presented to sched_ext, while the mutex owner is treated solely as the execution context selected internally by the core scheduler. Scheduling state is accounted against rq->donor where appropriate, while rq->curr identifies the execution context. The internal owner substitution does not generate synthetic sched_ext callbacks for a task that BPF did not dispatch.

Scheduler-facing current-task kfuncs follow the same rule:
scx_bpf_task_running(), scx_bpf_cpu_curr() and scx_bpf_cid_curr() report the donor selected by the scheduler rather than the mutex owner whose code happens to execute on its behalf.

The sched_ext callback bookkeeping is adjusted accordingly. A blocked donor enters a tracked ops.running()/ops.stopping() session only after proxy resolution succeeds, and ops.stopping() is only called for tasks that entered such a session. This state is independent of physical rq->curr execution, which may refer to the mutex owner instead of the donor.

NOHZ CFS bandwidth checks also follow rq->donor rather than rq->curr and use the number of queued FAIR tasks when deciding whether the tick can stop. This keeps bandwidth enforcement active for a constrained FAIR donor even when its proxy owner remains runnable in another scheduling class.

Scheduler ownership changes need special handling because a donor may already be blocked when a root scheduler is enabled, when a task moves between root and sub-schedulers, or when a class transition moves it into or out of EXT. Start every scheduling-class or BPF-scheduler ownership change from clean task state:
fully deactivate a retained donor before the transition and let the incoming scheduler apply its own admission policy the next time the task blocks. This conservative rule leaves retaining proxy state across compatible RT/DL PI transitions as a future improvement.

Remote DSQ transfers recheck proxy-sensitive state after locking the source runqueue. If a transfer races with proxy execution, the task is parked on the source runqueue's reject DSQ and returned to its owning BPF scheduler after proxy resolution settles. Ordinary migration-disabled tasks still reach the normal validation path, where an illegal BPF-directed migration is reported. If an affinity migration is pending, the task remains parked until the affinity machinery completes the operation. This preserves scheduler placement and sub-scheduler capability rules without migrating an active execution context.

scx_qmap has been modified with a -X option to enable queueing mutex-blocked tasks for proxy-exec. Blocked donors receive a fresh slice and are inserted at the head of an eligible local DSQ selected from the task's allowed cids and qmap's self cids, preferring the current cid. They request immediate preemption, and time-shared targets also carry SCX_ENQ_IMMED. This is an intentionally aggressive policy for making proxy-exec easy to observe.

A new kselftest (enq_blocked) is also introduced to validate proxy-exec with sched_ext. The test creates a priority inversion with a low-priority owner (nice +19), a high-priority donor (nice -20) and one nice 0 contender per available CPU. It exercises a same-CPU topology and a cross-CPU topology, where the donor and owner run on different CPUs. Each configuration runs with SCX_OPS_ENQ_BLOCKED first disabled and then enabled, counts blocked-donor enqueues by CPU and reports mutex hold/wait-time deltas. Access to the mutex is provided by a loadable kernel module built via TEST_GEN_MODS_DIR, with the test responsible for its lifecycle.

Example kselftest run:

===== START =====
TEST: enq_blocked
DESCRIPTION: Verify proxy donor admission under CPU-wide contention OUTPUT:

[topology=same-cpu SCX_OPS_ENQ_BLOCKED=disabled]
proxy_exec=enabled
donor_cpu=0
owner_cpu=0
nr_contenders=16
measured_trials=10
owner_nice=19
donor_nice=-20
contender_nice=0 mutex_hold_avg_ns=263587544 (263.587 ms, samples=10) mutex_wait_avg_ns=263597934 (263.597 ms, samples=10) nr_blocked_enqueues=0 nr_blocked_enqueues_donor_cpu=0 nr_blocked_enqueues_owner_cpu=0 nr_blocked_enqueues_other_cpu=0 nr_blocked_wakeups=0

[topology=same-cpu SCX_OPS_ENQ_BLOCKED=enabled]
proxy_exec=enabled
donor_cpu=0
owner_cpu=0
nr_contenders=16
measured_trials=10
owner_nice=19
donor_nice=-20
contender_nice=0 mutex_hold_avg_ns=251789178 (251.789 ms, samples=10) mutex_wait_avg_ns=209808599 (209.808 ms, samples=10) nr_blocked_enqueues=50 nr_blocked_enqueues_donor_cpu=50 nr_blocked_enqueues_owner_cpu=0 nr_blocked_enqueues_other_cpu=0 nr_blocked_wakeups=0

[topology=same-cpu delta: enabled - disabled]
mutex_hold_delta_ns=-11798366 (-4.48%)
mutex_wait_delta_ns=-53789335 (-20.41%)

[topology=cross-cpu SCX_OPS_ENQ_BLOCKED=disabled]
proxy_exec=enabled
donor_cpu=0
owner_cpu=1
nr_contenders=16
measured_trials=10
owner_nice=19
donor_nice=-20
contender_nice=0 mutex_hold_avg_ns=246794454 (246.794 ms, samples=10) mutex_wait_avg_ns=247298497 (247.298 ms, samples=10) nr_blocked_enqueues=0 nr_blocked_enqueues_donor_cpu=0 nr_blocked_enqueues_owner_cpu=0 nr_blocked_enqueues_other_cpu=0 nr_blocked_wakeups=0

[topology=cross-cpu SCX_OPS_ENQ_BLOCKED=enabled]
proxy_exec=enabled
donor_cpu=0
owner_cpu=1
nr_contenders=16
measured_trials=10
owner_nice=19
donor_nice=-20
contender_nice=0 mutex_hold_avg_ns=215486809 (215.486 ms, samples=10) mutex_wait_avg_ns=215498090 (215.498 ms, samples=10) nr_blocked_enqueues=80 nr_blocked_enqueues_donor_cpu=20 nr_blocked_enqueues_owner_cpu=60 nr_blocked_enqueues_other_cpu=0 nr_blocked_wakeups=0

[topology=cross-cpu delta: enabled - disabled]
mutex_hold_delta_ns=-31307645 (-12.69%)
mutex_wait_delta_ns=-31800407 (-12.86%)
ok 1 enq_blocked #
===== END =====

=============================

RESULTS:

PASSED:  1
SKIPPED: 0
FAILED:  0

References
==========

[1] https://lore.kernel.org/all/[email protected]

Git tree: git://git.kernel.org/pub/scm/linux/kernel/git/arighi/linux.git scx-proxy-exec

Changes in v13:
- Keep re-enqueue reason lifetime out of the generic DSQ unlink primitive; preserve it through ops.enqueue() and clear it before resolving the new placement (Tejun Heo)
- Immediately drain proxy-rejected tasks and request a later retry only when the task is still running or donating, using SCX_RQ_PROXY_RETRY to track the pending retry (Tejun Heo)
- Restructure proxy reject handling, rename the helpers to describe their roles and document why the locked DSQ move deliberately diagnoses ordinary migration-disabled tasks instead of treating them as proxy races (Tejun Heo)
- Introduce WF_ON_RQ to distinguish already-runnable wakeups from full activations, replacing SCX_TASK_ENQ_WAKEUP tracking, and gate the sched_ext reschedule path on proxy execution (Tejun Heo)
- Keep the conservative reset across scheduling-class transitions, retaining proxy state across compatible RT/DL PI transitions remains future work
- Dispatch scx_qmap blocked donors to an eligible self cid, preferring the current cid, carry SCX_ENQ_IMMED for time-shared targets and count reenqueued shared-DSQ dispatches in nr_enq_blocked
- Link to v12: https://lore.kernel.org/all/[email protected]/

Changes in v12:
- Preserve a fresh re-enqueue reason when ops.enqueue() synchronously rejects a task again and clear the consumed reason only after the task leaves the reject DSQ (sashiko)
- Immediately process tasks rejected after becoming migration-disabled during an rq-lock handoff, leaving only active proxy contexts parked (sashiko)
- Use the donor scheduling context for protected-slice admission, rescue preemption, slice save/restore and bypass handling, and charge rescue runtime to the rescued task's scheduler (sashiko)
- Refresh the NOHZ tick dependency after rq->donor changes in __schedule(), preventing a stale constrained FAIR donor from keeping the tick enabled for an incoming RT task (sashiko)
- Clarify that ops.running()/ops.stopping() track the donor's scheduling context including blocked donors after successful proxy resolution (sashiko)
- Restore the change that leaves mutex proxy-lock scopes before calling proxy_resched_idle() (required to avoid scheduling-class callbacks under the extra raw spinlocks)
- Dequeue waking proxy donors before replacing rq->donor references avoiding callbacks under blocked_lock and transient donor re-enqueueing
- Harden scx_qmap blocked-donor fast paths against missing eligible self cids
- Move SCX_TASK_ENQ_WAKEUP to bit 16, leaving room for task-state and re-enqueue-reason expansion
- Link to v11: https://lore.kernel.org/all/[email protected]/

Changes in v11:
- End retained proxy execution before every scheduling-class or BPF-scheduler ownership change, passing the incoming class to sched_change_begin() and removing the prepare_switch() class callback (Tejun Heo)
- Document the conservative class-transition rule and leave retaining proxy state across compatible RT/DL PI transitions for future work (Tejun Heo)
- Split reject-DSQ code movement from its generalization and carry the re-enqueue reason directly in p->scx.flags (Tejun Heo)
- Use one SCX_TASK_REENQ_PROXY reason for all proxy-raced remote transfers, clarify why proxy migration differs from BPF-directed migration and keep tasks parked while affinity migration is pending (Tejun Heo)
- Factor local-DSQ diversion cleanup into a shared helper, clear HEAD, IMMED, PREEMPT and carried-slice state for proxy rejection (Tejun Heo)
- Explain the scx_qmap SCX_OPS_ALWAYS_ENQ_IMMED re-enqueue fallback (Tejun Heo)
- Rename the blocked-donor command-line option to -X, since -B is now taken for the rescue bandwidth setting
- Drop the SCHED_FLAG_KEEP_PARAMS preparatory fix (this is addressed separately by https://lore.kernel.org/all/[email protected])
- Link to v10: https://lore.kernel.org/all/[email protected]/

Changes in v10:
- Check constrained FAIR proxy donors before the RT fast paths in sched_can_stop_tick(), so a throttled RT owner cannot bypass bandwidth enforcement (sashiko)
- Add a preparatory fix that sets DEQUEUE_CLASS only when SCHED_FLAG_KEEP_PARAMS permits the scheduling class to change, preventing class-transition callbacks from running without an actual transition (sashiko)
- Update try_to_block_task() comment to describe the sched_ext blocked-donor admission check (sashiko)
- Link to v9: https://lore.kernel.org/all/[email protected]/

Changes in v9:
- Add an incoming-class prepare_switch() callback and use it to block retained donors before transitions into sched_ext (Tejun Heo)
- Generalize the per-rq reject DSQ and use it for proxy-raced remote transfers instead of bouncing tasks to a global DSQ (Tejun Heo)
- Restrict migration-disabled warning exception to blocked donors whose scheduling context was actually proxy-migrated (sashiko)
- Keep ops.tick() and other donor accounting inside a tracked ops.running()/ops.stopping() session (sashiko)
- Prevent BPF-directed migration of an active proxy donor and use the donor as the current scheduling context during restore (sashiko)
- Update sched_fair_update_stop_tick() to use the number of queued FAIR tasks (sashiko)
- Fold the temporary local-DSQ donor placement into the final BPF-controlled admission change
- Link to v8: https://lore.kernel.org/all/[email protected]/

Changes in v8:
- Rework remote-DSQ migration check so the locked re-check handles only proxy-exec state changes (Tejun Heo)
- Use the current-donor helper when preventing migration of an active donor (sashiko)
- Handle blocked-donor reactivation into scx schedulers without SCX_OPS_ENQ_BLOCKED (John Stultz)
- Document sched_ext callback behavior under proxy execution (sashiko)
- Avoid redispatching a rejected blocked donor to the same local DSQ in scx_qmap (sashiko)
- Add a preparatory fix to avoid false migration warnings when proxy-exec moves a migration-disabled donor
- Skip the enq_blocked kselftest when proxy execution is unavailable
- Link to v7: https://lore.kernel.org/all/[email protected]/

Changes in v7:
- Move the remote CPU check out of the lockless remote-DSQ scan and perform it only after locking the source rq, retain the locked migration recheck immediately before moving the task (sashiko)
- Complete the scheduler-context conversion from rq->curr to rq->donor for capability revocation and preemption checks, make scx_bpf_task_running(), scx_bpf_cpu_curr() and scx_bpf_cid_curr() expose the donor selected by the scheduler (sashiko)
- Handle SCX_ENQ_REENQ before scx_qmap's blocked-donor fast path to avoid an infinite reject/re-enqueue loop (sashiko)
- Add scx_qmap stat for SCX_ENQ_BLOCKED dispatches
- Link to v6: https://lore.kernel.org/all/[email protected]/

Changes in v6:
- Add a scheduler-core fix to make NOHZ CFS bandwidth checks follow the proxy donor instead of the physical execution context (sashiko)
- Block retained donors when entering sched_ext through PI de-boosting or global scheduler activation (sashiko)
- Distinguish ordinary wakeup activations from retained-donor admissions so SCX_ENQ_BLOCKED is not reported for normal wakeups; extend the selftest to detect this regression (sashiko)
- Validate unexpected-CPU blocked-donor enqueues in both same-CPU and cross-CPU test topologies (sashiko)
- Link to v5: https://lore.kernel.org/all/[email protected]/

Changes in v5:
- Split retained-donor deactivation and sched_ext's default rejection into preparatory patches so the scheduler-core changes can be routed separately
- Drop the proxy destination query kfuncs and the preparatory mutex lock-scope change (John Stultz)
- Use p->is_blocked instead of task_is_blocked() to fix a WARN triggered during scx_pair testing (John Stultz)
- Rename SCX_TASK_IS_RUNNING to SCX_TASK_RUN_TRACKED and document that it tracks an ops.running()/stopping() session rather than physical rq->curr execution
- Keep a proxy-migrated blocked donor on the owner's rq until wakeup instead of allowing BPF-directed migration to pull it back to wake_cpu and cause repeated donor migration
- Extend kselftest to cover same-CPU and cross-CPU proxy-exec switches
- Make scx_qmap insert blocked donors at the head of their current cid's local DSQ and request immediate preemption
- Link to v4: https://lore.kernel.org/all/[email protected]/

Changes in v4:
- Harden scx_bpf_task_proxy_cpu() locking and blocked-state validation, return -ENOENT for unrunnable owners and drop unnecessary donor-affinity checks (K Prateek Nayak, sashiko)
- Reschedule remote CPUs after donor deactivation, handle sched_setscheduler() admission and assert scheduler-change locking (sashiko)
- Avoid a potential KCSAN data-race report in the lockless blocked-donor migration check by using rcu_access_pointer() for rq->donor (sashiko)
- Fix tick dependency updates for incoming EXT contexts and keep the tick enabled for blocked donors (sashiko)
- Dump EXT donors in scx_dump_state() (sashiko)
- Reordered the preparatory sched_ext changes so real running-state tracking is established before donor-based accounting, and split the proxy destination query kfuncs into a separate patch
- Add compatibility wrappers for the proxy CPU/cid kfuncs and document their results as scheduling hints
- Link to v3: https://lore.kernel.org/all/[email protected]/

Changes in v3:
- Dropped the core restrictions on proxy-migrating migration-disabled and single-CPU donors: proxy execution moves the scheduling context, not the task's execution context. (Peter Zijlstra, K Prateek Nayak)
- Dropped the sched_ext-specific put_prev_task()/set_next_task() exception and fixed ops.running()/ops.stopping() pairing inside sched_ext instead; track a real running transition even when either callback is absent. (Peter Zijlstra)
- Dropped the kf_tasks[] nesting and nested ops.runnable() patches from v2; extensive proxy-exec testing did not reproduce task-op re-entry, so retain the existing non-nesting invariant.
- Replaced scx_bpf_task_is_blocked() with SCX_ENQ_BLOCKED in ops.enqueue() flags, identifying blocked-donor admission directly at enqueue time.
- Expanded the curr/donor description to cover mixed scheduling classes and fixed local preemption to expire rq->donor's slice. (Aiqun Maria Yu)
- Allow inactive blocked donors to be placed on a remote local DSQ while preventing normal migration of an active rq donor. (Aiqun Maria Yu)
- Deactivate retained donors when ownership changes to a root or sub-scheduler without SCX_OPS_ENQ_BLOCKED, and extend the selftest to cover attaching a scheduler after the donor blocks. (K Prateek Nayak)
- Harden remote DSQ consumption by rejecting active tasks and rechecking migration eligibility after locking the source rq. Fall back to the global DSQ without treating an eligibility change during the lock handoff as a BPF scheduler error.
- scx_qmap has a command line option (-B) to enable blocked-donor queueing
- Link to v2: https://lore.kernel.org/all/[email protected]/

Changes in v2:
- Rebased onto sched_ext/for-7.3 and adapted the series to the split sched_ext implementation and cid-form scheduler interfaces.
- Replaced the global sched_proxy_exec_scx boot-time opt-in with the per-scheduler SCX_OPS_ENQ_BLOCKED capability, allowing BPF to control donor admission and ordering through ops.enqueue().
- Added scx_bpf_task_is_blocked(), scx_bpf_task_proxy_cpu(), and scx_bpf_task_proxy_cid(); enforce CPU/cid API separation for cid-form schedulers.
- Added proxy exec support to scx_qmap, including optional owner-cid steering, affinity validation, and fallback to the donor's current cid.
- Added a kselftest with a kernel mutex test and a three-task priority inversion workload, test is executed with blocked task admission disabled and enabled, validates the behavior, and reports hold/wait-time deltas.
- Link to v1: https://lore.kernel.org/all/[email protected]/

Andrea Righi (18):
sched/core: Drop mutex locks before proxy rescheduling sched/core: Dequeue waking proxy donors before reset
sched: Make NOHZ CFS bandwidth checks follow proxy donor sched/core: Avoid false migration warning for proxy donors
sched: Pass next class to sched_change_begin()
sched: Add helper to block retained proxy donors
sched: Add sched_ext hooks for proxy execution
sched: Introduce WF_ON_RQ wake flag
sched_ext: Block proxy donors across scheduler transitions
sched_ext: Fix ops.running/stopping() pairing for proxy-exec donors
sched_ext: Move reject DSQ draining into core
sched_ext: Generalize the reject DSQ reenqueue path
sched_ext: Handle proxy-exec races in remote DSQ transfers
sched_ext: Split curr|donor references properly
sched_ext: Delegate proxy donor admission to BPF schedulers
sched_ext: Add selftest for blocked donor admission
sched_ext: scx_qmap: Add proxy execution support
sched: Allow enabling proxy exec with sched_ext

 Documentation/scheduler/sched-ext.rst              |  13 +
 include/linux/sched/ext.h                          |   3 +
 init/Kconfig                                       |   2 -
 kernel/sched/core.c                                | 147 +++-
 kernel/sched/ext/ext.c                             | 527 +++++++++---
 kernel/sched/ext/ext.h                             |   6 +
 kernel/sched/ext/internal.h                        |  37 +-
 kernel/sched/ext/sub.c                             |  68 +-
 kernel/sched/ext/sub.h                             |  13 +-
 kernel/sched/fair.c                                |  12 +-
 kernel/sched/sched.h                               |  19 +-
 kernel/sched/syscalls.c                            |   4 +-
 tools/sched_ext/include/scx/compat.h               |   1 +
 tools/sched_ext/include/scx/enum_defs.autogen.h    |   3 +
 tools/sched_ext/include/scx/enums.autogen.bpf.h    |   3 +
 tools/sched_ext/include/scx/enums.autogen.h        |   1 +
 tools/sched_ext/scx_qmap.bpf.c                     |  53 +-
 tools/sched_ext/scx_qmap.c                         |  13 +-
 tools/sched_ext/scx_qmap.h                         |   1 +
 tools/testing/selftests/sched_ext/.gitignore       |   4 +
 tools/testing/selftests/sched_ext/Makefile         |   2 +
 tools/testing/selftests/sched_ext/config           |   2 +
 .../testing/selftests/sched_ext/enq_blocked.bpf.c  | 116 +++
 tools/testing/selftests/sched_ext/enq_blocked.c    | 917 +++++++++++++++++++++
 tools/testing/selftests/sched_ext/enq_blocked.h    |  28 +
 .../selftests/sched_ext/test_modules/Makefile      |  13 +
 .../sched_ext/test_modules/scx_enq_blocked_test.c  | 195 +++++
 27 files changed, 1985 insertions(+), 218 deletions(-)
 create mode 100644 tools/testing/selftests/sched_ext/enq_blocked.bpf.c
 create mode 100644 tools/testing/selftests/sched_ext/enq_blocked.c
 create mode 100644 tools/testing/selftests/sched_ext/enq_blocked.h
 create mode 100644 tools/testing/selftests/sched_ext/test_modules/Makefile
 create mode 100644 tools/testing/selftests/sched_ext/test_modules/scx_enq_blocked_test.c
author reviewer
#1 lore raw

PATCH 1/18 sched/core: Drop mutex locks before proxy rescheduling

find_proxy_task() can call proxy_resched_idle() from return expressions inside mutex wait_lock and blocked task lock guard scopes. Return expressions are evaluated before scoped guards are released, so proxy_resched_idle() invokes scheduling-class callbacks while both raw spinlocks are held.

Route these cases through a common label outside the guard scopes. The common label is required because a direct return would evaluate proxy_resched_idle() before guard cleanup and run the class callbacks under the proxy-exec locks again.

This is a preparatory change to support proxy execution with sched_ext.

---
 kernel/sched/core.c | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)
kernel/sched/core.c +6 -4
index f78275192036b..5817d1a4cea2c 100644
@@ -6965,7 +6965,7 @@ find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf)
 		if (!READ_ONCE(owner->on_rq) || owner->se.sched_delayed) {
 			/* XXX Don't handle blocked owners/delayed dequeue yet */
 			if (curr_in_chain)
-				return proxy_resched_idle(rq);
+				goto resched_idle;
 			__clear_task_blocked_on(p, NULL);
 			goto deactivate;
 		}
@@ -6977,7 +6977,7 @@ find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf)
 			 * and leave that CPU to sort things out.
 			 */
 			if (curr_in_chain)
-				return proxy_resched_idle(rq);
+				goto resched_idle;
 			goto migrate_task;
 		}
 
@@ -6990,7 +6990,7 @@ find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf)
 			 * case we should end up back in find_proxy_task(), this time
 			 * hopefully with all relevant tasks already enqueued.
 			 */
-			return proxy_resched_idle(rq);
+			goto resched_idle;
 		}
 
 		/*
@@ -7027,7 +7027,7 @@ find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf)
 			 * So schedule rq->idle so that ttwu_runnable() can get the rq
 			 * lock and mark owner as running.
 			 */
-			return proxy_resched_idle(rq);
+			goto resched_idle;
 		}
 		/*
 		 * OK, now we're absolutely sure @owner is on this
@@ -7039,6 +7039,8 @@ find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf)
 	WARN_ON_ONCE(owner && !owner->on_rq);
 	return owner;
 
+resched_idle:
+	return proxy_resched_idle(rq);
 deactivate:
 	proxy_deactivate(rq, p);
 	return NULL;
Signature · 1 line
2.55.0
author reviewer
#2 lore raw

PATCH 2/18 sched/core: Dequeue waking proxy donors before reset

In reply to Andrea Righi

proxy_needs_return() resets an active donor while holding blocked_lock. proxy_reset_donor() invokes scheduling-class callbacks, adding an unnecessary raw-spinlock nesting. It also presents the waking donor to put_prev_task() as still runnable immediately before block_task() removes it from the runqueue.

Split block_task() so the waking donor can first be dequeued from its scheduling class. Release blocked_lock, dequeue the donor while its generic on_rq state still prevents migration, replace all donor references, and only then complete the generic runqueue removal. This follows the normal sleep ordering and avoids transiently re-enqueuing the waking donor.

This is a preparatory change to support proxy execution with sched_ext.

---
 kernel/sched/core.c | 30 +++++++++++++++++++++++-------
 1 file changed, 23 insertions(+), 7 deletions(-)
kernel/sched/core.c +23 -7
index 5817d1a4cea2c..237d216382f46 100644
@@ -2252,7 +2252,8 @@ void deactivate_task(struct rq *rq, struct task_struct *p, int flags)
 	dequeue_task(rq, p, flags);
 }
 
-static void block_task(struct rq *rq, struct task_struct *p, unsigned long task_state)
+static bool dequeue_block_task(struct rq *rq, struct task_struct *p,
+			       unsigned long task_state)
 {
 	int flags = DEQUEUE_NOCLOCK;
 
@@ -2273,9 +2274,15 @@ static void block_task(struct rq *rq, struct task_struct *p, unsigned long task_
 	 *
 	 * Where __schedule() and ttwu() have matching control dependencies.
 	 *
-	 * After this, schedule() must not care about p->state any more.
+	 * Once the caller invokes __block_task(), schedule() must not care about
+	 * p->state any more.
 	 */
-	if (dequeue_task(rq, p, DEQUEUE_SLEEP | flags))
+	return dequeue_task(rq, p, DEQUEUE_SLEEP | flags);
+}
+
+static void block_task(struct rq *rq, struct task_struct *p, unsigned long task_state)
+{
+	if (dequeue_block_task(rq, p, task_state))
 		__block_task(rq, p);
 }
 
@@ -3774,6 +3781,9 @@ static inline void proxy_reset_donor(struct rq *rq)
  */
 static inline bool proxy_needs_return(struct rq *rq, struct task_struct *p)
 {
+	bool reset_donor = false;
+	bool dequeued;
+
 	/*
 	 * Typically per __set_task_cpu(), task_cpu(p) == p->wake_cpu.
 	 *
@@ -3797,11 +3807,17 @@ static inline bool proxy_needs_return(struct rq *rq, struct task_struct *p)
 		if (task_current(rq, p))
 			return false;
 
-		/* If we're return migrating the rq->donor, switch it out for idle */
-		if (task_current_donor(rq, p))
-			proxy_reset_donor(rq);
+		reset_donor = task_current_donor(rq, p);
 	}
-	block_task(rq, p, TASK_WAKING);
+
+	dequeued = dequeue_block_task(rq, p, TASK_WAKING);
+
+	/* Keep on_rq set until all donor references have been replaced. */
+	if (reset_donor)
+		proxy_reset_donor(rq);
+
+	if (dequeued)
+		__block_task(rq, p);
 	return true;
 }
 #else /* !CONFIG_SCHED_PROXY_EXEC */
Signature · 1 line
2.55.0
#3 lore raw

Re: [PATCH 02/18] sched/core: Dequeue waking proxy donors before reset

Hello Andrea,

Quoted 5 lines · Andrea Righi

proxy_needs_return() resets an active donor while holding blocked_lock. proxy_reset_donor() invokes scheduling-class callbacks, adding an unnecessary raw-spinlock nesting. It also presents the waking donor to put_prev_task() as still runnable immediately before block_task() removes it from the runqueue.

Is that an issue for scx?

Quoted 57 lines

Split block_task() so the waking donor can first be dequeued from its scheduling class. Release blocked_lock, dequeue the donor while its generic on_rq state still prevents migration, replace all donor references, and only then complete the generic runqueue removal. This follows the normal sleep ordering and avoids transiently re-enqueuing the waking donor.

This is a preparatory change to support proxy execution with sched_ext.

---
 kernel/sched/core.c | 30 +++++++++++++++++++++++-------
 1 file changed, 23 insertions(+), 7 deletions(-)
kernel/sched/core.c +13 -3
index 5817d1a4cea2c..237d216382f46 100644
@@ -2252,7 +2252,8 @@ void deactivate_task(struct rq *rq, struct task_struct *p, int flags)
 	dequeue_task(rq, p, flags);
 }
 
-static void block_task(struct rq *rq, struct task_struct *p, unsigned long task_state)
+static bool dequeue_block_task(struct rq *rq, struct task_struct *p,
+			       unsigned long task_state)
 {
 	int flags = DEQUEUE_NOCLOCK;
 
@@ -2273,9 +2274,15 @@ static void block_task(struct rq *rq, struct task_struct *p, unsigned long task_
 	 *
 	 * Where __schedule() and ttwu() have matching control dependencies.
 	 *
-	 * After this, schedule() must not care about p->state any more.
+	 * Once the caller invokes __block_task(), schedule() must not care about
+	 * p->state any more.
 	 */
-	if (dequeue_task(rq, p, DEQUEUE_SLEEP | flags))
+	return dequeue_task(rq, p, DEQUEUE_SLEEP | flags);
+}
+
+static void block_task(struct rq *rq, struct task_struct *p, unsigned long task_state)
+{
+	if (dequeue_block_task(rq, p, task_state))
 		__block_task(rq, p);
 }
 
@@ -3774,6 +3781,9 @@ static inline void proxy_reset_donor(struct rq *rq)
  */
 static inline bool proxy_needs_return(struct rq *rq, struct task_struct *p)
 {
+	bool reset_donor = false;
+	bool dequeued;
+
 	/*
 	 * Typically per __set_task_cpu(), task_cpu(p) == p->wake_cpu.
 	 *

kernel/sched/core.c static inline bool proxy_needs_return(struct rq *rq, struct task_struct *p) · line 3810

Quoted 7 lines
kernel/sched/core.c +1 -3
@@ -3797,11 +3807,17 @@ static inline bool proxy_needs_return(struct rq *rq, struct task_struct *p)
 		if (task_current(rq, p))
 			return false;
 
-		/* If we're return migrating the rq->donor, switch it out for idle */
-		if (task_current_donor(rq, p))
-			proxy_reset_donor(rq);
+		reset_donor = task_current_donor(rq, p);

nit.

Since proxy_needs_return() holds the rq_lock, you can check this outside the blocked_lock safely, even after the dequeue. There is no need to stash "reset_donor".

kernel/sched/core.c static inline bool proxy_needs_return(struct rq *rq, struct task_struct *p) · line 3817

Quoted 8 lines
kernel/sched/core.c +6 -1
 	}
-	block_task(rq, p, TASK_WAKING);
+
+	dequeued = dequeue_block_task(rq, p, TASK_WAKING);
+
+	/* Keep on_rq set until all donor references have been replaced. */
+	if (reset_donor)
+		proxy_reset_donor(rq);

Since TASK_WAKING is guaranteed to block the task by adding DEQUEUE_SPECIAL, you can just move the proxy_reset_donor() bit out the blocked lock and keep everything else the same right?

I'm not sure I understand "transiently re-enqueuing the waking donor" bit. How is that possible if we just do:

    /* __task_rq_lock is held throughout. */

    if (task_current_donor(rq, p))
        proxy_reset_donor(rq, p);

    block_task(rq, p, TASK_WAKING);

Is the split because dequeue_task_scx() needs a correct rq->donor reference or does ext requires dequeue_task_scx() to be called before doing put_prev_task_scx() always?

Quoted 6 lines
kernel/sched/core.c +3 -0
+
+	if (dequeued)
+		__block_task(rq, p);
 	return true;
 }
 #else /* !CONFIG_SCHED_PROXY_EXEC */
Signature · 2 lines
Thanks and Regards,
Prateek
author reviewer
#4 lore raw

Re: [PATCH 02/18] sched/core: Dequeue waking proxy donors before reset

Hi Prateek,

Quoted 10 lines · K Prateek Nayak

Hello Andrea,

Quoted 5 lines · Andrea Righi

proxy_needs_return() resets an active donor while holding blocked_lock. proxy_reset_donor() invokes scheduling-class callbacks, adding an unnecessary raw-spinlock nesting. It also presents the waking donor to put_prev_task() as still runnable immediately before block_task() removes it from the runqueue.

Is that an issue for scx?

Yes. When a proxy-migrated donor wakes while it's still rq->donor, proxy_needs_return() has to reset the rq's donor before clearing the task's generic on_rq state and returning it through the full wakeup path.

proxy_reset_donor() calls put_prev_set_next_task(), which invokes put_prev_task_scx() for an EXT donor. If this happens before the scheduling-class dequeue, SCX_TASK_QUEUED and is_blocked are still set, so put_prev_task_scx() re-enqueues the donor through scx_do_enqueue_task() and the following block_task() immediately dequeues it again.

I can clarify this sched_ext-specific ordering better in the patch description.

Quoted 68 lines
Quoted 58 lines

Split block_task() so the waking donor can first be dequeued from its scheduling class. Release blocked_lock, dequeue the donor while its generic on_rq state still prevents migration, replace all donor references, and only then complete the generic runqueue removal. This follows the normal sleep ordering and avoids transiently re-enqueuing the waking donor.

This is a preparatory change to support proxy execution with sched_ext.

---
 kernel/sched/core.c | 30 +++++++++++++++++++++++-------
 1 file changed, 23 insertions(+), 7 deletions(-)
kernel/sched/core.c +13 -3
index 5817d1a4cea2c..237d216382f46 100644
@@ -2252,7 +2252,8 @@ void deactivate_task(struct rq *rq, struct task_struct *p, int flags)
 	dequeue_task(rq, p, flags);
 }
 
-static void block_task(struct rq *rq, struct task_struct *p, unsigned long task_state)
+static bool dequeue_block_task(struct rq *rq, struct task_struct *p,
+			       unsigned long task_state)
 {
 	int flags = DEQUEUE_NOCLOCK;
 
@@ -2273,9 +2274,15 @@ static void block_task(struct rq *rq, struct task_struct *p, unsigned long task_
 	 *
 	 * Where __schedule() and ttwu() have matching control dependencies.
 	 *
-	 * After this, schedule() must not care about p->state any more.
+	 * Once the caller invokes __block_task(), schedule() must not care about
+	 * p->state any more.
 	 */
-	if (dequeue_task(rq, p, DEQUEUE_SLEEP | flags))
+	return dequeue_task(rq, p, DEQUEUE_SLEEP | flags);
+}
+
+static void block_task(struct rq *rq, struct task_struct *p, unsigned long task_state)
+{
+	if (dequeue_block_task(rq, p, task_state))
 		__block_task(rq, p);
 }
 
@@ -3774,6 +3781,9 @@ static inline void proxy_reset_donor(struct rq *rq)
  */
 static inline bool proxy_needs_return(struct rq *rq, struct task_struct *p)
 {
+	bool reset_donor = false;
+	bool dequeued;
+
 	/*
 	 * Typically per __set_task_cpu(), task_cpu(p) == p->wake_cpu.
 	 *

kernel/sched/core.c static inline bool proxy_needs_return(struct rq *rq, struct task_struct *p) · line 3810

Quoted 7 lines
kernel/sched/core.c +1 -3
@@ -3797,11 +3807,17 @@ static inline bool proxy_needs_return(struct rq *rq, struct task_struct *p)
 		if (task_current(rq, p))
 			return false;
 
-		/* If we're return migrating the rq->donor, switch it out for idle */
-		if (task_current_donor(rq, p))
-			proxy_reset_donor(rq);
+		reset_donor = task_current_donor(rq, p);

nit.

Quoted 3 lines

Since proxy_needs_return() holds the rq_lock, you can check this outside the blocked_lock safely, even after the dequeue. There is no need to stash "reset_donor".

Agreed, rq->donor is stable while the rq lock is held, so this can use task_current_donor(rq, p) directly after dequeue_block_task() and remove reset_donor.

Quoted 13 lines

kernel/sched/core.c static inline bool proxy_needs_return(struct rq *rq, struct task_struct *p) · line 3817

Quoted 9 lines
kernel/sched/core.c +6 -1
 	}
-	block_task(rq, p, TASK_WAKING);
+
+	dequeued = dequeue_block_task(rq, p, TASK_WAKING);
+
+	/* Keep on_rq set until all donor references have been replaced. */
+	if (reset_donor)
+		proxy_reset_donor(rq);

Since TASK_WAKING is guaranteed to block the task by adding DEQUEUE_SPECIAL, you can just move the proxy_reset_donor() bit out the blocked lock and keep everything else the same right?

We still need to perform the sched-class dequeue before proxy_reset_donor().

If proxy_reset_donor() runs first, put_prev_task_scx() sees the donor with SCX_TASK_QUEUED and is_blocked still set and reenqueues it through scx_do_enqueue_task(). Then the following block_task() dequeues it again.

Quoted 10 lines

I'm not sure I understand "transiently re-enqueuing the waking donor" bit. How is that possible if we just do:

    /* __task_rq_lock is held throughout. */

    if (task_current_donor(rq, p))
        proxy_reset_donor(rq, p);

    block_task(rq, p, TASK_WAKING);

And the transient reenqueue happens inside proxy_reset_donor():

proxy_reset_donor()

    put_prev_set_next_task()
      put_prev_task_scx()
        scx_do_enqueue_task()
          ops.enqueue()

__clear_task_blocked_on() clears the blocked_on relationship, but p->is_blocked remains set until ttwu_do_wakeup(). As SCX_TASK_QUEUED is also still set, put_prev_task_scx() takes its retained-blocked-donor path.

The subsequent block_task() would then immediately undo that enqueue.

Quoted 3 lines

Is the split because dequeue_task_scx() needs a correct rq->donor reference or does ext requires dequeue_task_scx() to be called before doing put_prev_task_scx() always?

Both are aspects of the same ordering requirement here.

dequeue_task_scx() must run with rq->donor == p so it recognizes p as the current scheduling context, emits the appropriate stopping transition and clears SCX_TASK_QUEUED. Then put_prev_task_scx() can run as part of proxy_reset_donor() without placing the blocked donor back into a DSQ or calling ops.enqueue() again.

And this is specific to resetting a retained donor, it's not a general requirement that dequeue_task_scx() always precede put_prev_task_scx().

I'll remove the unnecessary reset_donor variable and expand the comment to make this ordering more explicit.

Thanks!
-Andrea

Quoted 12 lines
Quoted 7 lines
kernel/sched/core.c +3 -0
+
+	if (dequeued)
+		__block_task(rq, p);
 	return true;
 }
 #else /* !CONFIG_SCHED_PROXY_EXEC */
Signature · 2 lines
Thanks and Regards,
Prateek
author reviewer
#5 lore raw

PATCH 3/18 sched: Make NOHZ CFS bandwidth checks follow proxy donor

In reply to Andrea Righi

Proxy execution separates the scheduling context in rq->donor from the physical execution context in rq->curr. sched_can_stop_tick() checks the latter for CFS bandwidth constraints and only does so when nr_running is one.

A retained proxy donor keeps both the donor and mutex owner queued. The check therefore misses a constrained FAIR donor and may stop the tick while its runtime still needs to be enforced.

Check the selected donor instead and remove the nr_running restriction. The donor being a queued FAIR task is sufficient to require bandwidth accounting regardless of other runnable tasks.

Fixes af0c8b2bf67b ("sched: Split scheduler and execution contexts")
---
 kernel/sched/core.c | 35 ++++++++++++++++++-----------------
 kernel/sched/fair.c | 12 +++++++-----
 2 files changed, 25 insertions(+), 22 deletions(-)
kernel/sched/core.c +18 -17
index 237d216382f46..14d0d5c884393 100644
@@ -1419,11 +1419,8 @@ static void nohz_csd_func(void *info)
 #endif /* CONFIG_NO_HZ_COMMON */
 
 #ifdef CONFIG_NO_HZ_FULL
-static inline bool __need_bw_check(struct rq *rq, struct task_struct *p)
+static inline bool __need_bw_check(struct task_struct *p)
 {
-	if (rq->nr_running != 1)
-		return false;
-
 	if (p->sched_class != &fair_sched_class)
 		return false;
 
@@ -1441,6 +1438,14 @@ bool sched_can_stop_tick(struct rq *rq)
 	if (rq->dl.dl_nr_running)
 		return false;
 
+	/*
+	 * The selected scheduling context can be a constrained FAIR donor even
+	 * when rq->curr is an RT task. Check it before the RT fast paths below,
+	 * which may report that the tick can stop for a throttled RT context.
+	 */
+	if (__need_bw_check(rq->donor) && cfs_task_bw_constrained(rq->donor))
+		return false;
+
 	/*
 	 * If there are more than one RR tasks, we need the tick to affect the
 	 * actual RR behaviour.
@@ -1471,18 +1476,6 @@ bool sched_can_stop_tick(struct rq *rq)
 	if (rq->cfs.h_nr_queued > 1)
 		return false;
 
-	/*
-	 * If there is one task and it has CFS runtime bandwidth constraints
-	 * and it's on the cpu now we don't want to stop the tick.
-	 * This check prevents clearing the bit if a newly enqueued task here is
-	 * dequeued by migrating while the constrained task continues to run.
-	 * E.g. going from 2->1 without going through pick_next_task().
-	 */
-	if (__need_bw_check(rq, rq->curr)) {
-		if (cfs_task_bw_constrained(rq->curr))
-			return false;
-	}
-
 	return true;
 }
 #endif /* CONFIG_NO_HZ_FULL */
@@ -7114,7 +7107,7 @@ find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf)
  */
 static void __sched notrace __schedule(int sched_mode)
 {
-	struct task_struct *prev, *next;
+	struct task_struct *prev, *next, *tick_donor;
 	/*
 	 * On PREEMPT_RT kernel, SM_RTLOCK_WAIT is noted
 	 * as a preemption by schedule_debug() and RCU.
@@ -7168,6 +7161,7 @@ static void __sched notrace __schedule(int sched_mode)
 	rq->clock_update_flags <<= 1;
 	update_rq_clock(rq);
 	rq->clock_update_flags = RQCF_UPDATED;
+	tick_donor = rq->donor;
 
 	switch_count = &prev->nivcsw;
 
@@ -7243,6 +7237,13 @@ static void __sched notrace __schedule(int sched_mode)
 	clear_tsk_need_resched(prev);
 	clear_preempt_need_resched();
 keep_resched:
+	/*
+	 * Enqueue and dequeue updates can evaluate the outgoing donor. Refresh
+	 * the dependency after selecting a different scheduling context.
+	 */
+	if (rq->donor != tick_donor)
+		sched_update_tick_dependency(rq);
+
 	rq->last_seen_need_resched_ns = 0;
 
 	is_switch = prev != next;
kernel/sched/fair.c +7 -5
index 8dff37059faf7..1842a748641b7 100644
@@ -7703,14 +7703,16 @@ static void sched_fair_update_stop_tick(struct rq *rq, struct task_struct *p)
 	if (!tick_nohz_full_cpu(cpu))
 		return;
 
-	if (rq->nr_running != 1)
+	if (rq->cfs.h_nr_queued != 1)
 		return;
 
 	/*
-	 *  We know there is only one task runnable and we've just picked it. The
-	 *  normal enqueue path will have cleared TICK_DEP_BIT_SCHED if we will
-	 *  be otherwise able to stop the tick. Just need to check if we are using
-	 *  bandwidth control.
+	 * We know there is only one FAIR task queued and we've just picked it.
+	 * The normal enqueue path will have cleared TICK_DEP_BIT_SCHED if we
+	 * will be otherwise able to stop the tick. With proxy execution, its
+	 * lock owner may remain runnable in another scheduling class, so
+	 * rq->nr_running can be greater than one. Just need to check if we are
+	 * using bandwidth control.
 	 */
 	if (cfs_task_bw_constrained(p))
 		tick_nohz_dep_set_cpu(cpu, TICK_DEP_BIT_SCHED);
Signature · 1 line
2.55.0
maintainer
#6 lore raw

Re: [PATCH 03/18] sched: Make NOHZ CFS bandwidth checks follow proxy donor

Quoted 41 lines · Andrea Righi

Proxy execution separates the scheduling context in rq->donor from the physical execution context in rq->curr. sched_can_stop_tick() checks the latter for CFS bandwidth constraints and only does so when nr_running is one.

A retained proxy donor keeps both the donor and mutex owner queued. The check therefore misses a constrained FAIR donor and may stop the tick while its runtime still needs to be enforced.

Check the selected donor instead and remove the nr_running restriction. The donor being a queued FAIR task is sufficient to require bandwidth accounting regardless of other runnable tasks.

Fixes af0c8b2bf67b ("sched: Split scheduler and execution contexts")
---
 kernel/sched/core.c | 35 ++++++++++++++++++-----------------
 kernel/sched/fair.c | 12 +++++++-----
 2 files changed, 25 insertions(+), 22 deletions(-)
kernel/sched/core.c +1 -4
index 237d216382f46..14d0d5c884393 100644
@@ -1419,11 +1419,8 @@ static void nohz_csd_func(void *info)
 #endif /* CONFIG_NO_HZ_COMMON */
 
 #ifdef CONFIG_NO_HZ_FULL
-static inline bool __need_bw_check(struct rq *rq, struct task_struct *p)
+static inline bool __need_bw_check(struct task_struct *p)
 {
-	if (rq->nr_running != 1)
-		return false;
-
 	if (p->sched_class != &fair_sched_class)
 		return false;
 

kernel/sched/core.c bool sched_can_stop_tick(struct rq *rq) · line 1445

Quoted 8 lines
kernel/sched/core.c +5 -0
@@ -1441,6 +1438,14 @@ bool sched_can_stop_tick(struct rq *rq)
 	if (rq->dl.dl_nr_running)
 		return false;
 
+	/*
+	 * The selected scheduling context can be a constrained FAIR donor even
+	 * when rq->curr is an RT task. Check it before the RT fast paths below,
+	 * which may report that the tick can stop for a throttled RT context.
+	 */

I am most confused... how can we ever have rq->donor be FAIR and rq->curr be RT? That makes no sense. If an RT task is runnable, pick should just straight up pick that.

Quoted 23 lines
kernel/sched/core.c +5 -1
+	if (__need_bw_check(rq->donor) && cfs_task_bw_constrained(rq->donor))
+		return false;
+
 
 
@@ -7114,7 +7107,7 @@ find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf)
  */
 static void __sched notrace __schedule(int sched_mode)
 {
-	struct task_struct *prev, *next;
+	struct task_struct *prev, *next, *tick_donor;
 	/*
 	 * On PREEMPT_RT kernel, SM_RTLOCK_WAIT is noted
 	 * as a preemption by schedule_debug() and RCU.
@@ -7168,6 +7161,7 @@ static void __sched notrace __schedule(int sched_mode)
 	rq->clock_update_flags <<= 1;
 	update_rq_clock(rq);
 	rq->clock_update_flags = RQCF_UPDATED;
+	tick_donor = rq->donor;
 
 	switch_count = &prev->nivcsw;
 

kernel/sched/core.c static void __sched notrace __schedule(int sched_mode) · line 7245

Quoted 9 lines
kernel/sched/core.c +6 -0
@@ -7243,6 +7237,13 @@ static void __sched notrace __schedule(int sched_mode)
 	clear_tsk_need_resched(prev);
 	clear_preempt_need_resched();
 keep_resched:
+	/*
+	 * Enqueue and dequeue updates can evaluate the outgoing donor. Refresh
+	 * the dependency after selecting a different scheduling context.
+	 */
+	if (rq->donor != tick_donor)
+		sched_update_tick_dependency(rq);

Please keep all the proxy specific bits inside the one sched_proxy_exec() branch above.

author reviewer
#7 lore raw

Re: [PATCH 03/18] sched: Make NOHZ CFS bandwidth checks follow proxy donor

Hi Peter,

thanks for looking at this and sorry for my late response! Comments below.

Quoted 45 lines · Peter Zijlstra
Quoted 44 lines · Andrea Righi

Proxy execution separates the scheduling context in rq->donor from the physical execution context in rq->curr. sched_can_stop_tick() checks the latter for CFS bandwidth constraints and only does so when nr_running is one.

A retained proxy donor keeps both the donor and mutex owner queued. The check therefore misses a constrained FAIR donor and may stop the tick while its runtime still needs to be enforced.

Check the selected donor instead and remove the nr_running restriction. The donor being a queued FAIR task is sufficient to require bandwidth accounting regardless of other runnable tasks.

Fixes af0c8b2bf67b ("sched: Split scheduler and execution contexts")
---
 kernel/sched/core.c | 35 ++++++++++++++++++-----------------
 kernel/sched/fair.c | 12 +++++++-----
 2 files changed, 25 insertions(+), 22 deletions(-)
kernel/sched/core.c +1 -4
index 237d216382f46..14d0d5c884393 100644
@@ -1419,11 +1419,8 @@ static void nohz_csd_func(void *info)
 #endif /* CONFIG_NO_HZ_COMMON */
 
 #ifdef CONFIG_NO_HZ_FULL
-static inline bool __need_bw_check(struct rq *rq, struct task_struct *p)
+static inline bool __need_bw_check(struct task_struct *p)
 {
-	if (rq->nr_running != 1)
-		return false;
-
 	if (p->sched_class != &fair_sched_class)
 		return false;
 
@@ -1441,6 +1438,14 @@ bool sched_can_stop_tick(struct rq *rq)
 	if (rq->dl.dl_nr_running)
 		return false;
 
Quoted 9 lines

kernel/sched/core.c bool sched_can_stop_tick(struct rq *rq) · line 1445

Quoted 5 lines
kernel/sched/core.c +5 -0
+	/*
+	 * The selected scheduling context can be a constrained FAIR donor even
+	 * when rq->curr is an RT task. Check it before the RT fast paths below,
+	 * which may report that the tick can stop for a throttled RT context.
+	 */

I am most confused... how can we ever have rq->donor be FAIR and rq->curr be RT? That makes no sense. If an RT task is runnable, pick should just straight up pick that.

Yeah, the RT example is bogus.

The relevant case is a FAIR donor belonging to a quota-controlled cgroup which still has runtime available, proxy-executing a lower-class mutex owner (specifically an EXT owner with the other patches in the series).

The owner's execution is charged to the donor's scheduling context. The retained donor and runnable owner make rq->nr_running greater than one, while rq->curr identifies the EXT owner rather than the quota-controlled FAIR context. There is also only one queued FAIR task, so the normal CFS multiplicity check does not retain the tick.

The bandwidth check must therefore follow rq->donor so that the donor's remaining CFS quota is enforced.

Quoted 23 lines
Quoted 23 lines
kernel/sched/core.c +5 -1
+	if (__need_bw_check(rq->donor) && cfs_task_bw_constrained(rq->donor))
+		return false;
+
 
 
@@ -7114,7 +7107,7 @@ find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf)
  */
 static void __sched notrace __schedule(int sched_mode)
 {
-	struct task_struct *prev, *next;
+	struct task_struct *prev, *next, *tick_donor;
 	/*
 	 * On PREEMPT_RT kernel, SM_RTLOCK_WAIT is noted
 	 * as a preemption by schedule_debug() and RCU.
@@ -7168,6 +7161,7 @@ static void __sched notrace __schedule(int sched_mode)
 	rq->clock_update_flags <<= 1;
 	update_rq_clock(rq);
 	rq->clock_update_flags = RQCF_UPDATED;
+	tick_donor = rq->donor;
 
 	switch_count = &prev->nivcsw;
 
Quoted 13 lines

kernel/sched/core.c static void __sched notrace __schedule(int sched_mode) · line 7245

Quoted 10 lines
kernel/sched/core.c +6 -0
@@ -7243,6 +7237,13 @@ static void __sched notrace __schedule(int sched_mode)
 	clear_tsk_need_resched(prev);
 	clear_preempt_need_resched();
 keep_resched:
+	/*
+	 * Enqueue and dequeue updates can evaluate the outgoing donor. Refresh
+	 * the dependency after selecting a different scheduling context.
+	 */
+	if (rq->donor != tick_donor)
+		sched_update_tick_dependency(rq);

Please keep all the proxy specific bits inside the one sched_proxy_exec() branch above.

Sure, I'll move the donor-change tracking and tick dependency update into the sched_proxy_exec() branch.

Thanks,
-Andrea

author reviewer
#8 lore raw

PATCH 4/18 sched/core: Avoid false migration warning for proxy donors

In reply to Andrea Righi

Proxy execution can move a blocked donor's scheduling context to the lock owner's CPU even when the donor is migration-disabled. The donor does not execute there, and its original execution CPU remains recorded in wake_cpu.

set_task_cpu() warns unconditionally for migration-disabled tasks, so a subsequent proxy migration or the wakeup path returning the donor home triggers a false positive. Moving a blocked scheduling context does not violate the migration-disabled execution context.

Exclude blocked proxy donors from the warning. The proxy wakeup path restores an executable placement before clearing the blocked state.

This is a preparatory fix for enabling proxy execution together with sched_ext.

---
 kernel/sched/core.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)
kernel/sched/core.c +8 -1
index 14d0d5c884393..0a2eabadfee02 100644
@@ -3351,6 +3351,8 @@ void relax_compatible_cpus_allowed_ptr(struct task_struct *p)
 void set_task_cpu(struct task_struct *p, unsigned int new_cpu)
 {
 	unsigned int state = READ_ONCE(p->__state);
+	bool proxy_migrated = sched_proxy_exec() && p->is_blocked &&
+			      task_cpu(p) != p->wake_cpu;
 
 	/*
 	 * We should never call set_task_cpu() on a blocked task,
@@ -3386,7 +3388,12 @@ void set_task_cpu(struct task_struct *p, unsigned int new_cpu)
 	 */
 	WARN_ON_ONCE(!cpu_online(new_cpu));
 
-	WARN_ON_ONCE(is_migration_disabled(p));
+	/*
+	 * Proxy execution can move a blocked task's scheduling context to any
+	 * CPU without moving its migration-disabled execution context. The
+	 * wakeup path will return the task to a CPU where it can execute.
+	 */
+	WARN_ON_ONCE(is_migration_disabled(p) && !proxy_migrated);
 
 	trace_sched_migrate_task(p, new_cpu);
 
Signature · 1 line
2.55.0
maintainer
#9 lore raw

Re: [PATCH 04/18] sched/core: Avoid false migration warning for proxy donors

Quoted 38 lines · Andrea Righi

Proxy execution can move a blocked donor's scheduling context to the lock owner's CPU even when the donor is migration-disabled. The donor does not execute there, and its original execution CPU remains recorded in wake_cpu.

set_task_cpu() warns unconditionally for migration-disabled tasks, so a subsequent proxy migration or the wakeup path returning the donor home triggers a false positive. Moving a blocked scheduling context does not violate the migration-disabled execution context.

Exclude blocked proxy donors from the warning. The proxy wakeup path restores an executable placement before clearing the blocked state.

This is a preparatory fix for enabling proxy execution together with sched_ext.

---
 kernel/sched/core.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)
kernel/sched/core.c +2 -0
index 14d0d5c884393..0a2eabadfee02 100644
@@ -3351,6 +3351,8 @@ void relax_compatible_cpus_allowed_ptr(struct task_struct *p)
 void set_task_cpu(struct task_struct *p, unsigned int new_cpu)
 {
 	unsigned int state = READ_ONCE(p->__state);
+	bool proxy_migrated = sched_proxy_exec() && p->is_blocked &&
+			      task_cpu(p) != p->wake_cpu;
 
 	/*
 	 * We should never call set_task_cpu() on a blocked task,
@@ -3386,7 +3388,12 @@ void set_task_cpu(struct task_struct *p, unsigned int new_cpu)
 	 */
 	WARN_ON_ONCE(!cpu_online(new_cpu));

kernel/sched/core.c void set_task_cpu(struct task_struct *p, unsigned int new_cpu) · line 3398

Quoted 10 lines
kernel/sched/core.c +6 -1
 
-	WARN_ON_ONCE(is_migration_disabled(p));
+	/*
+	 * Proxy execution can move a blocked task's scheduling context to any
+	 * CPU without moving its migration-disabled execution context. The
+	 * wakeup path will return the task to a CPU where it can execute.
+	 */
+	WARN_ON_ONCE(is_migration_disabled(p) && !proxy_migrated);
 
 	trace_sched_migrate_task(p, new_cpu);

I'm not sure why we're not hitting this upstream?

author reviewer
#10 lore raw

Re: [PATCH 04/18] sched/core: Avoid false migration warning for proxy donors

Quoted 48 lines · Peter Zijlstra
Quoted 47 lines · Andrea Righi

Proxy execution can move a blocked donor's scheduling context to the lock owner's CPU even when the donor is migration-disabled. The donor does not execute there, and its original execution CPU remains recorded in wake_cpu.

set_task_cpu() warns unconditionally for migration-disabled tasks, so a subsequent proxy migration or the wakeup path returning the donor home triggers a false positive. Moving a blocked scheduling context does not violate the migration-disabled execution context.

Exclude blocked proxy donors from the warning. The proxy wakeup path restores an executable placement before clearing the blocked state.

This is a preparatory fix for enabling proxy execution together with sched_ext.

---
 kernel/sched/core.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)
kernel/sched/core.c +8 -1
index 14d0d5c884393..0a2eabadfee02 100644
@@ -3351,6 +3351,8 @@ void relax_compatible_cpus_allowed_ptr(struct task_struct *p)
 void set_task_cpu(struct task_struct *p, unsigned int new_cpu)
 {
 	unsigned int state = READ_ONCE(p->__state);
+	bool proxy_migrated = sched_proxy_exec() && p->is_blocked &&
+			      task_cpu(p) != p->wake_cpu;
 
 	/*
 	 * We should never call set_task_cpu() on a blocked task,
@@ -3386,7 +3388,12 @@ void set_task_cpu(struct task_struct *p, unsigned int new_cpu)
 	 */
 	WARN_ON_ONCE(!cpu_online(new_cpu));
 
-	WARN_ON_ONCE(is_migration_disabled(p));
+	/*
+	 * Proxy execution can move a blocked task's scheduling context to any
+	 * CPU without moving its migration-disabled execution context. The
+	 * wakeup path will return the task to a CPU where it can execute.
+	 */
+	WARN_ON_ONCE(is_migration_disabled(p) && !proxy_migrated);
 
Quoted 3 lines

kernel/sched/core.c void set_task_cpu(struct task_struct *p, unsigned int new_cpu) · line 3398

Quoted 1 line
kernel/sched/core.c +0 -0
 	trace_sched_migrate_task(p, new_cpu);

I'm not sure why we're not hitting this upstream?

I think this can be hit upstream as well. I'll try to create a reproducer.

-Andrea

author reviewer
#11 lore raw

Re: [PATCH 04/18] sched/core: Avoid false migration warning for proxy donors

Quoted 49 lines · Andrea Righi
Quoted 48 lines · Peter Zijlstra
Quoted 47 lines · Andrea Righi

Proxy execution can move a blocked donor's scheduling context to the lock owner's CPU even when the donor is migration-disabled. The donor does not execute there, and its original execution CPU remains recorded in wake_cpu.

set_task_cpu() warns unconditionally for migration-disabled tasks, so a subsequent proxy migration or the wakeup path returning the donor home triggers a false positive. Moving a blocked scheduling context does not violate the migration-disabled execution context.

Exclude blocked proxy donors from the warning. The proxy wakeup path restores an executable placement before clearing the blocked state.

This is a preparatory fix for enabling proxy execution together with sched_ext.

---
 kernel/sched/core.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)
kernel/sched/core.c +8 -1
index 14d0d5c884393..0a2eabadfee02 100644
@@ -3351,6 +3351,8 @@ void relax_compatible_cpus_allowed_ptr(struct task_struct *p)
 void set_task_cpu(struct task_struct *p, unsigned int new_cpu)
 {
 	unsigned int state = READ_ONCE(p->__state);
+	bool proxy_migrated = sched_proxy_exec() && p->is_blocked &&
+			      task_cpu(p) != p->wake_cpu;
 
 	/*
 	 * We should never call set_task_cpu() on a blocked task,
@@ -3386,7 +3388,12 @@ void set_task_cpu(struct task_struct *p, unsigned int new_cpu)
 	 */
 	WARN_ON_ONCE(!cpu_online(new_cpu));
 
-	WARN_ON_ONCE(is_migration_disabled(p));
+	/*
+	 * Proxy execution can move a blocked task's scheduling context to any
+	 * CPU without moving its migration-disabled execution context. The
+	 * wakeup path will return the task to a CPU where it can execute.
+	 */
+	WARN_ON_ONCE(is_migration_disabled(p) && !proxy_migrated);
 
Quoted 5 lines
Quoted 3 lines

kernel/sched/core.c void set_task_cpu(struct task_struct *p, unsigned int new_cpu) · line 3398

Quoted 1 line
kernel/sched/core.c +0 -0
 	trace_sched_migrate_task(p, new_cpu);

I'm not sure why we're not hitting this upstream?

I think this can be hit upstream as well. I'll try to create a reproducer.

And I can reproduce this with the latest tip/master:

proxy_migrate_repro: donor blocking on CPU0 with migration disabled
proxy_migrate_repro: donor moved from CPU0 to CPU1

  WARNING: kernel/sched/core.c:3389 at set_task_cpu+0x1d3/0x280
  ...
  Call Trace:
   try_to_wake_up+0x43f/0x780
   __mutex_unlock_slowpath+0x330/0x540
   owner_fn+0x9f/0xc0 [proxy_migrate_repro]
  ...
  proxy_migrate_repro: donor woke on CPU0, task_cpu=0
  proxy_migrate_repro: completed

The reproducer creates a mutex owner on CPU1 and a migration-disabled waiter on CPU0. The waiter is proxy-migrated to CPU1 while blocked. When the owner releases the mutex, try_to_wake_up() returns the waiter to its pinned CPU0 through set_task_cpu(), triggering the warning.

I guess I can send this as a separate fix, it doesn't necessarily need to be in this series.

Thanks,
-Andrea

maintainer
#12 lore raw

Re: [PATCH 04/18] sched/core: Avoid false migration warning for proxy donors

Quoted 18 lines · Andrea Righi
Quoted 3 lines
Quoted 1 line

I'm not sure why we're not hitting this upstream?

I think this can be hit upstream as well. I'll try to create a reproducer.

And I can reproduce this with the latest tip/master:

proxy_migrate_repro: donor blocking on CPU0 with migration disabled
proxy_migrate_repro: donor moved from CPU0 to CPU1

  WARNING: kernel/sched/core.c:3389 at set_task_cpu+0x1d3/0x280
  ...
  Call Trace:
   try_to_wake_up+0x43f/0x780
   __mutex_unlock_slowpath+0x330/0x540
   owner_fn+0x9f/0xc0 [proxy_migrate_repro]
  ...
  proxy_migrate_repro: donor woke on CPU0, task_cpu=0
  proxy_migrate_repro: completed
Quoted 7 lines

The reproducer creates a mutex owner on CPU1 and a migration-disabled waiter on CPU0. The waiter is proxy-migrated to CPU1 while blocked. When the owner releases the mutex, try_to_wake_up() returns the waiter to its pinned CPU0 through set_task_cpu(), triggering the warning.

I guess I can send this as a separate fix, it doesn't necessarily need to be in this series.

Right, or at the head with a Fixes tag on will do too. Thanks for checking!

author reviewer
#13 lore raw

PATCH 5/18 sched: Pass next class to sched_change_begin()

In reply to Andrea Righi

sched_change_begin() currently only receives the task whose scheduling state is being changed. It cannot distinguish a class transition from a same-class update before recording the task queueing state.

Pass the incoming scheduling class to sched_change_begin() and update all callers. This allows transition handling to run before the normal dequeue and class change.

This is a preparatory change to support proxy execution with sched_ext.

---
 kernel/sched/core.c     | 12 +++++++-----
 kernel/sched/ext/ext.c  |  7 ++++---
 kernel/sched/ext/sub.c  |  9 ++++++---
 kernel/sched/sched.h    |  9 ++++++---
 kernel/sched/syscalls.c |  4 ++--
 5 files changed, 25 insertions(+), 16 deletions(-)
kernel/sched/core.c +7 -5
index 0a2eabadfee02..26b8da8c8a2c5 100644
@@ -2802,7 +2802,7 @@ void set_cpus_allowed_common(struct task_struct *p, struct affinity_context *ctx
 static void
 do_set_cpus_allowed(struct task_struct *p, struct affinity_context *ctx)
 {
-	scoped_guard (sched_change, p, DEQUEUE_SAVE)
+	scoped_guard (sched_change, p, p->sched_class, DEQUEUE_SAVE)
 		p->sched_class->set_cpus_allowed(p, ctx);
 }
 
@@ -7753,7 +7753,7 @@ void rt_mutex_setprio(struct task_struct *p, struct task_struct *pi_task)
 	if (prev_class != next_class)
 		queue_flag |= DEQUEUE_CLASS;
 
-	scoped_guard (sched_change, p, queue_flag) {
+	scoped_guard (sched_change, p, next_class, queue_flag) {
 		/*
 		 * Boosting condition are:
 		 * 1. -rt task is running and holds mutex A
@@ -8430,7 +8430,7 @@ int migrate_task_to(struct task_struct *p, int target_cpu)
 void sched_setnuma(struct task_struct *p, int nid)
 {
 	guard(task_rq_lock)(p);
-	scoped_guard (sched_change, p, DEQUEUE_SAVE)
+	scoped_guard (sched_change, p, p->sched_class, DEQUEUE_SAVE)
 		p->numa_preferred_nid = nid;
 }
 #endif /* CONFIG_NUMA_BALANCING */
@@ -9548,7 +9548,7 @@ void sched_move_task(struct task_struct *tsk, bool for_autogroup)
 	CLASS(task_rq_lock, rq_guard)(tsk);
 	rq = rq_guard.rq;
 
-	scoped_guard (sched_change, tsk, queue_flags) {
+	scoped_guard (sched_change, tsk, tsk->sched_class, queue_flags) {
 		sched_change_group(tsk);
 		if (!for_autogroup)
 			scx_cgroup_move_task(tsk);
@@ -11252,7 +11252,9 @@ static inline void sched_mm_cid_fork(struct task_struct *t) { }
 
 static DEFINE_PER_CPU(struct sched_change_ctx, sched_change_ctx);
 
-struct sched_change_ctx *sched_change_begin(struct task_struct *p, unsigned int flags)
+struct sched_change_ctx *
+sched_change_begin(struct task_struct *p, const struct sched_class *next_class,
+		   unsigned int flags)
 {
 	struct sched_change_ctx *ctx = this_cpu_ptr(&sched_change_ctx);
 	struct rq *rq = task_rq(p);
kernel/sched/ext/ext.c +4 -3
index 10af28a9f2c0c..00c368e2fc82d 100644
@@ -6159,7 +6159,8 @@ void scx_bypass(struct scx_sched *sch, bool bypass)
 				scx_task_slice_ended(rq, p);
 
 			/* cycling deq/enq is enough, see the function comment */
-			scoped_guard (sched_change, p, DEQUEUE_SAVE | DEQUEUE_MOVE) {
+			scoped_guard (sched_change, p, p->sched_class,
+				      DEQUEUE_SAVE | DEQUEUE_MOVE) {
 				/* nothing */ ;
 			}
 		}
@@ -6441,7 +6442,7 @@ static void scx_root_disable(struct scx_sched *sch)
 		if (old_class != new_class)
 			queue_flags |= DEQUEUE_CLASS;
 
-		scoped_guard (sched_change, p, queue_flags) {
+		scoped_guard (sched_change, p, new_class, queue_flags) {
 			p->sched_class = new_class;
 		}
 
@@ -7802,7 +7803,7 @@ static void scx_root_enable_workfn(struct kthread_work *work)
 		if (old_class != new_class)
 			queue_flags |= DEQUEUE_CLASS;
 
-		scoped_guard (sched_change, p, queue_flags) {
+		scoped_guard (sched_change, p, new_class, queue_flags) {
 			scx_set_task_slice(p, READ_ONCE(sch->slice_dfl));
 			p->sched_class = new_class;
 		}
kernel/sched/ext/sub.c +6 -3
index 0554448835bd0..c3955c8163a03 100644
@@ -1255,7 +1255,8 @@ static void scx_rehome_task(struct scx_sched *to, struct task_struct *p)
 	lockdep_assert_held(&p->pi_lock);
 	lockdep_assert_rq_held(task_rq(p));
 
-	scoped_guard (sched_change, p, DEQUEUE_SAVE | DEQUEUE_MOVE) {
+	scoped_guard (sched_change, p, p->sched_class,
+		      DEQUEUE_SAVE | DEQUEUE_MOVE) {
 		scx_disable_and_exit_task(scx_task_sched(p), p);
 		scx_set_task_state(p, SCX_TASK_INIT_BEGIN);
 		scx_set_task_state(p, SCX_TASK_INIT);
@@ -1285,7 +1286,8 @@ static void scx_punt_task(struct scx_sched *to, struct task_struct *p)
 	lockdep_assert_rq_held(task_rq(p));
 	WARN_ON_ONCE(!READ_ONCE(to->bypass_depth));
 
-	scoped_guard (sched_change, p, DEQUEUE_SAVE | DEQUEUE_MOVE) {
+	scoped_guard (sched_change, p, p->sched_class,
+		      DEQUEUE_SAVE | DEQUEUE_MOVE) {
 		scx_disable_and_exit_task(scx_task_sched(p), p);
 		scx_set_task_sched(p, to);
 	}
@@ -1939,7 +1941,8 @@ void scx_sub_enable_workfn(struct kthread_work *work)
 		if (!(p->scx.flags & SCX_TASK_SUB_INIT))
 			continue;
 
-		scoped_guard (sched_change, p, DEQUEUE_SAVE | DEQUEUE_MOVE) {
+		scoped_guard (sched_change, p, p->sched_class,
+			      DEQUEUE_SAVE | DEQUEUE_MOVE) {
 			/*
 			 * $p must be either READY or ENABLED. If ENABLED,
 			 * __scx_disabled_and_exit_task() first disables and
kernel/sched/sched.h +6 -3
index e656c7059bf86..6bba1f3b5d4cd 100644
@@ -4240,13 +4240,16 @@ struct sched_change_ctx {
 	bool			running;
 };
 
-struct sched_change_ctx *sched_change_begin(struct task_struct *p, unsigned int flags);
+struct sched_change_ctx *
+sched_change_begin(struct task_struct *p, const struct sched_class *next_class,
+		   unsigned int flags);
 void sched_change_end(struct sched_change_ctx *ctx);
 
 DEFINE_CLASS(sched_change, struct sched_change_ctx *,
 	     sched_change_end(_T),
-	     sched_change_begin(p, flags),
-	     struct task_struct *p, unsigned int flags)
+	     sched_change_begin(p, next_class, flags),
+	     struct task_struct *p, const struct sched_class *next_class,
+	     unsigned int flags)
 
 DEFINE_CLASS_IS_UNCONDITIONAL(sched_change)
 
kernel/sched/syscalls.c +2 -2
index b215b0ead9a60..bc32ce76ff4fe 100644
@@ -85,7 +85,7 @@ void set_user_nice(struct task_struct *p, long nice)
 		return;
 	}
 
-	scoped_guard (sched_change, p, DEQUEUE_SAVE) {
+	scoped_guard (sched_change, p, p->sched_class, DEQUEUE_SAVE) {
 		p->static_prio = NICE_TO_PRIO(nice);
 		set_load_weight(p, true);
 		old_prio = p->prio;
@@ -678,7 +678,7 @@ int __sched_setscheduler(struct task_struct *p,
 	if (prev_class != next_class)
 		queue_flags |= DEQUEUE_CLASS;
 
-	scoped_guard (sched_change, p, queue_flags) {
+	scoped_guard (sched_change, p, next_class, queue_flags) {
 
 		if (!(attr->sched_flags & SCHED_FLAG_KEEP_PARAMS)) {
 			__setscheduler_params(p, attr);
Signature · 1 line
2.55.0
maintainer
#14 lore raw

Re: [PATCH 05/18] sched: Pass next class to sched_change_begin()

Quoted 3 lines · Andrea Righi

sched_change_begin() currently only receives the task whose scheduling state is being changed. It cannot distinguish a class transition from a same-class update before recording the task queueing state.

Uh, yes it can. That's what {EN,DE}QUEUE_CLASS are for, no?

author reviewer
#15 lore raw

Re: [PATCH 05/18] sched: Pass next class to sched_change_begin()

Quoted 6 lines · Peter Zijlstra
Quoted 3 lines · Andrea Righi

sched_change_begin() currently only receives the task whose scheduling state is being changed. It cannot distinguish a class transition from a same-class update before recording the task queueing state.

Uh, yes it can. That's what {EN,DE}QUEUE_CLASS are for, no?

Right, I overlooked DEQUEUE_CLASS. I'll drop this patch and make the later proxy-donor handling conditional on DEQUEUE_CLASS directly.

Thanks,
-Andrea

author reviewer
#16 lore raw

PATCH 6/18 sched: Add helper to block retained proxy donors

In reply to Andrea Righi

Scheduler ownership changes may need to turn a mutex-blocked task retained on the runqueue for proxy execution back into a normally blocked task.

Add sched_proxy_block_task() to perform that transition while holding p->pi_lock and the task's runqueue lock. Reset an active donor to the physical current task before blocking it. If DELAY_DEQUEUE keeps a FAIR donor queued, the immediately following sched_change_begin() completes the dequeue through switching_from_fair() before recording its queued state.

This is a preparatory change to support proxy execution with sched_ext.

---
 kernel/sched/core.c  | 28 ++++++++++++++++++++++++++++
 kernel/sched/sched.h |  6 ++++++
 2 files changed, 34 insertions(+)
kernel/sched/core.c +28 -0
index 26b8da8c8a2c5..bb79fcfa21cd2 100644
@@ -6822,6 +6822,34 @@ static void proxy_deactivate(struct rq *rq, struct task_struct *donor)
 	block_task(rq, donor, state);
 }
 
+/*
+ * Remove a retained proxy donor before changing its scheduler ownership.
+ * The caller holds p->pi_lock, so p cannot wake and migrate if block_task()
+ * drops it from the runqueue. If DELAY_DEQUEUE keeps a blocked fair task
+ * queued, switching_from_fair() completes the dequeue in the immediately
+ * following sched_change_begin().
+ */
+void sched_proxy_block_task(struct rq *rq, struct task_struct *p)
+{
+	unsigned long state = READ_ONCE(p->__state);
+
+	lockdep_assert_held(&p->pi_lock);
+	lockdep_assert_rq_held(rq);
+
+	if (!p->is_blocked || !task_on_rq_queued(p))
+		return;
+	if (WARN_ON_ONCE(state == TASK_RUNNING))
+		return;
+
+	if (task_current_donor(rq, p))
+		proxy_reset_donor(rq);
+
+	if (!p->se.sched_delayed)
+		block_task(rq, p, state);
+
+	WARN_ON_ONCE(task_on_rq_queued(p) && !p->se.sched_delayed);
+}
+
 static inline void proxy_release_rq_lock(struct rq *rq, struct rq_flags *rf)
 	__releases(__rq_lockp(rq))
 {
kernel/sched/sched.h +6 -0
index 6bba1f3b5d4cd..56ac104f1f825 100644
@@ -2508,6 +2508,12 @@ static inline bool task_is_blocked(struct task_struct *p)
 	return !!p->blocked_on;
 }
 
+#ifdef CONFIG_SCHED_PROXY_EXEC
+void sched_proxy_block_task(struct rq *rq, struct task_struct *p);
+#else
+static inline void sched_proxy_block_task(struct rq *rq, struct task_struct *p) {}
+#endif
+
 static inline int task_on_cpu(struct rq *rq, struct task_struct *p)
 {
 	return p->on_cpu;
Signature · 1 line
2.55.0
author reviewer
#17 lore raw

PATCH 7/18 sched: Add sched_ext hooks for proxy execution

In reply to Andrea Righi

Proxy execution splits the scheduling context (the donor) from the execution context (the lock owner). sched_ext needs to observe that split at three points in __schedule():

- whether a blocked EXT task can be retained on the runqueue as a donor,
- when a donor's scheduling context starts driving a lock owner,
- once proxy resolution has settled for the current pick.

Introduce scx_allow_proxy_exec(), scx_proxy_donor_start() and scx_proxy_resolved(), and add their call sites in __schedule(). The implementations are empty here and are filled in by the sched_ext changes that follow, so that all the sched core changes needed by proxy execution stay together in the preparatory patches.

SCHED_PROXY_EXEC still depends on !SCHED_CLASS_EXT, so the new hooks are
inert: they are compiled out with CONFIG_SCHED_CLASS_EXT=n and unreachable otherwise.

This is a preparatory change to support proxy execution with sched_ext. No functional change.

---
 kernel/sched/core.c    | 12 +++++++-----
 kernel/sched/ext/ext.c | 13 +++++++++++++
 kernel/sched/ext/ext.h |  6 ++++++
 3 files changed, 26 insertions(+), 5 deletions(-)
kernel/sched/core.c +7 -5
index bb79fcfa21cd2..d578dd635f2eb 100644
@@ -7217,13 +7217,12 @@ static void __sched notrace __schedule(int sched_mode)
 		}
 	} else if (!preempt && prev_state) {
 		/*
-		 * We pass task_is_blocked() as the should_block arg
-		 * in order to keep mutex-blocked tasks on the runqueue
-		 * for slection with proxy-exec (without proxy-exec
-		 * task_is_blocked() will always be false).
+		 * Keep mutex-blocked tasks on the runqueue for proxy execution
+		 * only when their scheduling class allows it. Without proxy
+		 * execution, task_is_blocked() always returns false.
 		 */
 		try_to_block_task(rq, prev, &prev_state,
-				  !task_is_blocked(prev));
+				  !task_is_blocked(prev) || !scx_allow_proxy_exec(prev));
 		switch_count = &prev->nvcsw;
 	}
 
@@ -7244,6 +7243,7 @@ static void __sched notrace __schedule(int sched_mode)
 			}
 			if (next == rq->idle) {
 				zap_balance_callbacks(rq);
+				scx_proxy_resolved(rq);
 				goto keep_resched;
 			}
 		}
@@ -7264,6 +7264,8 @@ static void __sched notrace __schedule(int sched_mode)
 			donor->sched_class->put_prev_task(rq, donor, donor);
 			donor->sched_class->set_next_task(rq, donor, true);
 		}
+		scx_proxy_donor_start(rq);
+		scx_proxy_resolved(rq);
 	} else {
 		rq_set_donor(rq, next);
 	}
kernel/sched/ext/ext.c +13 -0
index 00c368e2fc82d..f3e59f4fad4ac 100644
@@ -24,6 +24,11 @@
 
 DEFINE_RAW_SPINLOCK(scx_sched_lock);
 
+bool scx_allow_proxy_exec(const struct task_struct *p)
+{
+	return true;
+}
+
 /*
  * NOTE: sched_ext is in the process of growing multiple scheduler support and
  * scx_root usage is in a transitional state. Naked dereferences are safe if the
@@ -1079,6 +1084,10 @@ static void schedule_deferred_locked(struct rq *rq)
 	schedule_deferred(rq);
 }
 
+void scx_proxy_resolved(struct rq *rq)
+{
+}
+
 void schedule_dsq_reenq(struct scx_sched *sch, struct scx_dispatch_q *dsq,
 			u64 reenq_flags, struct rq *locked_rq)
 {
@@ -3062,6 +3071,10 @@ static void set_next_task_scx(struct rq *rq, struct task_struct *p, bool first)
 	}
 }
 
+void scx_proxy_donor_start(struct rq *rq)
+{
+}
+
 static enum scx_cpu_preempt_reason
 preempt_reason_from_class(const struct sched_class *class)
 {
kernel/sched/ext/ext.h +6 -0
index 0b7fc46aee08c..859b9261a8a6b 100644
@@ -20,6 +20,9 @@ void scx_rq_deactivate(struct rq *rq);
 int scx_check_setscheduler(struct task_struct *p, int policy);
 bool task_should_scx(int policy);
 bool scx_allow_ttwu_queue(const struct task_struct *p);
+bool scx_allow_proxy_exec(const struct task_struct *p);
+void scx_proxy_donor_start(struct rq *rq);
+void scx_proxy_resolved(struct rq *rq);
 void init_sched_ext_class(void);
 
 static inline u32 scx_cpuperf_target(s32 cpu)
@@ -54,6 +57,9 @@ static inline void scx_rq_deactivate(struct rq *rq) {}
 static inline int scx_check_setscheduler(struct task_struct *p, int policy) { return 0; }
 static inline bool task_on_scx(const struct task_struct *p) { return false; }
 static inline bool scx_allow_ttwu_queue(const struct task_struct *p) { return true; }
+static inline bool scx_allow_proxy_exec(const struct task_struct *p) { return true; }
+static inline void scx_proxy_donor_start(struct rq *rq) {}
+static inline void scx_proxy_resolved(struct rq *rq) {}
 static inline void init_sched_ext_class(void) {}
 
 #endif	/* CONFIG_SCHED_CLASS_EXT */
Signature · 1 line
2.55.0
maintainer
#18 lore raw

Re: [PATCH 07/18] sched: Add sched_ext hooks for proxy execution

Quoted 60 lines · Andrea Righi

Proxy execution splits the scheduling context (the donor) from the execution context (the lock owner). sched_ext needs to observe that split at three points in __schedule():

- whether a blocked EXT task can be retained on the runqueue as a donor,
- when a donor's scheduling context starts driving a lock owner,
- once proxy resolution has settled for the current pick.

Introduce scx_allow_proxy_exec(), scx_proxy_donor_start() and scx_proxy_resolved(), and add their call sites in __schedule(). The implementations are empty here and are filled in by the sched_ext changes that follow, so that all the sched core changes needed by proxy execution stay together in the preparatory patches.

SCHED_PROXY_EXEC still depends on !SCHED_CLASS_EXT, so the new hooks are
inert: they are compiled out with CONFIG_SCHED_CLASS_EXT=n and unreachable otherwise.

This is a preparatory change to support proxy execution with sched_ext. No functional change.

---
 kernel/sched/core.c    | 12 +++++++-----
 kernel/sched/ext/ext.c | 13 +++++++++++++
 kernel/sched/ext/ext.h |  6 ++++++
 3 files changed, 26 insertions(+), 5 deletions(-)
kernel/sched/core.c +5 -5
index bb79fcfa21cd2..d578dd635f2eb 100644
@@ -7217,13 +7217,12 @@ static void __sched notrace __schedule(int sched_mode)
 		}
 	} else if (!preempt && prev_state) {
 		/*
-		 * We pass task_is_blocked() as the should_block arg
-		 * in order to keep mutex-blocked tasks on the runqueue
-		 * for slection with proxy-exec (without proxy-exec
-		 * task_is_blocked() will always be false).
+		 * Keep mutex-blocked tasks on the runqueue for proxy execution
+		 * only when their scheduling class allows it. Without proxy
+		 * execution, task_is_blocked() always returns false.
 		 */
 		try_to_block_task(rq, prev, &prev_state,
-				  !task_is_blocked(prev));
+				  !task_is_blocked(prev) || !scx_allow_proxy_exec(prev));
 		switch_count = &prev->nvcsw;
 	}
 
@@ -7244,6 +7243,7 @@ static void __sched notrace __schedule(int sched_mode)
 			}
 			if (next == rq->idle) {
 				zap_balance_callbacks(rq);
+				scx_proxy_resolved(rq);
 				goto keep_resched;
 			}
 		}

kernel/sched/core.c static void __sched notrace __schedule(int sched_mode) · line 7271

Quoted 8 lines
kernel/sched/core.c +2 -0
@@ -7264,6 +7264,8 @@ static void __sched notrace __schedule(int sched_mode)
 			donor->sched_class->put_prev_task(rq, donor, donor);
 			donor->sched_class->set_next_task(rq, donor, true);
 		}
+		scx_proxy_donor_start(rq);
+		scx_proxy_resolved(rq);
 	} else {
 		rq_set_donor(rq, next);
 	}

Can you expand on the need for scx_proxy_resolved() ? I understand the other two, but this one I'm struggling with a bit.

author reviewer
#19 lore raw

Re: [PATCH 07/18] sched: Add sched_ext hooks for proxy execution

Quoted 64 lines · Peter Zijlstra
Quoted 63 lines · Andrea Righi

Proxy execution splits the scheduling context (the donor) from the execution context (the lock owner). sched_ext needs to observe that split at three points in __schedule():

- whether a blocked EXT task can be retained on the runqueue as a donor,
- when a donor's scheduling context starts driving a lock owner,
- once proxy resolution has settled for the current pick.

Introduce scx_allow_proxy_exec(), scx_proxy_donor_start() and scx_proxy_resolved(), and add their call sites in __schedule(). The implementations are empty here and are filled in by the sched_ext changes that follow, so that all the sched core changes needed by proxy execution stay together in the preparatory patches.

SCHED_PROXY_EXEC still depends on !SCHED_CLASS_EXT, so the new hooks are
inert: they are compiled out with CONFIG_SCHED_CLASS_EXT=n and unreachable otherwise.

This is a preparatory change to support proxy execution with sched_ext. No functional change.

---
 kernel/sched/core.c    | 12 +++++++-----
 kernel/sched/ext/ext.c | 13 +++++++++++++
 kernel/sched/ext/ext.h |  6 ++++++
 3 files changed, 26 insertions(+), 5 deletions(-)
kernel/sched/core.c +5 -5
index bb79fcfa21cd2..d578dd635f2eb 100644
@@ -7217,13 +7217,12 @@ static void __sched notrace __schedule(int sched_mode)
 		}
 	} else if (!preempt && prev_state) {
 		/*
-		 * We pass task_is_blocked() as the should_block arg
-		 * in order to keep mutex-blocked tasks on the runqueue
-		 * for slection with proxy-exec (without proxy-exec
-		 * task_is_blocked() will always be false).
+		 * Keep mutex-blocked tasks on the runqueue for proxy execution
+		 * only when their scheduling class allows it. Without proxy
+		 * execution, task_is_blocked() always returns false.
 		 */
 		try_to_block_task(rq, prev, &prev_state,
-				  !task_is_blocked(prev));
+				  !task_is_blocked(prev) || !scx_allow_proxy_exec(prev));
 		switch_count = &prev->nvcsw;
 	}
 
@@ -7244,6 +7243,7 @@ static void __sched notrace __schedule(int sched_mode)
 			}
 			if (next == rq->idle) {
 				zap_balance_callbacks(rq);
+				scx_proxy_resolved(rq);
 				goto keep_resched;
 			}
 		}
@@ -7264,6 +7264,8 @@ static void __sched notrace __schedule(int sched_mode)
 			donor->sched_class->put_prev_task(rq, donor, donor);
 			donor->sched_class->set_next_task(rq, donor, true);
 		}
Quoted 8 lines

kernel/sched/core.c static void __sched notrace __schedule(int sched_mode) · line 7271

Quoted 5 lines
kernel/sched/core.c +2 -0
+		scx_proxy_donor_start(rq);
+		scx_proxy_resolved(rq);
 	} else {
 		rq_set_donor(rq, next);
 	}

Can you expand on the need for scx_proxy_resolved() ? I understand the other two, but this one I'm struggling with a bit.

Yeah and the name is a poor choice, it should renamed scx_proxy_reenqueue_retry() or something similar.

It's the retry point for a task that sched_ext couldn't move while processing a dispatch from a remote DSQ (used later in the series).

When sched_ext consumes a task dispatched to a CPU other than the one whose rq currently owns it, it may find (after locking the task's source rq) that proxy exec has made the task either the physical current task or the active donor. It can't migrate the task in that state, so it parks the task on the source rq's reject DSQ.

If the deferred reject-DSQ drain runs while the task is still current or donating, the task must remain parked. Retrying immediately could spin until the proxy relationship changes. The hook provides the notification that proxy selection has run again, allowing sched_ext to schedule another deferred drain after the current context switch.

I couldn't find an existing event that covers this transition without periodically retrying the reject-DSQ drain. Is there a better place to trigger this retry?

Thanks,
-Andrea

maintainer
#20 lore raw

Re: [PATCH 07/18] sched: Add sched_ext hooks for proxy execution

Quoted 33 lines · Andrea Righi
Quoted 4 lines · Peter Zijlstra
Quoted 4 lines
kernel/sched/core.c +0 -0
@@ -7264,6 +7264,8 @@ static void __sched notrace __schedule(int sched_mode)
 			donor->sched_class->put_prev_task(rq, donor, donor);
 			donor->sched_class->set_next_task(rq, donor, true);
 		}
Quoted 8 lines

kernel/sched/core.c static void __sched notrace __schedule(int sched_mode) · line 7271

Quoted 5 lines
kernel/sched/core.c +2 -0
+		scx_proxy_donor_start(rq);
+		scx_proxy_resolved(rq);
 	} else {
 		rq_set_donor(rq, next);
 	}

Can you expand on the need for scx_proxy_resolved() ? I understand the other two, but this one I'm struggling with a bit.

Yeah and the name is a poor choice, it should renamed scx_proxy_reenqueue_retry() or something similar.

It's the retry point for a task that sched_ext couldn't move while processing a dispatch from a remote DSQ (used later in the series).

When sched_ext consumes a task dispatched to a CPU other than the one whose rq currently owns it, it may find (after locking the task's source rq) that proxy exec has made the task either the physical current task or the active donor. It can't migrate the task in that state, so it parks the task on the source rq's reject DSQ.

If the deferred reject-DSQ drain runs while the task is still current or donating, the task must remain parked. Retrying immediately could spin until the proxy relationship changes. The hook provides the notification that proxy selection has run again, allowing sched_ext to schedule another deferred drain after the current context switch.

Quoted 3 lines

I couldn't find an existing event that covers this transition without periodically retrying the reject-DSQ drain. Is there a better place to trigger this retry?

Ah, so its a little like that problem we had with ->balance() and ->pick_task(). Where a task gets taken off the DSQ and moved to the local queue, but when not picked, it must be moved back.

In this case, ->donor is visible to ext and all is well, but ->curr is not so easy.

I'm still a little confused though, why not leave blocked tasks on this reject queue. If they're needed, the proxy mechanism will move them around. They won't actually ever run except through proxy.

The point where they will become runnable again, is through wakeup. So why not delay everything until that point?

Or am I not understanding ext again?

author reviewer
#21 lore raw

Re: [PATCH 07/18] sched: Add sched_ext hooks for proxy execution

Hi Peter,

Quoted 34 lines · Peter Zijlstra
Quoted 33 lines · Andrea Righi
Quoted 4 lines · Peter Zijlstra
Quoted 4 lines
kernel/sched/core.c +0 -0
@@ -7264,6 +7264,8 @@ static void __sched notrace __schedule(int sched_mode)
 			donor->sched_class->put_prev_task(rq, donor, donor);
 			donor->sched_class->set_next_task(rq, donor, true);
 		}
Quoted 8 lines

kernel/sched/core.c static void __sched notrace __schedule(int sched_mode) · line 7271

Quoted 5 lines
kernel/sched/core.c +2 -0
+		scx_proxy_donor_start(rq);
+		scx_proxy_resolved(rq);
 	} else {
 		rq_set_donor(rq, next);
 	}

Can you expand on the need for scx_proxy_resolved() ? I understand the other two, but this one I'm struggling with a bit.

Yeah and the name is a poor choice, it should renamed scx_proxy_reenqueue_retry() or something similar.

It's the retry point for a task that sched_ext couldn't move while processing a dispatch from a remote DSQ (used later in the series).

When sched_ext consumes a task dispatched to a CPU other than the one whose rq currently owns it, it may find (after locking the task's source rq) that proxy exec has made the task either the physical current task or the active donor. It can't migrate the task in that state, so it parks the task on the source rq's reject DSQ.

If the deferred reject-DSQ drain runs while the task is still current or donating, the task must remain parked. Retrying immediately could spin until the proxy relationship changes. The hook provides the notification that proxy selection has run again, allowing sched_ext to schedule another deferred drain after the current context switch.

Quoted 7 lines
Quoted 3 lines

I couldn't find an existing event that covers this transition without periodically retrying the reject-DSQ drain. Is there a better place to trigger this retry?

Ah, so its a little like that problem we had with ->balance() and ->pick_task(). Where a task gets taken off the DSQ and moved to the local queue, but when not picked, it must be moved back.

Exactly, that is the same general kind of problem.

Quoted 4 lines

In this case, ->donor is visible to ext and all is well, but ->curr is not so easy.

Quoted 8 lines

I'm still a little confused though, why not leave blocked tasks on this reject queue. If they're needed, the proxy mechanism will move them around. They won't actually ever run except through proxy.

The point where they will become runnable again, is through wakeup. So why not delay everything until that point?

Or am I not understanding ext again?

I think the missing detail is that the task rejected by the remote DSQ transfer isn't necessarily blocked.

Consider a blocked donor D proxy-executing a runnable EXT mutex owner O on CPU0. O is rq->curr and D is rq->donor. Since O is executing using D's scheduling context, O's own EXT scheduling entity may still be on a non-local DSQ.

CPU1 could concurrently consume O from that non-local DSQ and start transferring it to CPU1. After dropping the DSQ lock and acquiring O's source rq lock, it discovers that O has become rq->curr on CPU0 and therefore can't be migrated. O is then parked on CPU0's reject DSQ.

When proxy exec switches away from O, O stops being rq->curr but remains runnable. There's no wakeup in this case, because O never blocked. If it remains on the reject DSQ, it's no longer visible to the BPF scheduler and can't subsequently run using its own scheduling context.

The rq->donor case has a similar issue even though that task is blocked: a blocked donor can stop being the active donor before the mutex wakes it (for example, if another scheduling context is selected). The proxy exec mechanism follows the owner chain after the donor has been selected, it doesn't select tasks from the internal reject DSQ. Leaving the donor there until wakeup would prevent it from being selected again for proxy execution in the meantime.

This is why the retry is tied to another proxy selection point, once the task is no longer rq->curr or rq->donor, it can be re-enqueued to through its BPF scheduler.

Thanks,
-Andrea

maintainer
#22 lore raw

Re: [PATCH 07/18] sched: Add sched_ext hooks for proxy execution

Quoted 25 lines · Andrea Righi
Quoted 11 lines

In this case, ->donor is visible to ext and all is well, but ->curr is not so easy.

I'm still a little confused though, why not leave blocked tasks on this reject queue. If they're needed, the proxy mechanism will move them around. They won't actually ever run except through proxy.

The point where they will become runnable again, is through wakeup. So why not delay everything until that point?

Or am I not understanding ext again?

I think the missing detail is that the task rejected by the remote DSQ transfer isn't necessarily blocked.

Consider a blocked donor D proxy-executing a runnable EXT mutex owner O on CPU0. O is rq->curr and D is rq->donor. Since O is executing using D's scheduling context, O's own EXT scheduling entity may still be on a non-local DSQ.

CPU1 could concurrently consume O from that non-local DSQ and start transferring it to CPU1. After dropping the DSQ lock and acquiring O's source rq lock, it discovers that O has become rq->curr on CPU0 and therefore can't be migrated. O is then parked on CPU0's reject DSQ.

Quoted 5 lines

When proxy exec switches away from O, O stops being rq->curr but remains runnable. There's no wakeup in this case, because O never blocked. If it remains on the reject DSQ, it's no longer visible to the BPF scheduler and can't subsequently run using its own scheduling context.

Argh, right! I got D and O switched around :-/

Quoted 7 lines

The rq->donor case has a similar issue even though that task is
blocked: a blocked donor can stop being the active donor before the mutex wakes it (for example, if another scheduling context is selected). The proxy exec mechanism follows the owner chain after the donor has been selected, it doesn't select tasks from the internal reject DSQ. Leaving the donor there until wakeup would prevent it from being selected again for proxy execution in the meantime.

Ah, but rq->donor gets a put_prev_task() to sort it out. While there is no such things for rq->curr.

Quoted 3 lines

This is why the retry is tied to another proxy selection point, once the task is no longer rq->curr or rq->donor, it can be re-enqueued to through its BPF scheduler.

Fair enough.

author reviewer
#23 lore raw

PATCH 8/18 sched: Introduce WF_ON_RQ wake flag

In reply to Andrea Righi

ttwu_runnable() handles wakeups for tasks which are already on the runqueue, but scheduling classes cannot distinguish that path from a full wakeup activation in wakeup_preempt().

Pass WF_ON_RQ to wakeup_preempt() from ttwu_runnable() so scheduling classes can handle already-runnable wakeups separately.

This is a preparatory change to support proxy execution with sched_ext. No functional change.

---
 kernel/sched/core.c  | 2 +-
 kernel/sched/sched.h | 1 +
 2 files changed, 2 insertions(+), 1 deletion(-)
kernel/sched/core.c +1 -1
index d578dd635f2eb..f142ab455b797 100644
@@ -3908,7 +3908,7 @@ static int ttwu_runnable(struct task_struct *p, int wake_flags)
 		 * When on_rq && !on_cpu the task is preempted, see if
 		 * it should preempt the task that is current now.
 		 */
-		wakeup_preempt(rq, p, wake_flags);
+		wakeup_preempt(rq, p, wake_flags | WF_ON_RQ);
 	}
 	ttwu_do_wakeup(p);
 	return 1;
kernel/sched/sched.h +1 -0
index 56ac104f1f825..f30d122909e7b 100644
@@ -2538,6 +2538,7 @@ static inline int task_on_rq_migrating(struct task_struct *p)
 #define WF_MIGRATED		0x20 /* Internal use, task got migrated */
 #define WF_CURRENT_CPU		0x40 /* Prefer to move the wakee to the current CPU. */
 #define WF_RQ_SELECTED		0x80 /* ->select_task_rq() was called */
+#define WF_ON_RQ		0x100 /* Wakeup of an already runnable task */
 
 static_assert(WF_EXEC == SD_BALANCE_EXEC);
 static_assert(WF_FORK == SD_BALANCE_FORK);
Signature · 1 line
2.55.0
maintainer
#24 lore raw

Re: [PATCH 08/18] sched: Introduce WF_ON_RQ wake flag

Quoted 34 lines · Andrea Righi

ttwu_runnable() handles wakeups for tasks which are already on the runqueue, but scheduling classes cannot distinguish that path from a full wakeup activation in wakeup_preempt().

Pass WF_ON_RQ to wakeup_preempt() from ttwu_runnable() so scheduling classes can handle already-runnable wakeups separately.

This is a preparatory change to support proxy execution with sched_ext. No functional change.

---
 kernel/sched/core.c  | 2 +-
 kernel/sched/sched.h | 1 +
 2 files changed, 2 insertions(+), 1 deletion(-)
kernel/sched/core.c +1 -1
index d578dd635f2eb..f142ab455b797 100644
@@ -3908,7 +3908,7 @@ static int ttwu_runnable(struct task_struct *p, int wake_flags)
 		 * When on_rq && !on_cpu the task is preempted, see if
 		 * it should preempt the task that is current now.
 		 */
-		wakeup_preempt(rq, p, wake_flags);
+		wakeup_preempt(rq, p, wake_flags | WF_ON_RQ);
 	}
 	ttwu_do_wakeup(p);
 	return 1;
kernel/sched/sched.h +0 -0
index 56ac104f1f825..f30d122909e7b 100644

kernel/sched/sched.h static inline int task_on_rq_migrating(struct task_struct *p) · line 2541

Quoted 4 lines
kernel/sched/sched.h +1 -0
@@ -2538,6 +2538,7 @@ static inline int task_on_rq_migrating(struct task_struct *p)
 #define WF_MIGRATED		0x20 /* Internal use, task got migrated */
 #define WF_CURRENT_CPU		0x40 /* Prefer to move the wakee to the current CPU. */
 #define WF_RQ_SELECTED		0x80 /* ->select_task_rq() was called */
+#define WF_ON_RQ		0x100 /* Wakeup of an already runnable task */

I are confused again... help?

wakeup_preempt() can only ever be called for tasks that are on the runqueue. That is more or less the point of wakeup-preemption.

Even the already-runnable distinction would mean things like move_queued_task() should also have this. The task is new to this runqueue, but was definitely already runnable before.

What actual distinction are you needing?

author reviewer
#25 lore raw

Re: [PATCH 08/18] sched: Introduce WF_ON_RQ wake flag

Quoted 42 lines · Peter Zijlstra
Quoted 34 lines · Andrea Righi

ttwu_runnable() handles wakeups for tasks which are already on the runqueue, but scheduling classes cannot distinguish that path from a full wakeup activation in wakeup_preempt().

Pass WF_ON_RQ to wakeup_preempt() from ttwu_runnable() so scheduling classes can handle already-runnable wakeups separately.

This is a preparatory change to support proxy execution with sched_ext. No functional change.

---
 kernel/sched/core.c  | 2 +-
 kernel/sched/sched.h | 1 +
 2 files changed, 2 insertions(+), 1 deletion(-)
kernel/sched/core.c +1 -1
index d578dd635f2eb..f142ab455b797 100644
@@ -3908,7 +3908,7 @@ static int ttwu_runnable(struct task_struct *p, int wake_flags)
 		 * When on_rq && !on_cpu the task is preempted, see if
 		 * it should preempt the task that is current now.
 		 */
-		wakeup_preempt(rq, p, wake_flags);
+		wakeup_preempt(rq, p, wake_flags | WF_ON_RQ);
 	}
 	ttwu_do_wakeup(p);
 	return 1;
kernel/sched/sched.h +0 -0
index 56ac104f1f825..f30d122909e7b 100644

kernel/sched/sched.h static inline int task_on_rq_migrating(struct task_struct *p) · line 2541

Quoted 4 lines
kernel/sched/sched.h +1 -0
@@ -2538,6 +2538,7 @@ static inline int task_on_rq_migrating(struct task_struct *p)
 #define WF_MIGRATED		0x20 /* Internal use, task got migrated */
 #define WF_CURRENT_CPU		0x40 /* Prefer to move the wakee to the current CPU. */
 #define WF_RQ_SELECTED		0x80 /* ->select_task_rq() was called */
+#define WF_ON_RQ		0x100 /* Wakeup of an already runnable task */

I are confused again... help?

Quoted 8 lines

wakeup_preempt() can only ever be called for tasks that are on the runqueue. That is more or less the point of wakeup-preemption.

Even the already-runnable distinction would mean things like move_queued_task() should also have this. The task is new to this runqueue, but was definitely already runnable before.

What actual distinction are you needing?

Hm... yeah, WF_ON_RQ describes the wrong distinction.

What sched_ext actually needs to know is: did this wakeup use ttwu_runnable(), changing the task back to TASK_RUNNING without calling activate_task() / enqueue_task()?

We need the distinction because a retained proxy donor remains on the runqueue while blocked. When it wakes, ttwu_runnable() can clear its blocked state without calling enqueue_task_scx() again.

sched_ext must therefore request a reschedule so the newly unblocked task is reconsidered for dispatch. We don't want this extra reschedule for a normal wakeup because that path already called enqueue_task_scx() and performed the required sched_ext bookkeeping.

Thanks,
-Andrea

#26 lore raw

Re: [PATCH 08/18] sched: Introduce WF_ON_RQ wake flag

Quoted 58 lines · Andrea Righi
Quoted 42 lines · Peter Zijlstra
Quoted 34 lines · Andrea Righi

ttwu_runnable() handles wakeups for tasks which are already on the runqueue, but scheduling classes cannot distinguish that path from a full wakeup activation in wakeup_preempt().

Pass WF_ON_RQ to wakeup_preempt() from ttwu_runnable() so scheduling classes can handle already-runnable wakeups separately.

This is a preparatory change to support proxy execution with sched_ext. No functional change.

---
 kernel/sched/core.c  | 2 +-
 kernel/sched/sched.h | 1 +
 2 files changed, 2 insertions(+), 1 deletion(-)
kernel/sched/core.c +1 -1
index d578dd635f2eb..f142ab455b797 100644
@@ -3908,7 +3908,7 @@ static int ttwu_runnable(struct task_struct *p, int wake_flags)
             * When on_rq && !on_cpu the task is preempted, see if
             * it should preempt the task that is current now.
             */
-           wakeup_preempt(rq, p, wake_flags);
+           wakeup_preempt(rq, p, wake_flags | WF_ON_RQ);
    }
    ttwu_do_wakeup(p);
    return 1;
kernel/sched/sched.h +0 -0
index 56ac104f1f825..f30d122909e7b 100644

kernel/sched/sched.h static inline int task_on_rq_migrating(struct task_struct *p) · line 2541

Quoted 4 lines
kernel/sched/sched.h +1 -0
@@ -2538,6 +2538,7 @@ static inline int task_on_rq_migrating(struct task_struct *p)
 #define WF_MIGRATED                0x20 /* Internal use, task got migrated */
 #define WF_CURRENT_CPU             0x40 /* Prefer to move the wakee to the current CPU. */
 #define WF_RQ_SELECTED             0x80 /* ->select_task_rq() was called */
+#define WF_ON_RQ           0x100 /* Wakeup of an already runnable task */

I are confused again... help?

Quoted 8 lines

wakeup_preempt() can only ever be called for tasks that are on the runqueue. That is more or less the point of wakeup-preemption.

Even the already-runnable distinction would mean things like move_queued_task() should also have this. The task is new to this runqueue, but was definitely already runnable before.

What actual distinction are you needing?

Hm... yeah, WF_ON_RQ describes the wrong distinction.

What sched_ext actually needs to know is: did this wakeup use ttwu_runnable(), changing the task back to TASK_RUNNING without calling activate_task() / enqueue_task()?

Quoted 8 lines

We need the distinction because a retained proxy donor remains on the runqueue while blocked. When it wakes, ttwu_runnable() can clear its blocked state without calling enqueue_task_scx() again.

sched_ext must therefore request a reschedule so the newly unblocked task is reconsidered for dispatch. We don't want this extra reschedule for a normal wakeup because that path already called enqueue_task_scx() and performed the required sched_ext bookkeeping.

Would it be sufficient to just drop the if (task_cpu(p) == p->wake_cpu) shortcut in proxy_needs_return()? (or conditionalize it on the sched_class?)

thanks
-john

maintainer
#27 lore raw

Re: [PATCH 08/18] sched: Introduce WF_ON_RQ wake flag

Quoted 10 lines · John Stultz
Quoted 8 lines · Andrea Righi

What sched_ext actually needs to know is: did this wakeup use ttwu_runnable(), changing the task back to TASK_RUNNING without calling activate_task() / enqueue_task()?

We need the distinction because a retained proxy donor remains on the runqueue while blocked. When it wakes, ttwu_runnable() can clear its blocked state without calling enqueue_task_scx() again.

Quoted 8 lines
Quoted 4 lines

sched_ext must therefore request a reschedule so the newly unblocked task is reconsidered for dispatch. We don't want this extra reschedule for a normal wakeup because that path already called enqueue_task_scx() and performed the required sched_ext bookkeeping.

Would it be sufficient to just drop the if (task_cpu(p) == p->wake_cpu) shortcut in proxy_needs_return()? (or conditionalize it on the sched_class?)

One of the things I have on the TODO list was look at getting rid of p->se.sched_delayed usage in the core (That's bugged me ever since I introduced it.)

We're part-way there with the introduction of p->is_blocked.

The tentative plan -- but I've not tried, or even thought overly much about is -- is to do something like the *COMPLETELY*UNTESTED* below.

This would get the class methods {EN,DE}QUEUE_BLOCKED calls (which are unhandled except for fair, so that needs fixing at the very least).

I *think* this would allow ext to do the right thing, but I'm not saying we have to do this now, this might turn out to be a pain in the arse -- as these things tend to be.

---

kernel/sched/core.c +8 -11
index 91f059a55695..9a583e3cc528 100644
@@ -305,7 +305,7 @@ static inline int rb_sched_core_cmp(const void *key, const struct rb_node *node)
 
 void sched_core_enqueue(struct rq *rq, struct task_struct *p)
 {
-	if (p->se.sched_delayed)
+	if (p->is_blocked)
 		return;
 
 	rq->core->core_task_seq++;
@@ -318,7 +318,7 @@ void sched_core_enqueue(struct rq *rq, struct task_struct *p)
 
 void sched_core_dequeue(struct rq *rq, struct task_struct *p, int flags)
 {
-	if (p->se.sched_delayed)
+	if (p->is_blocked)
 		return;
 
 	rq->core->core_task_seq++;
@@ -1880,7 +1880,7 @@ static inline void uclamp_rq_inc(struct rq *rq, struct task_struct *p, int flags
 		return;
 
 	/* Only inc the delayed task which being woken up. */
-	if (p->se.sched_delayed && !(flags & ENQUEUE_DELAYED))
+	if (p->is_blocked && !(flags & ENQUEUE_BLOCKED))
 		return;
 
 	for_each_clamp_id(clamp_id)
@@ -1907,7 +1907,7 @@ static inline void uclamp_rq_dec(struct rq *rq, struct task_struct *p)
 	if (unlikely(!p->sched_class->uclamp_enabled))
 		return;
 
-	if (p->se.sched_delayed)
+	if (p->is_blocked)
 		return;
 
 	for_each_clamp_id(clamp_id)
@@ -2393,8 +2393,8 @@ unsigned long wait_task_inactive(struct task_struct *p, unsigned int match_state
 		 * If task is sched_delayed, force dequeue it, to avoid always
 		 * hitting the tick timeout in the queued case
 		 */
-		if (p->se.sched_delayed)
-			dequeue_task(rq, p, DEQUEUE_SLEEP | DEQUEUE_DELAYED);
+		if (p->is_blocked)
+			dequeue_task(rq, p, DEQUEUE_SLEEP | DEQUEUE_BLOCKED);
 		trace_sched_wait_task(p);
 		running = task_on_cpu(rq, p);
 		queued = task_on_rq_queued(p);
@@ -3889,8 +3889,7 @@ static int ttwu_runnable(struct task_struct *p, int wake_flags)
 
 	update_rq_clock(rq);
 	if (p->is_blocked) {
-		if (p->se.sched_delayed)
-			enqueue_task(rq, p, ENQUEUE_NOCLOCK | ENQUEUE_DELAYED);
+		enqueue_task(rq, p, ENQUEUE_NOCLOCK | ENQUEUE_BLOCKED);
 		if (proxy_needs_return(rq, p))
 			return 0;
 	}
@@ -4288,7 +4287,6 @@ int try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags)
 		 *  - we're serialized against set_special_state() by virtue of
 		 *    it disabling IRQs (this allows not taking ->pi_lock).
 		 */
-		WARN_ON_ONCE(p->se.sched_delayed);
 		WARN_ON_ONCE(p->is_blocked);
 		/* If p is current, we know we can run here, so clear blocked_on */
 		clear_task_blocked_on(p, NULL);
@@ -4592,7 +4590,6 @@ static void __sched_fork(u64 clone_flags, struct task_struct *p)
 	INIT_LIST_HEAD(&p->se.group_node);
 
 	/* A delayed task cannot be in clone(). */
-	WARN_ON_ONCE(p->se.sched_delayed);
 	WARN_ON_ONCE(p->is_blocked);
 
 #ifdef CONFIG_FAIR_GROUP_SCHED
@@ -7288,7 +7285,7 @@ static void __sched notrace __schedule(int sched_mode)
 
 		psi_account_irqtime(rq, prev, next);
 		psi_sched_switch(prev, next, !task_on_rq_queued(prev) ||
-					     prev->se.sched_delayed);
+					     prev->is_blocked);
 
 		trace_sched_switch(preempt, prev, next, prev_state);
 
kernel/sched/fair.c +13 -11
index 4d0b94465d19..ad72f428565f 100644
@@ -6439,7 +6439,7 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
 		if (task_on_rq_migrating(task_of(se)))
 			action |= DO_DETACH;
 
-		if ((flags & DEQUEUE_SLEEP) && !(flags & DEQUEUE_DELAYED))
+		if ((flags & DEQUEUE_SLEEP) && !(flags & DEQUEUE_BLOCKED))
 			action |= UPDATE_UTIL_EST;
 	}
 
@@ -6525,7 +6525,7 @@ pick_next_entity(struct rq *rq, bool protect)
 
 	se = pick_eevdf(cfs_rq, protect);
 	if (se->sched_delayed) {
-		__dequeue_task(rq, task_of(se), DEQUEUE_SLEEP | DEQUEUE_DELAYED);
+		__dequeue_task(rq, task_of(se), DEQUEUE_SLEEP | DEQUEUE_BLOCKED);
 		/*
 		 * Must not reference @se again, see __block_task().
 		 */
@@ -8007,13 +8007,14 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags)
 	 * Let's add the task's estimated utilization to the cfs_rq's
 	 * estimated utilization, before we update schedutil.
 	 */
-	if (!p->se.sched_delayed || (flags & ENQUEUE_DELAYED))
+	if (!p->se.sched_delayed || (flags & ENQUEUE_BLOCKED))
 		util_est_enqueue(cfs_rq, p);
 
 	update_curr_eevdf(cfs_rq);
 
-	if (flags & ENQUEUE_DELAYED) {
-		requeue_delayed_entity(cfs_rq, se);
+	if (flags & ENQUEUE_BLOCKED) {
+		if (se->sched_delayed)
+			requeue_delayed_entity(cfs_rq, se);
 		return;
 	}
 
@@ -8075,7 +8076,7 @@ static void dequeue_hierarchy(struct task_struct *p, int flags)
 {
 	struct sched_entity *se = &p->se;
 	bool task_sleep = flags & DEQUEUE_SLEEP;
-	bool task_delayed = flags & DEQUEUE_DELAYED;
+	bool task_delayed = flags & DEQUEUE_BLOCKED;
 	bool task_throttled = flags & DEQUEUE_THROTTLE;
 	int h_nr_runnable = 0;
 	int h_nr_idle = task_has_idle_policy(p);
@@ -8111,7 +8112,7 @@ static void dequeue_hierarchy(struct task_struct *p, int flags)
 			record_throttle_clock(cfs_rq);
 
 		flags |= DEQUEUE_SLEEP;
-		flags &= ~(DEQUEUE_DELAYED | DEQUEUE_SPECIAL);
+		flags &= ~(DEQUEUE_BLOCKED | DEQUEUE_SPECIAL);
 	}
 }
 
@@ -8128,15 +8129,16 @@ static bool __dequeue_task(struct rq *rq, struct task_struct *p, int flags)
 	struct cfs_rq *cfs_rq = &rq->cfs;
 	bool was_sched_idle = sched_idle_rq(rq);
 	bool task_sleep = flags & DEQUEUE_SLEEP;
-	bool task_delayed = flags & DEQUEUE_DELAYED;
+	bool task_delayed = flags & DEQUEUE_BLOCKED;
 
 	clear_buddies(cfs_rq, se);
 
 	update_curr_eevdf(cfs_rq);
 	update_entity_lag(cfs_rq, se);
 
-	if (flags & DEQUEUE_DELAYED) {
-		WARN_ON_ONCE(!se->sched_delayed);
+	if (flags & DEQUEUE_BLOCKED) {
+		if (!se->sched_delayed)
+			return true;
 	} else {
 		bool delay = task_sleep;
 		/*
@@ -15213,7 +15215,7 @@ static void attach_task_cfs_rq(struct task_struct *p)
 static void switching_from_fair(struct rq *rq, struct task_struct *p)
 {
 	if (p->se.sched_delayed)
-		dequeue_task(rq, p, DEQUEUE_SLEEP | DEQUEUE_DELAYED | DEQUEUE_NOCLOCK);
+		dequeue_task(rq, p, DEQUEUE_SLEEP | DEQUEUE_BLOCKED | DEQUEUE_NOCLOCK);
 }
 
 static void switched_from_fair(struct rq *rq, struct task_struct *p)
kernel/sched/sched.h +3 -3
index 6c3ad70e58b8..bd02ae83719c 100644
@@ -2578,7 +2578,7 @@ extern const u32		sched_prio_to_wmult[40];
  *
  * MIGRATION - p->on_rq == TASK_ON_RQ_MIGRATING (used for DEADLINE)
  *
- * DELAYED - de/re-queue a sched_delayed task
+ * BLOCKED - de/re-queue a sched_delayed task
  *
  * CLASS - going to update p->sched_class; makes sched_change call the
  *         various switch methods.
@@ -2598,7 +2598,7 @@ extern const u32		sched_prio_to_wmult[40];
 #define DEQUEUE_NOCLOCK		0x0008 /* Matches ENQUEUE_NOCLOCK */
 
 #define DEQUEUE_MIGRATING	0x0010 /* Matches ENQUEUE_MIGRATING */
-#define DEQUEUE_DELAYED		0x0020 /* Matches ENQUEUE_DELAYED */
+#define DEQUEUE_BLOCKED		0x0020 /* Matches ENQUEUE_DELAYED */
 #define DEQUEUE_CLASS		0x0040 /* Matches ENQUEUE_CLASS */
 
 #define DEQUEUE_SPECIAL		0x00010000
@@ -2610,7 +2610,7 @@ extern const u32		sched_prio_to_wmult[40];
 #define ENQUEUE_NOCLOCK		0x0008
 
 #define ENQUEUE_MIGRATING	0x0010
-#define ENQUEUE_DELAYED		0x0020
+#define ENQUEUE_BLOCKED		0x0020
 #define ENQUEUE_CLASS		0x0040
 
 #define ENQUEUE_HEAD		0x00010000
#28 lore raw

Re: [PATCH 08/18] sched: Introduce WF_ON_RQ wake flag

Hello Peter,

Quoted 14 lines · Peter Zijlstra
kernel/sched/core.c +1 -1
index 91f059a55695..9a583e3cc528 100644
@@ -305,7 +305,7 @@ static inline int rb_sched_core_cmp(const void *key, const struct rb_node *node)
 
 void sched_core_enqueue(struct rq *rq, struct task_struct *p)
 {
-	if (p->se.sched_delayed)
+	if (p->is_blocked)
 		return;
 
 	rq->core->core_task_seq++;

kernel/sched/core.c void sched_core_enqueue(struct rq *rq, struct task_struct *p) · line 324

Quoted 8 lines
kernel/sched/core.c +1 -1
@@ -318,7 +318,7 @@ void sched_core_enqueue(struct rq *rq, struct task_struct *p)
 
 void sched_core_dequeue(struct rq *rq, struct task_struct *p, int flags)
 {
-	if (p->se.sched_delayed)
+	if (p->is_blocked)
 		return;
 
 	rq->core->core_task_seq++;

Although we don't support proxy with mis-matched cookie, this might still be required down the line when we add support for it.

I can add it back then if that is a better option.

Quoted 19 lines
kernel/sched/core.c +2 -2
@@ -1880,7 +1880,7 @@ static inline void uclamp_rq_inc(struct rq *rq, struct task_struct *p, int flags
 		return;
 
 	/* Only inc the delayed task which being woken up. */
-	if (p->se.sched_delayed && !(flags & ENQUEUE_DELAYED))
+	if (p->is_blocked && !(flags & ENQUEUE_BLOCKED))
 		return;
 
 	for_each_clamp_id(clamp_id)
@@ -1907,7 +1907,7 @@ static inline void uclamp_rq_dec(struct rq *rq, struct task_struct *p)
 	if (unlikely(!p->sched_class->uclamp_enabled))
 		return;
 
-	if (p->se.sched_delayed)
+	if (p->is_blocked)
 		return;
 
 	for_each_clamp_id(clamp_id)

kernel/sched/core.c unsigned long wait_task_inactive(struct task_struct *p, unsigned int match_state · line 2397

Quoted 7 lines
kernel/sched/core.c +2 -2
@@ -2393,8 +2393,8 @@ unsigned long wait_task_inactive(struct task_struct *p, unsigned int match_state
 		 * If task is sched_delayed, force dequeue it, to avoid always
 		 * hitting the tick timeout in the queued case
 		 */
-		if (p->se.sched_delayed)
-			dequeue_task(rq, p, DEQUEUE_SLEEP | DEQUEUE_DELAYED);
+		if (p->is_blocked)
+			dequeue_task(rq, p, DEQUEUE_SLEEP | DEQUEUE_BLOCKED);

For proxy donors, this will simply dequeue the task without transitioning p->on_rq to 0.

If we want to break proxy at this point, maybe we want to bring that __block_task() into dequeue task and do it for DEQUEUE_BLOCKED?

kernel/sched/core.c static int ttwu_runnable(struct task_struct *p, int wake_flags) · line 3892

Quoted 10 lines
kernel/sched/core.c +1 -2
 		trace_sched_wait_task(p);
 		running = task_on_cpu(rq, p);
 		queued = task_on_rq_queued(p);
@@ -3889,8 +3889,7 @@ static int ttwu_runnable(struct task_struct *p, int wake_flags)
 
 	update_rq_clock(rq);
 	if (p->is_blocked) {
-		if (p->se.sched_delayed)
-			enqueue_task(rq, p, ENQUEUE_NOCLOCK | ENQUEUE_DELAYED);
+		enqueue_task(rq, p, ENQUEUE_NOCLOCK | ENQUEUE_BLOCKED);

For proxy donors on rq that don't belong to fair, this will enqueue an already enqueued task.

Although the sched call callbacks may be able to handle it, psi_enqueue() will do double accounting since block_task() was skipped for proxy donors.

Quoted 3 lines
kernel/sched/core.c +0 -0
 		if (proxy_needs_return(rq, p))
 			return 0;
 	}
Signature · 2 lines
Thanks and Regards,
Prateek
maintainer
#29 lore raw

Re: [PATCH 08/18] sched: Introduce WF_ON_RQ wake flag

Quoted 21 lines · K Prateek Nayak

kernel/sched/core.c unsigned long wait_task_inactive(struct task_struct *p, unsigned int match_state · line 2397

Quoted 8 lines
kernel/sched/core.c +2 -2
@@ -2393,8 +2393,8 @@ unsigned long wait_task_inactive(struct task_struct *p, unsigned int match_state
 		 * If task is sched_delayed, force dequeue it, to avoid always
 		 * hitting the tick timeout in the queued case
 		 */
-		if (p->se.sched_delayed)
-			dequeue_task(rq, p, DEQUEUE_SLEEP | DEQUEUE_DELAYED);
+		if (p->is_blocked)
+			dequeue_task(rq, p, DEQUEUE_SLEEP | DEQUEUE_BLOCKED);

For proxy donors, this will simply dequeue the task without transitioning p->on_rq to 0.

If we want to break proxy at this point, maybe we want to bring that __block_task() into dequeue task and do it for DEQUEUE_BLOCKED?

Quoted 5 lines
kernel/sched/core.c +0 -0
 		trace_sched_wait_task(p);
 		running = task_on_cpu(rq, p);
 		queued = task_on_rq_queued(p);
@@ -3889,8 +3889,7 @@ static int ttwu_runnable(struct task_struct *p, int wake_flags)
 
Quoted 8 lines

kernel/sched/core.c static int ttwu_runnable(struct task_struct *p, int wake_flags) · line 3892

Quoted 5 lines
kernel/sched/core.c +1 -2
 	update_rq_clock(rq);
 	if (p->is_blocked) {
-		if (p->se.sched_delayed)
-			enqueue_task(rq, p, ENQUEUE_NOCLOCK | ENQUEUE_DELAYED);
+		enqueue_task(rq, p, ENQUEUE_NOCLOCK | ENQUEUE_BLOCKED);

For proxy donors on rq that don't belong to fair, this will enqueue an already enqueued task.

For both the idea would be that {EN,DE}QUEUE_BLOCKED would be ignored when inappropriate.

Eg. fair will no-op when !sched_delayed.

I did mention none of the other classes behave properly in this regard, although the note was perhaps a little cryptic.

#30 lore raw

Re: [PATCH 08/18] sched: Introduce WF_ON_RQ wake flag

Quoted 32 lines · Peter Zijlstra
Quoted 21 lines · K Prateek Nayak

kernel/sched/core.c unsigned long wait_task_inactive(struct task_struct *p, unsigned int match_state · line 2397

Quoted 8 lines
kernel/sched/core.c +2 -2
@@ -2393,8 +2393,8 @@ unsigned long wait_task_inactive(struct task_struct *p, unsigned int match_state
 		 * If task is sched_delayed, force dequeue it, to avoid always
 		 * hitting the tick timeout in the queued case
 		 */
-		if (p->se.sched_delayed)
-			dequeue_task(rq, p, DEQUEUE_SLEEP | DEQUEUE_DELAYED);
+		if (p->is_blocked)
+			dequeue_task(rq, p, DEQUEUE_SLEEP | DEQUEUE_BLOCKED);

For proxy donors, this will simply dequeue the task without transitioning p->on_rq to 0.

If we want to break proxy at this point, maybe we want to bring that __block_task() into dequeue task and do it for DEQUEUE_BLOCKED?

Quoted 5 lines
kernel/sched/core.c +0 -0
 		trace_sched_wait_task(p);
 		running = task_on_cpu(rq, p);
 		queued = task_on_rq_queued(p);
@@ -3889,8 +3889,7 @@ static int ttwu_runnable(struct task_struct *p, int wake_flags)
 
Quoted 8 lines

kernel/sched/core.c static int ttwu_runnable(struct task_struct *p, int wake_flags) · line 3892

Quoted 5 lines
kernel/sched/core.c +1 -2
 	update_rq_clock(rq);
 	if (p->is_blocked) {
-		if (p->se.sched_delayed)
-			enqueue_task(rq, p, ENQUEUE_NOCLOCK | ENQUEUE_DELAYED);
+		enqueue_task(rq, p, ENQUEUE_NOCLOCK | ENQUEUE_BLOCKED);

For proxy donors on rq that don't belong to fair, this will enqueue an already enqueued task.

Quoted 7 lines

For both the idea would be that {EN,DE}QUEUE_BLOCKED would be ignored when inappropriate.

Eg. fair will no-op when !sched_delayed.

I did mention none of the other classes behave properly in this regard, although the note was perhaps a little cryptic.

My bad! But what do you think about having a bitmap of allowed flags that lives in sched_class->flags and is checked at enqueue / dequeue?

Only the lower nibble is set conditionally for matching flags and the upper ones are set unconditionally. Only user would be fair for DEQUEUE_BLOCKED / ENQUEUE_BLOCKED but is kinda extensible if more class specific flags emerge.

Signature · 2 lines
Thanks and Regards,
Prateek
maintainer
#31 lore raw

Re: [PATCH 08/18] sched: Introduce WF_ON_RQ wake flag

Quoted 2 lines · K Prateek Nayak

My bad! But what do you think about having a bitmap of allowed flags that lives in sched_class->flags and is checked at enqueue / dequeue?

I've considered that a number of times, but never quite done it. If it helps sure.

author reviewer
#32 lore raw

PATCH 9/18 sched_ext: Block proxy donors across scheduler transitions

In reply to Andrea Righi

Proxy execution retains mutex-blocked donors on the runqueue so their scheduling context can execute a lock owner. sched_ext cannot safely retain such donors unless the BPF scheduler explicitly participates in their admission and ordering.

Make sched_ext reject retained donors by default. Implement scx_allow_proxy_exec() to force blocked EXT tasks through the regular block path in schedule().

Also fully deactivate any retained proxy donor in sched_change_begin() before changing its scheduling class. This prevents sched_setscheduler(), PI transitions and sched_ext activation from carrying an existing proxy session into the new class.

Some RT/DL PI transitions could usefully preserve the proxy session. For example, an RT waiter can boost a FAIR donor which is itself blocked on a non-PI mutex; retaining the donor would allow the boosted context to keep proxy-executing that mutex owner. Doing so safely requires defining which scheduling classes can carry retained proxy state across a transition. Keep the conservative reset for now and leave compatible RT/DL PI chains for future work.

This is a preparatory change to support proxy execution with sched_ext.

---
 kernel/sched/core.c    | 9 +++++++++
 kernel/sched/ext/ext.c | 2 +-
 2 files changed, 10 insertions(+), 1 deletion(-)
kernel/sched/core.c +9 -0
index f142ab455b797..4bd956182af8d 100644
@@ -11302,6 +11302,15 @@ sched_change_begin(struct task_struct *p, const struct sched_class *next_class,
 		flags |= DEQUEUE_NOCLOCK;
 	}
 
+	/*
+	 * Don't carry retained proxy state across scheduling class changes.
+	 * Compatible RT/DL PI transitions could preserve the session so that a
+	 * boosted donor continues proxy-executing its lock owner. Defining which
+	 * class transitions can safely retain that state is left for future work.
+	 */
+	if ((flags & DEQUEUE_CLASS) && next_class != p->sched_class)
+		sched_proxy_block_task(rq, p);
+
 	if ((flags & DEQUEUE_CLASS) && p->sched_class->switching_from)
 		p->sched_class->switching_from(rq, p);
 
kernel/sched/ext/ext.c +1 -1
index f3e59f4fad4ac..d5ed996ebfb4c 100644
@@ -26,7 +26,7 @@ DEFINE_RAW_SPINLOCK(scx_sched_lock);
 
 bool scx_allow_proxy_exec(const struct task_struct *p)
 {
-	return true;
+	return p->sched_class != &ext_sched_class;
 }
 
 /*
Signature · 1 line
2.55.0
maintainer
#33 lore raw

Re: [PATCH 09/18] sched_ext: Block proxy donors across scheduler transitions

Quoted 35 lines · Andrea Righi

Proxy execution retains mutex-blocked donors on the runqueue so their scheduling context can execute a lock owner. sched_ext cannot safely retain such donors unless the BPF scheduler explicitly participates in their admission and ordering.

Make sched_ext reject retained donors by default. Implement scx_allow_proxy_exec() to force blocked EXT tasks through the regular block path in schedule().

Also fully deactivate any retained proxy donor in sched_change_begin() before changing its scheduling class. This prevents sched_setscheduler(), PI transitions and sched_ext activation from carrying an existing proxy session into the new class.

Some RT/DL PI transitions could usefully preserve the proxy session. For example, an RT waiter can boost a FAIR donor which is itself blocked on a non-PI mutex; retaining the donor would allow the boosted context to keep proxy-executing that mutex owner. Doing so safely requires defining which scheduling classes can carry retained proxy state across a transition. Keep the conservative reset for now and leave compatible RT/DL PI chains for future work.

This is a preparatory change to support proxy execution with sched_ext.

---
 kernel/sched/core.c    | 9 +++++++++
 kernel/sched/ext/ext.c | 2 +-
 2 files changed, 10 insertions(+), 1 deletion(-)
kernel/sched/core.c +0 -0
index f142ab455b797..4bd956182af8d 100644

kernel/sched/core.c sched_change_begin(struct task_struct *p, const struct sched_class *next_class, · line 11311

Quoted 10 lines
kernel/sched/core.c +7 -0
@@ -11302,6 +11302,15 @@ sched_change_begin(struct task_struct *p, const struct sched_class *next_class,
 		flags |= DEQUEUE_NOCLOCK;
 	}
 
+	/*
+	 * Don't carry retained proxy state across scheduling class changes.
+	 * Compatible RT/DL PI transitions could preserve the session so that a
+	 * boosted donor continues proxy-executing its lock owner. Defining which
+	 * class transitions can safely retain that state is left for future work.
+	 */
+	if ((flags & DEQUEUE_CLASS) && next_class != p->sched_class)

That's a tautology, having DEQUEUE_CLASS means next_class != ->sched_class.

Quoted 10 lines
kernel/sched/core.c +2 -0
+		sched_proxy_block_task(rq, p);
+
 	if ((flags & DEQUEUE_CLASS) && p->sched_class->switching_from)
 		p->sched_class->switching_from(rq, p);
 
kernel/sched/ext/ext.c +0 -0
index f3e59f4fad4ac..d5ed996ebfb4c 100644

kernel/sched/ext/ext.c DEFINE_RAW_SPINLOCK(scx_sched_lock); · line 30

Quoted 6 lines
kernel/sched/ext/ext.c +1 -1
@@ -26,7 +26,7 @@ DEFINE_RAW_SPINLOCK(scx_sched_lock);
 
 bool scx_allow_proxy_exec(const struct task_struct *p)
 {
-	return true;
+	return p->sched_class != &ext_sched_class;
 }

Hmmmm... I don't like this. I was expecting this sched_proxy_block_task() thing to be called in the big switcheroo function when loading one of these ext thing. (scx_root_enable_ or somesuch).

We most certainly don't want to do this on every sched class change. It isn't even gated by scx_allow_proxy_exec().

maintainer
#34 lore raw

Re: [PATCH 09/18] sched_ext: Block proxy donors across scheduler transitions

Quoted 32 lines · Peter Zijlstra
Quoted 5 lines · Andrea Righi
kernel/sched/core.c +0 -0
index f142ab455b797..4bd956182af8d 100644

kernel/sched/core.c sched_change_begin(struct task_struct *p, const struct sched_class *next_class, · line 11311

Quoted 10 lines
kernel/sched/core.c +7 -0
@@ -11302,6 +11302,15 @@ sched_change_begin(struct task_struct *p, const struct sched_class *next_class,
 		flags |= DEQUEUE_NOCLOCK;
 	}
 
+	/*
+	 * Don't carry retained proxy state across scheduling class changes.
+	 * Compatible RT/DL PI transitions could preserve the session so that a
+	 * boosted donor continues proxy-executing its lock owner. Defining which
+	 * class transitions can safely retain that state is left for future work.
+	 */
+	if ((flags & DEQUEUE_CLASS) && next_class != p->sched_class)

That's a tautology, having DEQUEUE_CLASS means next_class != ->sched_class.

kernel/sched/core.c sched_change_begin(struct task_struct *p, const struct sched_class *next_class, · line 11316

Quoted 5 lines
kernel/sched/core.c +2 -0
+		sched_proxy_block_task(rq, p);
+
 	if ((flags & DEQUEUE_CLASS) && p->sched_class->switching_from)
 		p->sched_class->switching_from(rq, p);
 

Hmmmm... I don't like this. I was expecting this sched_proxy_block_task() thing to be called in the big switcheroo function when loading one of these ext thing. (scx_root_enable_ or somesuch).

Quoted 2 lines

We most certainly don't want to do this on every sched class change. It isn't even gated by scx_allow_proxy_exec().

Why not do:

switching_to_scx():

+ if (!scx_allow_proxy_exec(p))
+ sched_proxy_block_task(rq, p);

author reviewer
#35 lore raw

Re: [PATCH 09/18] sched_ext: Block proxy donors across scheduler transitions

Quoted 22 lines · Peter Zijlstra
Quoted 21 lines · Peter Zijlstra
Quoted 5 lines · Andrea Righi
kernel/sched/core.c +0 -0
index f142ab455b797..4bd956182af8d 100644

kernel/sched/core.c sched_change_begin(struct task_struct *p, const struct sched_class *next_class, · line 11311

Quoted 10 lines
kernel/sched/core.c +7 -0
@@ -11302,6 +11302,15 @@ sched_change_begin(struct task_struct *p, const struct sched_class *next_class,
 		flags |= DEQUEUE_NOCLOCK;
 	}
 
+	/*
+	 * Don't carry retained proxy state across scheduling class changes.
+	 * Compatible RT/DL PI transitions could preserve the session so that a
+	 * boosted donor continues proxy-executing its lock owner. Defining which
+	 * class transitions can safely retain that state is left for future work.
+	 */
+	if ((flags & DEQUEUE_CLASS) && next_class != p->sched_class)

That's a tautology, having DEQUEUE_CLASS means next_class != ->sched_class.

Quoted 10 lines
Quoted 10 lines

kernel/sched/core.c sched_change_begin(struct task_struct *p, const struct sched_class *next_class, · line 11316

Quoted 5 lines
kernel/sched/core.c +2 -0
+		sched_proxy_block_task(rq, p);
+
 	if ((flags & DEQUEUE_CLASS) && p->sched_class->switching_from)
 		p->sched_class->switching_from(rq, p);
 

Hmmmm... I don't like this. I was expecting this sched_proxy_block_task() thing to be called in the big switcheroo function when loading one of these ext thing. (scx_root_enable_ or somesuch).

Agreed. The cleanup is required when sched_ext takes ownership of a task, not for every scheduling-class transition.

Calling it from sched_change_begin() also breaks proxy exec across FAIR/RT/DL transitions. I'll move it into the sched_ext enable and task-ownership transition paths and test that approach.

Review

Quoted 10 lines
Quoted 3 lines

We most certainly don't want to do this on every sched class change. It isn't even gated by scx_allow_proxy_exec().

Why not do:

switching_to_scx():

(quoted diff) +2 -0
+	if (!scx_allow_proxy_exec(p))
+		sched_proxy_block_task(rq, p);

The scx_allow_proxy_exec() gate makes sense, but switching_to_scx() is called from sched_change_end(). At that point, sched_change_begin() has already saved ctx->queued = true for the retained donor and removed it from the old class.

Calling sched_proxy_block_task() there is too late: sched_change_end() would still see the saved ctx->queued value and enqueue the blocked task in to EXT. Also, p->sched_class already points to EXT even though the task has not yet been enqueued there, while sched_proxy_block_task() dequeues through the current class.

I think the cleanup needs to happen in the sched_ext ownership-transition path immediately before sched_change_begin(). In this way ctx->queued is recorded as false and the task is not restored into EXT.

I'll rework it along these lines and test the sched_ext enable / ownership-transition paths.

Thanks,
-Andrea

author reviewer
#36 lore raw

PATCH 10/18 sched_ext: Fix ops.running/stopping() pairing for proxy-exec donors

In reply to Andrea Righi

With proxy execution, pick_next_task() can select a blocked task as the scheduling context before find_proxy_task() resolves the execution context.

From the BPF scheduler perspective, that donor is running while its scheduling context drives the lock owner; ops.tick() and other accounting must therefore remain enclosed by a matching ops.running()/ops.stopping() session.

In this scenario, the session boundaries do not always match physical task switches. Keep the "running" session open when the same donor continues on the same CPU. When proxy execution migrates a donor's scheduling context to another CPU, end its running session on the source CPU and start a new session on the destination CPU only after proxy resolution succeeds. This prevents a failed resolution from exposing a provisional ops.running() event.

Track these sessions with a new SCX_TASK_RUN_TRACKED flag. The explicit running-state tracking is also required by later donor-based accounting:
it prevents an EXT owner executing for a non-EXT donor from being treated as the active EXT scheduling context when it is dequeued.

This is a preparatory change for enabling proxy execution together with sched_ext.

---
 include/linux/sched/ext.h                     |  1 +
 kernel/sched/ext/ext.c                        | 70 ++++++++++++++-----
 kernel/sched/ext/internal.h                   |  6 ++
 .../sched_ext/include/scx/enum_defs.autogen.h |  1 +
 4 files changed, 62 insertions(+), 16 deletions(-)
include/linux/sched/ext.h +1 -0
index 582d7cd4a9839..9912ad0c2d445 100644
@@ -104,6 +104,7 @@ enum scx_ent_flags {
 	SCX_TASK_SUB_INIT	= 1 << 4, /* task being initialized for a sub sched */
 	SCX_TASK_IMMED		= 1 << 5, /* task is on local DSQ with %SCX_ENQ_IMMED */
 	SCX_TASK_PROTECTED	= 1 << 6, /* slice and DSQ head position protected */
+	SCX_TASK_RUN_TRACKED	= 1 << 7, /* task is in an ops.running()/stopping() session */
 
 	/*
 	 * Bits 8 to 10 are used to carry task state:
kernel/sched/ext/ext.c +54 -16

109 lines not on this page · raw message

kernel/sched/ext/internal.h +6 -0
index 27bbf5e04d900..0d7efbee2862c 100644
@@ -449,6 +449,12 @@ struct sched_ext_ops {
 	 * Therefore, always use scx_bpf_task_cpu(@p) to determine the
 	 * target CPU the task is going to use.
 	 *
+	 * Under proxy execution, the BPF scheduler continues to observe the
+	 * donor as the current scheduling context. A blocked donor enters a
+	 * ->running()/->stopping() session while its scheduling context drives
+	 * the lock owner. The lock owner executing on its behalf is intentionally
+	 * not reported through these callbacks.
+	 *
 	 * See ->runnable() for explanation on the task state notifiers.
 	 */
 	void (*running)(struct task_struct *p);
tools/sched_ext/include/scx/enum_defs.autogen.h +1 -0
index 19aa1de3e7005..cccc0c3987b85 100644
@@ -101,6 +101,7 @@
 #define HAVE_SCX_TASK_SUB_INIT
 #define HAVE_SCX_TASK_IMMED
 #define HAVE_SCX_TASK_PROTECTED
+#define HAVE_SCX_TASK_RUN_TRACKED
 #define HAVE_SCX_TASK_STATE_SHIFT
 #define HAVE_SCX_TASK_STATE_BITS
 #define HAVE_SCX_TASK_STATE_MASK
Signature · 1 line
2.55.0
author reviewer
#37 lore raw

PATCH 11/18 sched_ext: Move reject DSQ draining into core

In reply to Andrea Righi

The reject DSQ drain will be generalized for placement failures outside sub-scheduler capability rejection. Move the drain into the core sched_ext implementation first so the subsequent behavioral changes can be reviewed separately. No functional change.

---
 kernel/sched/ext/ext.c | 49 ++++++++++++++++++++++++++++++++++++++++++
 kernel/sched/ext/sub.c | 45 --------------------------------------
 kernel/sched/ext/sub.h |  2 --
 3 files changed, 49 insertions(+), 47 deletions(-)
kernel/sched/ext/ext.c +49 -0
index 307d2080b967a..967bf31a32754 100644
@@ -4674,6 +4674,55 @@ static void process_deferred_reenq_users(struct rq *rq)
 	}
 }
 
+#ifdef CONFIG_EXT_SUB_SCHED
+/*
+ * Drain @rq->scx.reject_dsq, reenqueueing each task so the BPF re-decides
+ * from p->scx.reenq_reason_*.
+ *
+ * A task can be re-rejected repeatedly. The reenqueue is bounded per task in
+ * scx_do_enqueue_task(), which ejects the owning sub past SCX_REENQ_MAX_REPEAT.
+ * Rejection can't happen for root.
+ */
+static void scx_reenq_reject(struct rq *rq)
+{
+	LIST_HEAD(tasks);
+	struct task_struct *p, *n;
+
+	lockdep_assert_rq_held(rq);
+
+	if (!scx_has_subs() || list_empty(&rq->scx.reject_dsq.list))
+		return;
+
+	/*
+	 * Move to a private list so a task re-rejected by the
+	 * scx_do_enqueue_task() below isn't revisited this round.
+	 */
+	list_for_each_entry_safe(p, n, &rq->scx.reject_dsq.list, scx.dsq_list.node) {
+		/* migration_pending tasks should have bypassed to local DSQ */
+		if (WARN_ON_ONCE(p->migration_pending))
+			continue;
+
+		scx_dispatch_dequeue(rq, p);
+
+		if (WARN_ON_ONCE(p->scx.flags & SCX_TASK_REENQ_REASON_MASK))
+			p->scx.flags &= ~SCX_TASK_REENQ_REASON_MASK;
+		p->scx.flags |= SCX_TASK_REENQ_CAP;
+
+		list_add_tail(&p->scx.dsq_list.node, &tasks);
+	}
+
+	list_for_each_entry_safe(p, n, &tasks, scx.dsq_list.node) {
+		list_del_init(&p->scx.dsq_list.node);
+
+		scx_do_enqueue_task(rq, p, SCX_ENQ_REENQ, -1);
+
+		p->scx.flags &= ~SCX_TASK_REENQ_REASON_MASK;
+	}
+}
+#else
+static void scx_reenq_reject(struct rq *rq) {}
+#endif
+
 static void run_deferred(struct rq *rq)
 {
 	process_ddsp_deferred_locals(rq);
kernel/sched/ext/sub.c +0 -45

51 lines not on this page · raw message

kernel/sched/ext/sub.h +0 -2
index f7bcdfda8dd85..8f2425bdb9530 100644
@@ -37,7 +37,6 @@ void scx_discard_stale_ecaps_syncs(void);
 struct scx_dispatch_q *scx_resolve_local_dsq(struct scx_sched *sch, struct rq *rq,
 					     struct task_struct *p, u64 *enq_flags);
 bool scx_task_reenq_on_cap_revoke(struct rq *rq, struct task_struct *p);
-void scx_reenq_reject(struct rq *rq);
 void scx_rescue_charge(struct rq *rq, s64 delta_exec);
 void scx_rescue_end(struct rq *rq);
 bool scx_rescue_keep(struct rq *rq, struct task_struct *p);
@@ -93,7 +92,6 @@ static inline void scx_discard_ecaps_to_sync(s32 cpu, struct scx_sched_pcpu *pcp
 static inline void scx_discard_stale_ecaps_syncs(void) {}
 static inline struct scx_dispatch_q *scx_resolve_local_dsq(struct scx_sched *sch, struct rq *rq, struct task_struct *p, u64 *enq_flags) { return &rq->scx.local_dsq; }
 static inline bool scx_task_reenq_on_cap_revoke(struct rq *rq, struct task_struct *p) { return false; }
-static inline void scx_reenq_reject(struct rq *rq) {}
 static inline void scx_rescue_charge(struct rq *rq, s64 delta_exec) {}
 static inline void scx_rescue_end(struct rq *rq) {}
 static inline bool scx_rescue_keep(struct rq *rq, struct task_struct *p) { return false; }
Signature · 1 line
2.55.0
author reviewer
#38 lore raw

PATCH 12/18 sched_ext: Generalize the reject DSQ reenqueue path

In reply to Andrea Righi

The reject DSQ is currently specific to sub-scheduler cap failures. Its storage and initialization depend on CONFIG_EXT_SUB_SCHED, and the drain path assumes every rejected task was rejected for SCX_TASK_REENQ_CAP.

Other transient placement failures need the same ability to park a task on its source rq and return it to the owning BPF scheduler. Make the reject DSQ unconditional and carry the reenqueue reason directly in p->scx.flags from the rejection site.

A reenqueue reason remains valid while ops.enqueue() runs so that BPF can inspect it. Clear it immediately after the callback returns and before resolving a direct dispatch. Paths which bypass ops.enqueue() clear the reason before the kernel-selected placement. A rejection from the new placement can therefore install its reason into a clear field.

This is a preparatory change to support proxy execution with sched_ext.

---
 kernel/sched/ext/ext.c | 52 ++++++++++++++++++------------------------
 kernel/sched/ext/sub.c |  2 ++
 kernel/sched/sched.h   |  2 +-
 3 files changed, 25 insertions(+), 31 deletions(-)
kernel/sched/ext/ext.c +22 -30
index 967bf31a32754..2c5ae2811c77e 100644
@@ -1599,11 +1599,10 @@ static void scx_dispatch_enqueue(struct scx_sched *sch, struct rq *rq,
 				 struct scx_dispatch_q *dsq, struct task_struct *p,
 				 u64 slice, u64 vtime, u64 enq_flags)
 {
-	bool is_rq_owned = false;
+	bool is_rq_owned = dsq_is_rq_owned(dsq);
 
 	if (dsq->id == SCX_DSQ_LOCAL) {
 		dsq = scx_resolve_local_dsq(sch, rq, p, &enq_flags);
-		is_rq_owned = true;
 	}
 
 	WARN_ON_ONCE(p->scx.dsq || !list_empty(&p->scx.dsq_list.node));
@@ -2017,6 +2016,7 @@ void scx_do_enqueue_task(struct rq *rq, struct task_struct *p, u64 enq_flags,
 			__scx_exit(sch, SCX_EXIT_ERROR_REENQ, 0, cpu_of(rq),
 				   "%s[%d] reenqueued %u times without running",
 				   p->comm, p->pid, p->scx.reenq_cnt);
+			p->scx.flags &= ~SCX_TASK_REENQ_REASON_MASK;
 			return;
 		}
 	}
@@ -2067,6 +2067,8 @@ void scx_do_enqueue_task(struct rq *rq, struct task_struct *p, u64 enq_flags,
 
 	SCX_CALL_OP_TASK(sch, enqueue, rq, p, enq_flags);
 
+	/* The reason is input to ops.enqueue(), not to the resulting placement. */
+	p->scx.flags &= ~SCX_TASK_REENQ_REASON_MASK;
 	*ddsp_taskp = NULL;
 	if (p->scx.ddsp_dsq_id != SCX_DSQ_INVALID)
 		goto direct;
@@ -2085,9 +2087,11 @@ void scx_do_enqueue_task(struct rq *rq, struct task_struct *p, u64 enq_flags,
 	return;
 
 direct:
+	p->scx.flags &= ~SCX_TASK_REENQ_REASON_MASK;
 	direct_dispatch(sch, p, enq_flags);
 	return;
 local_norefill:
+	p->scx.flags &= ~SCX_TASK_REENQ_REASON_MASK;
 	scx_dispatch_enqueue(sch, rq, &rq->scx.local_dsq, p, 0, 0, enq_flags);
 	return;
 local:
@@ -2101,6 +2105,7 @@ void scx_do_enqueue_task(struct rq *rq, struct task_struct *p, u64 enq_flags,
 	goto enqueue;
 
 enqueue:
+	p->scx.flags &= ~SCX_TASK_REENQ_REASON_MASK;
 	refill_task_slice_dfl(sch, p);
 	clear_direct_dispatch(p);
 	scx_dispatch_enqueue(sch, rq, dsq, p, 0, 0, enq_flags);
@@ -2323,6 +2328,7 @@ static bool dequeue_task_scx(struct rq *rq, struct task_struct *p, int core_deq_
 	sub_nr_running(rq, 1);
 
 	scx_dispatch_dequeue(rq, p);
+	p->scx.flags &= ~SCX_TASK_REENQ_REASON_MASK;
 
 	/* see scx_task_slice_ended() for the save/restore exception */
 	if (!((deq_flags & DEQUEUE_SAVE) && task_current(rq, p)))
@@ -3034,6 +3040,7 @@ static void set_next_task_scx(struct rq *rq, struct task_struct *p, bool first)
 		 */
 		ops_dequeue(rq, p, SCX_DEQ_CORE_SCHED_EXEC);
 		scx_dispatch_dequeue(rq, p);
+		p->scx.flags &= ~SCX_TASK_REENQ_REASON_MASK;
 	}
 
 	p->se.exec_start = rq_clock_task(rq);
@@ -3208,7 +3215,6 @@ static void put_prev_task_scx(struct rq *rq, struct task_struct *p,
 			if (p->scx.flags & SCX_TASK_IMMED) {
 				p->scx.flags |= SCX_TASK_REENQ_PREEMPTED;
 				scx_do_enqueue_task(rq, p, SCX_ENQ_REENQ, -1);
-				p->scx.flags &= ~SCX_TASK_REENQ_REASON_MASK;
 			} else {
 				u64 enq_flags = 0;
 
@@ -4481,8 +4487,7 @@ static u32 reenq_local(struct scx_sched *sch, struct rq *rq, u64 reenq_flags)
 
 		scx_dispatch_dequeue(rq, p);
 
-		if (WARN_ON_ONCE(p->scx.flags & SCX_TASK_REENQ_REASON_MASK))
-			p->scx.flags &= ~SCX_TASK_REENQ_REASON_MASK;
+		WARN_ON_ONCE(p->scx.flags & SCX_TASK_REENQ_REASON_MASK);
 		p->scx.flags |= reason;
 
 		list_add_tail(&p->scx.dsq_list.node, &tasks);
@@ -4493,7 +4498,6 @@ static u32 reenq_local(struct scx_sched *sch, struct rq *rq, u64 reenq_flags)
 
 		scx_do_enqueue_task(rq, p, SCX_ENQ_REENQ, -1);
 
-		p->scx.flags &= ~SCX_TASK_REENQ_REASON_MASK;
 		nr_enqueued++;
 	}
 
@@ -4605,14 +4609,11 @@ static void reenq_user(struct rq *rq, struct scx_dispatch_q *dsq, u64 reenq_flag
 		dispatch_dequeue_locked(p, dsq);
 		raw_spin_unlock(&dsq->lock);
 
-		if (WARN_ON_ONCE(p->scx.flags & SCX_TASK_REENQ_REASON_MASK))
-			p->scx.flags &= ~SCX_TASK_REENQ_REASON_MASK;
+		WARN_ON_ONCE(p->scx.flags & SCX_TASK_REENQ_REASON_MASK);
 		p->scx.flags |= reason;
 
 		scx_do_enqueue_task(task_rq, p, SCX_ENQ_REENQ, -1);
 
-		p->scx.flags &= ~SCX_TASK_REENQ_REASON_MASK;
-
 		if (!(++nr_enqueued % SCX_TASK_ITER_BATCH)) {
 			scx_rq_lock_drop(locked_rq);
 			raw_spin_rq_unlock(locked_rq);
@@ -4674,14 +4675,14 @@ static void process_deferred_reenq_users(struct rq *rq)
 	}
 }
 
-#ifdef CONFIG_EXT_SUB_SCHED
 /*
- * Drain @rq->scx.reject_dsq, reenqueueing each task so the BPF re-decides
- * from p->scx.reenq_reason_*.
+ * Drain @rq->scx.reject_dsq and reenqueue each task so that its owning BPF
+ * scheduler chooses placement again.
  *
- * A task can be re-rejected repeatedly. The reenqueue is bounded per task in
- * scx_do_enqueue_task(), which ejects the owning sub past SCX_REENQ_MAX_REPEAT.
- * Rejection can't happen for root.
+ * A task can be re-rejected repeatedly. Reenqueues are bounded per task by
+ * SCX_REENQ_MAX_REPEAT in scx_do_enqueue_task(), which ejects the owning
+ * scheduler. The private list below prevents a task from being revisited in
+ * the same round.
  */
 static void scx_reenq_reject(struct rq *rq)
 {
@@ -4690,24 +4691,20 @@ static void scx_reenq_reject(struct rq *rq)
 
 	lockdep_assert_rq_held(rq);
 
-	if (!scx_has_subs() || list_empty(&rq->scx.reject_dsq.list))
+	if (list_empty(&rq->scx.reject_dsq.list))
 		return;
 
 	/*
-	 * Move to a private list so a task re-rejected by the
+	 * Move tasks to a private list so a task re-rejected by
 	 * scx_do_enqueue_task() below isn't revisited this round.
 	 */
 	list_for_each_entry_safe(p, n, &rq->scx.reject_dsq.list, scx.dsq_list.node) {
 		/* migration_pending tasks should have bypassed to local DSQ */
-		if (WARN_ON_ONCE(p->migration_pending))
-			continue;
+		WARN_ON_ONCE(p->migration_pending);
+		WARN_ON_ONCE(!(p->scx.flags & SCX_TASK_REENQ_REASON_MASK));
 
 		scx_dispatch_dequeue(rq, p);
 
-		if (WARN_ON_ONCE(p->scx.flags & SCX_TASK_REENQ_REASON_MASK))
-			p->scx.flags &= ~SCX_TASK_REENQ_REASON_MASK;
-		p->scx.flags |= SCX_TASK_REENQ_CAP;
-
 		list_add_tail(&p->scx.dsq_list.node, &tasks);
 	}
 
@@ -4715,13 +4712,8 @@ static void scx_reenq_reject(struct rq *rq)
 		list_del_init(&p->scx.dsq_list.node);
 
 		scx_do_enqueue_task(rq, p, SCX_ENQ_REENQ, -1);
-
-		p->scx.flags &= ~SCX_TASK_REENQ_REASON_MASK;
 	}
 }
-#else
-static void scx_reenq_reject(struct rq *rq) {}
-#endif
 
 static void run_deferred(struct rq *rq)
 {
@@ -8780,8 +8772,8 @@ void __init init_sched_ext_class(void)
 
 		/* local_dsq's sch will be set during scx_root_enable() */
 		BUG_ON(scx_init_dsq(&rq->scx.local_dsq, SCX_DSQ_LOCAL, NULL));
-#ifdef CONFIG_EXT_SUB_SCHED
 		BUG_ON(scx_init_dsq(&rq->scx.reject_dsq, SCX_DSQ_REJECT, NULL));
+#ifdef CONFIG_EXT_SUB_SCHED
 		scx_rescue_init(rq);
 #endif
 
kernel/sched/ext/sub.c +2 -0
index a73593e828267..40a79028de8e6 100644
@@ -752,6 +752,8 @@ struct scx_dispatch_q *scx_resolve_local_dsq(struct scx_sched *sch, struct rq *r
 
 	p->scx.reenq_reason_caps = missing;
 	p->scx.reenq_reason_cid = cid;
+	WARN_ON_ONCE(p->scx.flags & SCX_TASK_REENQ_REASON_MASK);
+	p->scx.flags |= SCX_TASK_REENQ_CAP;
 
 	return &rq->scx.reject_dsq;
 }
kernel/sched/sched.h +1 -1
index f30d122909e7b..43f04989557b5 100644
@@ -810,8 +810,8 @@ struct scx_rq_rescue {
 
 struct scx_rq {
 	struct scx_dispatch_q	local_dsq;
+	struct scx_dispatch_q	reject_dsq;		/* staging for rejected tasks */
 #ifdef CONFIG_EXT_SUB_SCHED
-	struct scx_dispatch_q	reject_dsq;		/* staging for cap-rejected tasks */
 	struct scx_rq_rescue	rescue;
 #endif
 	struct list_head	runnable_list;		/* runnable tasks on this rq */
Signature · 1 line
2.55.0
reviewer
#39 lore raw

Re: [PATCH 12/18] sched_ext: Generalize the reject DSQ reenqueue path

Hello,

Nits follow.

kernel/sched/ext/ext.c static void scx_dispatch_enqueue(struct scx_sched *sch, struct rq *rq, · line 1606

Quoted 11 lines · Andrea Righi
kernel/sched/ext/ext.c +1 -2
@@ -1599,11 +1599,10 @@ static void scx_dispatch_enqueue(struct scx_sched *sch, struct rq *rq,
 				 struct scx_dispatch_q *dsq, struct task_struct *p,
 				 u64 slice, u64 vtime, u64 enq_flags)
 {
-	bool is_rq_owned = false;
+	bool is_rq_owned = dsq_is_rq_owned(dsq);
 
 	if (dsq->id == SCX_DSQ_LOCAL) {
 		dsq = scx_resolve_local_dsq(sch, rq, p, &enq_flags);
-		is_rq_owned = true;
 	}

No need for {}.

Quoted 30 lines
kernel/sched/ext/ext.c +5 -0
@@ -2017,6 +2016,7 @@ void scx_do_enqueue_task(struct rq *rq, struct task_struct *p, u64 enq_flags,
 			__scx_exit(sch, SCX_EXIT_ERROR_REENQ, 0, cpu_of(rq),
 				   "%s[%d] reenqueued %u times without running",
 				   p->comm, p->pid, p->scx.reenq_cnt);
+			p->scx.flags &= ~SCX_TASK_REENQ_REASON_MASK;
 			return;
 		}
 	}
@@ -2067,6 +2067,8 @@ void scx_do_enqueue_task(struct rq *rq, struct task_struct *p, u64 enq_flags,
 
 	SCX_CALL_OP_TASK(sch, enqueue, rq, p, enq_flags);
 
+	/* The reason is input to ops.enqueue(), not to the resulting placement. */
+	p->scx.flags &= ~SCX_TASK_REENQ_REASON_MASK;
 	*ddsp_taskp = NULL;
 	if (p->scx.ddsp_dsq_id != SCX_DSQ_INVALID)
 		goto direct;
@@ -2085,9 +2087,11 @@ void scx_do_enqueue_task(struct rq *rq, struct task_struct *p, u64 enq_flags,
 	return;
 
 direct:
+	p->scx.flags &= ~SCX_TASK_REENQ_REASON_MASK;
 	direct_dispatch(sch, p, enq_flags);
 	return;
 local_norefill:
+	p->scx.flags &= ~SCX_TASK_REENQ_REASON_MASK;
 	scx_dispatch_enqueue(sch, rq, &rq->scx.local_dsq, p, 0, 0, enq_flags);
 	return;
 local:

kernel/sched/ext/ext.c void scx_do_enqueue_task(struct rq *rq, struct task_struct *p, u64 enq_flags, · line 2111

Quoted 7 lines
kernel/sched/ext/ext.c +1 -0
@@ -2101,6 +2105,7 @@ void scx_do_enqueue_task(struct rq *rq, struct task_struct *p, u64 enq_flags,
 	goto enqueue;
 
 enqueue:
+	p->scx.flags &= ~SCX_TASK_REENQ_REASON_MASK;
 	refill_task_slice_dfl(sch, p);
 	clear_direct_dispatch(p);
 	scx_dispatch_enqueue(sch, rq, dsq, p, 0, 0, enq_flags);

This is too messy. The clearing is scattered all over and sometimes done twice. Can't you add one label that everyone jumps to for exit and clear it there?

Thanks.

Signature · 1 line
tejun
author reviewer
#40 lore raw

Re: [PATCH 12/18] sched_ext: Generalize the reject DSQ reenqueue path

Hi Tejun,

Quoted 61 lines · Tejun Heo

Hello,

Nits follow.

kernel/sched/ext/ext.c static void scx_dispatch_enqueue(struct scx_sched *sch, struct rq *rq, · line 1606

Quoted 11 lines · Andrea Righi
kernel/sched/ext/ext.c +1 -2
@@ -1599,11 +1599,10 @@ static void scx_dispatch_enqueue(struct scx_sched *sch, struct rq *rq,
 				 struct scx_dispatch_q *dsq, struct task_struct *p,
 				 u64 slice, u64 vtime, u64 enq_flags)
 {
-	bool is_rq_owned = false;
+	bool is_rq_owned = dsq_is_rq_owned(dsq);
 
 	if (dsq->id == SCX_DSQ_LOCAL) {
 		dsq = scx_resolve_local_dsq(sch, rq, p, &enq_flags);
-		is_rq_owned = true;
 	}

No need for {}.

Quoted 30 lines
kernel/sched/ext/ext.c +5 -0
@@ -2017,6 +2016,7 @@ void scx_do_enqueue_task(struct rq *rq, struct task_struct *p, u64 enq_flags,
 			__scx_exit(sch, SCX_EXIT_ERROR_REENQ, 0, cpu_of(rq),
 				   "%s[%d] reenqueued %u times without running",
 				   p->comm, p->pid, p->scx.reenq_cnt);
+			p->scx.flags &= ~SCX_TASK_REENQ_REASON_MASK;
 			return;
 		}
 	}
@@ -2067,6 +2067,8 @@ void scx_do_enqueue_task(struct rq *rq, struct task_struct *p, u64 enq_flags,
 
 	SCX_CALL_OP_TASK(sch, enqueue, rq, p, enq_flags);
 
+	/* The reason is input to ops.enqueue(), not to the resulting placement. */
+	p->scx.flags &= ~SCX_TASK_REENQ_REASON_MASK;
 	*ddsp_taskp = NULL;
 	if (p->scx.ddsp_dsq_id != SCX_DSQ_INVALID)
 		goto direct;
@@ -2085,9 +2087,11 @@ void scx_do_enqueue_task(struct rq *rq, struct task_struct *p, u64 enq_flags,
 	return;
 
 direct:
+	p->scx.flags &= ~SCX_TASK_REENQ_REASON_MASK;
 	direct_dispatch(sch, p, enq_flags);
 	return;
 local_norefill:
+	p->scx.flags &= ~SCX_TASK_REENQ_REASON_MASK;
 	scx_dispatch_enqueue(sch, rq, &rq->scx.local_dsq, p, 0, 0, enq_flags);
 	return;
 local:

kernel/sched/ext/ext.c void scx_do_enqueue_task(struct rq *rq, struct task_struct *p, u64 enq_flags, · line 2111

Quoted 7 lines
kernel/sched/ext/ext.c +1 -0
@@ -2101,6 +2105,7 @@ void scx_do_enqueue_task(struct rq *rq, struct task_struct *p, u64 enq_flags,
 	goto enqueue;
 
 enqueue:
+	p->scx.flags &= ~SCX_TASK_REENQ_REASON_MASK;
 	refill_task_slice_dfl(sch, p);
 	clear_direct_dispatch(p);
 	scx_dispatch_enqueue(sch, rq, dsq, p, 0, 0, enq_flags);

This is too messy. The clearing is scattered all over and sometimes done twice. Can't you add one label that everyone jumps to for exit and clear it there?

Quoted 1 line

Thanks.

Ack to all the nits, I'll clean them up.

Thanks,
-Andrea

author reviewer
#41 lore raw

PATCH 13/18 sched_ext: Handle proxy-exec races in remote DSQ transfers

In reply to Andrea Righi

Without proxy execution, the DSQ lock and holding_cpu handshake ensure that a task cannot be dequeued or start running during an rq-lock handoff without clearing holding_cpu.

Proxy execution is an exception: a task can start physically executing as a lock owner while its scheduling context remains on a DSQ; its on-CPU or migration-disabled state can therefore change without clearing holding_cpu.

Recheck these states after acquiring the source rq lock. If the transfer can no longer proceed, park the task on the source rq's reject DSQ and reenqueue it through its owning scheduler. This preserves the BPF scheduler's placement policy and keeps descendant tasks within their sub-scheduler's cap grants.

Queue a deferred drain for every rejection. If a drain finds a proxy-rejected task still running or donating, leave it parked and record a retry request on the rq. scx_proxy_resolved() consumes the request and schedules another drain when proxy state changes.

Without this change and proxy execution enabled, stress-ng --pipeherd can trigger this race and migrate an active execution context, leading to sleeping-while-atomic warnings and subsequent lockdep corruption.

This is a preparatory change to support proxy execution with sched_ext.

---
 include/linux/sched/ext.h                     |   2 +
 kernel/sched/ext/ext.c                        | 143 +++++++++++++++---
 kernel/sched/ext/internal.h                   |   8 +
 kernel/sched/ext/sub.c                        |   4 +-
 kernel/sched/sched.h                          |   1 +
 .../sched_ext/include/scx/enum_defs.autogen.h |   1 +
 6 files changed, 139 insertions(+), 20 deletions(-)
include/linux/sched/ext.h +2 -0
index 9912ad0c2d445..55c2665a6d37a 100644
@@ -137,6 +137,7 @@ enum scx_ent_flags {
 	 * IMMED	reenqueued due to failed ENQ_IMMED
 	 * PREEMPTED	preempted while running
 	 * CAP		sub-sched cap miss, see p->scx.reenq_reason_*
+	 * PROXY	proxy state prevented a remote DSQ transfer
 	 */
 	SCX_TASK_REENQ_REASON_SHIFT = 12,
 	SCX_TASK_REENQ_REASON_BITS = 3,
@@ -147,6 +148,7 @@ enum scx_ent_flags {
 	SCX_TASK_REENQ_IMMED	= 2 << SCX_TASK_REENQ_REASON_SHIFT,
 	SCX_TASK_REENQ_PREEMPTED = 3 << SCX_TASK_REENQ_REASON_SHIFT,
 	SCX_TASK_REENQ_CAP	= 4 << SCX_TASK_REENQ_REASON_SHIFT,
+	SCX_TASK_REENQ_PROXY	= 5 << SCX_TASK_REENQ_REASON_SHIFT,
 
 	/* iteration cursor, not a task */
 	SCX_TASK_CURSOR		= 1 << 31,
kernel/sched/ext/ext.c +126 -17

217 lines not on this page · raw message

kernel/sched/ext/internal.h +8 -0
index 0d7efbee2862c..5a1f6875e3d7d 100644
@@ -1747,6 +1747,14 @@ enum scx_enq_flags {
 	SCX_ENQ_SLICE_DFL	= 1LLU << 62,	/* carried slice is a default refill */
 };
 
+/* Strip priority and carried slice state when diverting from a local DSQ. */
+static inline void scx_divert_strip_flags(struct task_struct *p, u64 *enq_flags)
+{
+	*enq_flags &= ~(SCX_ENQ_IMMED | SCX_ENQ_PREEMPT | SCX_ENQ_HEAD |
+			SCX_ENQ_APPLY_SLICE | SCX_ENQ_SLICE_DFL);
+	p->scx.flags &= ~SCX_TASK_IMMED;
+}
+
 enum scx_deq_flags {
 	/* expose select DEQUEUE_* flags as enums */
 	SCX_DEQ_SLEEP		= DEQUEUE_SLEEP,
kernel/sched/ext/sub.c +1 -3
index 40a79028de8e6..7b61e3a3a11e8 100644
@@ -735,9 +735,7 @@ struct scx_dispatch_q *scx_resolve_local_dsq(struct scx_sched *sch, struct rq *r
 	 * or HEAD - a diversion has no priority and IMMED is not allowed on
 	 * non-local DSQs. Strip the enq and task flags along with the slice.
 	 */
-	*enq_flags &= ~(SCX_ENQ_IMMED | SCX_ENQ_PREEMPT | SCX_ENQ_HEAD |
-			SCX_ENQ_APPLY_SLICE | SCX_ENQ_SLICE_DFL);
-	p->scx.flags &= ~SCX_TASK_IMMED;
+	scx_divert_strip_flags(p, enq_flags);
 
 	/* the enqueuer opted for rescue instead of rejection and reenqueue */
 	if ((*enq_flags & SCX_ENQ_RESCUE) && likely(scx_rescue_bw_1024)) {
kernel/sched/sched.h +1 -0
index 43f04989557b5..8361a38c3c263 100644
@@ -791,6 +791,7 @@ enum scx_rq_flags {
 	SCX_RQ_BAL_CB_PENDING	= 1 << 6, /* must queue a cb after dispatching */
 	SCX_RQ_SUB_IDLE_RENOTIFY	= 1 << 7, /* sub-scheds are owed update_idle() */
 	SCX_RQ_ROOT_IDLE_RENOTIFY	= 1 << 8, /* the root is owed update_idle() */
+	SCX_RQ_PROXY_RETRY	= 1 << 9, /* proxy-rejected tasks need retry */
 
 	SCX_RQ_IN_WAKEUP	= 1 << 16,
 	SCX_RQ_IN_DISPATCH	= 1 << 17,
tools/sched_ext/include/scx/enum_defs.autogen.h +1 -0
index cccc0c3987b85..b1351f346e1d9 100644
@@ -119,6 +119,7 @@
 #define HAVE_SCX_TASK_REENQ_IMMED
 #define HAVE_SCX_TASK_REENQ_PREEMPTED
 #define HAVE_SCX_TASK_REENQ_CAP
+#define HAVE_SCX_TASK_REENQ_PROXY
 #define HAVE_SCX_TASK_CURSOR
 #define HAVE_SCX_ECODE_RSN_HOTPLUG
 #define HAVE_SCX_ECODE_RSN_CGROUP_OFFLINE
Signature · 1 line
2.55.0
author reviewer
#42 lore raw

PATCH 14/18 sched_ext: Split curr|donor references properly

In reply to Andrea Righi

With proxy execution, the task selected by the scheduler and the task physically executing can differ. A blocked mutex waiter donates its scheduling context to the lock owner:

D -----------------> M -------------> O ----------------> T [donor] blocked on [mutex] owned by [owner] preempted by [task]
\_________________________________^
donates scheduling context

where:

D = blocked donor
M = mutex
O = mutex owner
T = competing runnable task

During a proxy execution switch, D supplies the scheduling class, priority, and runtime budget, while O supplies the execution context: O is the task whose code physically executes. T is a competing runnable task which may preempt the D/O proxy execution.

Consider FAIR and EXT tasks with sched_ext running in partial mode. FAIR can be replaced with a higher scheduling class such as RT or deadline without changing the class interaction described here. The possible combinations are:

1. D is EXT, O is EXT, T is EXT

D can interrupt T according to BPF scheduling policy. O executes with D's EXT priority and runtime budget, while T waits in EXT.

2. D is EXT, O is EXT, T is FAIR

D is visible to the BPF scheduler, but cannot preempt T because EXT is below FAIR. Once T stops, BPF can dispatch D and O executes with D's EXT priority and runtime budget. If T becomes runnable again, it preempts the D/O proxy execution.

3. D is EXT, O is FAIR, T is EXT

This cannot represent T preempting O because EXT is below FAIR.

4. D is EXT, O is FAIR, T is FAIR

D cannot boost O above T because EXT is below FAIR. O and T continue competing under FAIR. Once O releases M, D wakes and resumes normal EXT scheduling.

5. D is FAIR, O is EXT, T is EXT

D preempts T as the higher-class scheduling context. O executes with D's FAIR priority and runtime budget, while T waits in EXT. D is not visible to the BPF scheduler.

6. D is FAIR, O is EXT, T is FAIR

D competes with T according to its FAIR deadline. When D is selected, O executes with D's FAIR priority and runtime budget. D is not visible to the BPF scheduler.

7. D is FAIR, O is FAIR, T is EXT

This cannot represent T preempting O because EXT is below FAIR.

8. D is FAIR, O is FAIR, T is FAIR

O, T, and D all have FAIR scheduling contexts. D remains runnable as a blocked proxy donor. When CFS selects D, O executes using D's FAIR scheduling context. When CFS selects O, O executes using its own FAIR context, and when CFS selects T, T executes normally. D is not visible to the BPF scheduler.

Thus, sched_ext policy and accounting must generally use rq->donor, the scheduler-selected task which supplies the scheduling context, rather than rq->curr, the task whose code physically executes. Without proxy execution they are the same task.

On nohz_full CPUs, a blocked proxy donor must retain the scheduler tick even when it has an infinite slice. Otherwise, a full dynticks CPU could stop the tick while rq->curr and rq->donor differ, violating assumptions made by the remote NOHZ tick path.

This is a conservative compromise that keeps the change local to sched_ext, at the cost of a periodic tick while a blocked proxy donor is selected. Allowing blocked proxy donors to run tickless would require making the core scheduler's remote tick handling aware that rq->curr and rq->donor can differ.

Moreover, extend scx_dump_state() to report both contexts. Each CPU record now includes a donor= line. If an EXT donor differs from rq->curr, also emit its detailed task record. The existing '*' marker continues to identify rq->curr, while the donor= line identifies the otherwise unmarked donor record.

Note that at this point in the series, CONFIG_SCHED_PROXY_EXEC still depends on !CONFIG_SCHED_CLASS_EXT, so proxy execution and sched_ext cannot be enabled together. The scheduling changes are therefore preparatory. A later patch removes this restriction.

---
 Documentation/scheduler/sched-ext.rst |  13 +++
 kernel/sched/ext/ext.c                | 128 +++++++++++++++++---------
 kernel/sched/ext/sub.c                |   5 +-
 kernel/sched/ext/sub.h                |  11 ++-
 4 files changed, 104 insertions(+), 53 deletions(-)
Documentation/scheduler/sched-ext.rst +13 -0
index 0e97fd0199940..38ffd6da516e5 100644
@@ -488,6 +488,19 @@ and edge cases, to name a few examples:
   class, in which case it will exit the tick-dispatch loop even though it is runnable
   and has a non-zero slice.
 
+* Under proxy execution, sched_ext continues to observe the donor as the
+  current scheduling context. Accordingly, ``ops.running()`` and
+  ``ops.stopping()`` report when the donor's scheduling context becomes active
+  and inactive, even when the donor is blocked and a lock owner executes on its
+  behalf. The physical execution context is intentionally not reported through
+  these callbacks.
+
+  A blocked donor enters a running session only after proxy resolution finds
+  an execution context. The session remains active if only the physical
+  execution context changes while the donor remains the same. Running sessions
+  are tracked so that ``ops.running()`` and ``ops.stopping()`` remain paired
+  and are not emitted recursively.
+
 See the "Scheduling Cycle" section for a more detailed description of how
 a freshly woken up task gets on a CPU.
 
kernel/sched/ext/ext.c +82 -46

342 lines not on this page · raw message

kernel/sched/ext/sub.c +3 -2
index 7b61e3a3a11e8..8c5b64b57b24a 100644
@@ -285,7 +285,8 @@ void scx_rescue_charge(struct rq *rq, s64 delta_exec)
 	rq->scx.rescue.budget -= delta_exec;
 
 	/* per-cpu usage average feeds the overload victim pick */
-	pcpu = per_cpu_ptr(scx_task_sched(rq->curr)->pcpu, cpu_of(rq));
+	pcpu = per_cpu_ptr(scx_task_sched(rq->scx.rescue.curr)->pcpu,
+			   cpu_of(rq));
 	pcpu->rescue_avg = scx_rescue_decay_avg(pcpu) + delta_exec;
 
 	if (!scx_rescue_slice_remaining(rq))
@@ -557,7 +558,7 @@ static void scx_rescue_timerfn(struct timer_list *timer)
 		scx_rescue_admit(rq, p, slice);
 		scx_move_local_task_to_local_dsq(scx_task_sched(p), p, SCX_ENQ_IGNORE_CAPS,
 						 &rq->scx.rescue.dsq, rq);
-		if (sched_class_above(&ext_sched_class, rq->curr->sched_class))
+		if (sched_class_above(&ext_sched_class, rq->donor->sched_class))
 			resched_curr(rq);
 	} else if (p->scx.dsq && rq->scx.rescue.budget > 2 * scx_rescue_quantum_ns) {
 		/*
kernel/sched/ext/sub.h +6 -5
index 8f2425bdb9530..24357d8c5e335 100644
@@ -167,17 +167,18 @@ static inline u64 scx_caps_for_task(struct task_struct *p)
 	return SCX_CAP_ENQ;
 }
 
-/* the cap @sch needs to preempt @rq's current task, 0 if none */
-static inline u64 scx_caps_for_preempt(struct scx_sched *sch, struct rq *rq, u64 enq_flags)
+/* the cap @sch needs to preempt @rq's current scheduling context, 0 if none */
+static inline u64 scx_caps_for_preempt(struct scx_sched *sch, struct rq *rq,
+				       u64 enq_flags)
 {
-	struct task_struct *curr = rq->curr;
+	struct task_struct *donor = rq->donor;
 
 	/* a kernel-forced placement preempts regardless of caps */
 	if (unlikely(enq_flags & SCX_ENQ_IGNORE_CAPS))
 		return 0;
 	/* a non-ext task can't be preempted by ext, own-subtree needs no cap */
-	if (curr->sched_class != &ext_sched_class ||
-	    scx_is_descendant(scx_task_sched(curr), sch))
+	if (donor->sched_class != &ext_sched_class ||
+	    scx_is_descendant(scx_task_sched(donor), sch))
 		return 0;
 	return SCX_CAP_PREEMPT;
 }
Signature · 1 line
2.55.0
#43 lore raw

Re: [PATCH 14/18] sched_ext: Split curr|donor references properly

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] The physical proxy execution owner is spuriously exposed to the BPF scheduler and loses its…

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] The physical proxy execution owner is spuriously exposed to the BPF scheduler and loses its personal slice during SAVE/RESTORE cycles. --

commit 9c0210f5a5bf78476570e1133b4136f7fc67852c
Author: Andrea Righi <[email protected]>

sched_ext: Split curr|donor references properly

This commit separates the physical execution context (rq->curr) from the scheduling context (rq->donor) to support proxy execution in sched_ext. It updates various scheduler paths to use the donor where appropriate.

Quoted 4 lines
kernel/sched/ext/ext.c +0 -0
index b2e6be4a887de..f23e9f1ee2f3c 100644

[ ... ]

Quoted 5 lines
kernel/sched/ext/ext.c +0 -1
@@ -2170,13 +2177,14 @@ static void enqueue_task_scx(struct rq *rq, struct task_struct *p, int core_enq_
 		rq->scx.flags |= SCX_RQ_IN_WAKEUP;
 
 	/*
-	 * Restoring a running task will be immediately followed by

kernel/sched/ext/ext.c static void enqueue_task_scx(struct rq *rq, struct task_struct *p, int core_enq_ · line 2187

Quoted 10 lines
kernel/sched/ext/ext.c +4 -2
-	 * set_next_task_scx() which expects the task to not be on the BPF
+	 * Restoring the current scheduling context will be immediately followed
+	 * by set_next_task_scx() which expects the task to not be on the BPF
 	 * scheduler as tasks can only start running through local DSQs. Force
 	 * direct-dispatch into the local DSQ by setting the sticky_cpu. Mark
 	 * IGNORE_CAPS to force entry into the local DSQ.
 	 */
-	if (unlikely(enq_flags & ENQUEUE_RESTORE) && task_current(rq, p)) {
+	if (unlikely(enq_flags & ENQUEUE_RESTORE) &&
+	    task_current_donor(rq, p)) {

[Severity: High]
Does this check fail to recognize the physical execution context during ENQUEUE_RESTORE?

During proxy execution, the currently executing task (rq->curr) and the donor (rq->donor) are different. If a property change (such as sched_setscheduler or setpriority) is requested on the physically executing task while it is actively running, it will undergo a DEQUEUE_SAVE / ENQUEUE_RESTORE cycle.

Because task_current_donor(rq, p) evaluates to false for the physically executing task, wouldn't this bypass the SCX_ENQ_IGNORE_CAPS fast-path and erroneously expose the actively executing task to ops.enqueue(), which violates the constraint that physical execution contexts are not exposed to the BPF scheduler?

[ ... ]

kernel/sched/ext/ext.c static bool dequeue_task_scx(struct rq *rq, struct task_struct *p, int core_deq_ · line 2354

Quoted 7 lines
kernel/sched/ext/ext.c +1 -1
@@ -2342,7 +2350,7 @@ static bool dequeue_task_scx(struct rq *rq, struct task_struct *p, int core_deq_
 	p->scx.flags &= ~SCX_TASK_REENQ_REASON_MASK;
 
 	/* see scx_task_slice_ended() for the save/restore exception */
-	if (!((deq_flags & DEQUEUE_SAVE) && task_current(rq, p)))
+	if (!((deq_flags & DEQUEUE_SAVE) && task_current_donor(rq, p)))
 		scx_task_slice_ended(rq, p);

[Severity: High]
Similarly, does this fail to recognize the physical execution context during DEQUEUE_SAVE?

If the physically executing task undergoes a DEQUEUE_SAVE cycle, task_current_donor(rq, p) evaluates to false. Could this spuriously trigger scx_task_slice_ended(), incorrectly zeroing out its personal slice while it is running on behalf of the donor?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=14
author reviewer
#44 lore raw

Re: [PATCH 14/18] sched_ext: Split curr|donor references properly

Quoted 62 lines · On Mon, Aug 31, 2026 at 05:49:07PM +0000, [email protected] wrote:

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] The physical proxy execution owner is spuriously exposed to the BPF scheduler and loses its personal slice during SAVE/RESTORE cycles. --

commit 9c0210f5a5bf78476570e1133b4136f7fc67852c
Author: Andrea Righi <[email protected]>

sched_ext: Split curr|donor references properly

This commit separates the physical execution context (rq->curr) from the scheduling context (rq->donor) to support proxy execution in sched_ext. It updates various scheduler paths to use the donor where appropriate.

Quoted 4 lines
kernel/sched/ext/ext.c +0 -0
index b2e6be4a887de..f23e9f1ee2f3c 100644

[ ... ]

Quoted 5 lines
kernel/sched/ext/ext.c +0 -1
@@ -2170,13 +2177,14 @@ static void enqueue_task_scx(struct rq *rq, struct task_struct *p, int core_enq_
 		rq->scx.flags |= SCX_RQ_IN_WAKEUP;
 
 	/*
-	 * Restoring a running task will be immediately followed by

kernel/sched/ext/ext.c static void enqueue_task_scx(struct rq *rq, struct task_struct *p, int core_enq_ · line 2187

Quoted 10 lines
kernel/sched/ext/ext.c +4 -2
-	 * set_next_task_scx() which expects the task to not be on the BPF
+	 * Restoring the current scheduling context will be immediately followed
+	 * by set_next_task_scx() which expects the task to not be on the BPF
 	 * scheduler as tasks can only start running through local DSQs. Force
 	 * direct-dispatch into the local DSQ by setting the sticky_cpu. Mark
 	 * IGNORE_CAPS to force entry into the local DSQ.
 	 */
-	if (unlikely(enq_flags & ENQUEUE_RESTORE) && task_current(rq, p)) {
+	if (unlikely(enq_flags & ENQUEUE_RESTORE) &&
+	    task_current_donor(rq, p)) {

[Severity: High]
Does this check fail to recognize the physical execution context during ENQUEUE_RESTORE?

During proxy execution, the currently executing task (rq->curr) and the donor (rq->donor) are different. If a property change (such as sched_setscheduler or setpriority) is requested on the physically executing task while it is actively running, it will undergo a DEQUEUE_SAVE / ENQUEUE_RESTORE cycle.

Because task_current_donor(rq, p) evaluates to false for the physically executing task, wouldn't this bypass the SCX_ENQ_IGNORE_CAPS fast-path and erroneously expose the actively executing task to ops.enqueue(), which violates the constraint that physical execution contexts are not exposed to the BPF scheduler?

[ ... ]

kernel/sched/ext/ext.c static bool dequeue_task_scx(struct rq *rq, struct task_struct *p, int core_deq_ · line 2354

Quoted 7 lines
kernel/sched/ext/ext.c +1 -1
@@ -2342,7 +2350,7 @@ static bool dequeue_task_scx(struct rq *rq, struct task_struct *p, int core_deq_
 	p->scx.flags &= ~SCX_TASK_REENQ_REASON_MASK;
 
 	/* see scx_task_slice_ended() for the save/restore exception */
-	if (!((deq_flags & DEQUEUE_SAVE) && task_current(rq, p)))
+	if (!((deq_flags & DEQUEUE_SAVE) && task_current_donor(rq, p)))
 		scx_task_slice_ended(rq, p);

[Severity: High]
Similarly, does this fail to recognize the physical execution context during DEQUEUE_SAVE?

Quoted 4 lines

If the physically executing task undergoes a DEQUEUE_SAVE cycle, task_current_donor(rq, p) evaluates to false. Could this spuriously trigger scx_task_slice_ended(), incorrectly zeroing out its personal slice while it is running on behalf of the donor?

I think these are all false positives, the use of task_current_donor() here is intentional. SAVE/RESTORE is preserving the current scheduling context, rather than the physical execution context.

-Andrea

maintainer
#45 lore raw

Re: [PATCH 14/18] sched_ext: Split curr|donor references properly

In reply to Andrea Righi
Quoted 43 lines · Andrea Righi

Consider FAIR and EXT tasks with sched_ext running in partial mode. FAIR can be replaced with a higher scheduling class such as RT or deadline without changing the class interaction described here. The possible combinations are:

1. D is EXT, O is EXT, T is EXT

D can interrupt T according to BPF scheduling policy. O executes with D's EXT priority and runtime budget, while T waits in EXT.

2. D is EXT, O is EXT, T is FAIR

D is visible to the BPF scheduler, but cannot preempt T because EXT is below FAIR. Once T stops, BPF can dispatch D and O executes with D's EXT priority and runtime budget. If T becomes runnable again, it preempts the D/O proxy execution.

3. D is EXT, O is FAIR, T is EXT

This cannot represent T preempting O because EXT is below FAIR.

4. D is EXT, O is FAIR, T is FAIR

D cannot boost O above T because EXT is below FAIR. O and T continue competing under FAIR. Once O releases M, D wakes and resumes normal EXT scheduling.

5. D is FAIR, O is EXT, T is EXT

D preempts T as the higher-class scheduling context. O executes with D's FAIR priority and runtime budget, while T waits in EXT. D is not visible to the BPF scheduler.

6. D is FAIR, O is EXT, T is FAIR

D competes with T according to its FAIR deadline. When D is selected, O executes with D's FAIR priority and runtime budget. D is not visible to the BPF scheduler.

7. D is FAIR, O is FAIR, T is EXT

This cannot represent T preempting O because EXT is below FAIR.

Quoted 7 lines

8. D is FAIR, O is FAIR, T is FAIR

O, T, and D all have FAIR scheduling contexts. D remains runnable as a blocked proxy donor. When CFS selects D, O executes using D's FAIR scheduling context. When CFS selects O, O executes using its own FAIR context, and when CFS selects T, T executes normally. D is not visible to the BPF scheduler.

I don't see how 3,4 can happen. If there is a runnable FAIR task, then pick will pick that directly.

Per the pick priority, the donor class must always be higher than the owner class.

author reviewer
#46 lore raw

Re: [PATCH 14/18] sched_ext: Split curr|donor references properly

Quoted 53 lines · Peter Zijlstra
Quoted 43 lines · Andrea Righi

Consider FAIR and EXT tasks with sched_ext running in partial mode. FAIR can be replaced with a higher scheduling class such as RT or deadline without changing the class interaction described here. The possible combinations are:

1. D is EXT, O is EXT, T is EXT

D can interrupt T according to BPF scheduling policy. O executes with D's EXT priority and runtime budget, while T waits in EXT.

2. D is EXT, O is EXT, T is FAIR

D is visible to the BPF scheduler, but cannot preempt T because EXT is below FAIR. Once T stops, BPF can dispatch D and O executes with D's EXT priority and runtime budget. If T becomes runnable again, it preempts the D/O proxy execution.

3. D is EXT, O is FAIR, T is EXT

This cannot represent T preempting O because EXT is below FAIR.

4. D is EXT, O is FAIR, T is FAIR

D cannot boost O above T because EXT is below FAIR. O and T continue competing under FAIR. Once O releases M, D wakes and resumes normal EXT scheduling.

5. D is FAIR, O is EXT, T is EXT

D preempts T as the higher-class scheduling context. O executes with D's FAIR priority and runtime budget, while T waits in EXT. D is not visible to the BPF scheduler.

6. D is FAIR, O is EXT, T is FAIR

D competes with T according to its FAIR deadline. When D is selected, O executes with D's FAIR priority and runtime budget. D is not visible to the BPF scheduler.

7. D is FAIR, O is FAIR, T is EXT

This cannot represent T preempting O because EXT is below FAIR.

Quoted 7 lines

8. D is FAIR, O is FAIR, T is FAIR

O, T, and D all have FAIR scheduling contexts. D remains runnable as a blocked proxy donor. When CFS selects D, O executes using D's FAIR scheduling context. When CFS selects O, O executes using its own FAIR context, and when CFS selects T, T executes normally. D is not visible to the BPF scheduler.

Quoted 6 lines

I don't see how 3,4 can happen. If there is a runnable FAIR task, then pick will pick that directly.

Per the pick priority, the donor class must always be higher than the owner class.

Right. I was trying to describe why those cases cannot result in the stated preemption, but listing them under "possible combinations" made that unclear. I'll remove those two cases and state this invariant explicitly.

Thanks,
-Andrea

author reviewer
#47 lore raw

PATCH 15/18 sched_ext: Delegate proxy donor admission to BPF schedulers

In reply to Andrea Righi

Proxy execution keeps a mutex-blocked donor runnable so that its scheduling context can execute the mutex owner.

Define SCX_OPS_ENQ_BLOCKED as the admission contract for retained donors. Schedulers without the flag block EXT donors normally. Schedulers with the flag continue to own blocked donors and receive them through ops.enqueue() with SCX_ENQ_BLOCKED, allowing BPF to choose their DSQ, CPU and ordering.

After BPF dispatches a donor, proxy execution may move its context to the rq of the mutex owner. This move is distinct from BPF placement:
proxy_set_task_cpu() changes task_cpu() while preserving the donor's pre-proxy CPU in wake_cpu as the starting point for normal wakeup placement. A proxy-migrated donor returns through the full wakeup activation path when the mutex is released.

Do not carry a retained proxy session across BPF scheduler ownership changes. Before root activation, parent-to-child takeover, rehome or punt, fully deactivate a retained donor unconditionally. The incoming scheduler then starts with clean task state and applies its admission policy the next time the task blocks.

A retained donor can wake through ttwu_runnable() while still on-rq, without another enqueue_task_scx() call. On a WF_ON_RQ wake, wakeup_preempt_scx() requests dispatch reconsideration after is_blocked is cleared. A full wakeup activation has already enqueued the task and does not need the additional reschedule.

Blocked-donor enqueueing takes precedence over the exiting and migration-disabled local-DSQ fallbacks, so that an opted-in scheduler sees every eligible donor request.

---
 kernel/sched/ext/ext.c                        | 121 +++++++++++++++---
 kernel/sched/ext/internal.h                   |  23 +++-
 kernel/sched/ext/sub.c                        |   9 +-
 tools/sched_ext/include/scx/compat.h          |   1 +
 .../sched_ext/include/scx/enum_defs.autogen.h |   1 +
 .../sched_ext/include/scx/enums.autogen.bpf.h |   3 +
 tools/sched_ext/include/scx/enums.autogen.h   |   1 +
 7 files changed, 139 insertions(+), 20 deletions(-)
kernel/sched/ext/ext.c +105 -16
index 5f4956cf7e369..aa6ba9e8d7cb0 100644
@@ -26,7 +26,28 @@ DEFINE_RAW_SPINLOCK(scx_sched_lock);
 
 bool scx_allow_proxy_exec(const struct task_struct *p)
 {
-	return p->sched_class != &ext_sched_class;
+	struct scx_sched *sch;
+
+	if (p->sched_class != &ext_sched_class)
+		return true;
+
+	sch = scx_task_sched(p);
+	return !sch || (sch->ops.flags & SCX_OPS_ENQ_BLOCKED);
+}
+
+/*
+ * End retained proxy execution before changing @p's BPF scheduler ownership.
+ * Called with @p's pi and rq locks held immediately before
+ * sched_change_begin(). The caller must pass DEQUEUE_NOCLOCK so the rq clock
+ * is updated only once.
+ */
+void scx_prepare_task_sched_change(struct task_struct *p)
+{
+	lockdep_assert_held(&p->pi_lock);
+	lockdep_assert_rq_held(task_rq(p));
+
+	update_rq_clock(task_rq(p));
+	sched_proxy_block_task(task_rq(p), p);
 }
 
 /*
@@ -2055,19 +2076,28 @@ void scx_do_enqueue_task(struct rq *rq, struct task_struct *p, u64 enq_flags,
 	if (p->scx.ddsp_dsq_id != SCX_DSQ_INVALID)
 		goto direct;
 
-	/* see %SCX_OPS_ENQ_EXITING */
-	if (!(sch->ops.flags & SCX_OPS_ENQ_EXITING) &&
-	    unlikely(p->flags & PF_EXITING)) {
-		__scx_add_event(sch, SCX_EV_ENQ_SKIP_EXITING, 1);
-		enq_flags |= SCX_ENQ_RESCUE;	/* avoid looping on cap rejection */
-		goto local;
-	}
+	/*
+	 * A full wakeup enqueue happens before is_blocked is cleared. Don't
+	 * report it as another blocked-donor admission.
+	 */
+	if ((sch->ops.flags & SCX_OPS_ENQ_BLOCKED) && p->is_blocked &&
+	    !(enq_flags & SCX_ENQ_WAKEUP)) {
+		enq_flags |= SCX_ENQ_BLOCKED;
+	} else {
+		/* see %SCX_OPS_ENQ_EXITING */
+		if (unlikely(p->flags & PF_EXITING) &&
+		    !(sch->ops.flags & SCX_OPS_ENQ_EXITING)) {
+			__scx_add_event(sch, SCX_EV_ENQ_SKIP_EXITING, 1);
+			enq_flags |= SCX_ENQ_RESCUE;	/* avoid looping on cap rejection */
+			goto local;
+		}
 
-	/* see %SCX_OPS_ENQ_MIGRATION_DISABLED */
-	if (!(sch->ops.flags & SCX_OPS_ENQ_MIGRATION_DISABLED) &&
-	    is_migration_disabled(p)) {
-		__scx_add_event(sch, SCX_EV_ENQ_SKIP_MIGRATION_DISABLED, 1);
-		goto local;
+		/* see %SCX_OPS_ENQ_MIGRATION_DISABLED */
+		if (!(sch->ops.flags & SCX_OPS_ENQ_MIGRATION_DISABLED) &&
+		    is_migration_disabled(p)) {
+			__scx_add_event(sch, SCX_EV_ENQ_SKIP_MIGRATION_DISABLED, 1);
+			goto local;
+		}
 	}
 
 	if (unlikely(!SCX_HAS_OP(sch, enqueue)))
@@ -2390,10 +2420,23 @@ static void wakeup_preempt_scx(struct rq *rq, struct task_struct *p, int wake_fl
 	/*
 	 * Preemption between SCX tasks is implemented by resetting the victim
 	 * task's slice to 0 and triggering reschedule on the target CPU.
-	 * Nothing to do.
-	 */
-	if (p->sched_class == &ext_sched_class)
+	 *
+	 * A retained proxy donor can wake through ttwu_runnable() without another
+	 * ops.enqueue(). WF_ON_RQ identifies this path. Request rescheduling so
+	 * that ops.dispatch() can reconsider the task after ttwu_runnable() clears
+	 * is_blocked. A full wakeup activation has already enqueued the task and
+	 * doesn't need the additional reschedule.
+	 */
+	if (p->sched_class == &ext_sched_class) {
+		if (sched_proxy_exec() && (wake_flags & WF_ON_RQ) &&
+		    p->is_blocked) {
+			struct scx_sched *sch = scx_task_sched(p);
+
+			if (sch && (sch->ops.flags & SCX_OPS_ENQ_BLOCKED))
+				resched_curr(rq);
+		}
 		return;
+	}
 
 	/*
 	 * Getting preempted by a higher-priority class. Reenqueue IMMED tasks.
@@ -2508,6 +2551,21 @@ static bool task_can_run_on_remote_rq(struct scx_sched *sch,
 
 	WARN_ON_ONCE(task_cpu(p) == cpu);
 
+	/*
+	 * proxy_set_task_cpu() preserves wake_cpu when moving a donor's scheduling
+	 * context to its lock owner's rq. For example, if BPF places donor D on
+	 * CPU0 while its lock owner runs on CPU1, proxy execution moves D to CPU1
+	 * but leaves D->wake_cpu pointing to CPU0. If D is later put on a shared
+	 * DSQ, CPU0 could otherwise consume it and move it back, only for proxy
+	 * execution to return it to CPU1 again.
+	 *
+	 * When task_cpu() == wake_cpu, the donor has not been proxy-migrated and
+	 * BPF may still choose its placement. An actively donating task is
+	 * rejected separately by task_proxy_running_or_donating().
+	 */
+	if (sched_proxy_exec() && p->is_blocked && task_cpu(p) != p->wake_cpu)
+		return false;
+
 	/*
 	 * If @p has migration disabled, @p->cpus_ptr is updated to contain only
 	 * the pinned CPU in migrate_disable_switch() while @p is being switched
@@ -3309,6 +3367,24 @@ static void put_prev_task_scx(struct rq *rq, struct task_struct *p,
 	if (p->scx.flags & SCX_TASK_QUEUED) {
 		set_task_runnable(rq, p);
 
+		/* Delegate retained donor admission to its owning BPF scheduler. */
+		if (p->is_blocked) {
+			/*
+			 * If the donor is the same and only the mutex owner
+			 * changes, avoid triggering another ops.enqueue(): the
+			 * BPF scheduler has already admitted the donor, so it
+			 * can continue running.
+			 */
+			if (next == p)
+				goto switch_class;
+
+			if (WARN_ON_ONCE(!sch))
+				goto switch_class;
+			WARN_ON_ONCE(!(sch->ops.flags & SCX_OPS_ENQ_BLOCKED));
+			scx_do_enqueue_task(rq, p, 0, -1);
+			goto switch_class;
+		}
+
 		/*
 		 * If @p has slice left and is being put, @p is getting
 		 * preempted by a higher priority scheduler class or core-sched
@@ -7640,6 +7716,11 @@ int scx_validate_ops(struct scx_sched *sch, const struct sched_ext_ops *ops)
 		return -EINVAL;
 	}
 
+	if ((ops->flags & SCX_OPS_ENQ_BLOCKED) && !ops->enqueue) {
+		scx_error(sch, "SCX_OPS_ENQ_BLOCKED requires ops.enqueue() to be implemented");
+		return -EINVAL;
+	}
+
 	/*
 	 * SCX_OPS_TID_TO_TASK is enabled by the root scheduler. A sub-sched
 	 * may set it to declare a dependency; reject if the root hasn't
@@ -8027,6 +8108,14 @@ static void scx_root_enable_workfn(struct kthread_work *work)
 
 		if (old_class != new_class)
 			queue_flags |= DEQUEUE_CLASS;
+		if (old_class == new_class && new_class == &ext_sched_class) {
+			/*
+			 * This is an EXT-to-EXT scheduler ownership change, so
+			 * sched_change_begin() won't end retained proxy execution.
+			 */
+			scx_prepare_task_sched_change(p);
+			queue_flags |= DEQUEUE_NOCLOCK;
+		}
 
 		scoped_guard (sched_change, p, new_class, queue_flags) {
 			scx_set_task_slice(p, READ_ONCE(sch->slice_dfl));
kernel/sched/ext/internal.h +22 -1
index 5a1f6875e3d7d..f830e76aa645a 100644
@@ -215,6 +215,19 @@ enum scx_ops_flags {
 	 */
 	SCX_OPS_TID_TO_TASK		= 1LLU << 8,
 
+	/*
+	 * If set, mutex-blocked tasks remain runnable as proxy donors and are
+	 * passed to ops.enqueue() with %SCX_ENQ_BLOCKED. The BPF scheduler controls
+	 * when donors are dispatched and whether they should preempt other work.
+	 *
+	 * If clear, mutex-blocked tasks are removed from the runqueue normally
+	 * and cannot donate their scheduling context through proxy execution.
+	 *
+	 * For blocked donors, this flag takes precedence over
+	 * %SCX_OPS_ENQ_EXITING and %SCX_OPS_ENQ_MIGRATION_DISABLED.
+	 */
+	SCX_OPS_ENQ_BLOCKED		= 1LLU << 9,
+
 	SCX_OPS_ALL_FLAGS		= SCX_OPS_KEEP_BUILTIN_IDLE |
 					  SCX_OPS_ENQ_LAST |
 					  SCX_OPS_ENQ_EXITING |
@@ -223,7 +236,8 @@ enum scx_ops_flags {
 					  SCX_OPS_SWITCH_PARTIAL |
 					  SCX_OPS_BUILTIN_IDLE_PER_NODE |
 					  SCX_OPS_ALWAYS_ENQ_IMMED |
-					  SCX_OPS_TID_TO_TASK,
+					  SCX_OPS_TID_TO_TASK |
+					  SCX_OPS_ENQ_BLOCKED,
 
 	/* high 8 bits are internal, don't include in SCX_OPS_ALL_FLAGS */
 	__SCX_OPS_INTERNAL_MASK		= 0xffLLU << 56,
@@ -1735,6 +1749,12 @@ enum scx_enq_flags {
 	 */
 	SCX_ENQ_LAST		= 1LLU << 41,
 
+	/*
+	 * The task is blocked on a mutex and is being kept runnable as a proxy
+	 * donor. Only passed to ops.enqueue() when %SCX_OPS_ENQ_BLOCKED is set.
+	 */
+	SCX_ENQ_BLOCKED		= 1LLU << 42,
+
 	/* high 8 bits are internal */
 	__SCX_ENQ_INTERNAL_MASK	= 0xffLLU << 56,
 
@@ -2044,6 +2064,7 @@ struct task_struct *scx_task_iter_next_locked(struct scx_task_iter *iter);
 bool scx_set_task_slice(struct task_struct *p, u64 slice);
 void scx_task_slice_ended(struct rq *rq, struct task_struct *p);
 void scx_task_unlink_from_dsq(struct task_struct *p, struct scx_dispatch_q *dsq);
+void scx_prepare_task_sched_change(struct task_struct *p);
 void scx_dispatch_dequeue(struct rq *rq, struct task_struct *p);
 void scx_do_enqueue_task(struct rq *rq, struct task_struct *p, u64 enq_flags,
 			 int sticky_cpu);
kernel/sched/ext/sub.c +6 -3
index 8c5b64b57b24a..f890ef2ee94d7 100644
@@ -1211,8 +1211,9 @@ static void scx_rehome_task(struct scx_sched *to, struct task_struct *p)
 	lockdep_assert_held(&p->pi_lock);
 	lockdep_assert_rq_held(task_rq(p));
 
+	scx_prepare_task_sched_change(p);
 	scoped_guard (sched_change, p, p->sched_class,
-		      DEQUEUE_SAVE | DEQUEUE_MOVE) {
+		      DEQUEUE_SAVE | DEQUEUE_MOVE | DEQUEUE_NOCLOCK) {
 		scx_disable_and_exit_task(scx_task_sched(p), p);
 		scx_set_task_state(p, SCX_TASK_INIT_BEGIN);
 		scx_set_task_state(p, SCX_TASK_INIT);
@@ -1242,8 +1243,9 @@ static void scx_punt_task(struct scx_sched *to, struct task_struct *p)
 	lockdep_assert_rq_held(task_rq(p));
 	WARN_ON_ONCE(!READ_ONCE(to->bypass_depth));
 
+	scx_prepare_task_sched_change(p);
 	scoped_guard (sched_change, p, p->sched_class,
-		      DEQUEUE_SAVE | DEQUEUE_MOVE) {
+		      DEQUEUE_SAVE | DEQUEUE_MOVE | DEQUEUE_NOCLOCK) {
 		scx_disable_and_exit_task(scx_task_sched(p), p);
 		scx_set_task_sched(p, to);
 	}
@@ -1897,8 +1899,9 @@ void scx_sub_enable_workfn(struct kthread_work *work)
 		if (!(p->scx.flags & SCX_TASK_SUB_INIT))
 			continue;
 
+		scx_prepare_task_sched_change(p);
 		scoped_guard (sched_change, p, p->sched_class,
-			      DEQUEUE_SAVE | DEQUEUE_MOVE) {
+			      DEQUEUE_SAVE | DEQUEUE_MOVE | DEQUEUE_NOCLOCK) {
 			/*
 			 * $p must be either READY or ENABLED. If ENABLED,
 			 * __scx_disabled_and_exit_task() first disables and
tools/sched_ext/include/scx/compat.h +1 -0
index d2e4384df5af7..8313a2f6d03ce 100644
@@ -117,6 +117,7 @@ static inline bool __COMPAT_struct_has_field(const char *type, const char *field
 #define SCX_OPS_ALLOW_QUEUED_WAKEUP SCX_OPS_FLAG(SCX_OPS_ALLOW_QUEUED_WAKEUP)
 #define SCX_OPS_BUILTIN_IDLE_PER_NODE SCX_OPS_FLAG(SCX_OPS_BUILTIN_IDLE_PER_NODE)
 #define SCX_OPS_ALWAYS_ENQ_IMMED SCX_OPS_FLAG(SCX_OPS_ALWAYS_ENQ_IMMED)
+#define SCX_OPS_ENQ_BLOCKED SCX_OPS_FLAG(SCX_OPS_ENQ_BLOCKED)
 
 #define SCX_PICK_IDLE_FLAG(name) __COMPAT_ENUM_OR_ZERO("scx_pick_idle_cpu_flags", #name)
 
tools/sched_ext/include/scx/enum_defs.autogen.h +1 -0
index b1351f346e1d9..d3e1185f00ac1 100644
@@ -85,6 +85,7 @@
 #define HAVE_SCX_ENQ_RESCUE
 #define HAVE_SCX_ENQ_REENQ
 #define HAVE_SCX_ENQ_LAST
+#define HAVE_SCX_ENQ_BLOCKED
 #define HAVE___SCX_ENQ_INTERNAL_MASK
 #define HAVE_SCX_ENQ_CLEAR_OPSS
 #define HAVE_SCX_ENQ_DSQ_PRIQ
tools/sched_ext/include/scx/enums.autogen.bpf.h +3 -0
index 7268131010de3..c45ce9f1c39ff 100644
@@ -133,6 +133,9 @@ const volatile u64 __SCX_ENQ_REENQ __weak;
 const volatile u64 __SCX_ENQ_LAST __weak;
 #define SCX_ENQ_LAST __SCX_ENQ_LAST
 
+const volatile u64 __SCX_ENQ_BLOCKED __weak;
+#define SCX_ENQ_BLOCKED __SCX_ENQ_BLOCKED
+
 const volatile u64 __SCX_ENQ_CLEAR_OPSS __weak;
 #define SCX_ENQ_CLEAR_OPSS __SCX_ENQ_CLEAR_OPSS
 
tools/sched_ext/include/scx/enums.autogen.h +1 -0
index e616326545172..52d64a67401ea 100644
@@ -48,6 +48,7 @@
 	SCX_ENUM_SET(skel, scx_enq_flags, SCX_ENQ_RESCUE); \
 	SCX_ENUM_SET(skel, scx_enq_flags, SCX_ENQ_REENQ); \
 	SCX_ENUM_SET(skel, scx_enq_flags, SCX_ENQ_LAST); \
+	SCX_ENUM_SET(skel, scx_enq_flags, SCX_ENQ_BLOCKED); \
 	SCX_ENUM_SET(skel, scx_enq_flags, SCX_ENQ_CLEAR_OPSS); \
 	SCX_ENUM_SET(skel, scx_enq_flags, SCX_ENQ_DSQ_PRIQ); \
 	SCX_ENUM_SET(skel, scx_deq_flags, SCX_DEQ_SCHED_CHANGE); \
Signature · 1 line
2.55.0
#48 lore raw

Re: [PATCH 15/18] sched_ext: Delegate proxy donor admission to BPF schedulers

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] The new BPF scheduler receives `ops.dequeue` before `ops.enable` or `ops.enqueue` for retained…

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] The new BPF scheduler receives ops.dequeue before ops.enable or ops.enqueue for retained proxy donors during root scheduler enablement, violating the BPF lifecycle contract. --

commit 2446f36cb7fab90783875259131bf6b0d4fa2d3d
Author: Andrea Righi <[email protected]>

sched_ext: Delegate proxy donor admission to BPF schedulers

Proxy execution keeps a mutex-blocked donor runnable so that its scheduling context can execute the mutex owner. This patch introduces SCX_OPS_ENQ_BLOCKED to allow schedulers to continue owning blocked donors. A proxy-migrated donor returns through the full wakeup activation path when the mutex is released.

Quoted 4 lines
kernel/sched/ext/ext.c +0 -0
index f23e9f1ee2f3c..06067fe504a24 100644

[ ... ]

kernel/sched/ext/ext.c static void scx_root_enable_workfn(struct kthread_work *work) · line 8118

Quoted 12 lines
kernel/sched/ext/ext.c +8 -0
@@ -8027,6 +8108,14 @@ static void scx_root_enable_workfn(struct kthread_work *work)
 
 		if (old_class != new_class)
 			queue_flags |= DEQUEUE_CLASS;
+		if (old_class == new_class && new_class == &ext_sched_class) {
+			/*
+			 * This is an EXT-to-EXT scheduler ownership change, so
+			 * sched_change_begin() won't end retained proxy execution.
+			 */
+			scx_prepare_task_sched_change(p);
+			queue_flags |= DEQUEUE_NOCLOCK;
+		}
                 ^

[Severity: High]
Does this code call ops.dequeue() on the new BPF scheduler before ops.enable() has been called?

Looking at scx_root_enable_workfn() in kernel/sched/ext/ext.c, the task's p->scx.sched is updated to the new scheduler in a previous loop.

When scx_prepare_task_sched_change(p) is called during an EXT-to-EXT scheduler ownership change, it triggers the following sequence:

scx_prepare_task_sched_change()
sched_proxy_block_task()
dequeue_task_scx()
ops.dequeue()

Because dequeue_task_scx() sees the SCX_TASK_IN_CUSTODY flag left over from the old scheduler, it executes ops.dequeue() on the new scheduler.

Since DEQUEUE_CLASS is omitted from queue_flags for EXT-to-EXT transitions, the switching_to hook is skipped and scx_enable_task() is never called to initialize the per-task state for the new scheduler.

Can this lead to the BPF program experiencing out-of-bounds map accesses or NULL pointer dereferences, as it relies on ops.enable() to initialize state before ops.dequeue() is called?

Quoted 3 lines
kernel/sched/ext/ext.c +0 -0
 		scoped_guard (sched_change, p, new_class, queue_flags) {
 			scx_set_task_slice(p, READ_ONCE(sch->slice_dfl));
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=15
author reviewer
#49 lore raw

Re: [PATCH 15/18] sched_ext: Delegate proxy donor admission to BPF schedulers

Quoted 54 lines · On Mon, Aug 31, 2026 at 06:08:37PM +0000, [email protected] wrote:

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] The new BPF scheduler receives ops.dequeue before ops.enable or ops.enqueue for retained proxy donors during root scheduler enablement, violating the BPF lifecycle contract. --

commit 2446f36cb7fab90783875259131bf6b0d4fa2d3d
Author: Andrea Righi <[email protected]>

sched_ext: Delegate proxy donor admission to BPF schedulers

Proxy execution keeps a mutex-blocked donor runnable so that its scheduling context can execute the mutex owner. This patch introduces SCX_OPS_ENQ_BLOCKED to allow schedulers to continue owning blocked donors. A proxy-migrated donor returns through the full wakeup activation path when the mutex is released.

Quoted 4 lines
kernel/sched/ext/ext.c +0 -0
index f23e9f1ee2f3c..06067fe504a24 100644

[ ... ]

kernel/sched/ext/ext.c static void scx_root_enable_workfn(struct kthread_work *work) · line 8118

Quoted 12 lines
kernel/sched/ext/ext.c +8 -0
@@ -8027,6 +8108,14 @@ static void scx_root_enable_workfn(struct kthread_work *work)
 
 		if (old_class != new_class)
 			queue_flags |= DEQUEUE_CLASS;
+		if (old_class == new_class && new_class == &ext_sched_class) {
+			/*
+			 * This is an EXT-to-EXT scheduler ownership change, so
+			 * sched_change_begin() won't end retained proxy execution.
+			 */
+			scx_prepare_task_sched_change(p);
+			queue_flags |= DEQUEUE_NOCLOCK;
+		}
                 ^

[Severity: High]
Does this code call ops.dequeue() on the new BPF scheduler before ops.enable() has been called?

Looking at scx_root_enable_workfn() in kernel/sched/ext/ext.c, the task's p->scx.sched is updated to the new scheduler in a previous loop.

When scx_prepare_task_sched_change(p) is called during an EXT-to-EXT scheduler ownership change, it triggers the following sequence:

scx_prepare_task_sched_change()
sched_proxy_block_task()
dequeue_task_scx()
ops.dequeue()

Because dequeue_task_scx() sees the SCX_TASK_IN_CUSTODY flag left over from the old scheduler, it executes ops.dequeue() on the new scheduler.

Quoted 7 lines

Since DEQUEUE_CLASS is omitted from queue_flags for EXT-to-EXT transitions, the switching_to hook is skipped and scx_enable_task() is never called to initialize the per-task state for the new scheduler.

Can this lead to the BPF program experiencing out-of-bounds map accesses or NULL pointer dereferences, as it relies on ops.enable() to initialize state before ops.dequeue() is called?

I think the described sequence can't occur during root scheduler enable:
sched_ext is globally disabled while tasks are initialized, so tasks always enter this path from SCX_TASK_NONE without custody from previous BPF scheduler.

And I think the EXT-to-EXT scenario is unreachable. Here's why: after __scx_enabled is set, a concurrent transition to EXT goes through switching_to_scx(), which calls scx_enable_task() and changes the task from SCX_TASK_READY to SCX_TASK_ENABLED. The second loop consequently skips that task. Otherwise, a task being moved to EXT still has its old non-EXT class, so DEQUEUE_CLASS handles the retained proxy state before switching_to_scx() enables it.

Therefore a SCX_TASK_READY task can't legitimately reach the old_class == new_class == &ext_sched_class case with SCX_TASK_IN_CUSTODY and ops.dequeue() can't be called on the new scheduler before ops.enable().

So the EXT-to-EXT special case in the root-enable loop can be removed.

-Andrea

maintainer
#50 lore raw

Re: [PATCH 15/18] sched_ext: Delegate proxy donor admission to BPF schedulers

In reply to Andrea Righi
Quoted 7 lines · Andrea Righi
kernel/sched/ext/ext.c +0 -0
index 5f4956cf7e369..aa6ba9e8d7cb0 100644
@@ -26,7 +26,28 @@ DEFINE_RAW_SPINLOCK(scx_sched_lock);
 
 bool scx_allow_proxy_exec(const struct task_struct *p)

kernel/sched/ext/ext.c DEFINE_RAW_SPINLOCK(scx_sched_lock); · line 36

Quoted 10 lines
kernel/sched/ext/ext.c +8 -1
 {
-	return p->sched_class != &ext_sched_class;
+	struct scx_sched *sch;
+
+	if (p->sched_class != &ext_sched_class)
+		return true;
+
+	sch = scx_task_sched(p);
+	return !sch || (sch->ops.flags & SCX_OPS_ENQ_BLOCKED);
+}

Can we have this thing either be inline or wrapped in an inline function that has scx_enabled() on?

author reviewer
#51 lore raw

Re: [PATCH 15/18] sched_ext: Delegate proxy donor admission to BPF schedulers

Quoted 8 lines · Peter Zijlstra
Quoted 6 lines · Andrea Righi
kernel/sched/ext/ext.c +0 -0
index 5f4956cf7e369..aa6ba9e8d7cb0 100644
@@ -26,7 +26,28 @@ DEFINE_RAW_SPINLOCK(scx_sched_lock);
 
Quoted 15 lines

kernel/sched/ext/ext.c DEFINE_RAW_SPINLOCK(scx_sched_lock); · line 36

Quoted 11 lines
kernel/sched/ext/ext.c +8 -1
 bool scx_allow_proxy_exec(const struct task_struct *p)
 {
-	return p->sched_class != &ext_sched_class;
+	struct scx_sched *sch;
+
+	if (p->sched_class != &ext_sched_class)
+		return true;
+
+	sch = scx_task_sched(p);
+	return !sch || (sch->ops.flags & SCX_OPS_ENQ_BLOCKED);
+}

Can we have this thing either be inline or wrapped in an inline function that has scx_enabled() on?

Yes, that makes sense. This is called from the scheduler blocking path, so the inactive sched_ext case should be handled through the scx_enabled() static key.

I'll add an inline scx_allow_proxy_exec() wrapper that returns true when sched_ext is disabled and calls __scx_allow_proxy_exec() otherwise.

Thanks,
-Andrea

maintainer
#52 lore raw

Re: [PATCH 15/18] sched_ext: Delegate proxy donor admission to BPF schedulers

In reply to Andrea Righi
Quoted 18 lines · Andrea Righi
kernel/sched/ext/ext.c +13 -0
+/*
+ * End retained proxy execution before changing @p's BPF scheduler ownership.
+ * Called with @p's pi and rq locks held immediately before
+ * sched_change_begin(). The caller must pass DEQUEUE_NOCLOCK so the rq clock
+ * is updated only once.
+ */
+void scx_prepare_task_sched_change(struct task_struct *p)
+{
+	lockdep_assert_held(&p->pi_lock);
+	lockdep_assert_rq_held(task_rq(p));
+
+	update_rq_clock(task_rq(p));
+	sched_proxy_block_task(task_rq(p), p);
 }
 
 /*

kernel/sched/ext/ext.c static void scx_root_enable_workfn(struct kthread_work *work) · line 8116

Quoted 9 lines
kernel/sched/ext/ext.c +6 -0
@@ -8027,6 +8108,14 @@ static void scx_root_enable_workfn(struct kthread_work *work)
 
 		if (old_class != new_class)
 			queue_flags |= DEQUEUE_CLASS;
+		if (old_class == new_class && new_class == &ext_sched_class) {
+			/*
+			 * This is an EXT-to-EXT scheduler ownership change, so
+			 * sched_change_begin() won't end retained proxy execution.
+			 */
+			scx_prepare_task_sched_change(p);

Here, see, this is what I was expecting for this thing. Just have it be always instead of for ext-ext only.

author reviewer
#53 lore raw

PATCH 16/18 sched_ext: Add selftest for blocked donor admission

In reply to Andrea Righi

SCX_OPS_ENQ_BLOCKED allows BPF schedulers to receive blocked proxy donors through ops.enqueue(). SCX_ENQ_BLOCKED identifies blocked-donor admission requests. Add selftest coverage for this interface.

Exercise a priority inversion using a weighted-vruntime BPF scheduler. A nice +19 owner holds a shared mutex, a nice -20 donor blocks on it, and nice 0 CPU contenders, one per allowed CPU, keep the system busy. Test both a same-CPU topology and a cross-CPU topology with the donor and owner on different CPUs.

Treat blocked donors according to the normal BPF ordering policy and assign the default slice on every enqueue, as for other tasks. Run each CPU placement configuration with SCX_OPS_ENQ_BLOCKED first disabled and then enabled, count blocked-donor enqueues by CPU and report average mutex hold and wait times. Verify that full wakeups are never reported as blocked-donor admissions by checking that SCX_ENQ_WAKEUP and SCX_ENQ_BLOCKED are not set together.

Proxy execution coverage requires CONFIG_SCHED_PROXY_EXEC=y, which the selftest config selects. Access to the kernel mutex is provided via a loadable kernel module, built through TEST_GEN_MODS_DIR and managed by the test.

---
 tools/testing/selftests/sched_ext/.gitignore  |   4 +
 tools/testing/selftests/sched_ext/Makefile    |   2 +
 tools/testing/selftests/sched_ext/config      |   2 +
 .../selftests/sched_ext/enq_blocked.bpf.c     | 116 +++
 .../testing/selftests/sched_ext/enq_blocked.c | 917 ++++++++++++++++++
 .../testing/selftests/sched_ext/enq_blocked.h |  28 +
 .../selftests/sched_ext/test_modules/Makefile |  13 +
 .../test_modules/scx_enq_blocked_test.c       | 195 ++++
 8 files changed, 1277 insertions(+)
 create mode 100644 tools/testing/selftests/sched_ext/enq_blocked.bpf.c
 create mode 100644 tools/testing/selftests/sched_ext/enq_blocked.c
 create mode 100644 tools/testing/selftests/sched_ext/enq_blocked.h
 create mode 100644 tools/testing/selftests/sched_ext/test_modules/Makefile
 create mode 100644 tools/testing/selftests/sched_ext/test_modules/scx_enq_blocked_test.c
tools/testing/selftests/sched_ext/.gitignore +4 -0
index ae5491a114c09..54a1fd2af713d 100644
@@ -4,3 +4,7 @@
 !Makefile
 !.gitignore
 !config
+!test_modules/
+!test_modules/scx_enq_blocked_test.c
+!test_modules/Makefile
+test_modules/*.mod.c
tools/testing/selftests/sched_ext/Makefile +2 -0
index 3cfe90e0f34fa..51a16b3d32d9b 100644
@@ -5,6 +5,7 @@ include ../../../scripts/Makefile.arch
 include ../../../scripts/Makefile.include
 
 TEST_GEN_PROGS := runner
+TEST_GEN_MODS_DIR := test_modules
 
 # override lib.mk's default rules
 OVERRIDE_TARGETS := 1
@@ -164,6 +165,7 @@ all_test_bpfprogs := $(foreach prog,$(wildcard *.bpf.c),$(INCLUDE_DIR)/$(patsubs
 auto-test-targets :=			\
 	create_dsq			\
 	dequeue				\
+	enq_blocked			\
 	enq_last_no_enq_fails		\
 	ddsp_bogus_dsq_fail		\
 	ddsp_vtimelocal_fail		\
tools/testing/selftests/sched_ext/config +2 -0
index aa901b05c8ad6..affa3cf33470a 100644
@@ -6,3 +6,5 @@ CONFIG_BPF=y
 CONFIG_BPF_SYSCALL=y
 CONFIG_DEBUG_INFO=y
 CONFIG_DEBUG_INFO_BTF=y
+CONFIG_EXPERT=y
+CONFIG_SCHED_PROXY_EXEC=y
tools/testing/selftests/sched_ext/enq_blocked.bpf.c new file +116 -0

116 lines not on this page · raw message

tools/testing/selftests/sched_ext/enq_blocked.c new file +917 -0

917 lines not on this page · raw message

tools/testing/selftests/sched_ext/enq_blocked.h new file +28 -0
new file mode 100644
index 0000000000000..ef1eb97feebe6
@@ -0,0 +1,28 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES */
+#ifndef __ENQ_BLOCKED_H
+#define __ENQ_BLOCKED_H
+
+#include <linux/ioctl.h>
+#include <linux/types.h>
+
+struct enq_blocked_stats {
+	__u64 hold_time_ns;
+	__u64 wait_time_ns;
+	__u64 nr_holds;
+	__u64 nr_waits;
+};
+
+#define ENQ_BLOCKED_IOCTL_OWNER	_IO('s', 1)
+#define ENQ_BLOCKED_IOCTL_DONOR	_IO('s', 2)
+#define ENQ_BLOCKED_IOCTL_RESET_STATS	_IO('s', 3)
+#define ENQ_BLOCKED_IOCTL_GET_STATS	_IOR('s', 4, struct enq_blocked_stats)
+#define ENQ_BLOCKED_IOCTL_PREP_ATTACH	_IO('s', 5)
+#define ENQ_BLOCKED_IOCTL_ATTACH_DONE	_IO('s', 6)
+#define ENQ_BLOCKED_IOCTL_DONOR_STATE	_IO('s', 7)
+#define ENQ_BLOCKED_IOCTL_PROXY_SUPPORTED _IO('s', 8)
+
+#define ENQ_BLOCKED_DONOR_SLEEPING	(1U << 0)
+#define ENQ_BLOCKED_DONOR_ON_RQ		(1U << 1)
+
+#endif /* __ENQ_BLOCKED_H */
tools/testing/selftests/sched_ext/test_modules/Makefile new file +13 -0
new file mode 100644
index 0000000000000..a0e9e9401ead6
@@ -0,0 +1,13 @@
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES
+
+TESTMODS_DIR := $(realpath $(dir $(abspath $(lastword $(MAKEFILE_LIST)))))
+KDIR ?= $(if $(O),$(O),$(realpath ../../../../..))
+
+obj-m += scx_enq_blocked_test.o
+
+all:
+	+$(Q)$(MAKE) -C $(KDIR) M=$(TESTMODS_DIR) modules
+
+clean:
+	+$(Q)$(MAKE) -C $(KDIR) M=$(TESTMODS_DIR) clean
tools/testing/selftests/sched_ext/test_modules/scx_enq_blocked_test.c new file +195 -0

195 lines not on this page · raw message

Signature · 1 line
2.55.0
author reviewer
#54 lore raw

PATCH 17/18 sched_ext: scx_qmap: Add proxy execution support

In reply to Andrea Righi

Add a -X option to opt scx_qmap into queueing mutex-blocked tasks for proxy execution. Without the option, SCX_OPS_ENQ_BLOCKED remains clear and mutex waiters block normally. With -X, blocked donors are passed to qmap_enqueue() with SCX_ENQ_BLOCKED.

When scx_qmap receives a blocked donor, select a cid allowed by the task and held by qmap, preferring its current cid. Dispatch the donor to that cid's local DSQ with a fresh slice and SCX_ENQ_PREEMPT, adding SCX_ENQ_IMMED when the cid is time-shared. This places the donor at the head of an eligible DSQ and requests an immediate reschedule, allowing the core proxy-exec path to run the mutex owner using the donor's scheduling context as soon as the donor is selected.

The blocked policy is intentionally unfair and can strongly prioritize tasks using contended mutexes, but scx_qmap is a demo scheduler and such aggressive behavior makes proxy-exec support easy to observe. Count all blocked dispatch attempts in nr_enq_blocked and report their per-interval delta.

---
 tools/sched_ext/scx_qmap.bpf.c | 53 +++++++++++++++++++++++++++++++++-
 tools/sched_ext/scx_qmap.c     | 13 +++++++--
 tools/sched_ext/scx_qmap.h     |  1 +
 3 files changed, 63 insertions(+), 4 deletions(-)
tools/sched_ext/scx_qmap.bpf.c +52 -1
index 5bb8b90a275a3..21f8a9be9650c 100644
@@ -439,6 +439,9 @@ void BPF_STRUCT_OPS(qmap_enqueue, struct task_struct *p, u64 enq_flags)
 	 */
 	taskc->core_sched_seq = qa.core_sched_tail_seqs[idx]++;
 
+	if (enq_flags & SCX_ENQ_BLOCKED)
+		__sync_fetch_and_add(&qa.nr_enq_blocked, 1);
+
 	/*
 	 * A task of ours that can run on none of our self cids - the parent
 	 * didn't grant them or we delegated them to children - would starve in
@@ -446,7 +449,8 @@ void BPF_STRUCT_OPS(qmap_enqueue, struct task_struct *p, u64 enq_flags)
 	 *
 	 * Force it onto its first allowed cid's local DSQ. If we hold that cid
 	 * it runs. Otherwise the insert carries SCX_ENQ_RESCUE and the kernel
-	 * diverts the task to its rescue path.
+	 * diverts the task to its rescue path. Do this before the blocked-donor
+	 * fast paths, which also require an eligible self cid to make progress.
 	 */
 	if (!cmask_intersects(&taskc->cpus_allowed, &qa.self_cids.mask)) {
 		s32 c = cmask_next_set_wrap(&taskc->cpus_allowed, 0);
@@ -460,6 +464,53 @@ void BPF_STRUCT_OPS(qmap_enqueue, struct task_struct *p, u64 enq_flags)
 		}
 	}
 
+	/*
+	 * SCX_OPS_ALWAYS_ENQ_IMMED makes the local insertion below implicitly
+	 * carry SCX_ENQ_IMMED. If the CPU can't run the blocked donor immediately,
+	 * the core returns it through ops.enqueue() with SCX_ENQ_REENQ. Inserting
+	 * it into the same local DSQ would repeat the IMMED handback until the
+	 * scheduler is ejected. Move reenqueued blocked donors to the shared DSQ,
+	 * which doesn't carry SCX_ENQ_IMMED, so another CPU can consume them.
+	 */
+	if ((enq_flags & (SCX_ENQ_BLOCKED | SCX_ENQ_REENQ)) ==
+	    (SCX_ENQ_BLOCKED | SCX_ENQ_REENQ)) {
+		taskc->force_local = false;
+		scx_bpf_dsq_insert(p, SHARED_DSQ, 0, enq_flags);
+		cid = cmask_next_and2_set_wrap(&taskc->cpus_allowed,
+					       &qa.idle_cids.mask,
+					       &qa.self_cids.mask, 0);
+		if (cid < scx_bpf_nr_cids())
+			scx_bpf_kick_cid(cid, SCX_KICK_IDLE);
+		return;
+	}
+
+	/*
+	 * Insert a blocked mutex donor at the head of an eligible local DSQ with
+	 * a fresh slice and %SCX_ENQ_PREEMPT, requesting an immediate reschedule.
+	 * The test above guarantees that cpus_allowed intersects self_cids, but
+	 * the donor's current cid may have been delegated to a child. Search the
+	 * intersection starting at the current cid, preserving it when qmap still
+	 * holds it and wrapping to another eligible self cid otherwise.
+	 *
+	 * A self cid may be held exclusively with SCX_CAP_ENQ or time-shared with
+	 * only SCX_CAP_ENQ_IMMED. Add needs_immed() so either kind can accept the
+	 * local insertion instead of rejecting and reenqueuing the donor for a
+	 * capability miss. Once selected, the core proxy-exec path can run the
+	 * mutex owner using the donor's scheduling context.
+	 *
+	 * This policy is intentionally unfair and can strongly prioritize tasks
+	 * using contended mutexes; scx_qmap is a demonstration scheduler and
+	 * this behavior makes proxy-exec support easy to observe.
+	 */
+	if (enq_flags & SCX_ENQ_BLOCKED) {
+		cid = cmask_next_and_set_wrap(&taskc->cpus_allowed,
+					      &qa.self_cids.mask,
+					      scx_bpf_task_cid(p));
+		scx_bpf_dsq_insert(p, SCX_DSQ_LOCAL_ON | cid, slice_ns,
+				   enq_flags | needs_immed(cid) | SCX_ENQ_PREEMPT);
+		return;
+	}
+
 	/*
 	 * Fault injection: deliberately dispatch one of our own tasks to a cid
 	 * we don't hold. The inserts carry SCX_ENQ_RESCUE and divert to the
tools/sched_ext/scx_qmap.c +10 -3
index 5bb5f687e579c..d5226e0716573 100644
@@ -46,7 +46,7 @@ const char help_fmt[] =
 "See the top-of-file comment in .bpf.c for the design.\n"
 "\n"
 "Usage: %s [-s SLICE_US] [-e COUNT] [-t COUNT] [-T COUNT] [-l COUNT] [-b COUNT]\n"
-"       [-N COUNT] [-P] [-M] [-H] [-c CG_PATH] [-d PID] [-D LEN] [-S] [-p] [-I]\n"
+"       [-N COUNT] [-P] [-M] [-H] [-c CG_PATH] [-d PID] [-D LEN] [-S] [-p] [-I] [-X]\n"
 "       [-F COUNT] [-i SEC] [-R MS] [-J MODE] [-v]\n"
 "\n"
 "  -s SLICE_US   Override slice duration\n"
@@ -65,6 +65,7 @@ const char help_fmt[] =
 "  -S            Suppress qmap-specific debug dump\n"
 "  -p            Switch only tasks on SCHED_EXT policy instead of all\n"
 "  -I            Turn on SCX_OPS_ALWAYS_ENQ_IMMED\n"
+"  -X            Turn on SCX_OPS_ENQ_BLOCKED\n"
 "  -F COUNT      IMMED stress: force every COUNT'th enqueue to a busy local DSQ (use with -I)\n"
 "  -C MODE       cid-override test (shuffle|bad-dup|bad-range|bad-mono)\n"
 "  -i SEC        Stats interval, seconds (default 5)\n"
@@ -107,6 +108,7 @@ struct hier_prev {
 	u64 nr_dsps[MAX_SUB_SCHEDS];
 	u64 nr_reenq_cap;
 	u64 nr_reenq_immed;
+	u64 nr_enq_blocked;
 	u64 nr_inject_attempts;
 	u64 nr_rescue_dsp;
 };
@@ -190,14 +192,16 @@ static void print_hier(struct qmap_arena *qa, struct hier_prev *prev, u64 own_cg
 	}
 
 	format_cid_ranges(qa, CID_SHARED, ranges, sizeof(ranges));
-	printf("hier   : nsub=%llu excl=%u shared=%s rr=%s reenq cap/immed +%llu/+%llu inj=+%llu rescue=+%llu\n",
+	printf("hier   : nsub=%llu excl=%u shared=%s rr=%s reenq cap/immed +%llu/+%llu blocked=+%llu inj=+%llu rescue=+%llu\n",
 	       (unsigned long long)qa->nr_sub_scheds, qa->part.nr_excl, ranges, rr,
 	       (unsigned long long)(qa->nr_reenq_cap - prev->nr_reenq_cap),
 	       (unsigned long long)(qa->nr_reenq_immed - prev->nr_reenq_immed),
+	       (unsigned long long)(qa->nr_enq_blocked - prev->nr_enq_blocked),
 	       (unsigned long long)(qa->nr_inject_attempts - prev->nr_inject_attempts),
 	       (unsigned long long)(qa->nr_rescue_dsp - prev->nr_rescue_dsp));
 	prev->nr_reenq_cap = qa->nr_reenq_cap;
 	prev->nr_reenq_immed = qa->nr_reenq_immed;
+	prev->nr_enq_blocked = qa->nr_enq_blocked;
 	prev->nr_inject_attempts = qa->nr_inject_attempts;
 	prev->nr_rescue_dsp = qa->nr_rescue_dsp;
 
@@ -263,7 +267,7 @@ int main(int argc, char **argv)
 	skel->rodata->max_tasks = 16384;
 
 	while ((opt = getopt(argc, argv,
-			     "s:e:t:T:l:b:N:PMHc:d:D:SpIF:C:i:R:J:B:q:vh")) != -1) {
+			     "s:e:t:T:l:b:N:PMHc:d:D:SpIXF:C:i:R:J:B:q:vh")) != -1) {
 		switch (opt) {
 		case 's':
 			skel->rodata->slice_ns = strtoull(optarg, NULL, 0) * 1000;
@@ -324,6 +328,9 @@ int main(int argc, char **argv)
 		case 'I':
 			skel->struct_ops.qmap_ops->flags |= SCX_OPS_ALWAYS_ENQ_IMMED;
 			break;
+		case 'X':
+			skel->struct_ops.qmap_ops->flags |= SCX_OPS_ENQ_BLOCKED;
+			break;
 		case 'F':
 			skel->rodata->immed_stress_nth = strtoul(optarg, NULL, 0);
 			break;
tools/sched_ext/scx_qmap.h +1 -0
index c78d61806b398..fc10b542fb477 100644
@@ -178,6 +178,7 @@ struct qmap_arena {
 	/* bpf -> userspace: stats */
 	u64 nr_reenq_cap;		/* SCX_TASK_REENQ_CAP bounces */
 	u64 nr_reenq_immed;		/* SCX_TASK_REENQ_IMMED bounces */
+	u64 nr_enq_blocked;		/* SCX_ENQ_BLOCKED dispatches */
 	u64 nr_inject_attempts;		/* fault-injection: dispatches to an unheld cid */
 	u64 nr_rescue_dsp;		/* SCX_ENQ_RESCUE dispatch attempts */
 	u32 inject_mode;		/* fault-injection mode (QMAP_INJ_*) */
Signature · 1 line
2.55.0
#55 lore raw

Re: [PATCH 17/18] sched_ext: scx_qmap: Add proxy execution support

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] Missing bounds check on `cid` returned from `cmask_next_and_set_wrap()` due to a TOCTOU race with…

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] Missing bounds check on cid returned from cmask_next_and_set_wrap() due to a TOCTOU race with qa.self_cids.mask modifications. --

commit 564b1258b3dc07e22341c55b61be164307b61870
Author: Andrea Righi <[email protected]>

sched_ext: scx_qmap: Add proxy execution support

Add a -X option to opt scx_qmap into queueing mutex-blocked tasks for proxy execution. When scx_qmap receives a blocked donor, it selects an eligible cid and dispatches the donor to that local DSQ.

Quoted 4 lines
tools/sched_ext/scx_qmap.bpf.c +0 -0
index 5bb8b90a275a3..21f8a9be9650c 100644

[ ... ]

Quoted 1 line
tools/sched_ext/scx_qmap.bpf.c +0 -0
@@ -460,6 +464,53 @@ void BPF_STRUCT_OPS(qmap_enqueue, struct task_struct *p, u64 enq_flags)

[ ... ]

Quoted 16 lines
tools/sched_ext/scx_qmap.bpf.c +16 -0
+	/*
+	 * Insert a blocked mutex donor at the head of an eligible local DSQ with
+	 * a fresh slice and %SCX_ENQ_PREEMPT, requesting an immediate reschedule.
+	 * The test above guarantees that cpus_allowed intersects self_cids, but
+	 * the donor's current cid may have been delegated to a child. Search the
+	 * intersection starting at the current cid, preserving it when qmap still
+	 * holds it and wrapping to another eligible self cid otherwise.
+	 *
+	 * A self cid may be held exclusively with SCX_CAP_ENQ or time-shared with
+	 * only SCX_CAP_ENQ_IMMED. Add needs_immed() so either kind can accept the
+	 * local insertion instead of rejecting and reenqueuing the donor for a
+	 * capability miss. Once selected, the core proxy-exec path can run the
+	 * mutex owner using the donor's scheduling context.
+	 *
+	 * This policy is intentionally unfair and can strongly prioritize tasks
+	 * using contended mutexes; scx_qmap is a demonstration scheduler and

tools/sched_ext/scx_qmap.bpf.c void BPF_STRUCT_OPS(qmap_enqueue, struct task_struct *p, u64 enq_flags) · line 512

Quoted 10 lines
tools/sched_ext/scx_qmap.bpf.c +10 -0
+	 * this behavior makes proxy-exec support easy to observe.
+	 */
+	if (enq_flags & SCX_ENQ_BLOCKED) {
+		cid = cmask_next_and_set_wrap(&taskc->cpus_allowed,
+					      &qa.self_cids.mask,
+					      scx_bpf_task_cid(p));
+		scx_bpf_dsq_insert(p, SCX_DSQ_LOCAL_ON | cid, slice_ns,
+				   enq_flags | needs_immed(cid) | SCX_ENQ_PREEMPT);
+		return;
+	}

[Severity: High]
Does this code risk an out-of-bounds access if the intersection becomes empty?

In qmap_enqueue(), there is an earlier check for cmask_intersects(&taskc->cpus_allowed, &qa.self_cids.mask).

However, if apply_partition() modifies and zeroes qa.self_cids.mask concurrently on another CPU after the check, the intersection might become empty before cmask_next_and_set_wrap() executes.

If this happens, cmask_next_and_set_wrap() returns the out-of-bounds sentinel scx_bpf_nr_cids().

This out-of-bounds cid is then used without a bounds check in:

    scx_bpf_dsq_insert(p, SCX_DSQ_LOCAL_ON | cid, ...)

which can cause scx_cpu_valid() to fail and forcefully eject the BPF scheduler. It could also lead to an out-of-bounds read in needs_immed(cid).

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=17
#56 lore raw

Re: [PATCH 17/18] sched_ext: scx_qmap: Add proxy execution support

In reply to Andrea Righi
Quoted 92 lines · Andrea Righi

Add a -X option to opt scx_qmap into queueing mutex-blocked tasks for proxy execution. Without the option, SCX_OPS_ENQ_BLOCKED remains clear and mutex waiters block normally. With -X, blocked donors are passed to qmap_enqueue() with SCX_ENQ_BLOCKED.

When scx_qmap receives a blocked donor, select a cid allowed by the task and held by qmap, preferring its current cid. Dispatch the donor to that cid's local DSQ with a fresh slice and SCX_ENQ_PREEMPT, adding SCX_ENQ_IMMED when the cid is time-shared. This places the donor at the head of an eligible DSQ and requests an immediate reschedule, allowing the core proxy-exec path to run the mutex owner using the donor's scheduling context as soon as the donor is selected.

The blocked policy is intentionally unfair and can strongly prioritize tasks using contended mutexes, but scx_qmap is a demo scheduler and such aggressive behavior makes proxy-exec support easy to observe. Count all blocked dispatch attempts in nr_enq_blocked and report their per-interval delta.

---
 tools/sched_ext/scx_qmap.bpf.c | 53 +++++++++++++++++++++++++++++++++-
 tools/sched_ext/scx_qmap.c     | 13 +++++++--
 tools/sched_ext/scx_qmap.h     |  1 +
 3 files changed, 63 insertions(+), 4 deletions(-)
tools/sched_ext/scx_qmap.bpf.c +42 -1
index 5bb8b90a275a3..21f8a9be9650c 100644
@@ -439,6 +439,9 @@ void BPF_STRUCT_OPS(qmap_enqueue, struct task_struct *p, u64 enq_flags)
 	 */
 	taskc->core_sched_seq = qa.core_sched_tail_seqs[idx]++;
 
+	if (enq_flags & SCX_ENQ_BLOCKED)
+		__sync_fetch_and_add(&qa.nr_enq_blocked, 1);
+
 	/*
 	 * A task of ours that can run on none of our self cids - the parent
 	 * didn't grant them or we delegated them to children - would starve in
@@ -446,7 +449,8 @@ void BPF_STRUCT_OPS(qmap_enqueue, struct task_struct *p, u64 enq_flags)
 	 *
 	 * Force it onto its first allowed cid's local DSQ. If we hold that cid
 	 * it runs. Otherwise the insert carries SCX_ENQ_RESCUE and the kernel
-	 * diverts the task to its rescue path.
+	 * diverts the task to its rescue path. Do this before the blocked-donor
+	 * fast paths, which also require an eligible self cid to make progress.
 	 */
 	if (!cmask_intersects(&taskc->cpus_allowed, &qa.self_cids.mask)) {
 		s32 c = cmask_next_set_wrap(&taskc->cpus_allowed, 0);
@@ -460,6 +464,53 @@ void BPF_STRUCT_OPS(qmap_enqueue, struct task_struct *p, u64 enq_flags)
 		}
 	}
 
+	/*
+	 * SCX_OPS_ALWAYS_ENQ_IMMED makes the local insertion below implicitly
+	 * carry SCX_ENQ_IMMED. If the CPU can't run the blocked donor immediately,
+	 * the core returns it through ops.enqueue() with SCX_ENQ_REENQ. Inserting
+	 * it into the same local DSQ would repeat the IMMED handback until the
+	 * scheduler is ejected. Move reenqueued blocked donors to the shared DSQ,
+	 * which doesn't carry SCX_ENQ_IMMED, so another CPU can consume them.
+	 */
+	if ((enq_flags & (SCX_ENQ_BLOCKED | SCX_ENQ_REENQ)) ==
+	    (SCX_ENQ_BLOCKED | SCX_ENQ_REENQ)) {
+		taskc->force_local = false;
+		scx_bpf_dsq_insert(p, SHARED_DSQ, 0, enq_flags);
+		cid = cmask_next_and2_set_wrap(&taskc->cpus_allowed,
+					       &qa.idle_cids.mask,
+					       &qa.self_cids.mask, 0);
+		if (cid < scx_bpf_nr_cids())
+			scx_bpf_kick_cid(cid, SCX_KICK_IDLE);
+		return;
+	}
+
+	/*
+	 * Insert a blocked mutex donor at the head of an eligible local DSQ with
+	 * a fresh slice and %SCX_ENQ_PREEMPT, requesting an immediate reschedule.
+	 * The test above guarantees that cpus_allowed intersects self_cids, but
+	 * the donor's current cid may have been delegated to a child. Search the
+	 * intersection starting at the current cid, preserving it when qmap still
+	 * holds it and wrapping to another eligible self cid otherwise.
+	 *
+	 * A self cid may be held exclusively with SCX_CAP_ENQ or time-shared with
+	 * only SCX_CAP_ENQ_IMMED. Add needs_immed() so either kind can accept the
+	 * local insertion instead of rejecting and reenqueuing the donor for a
+	 * capability miss. Once selected, the core proxy-exec path can run the
+	 * mutex owner using the donor's scheduling context.
+	 *
+	 * This policy is intentionally unfair and can strongly prioritize tasks
+	 * using contended mutexes; scx_qmap is a demonstration scheduler and
+	 * this behavior makes proxy-exec support easy to observe.

tools/sched_ext/scx_qmap.bpf.c void BPF_STRUCT_OPS(qmap_enqueue, struct task_struct *p, u64 enq_flags) · line 513

Quoted 10 lines
tools/sched_ext/scx_qmap.bpf.c +10 -0
+	 */
+	if (enq_flags & SCX_ENQ_BLOCKED) {
+		cid = cmask_next_and_set_wrap(&taskc->cpus_allowed,
+					      &qa.self_cids.mask,
+					      scx_bpf_task_cid(p));
+		scx_bpf_dsq_insert(p, SCX_DSQ_LOCAL_ON | cid, slice_ns,
+				   enq_flags | needs_immed(cid) | SCX_ENQ_PREEMPT);
+		return;
+	}
+

Hi Andrea,

Do we need to do serialization between ops.enqueue() and the paths which rebuild qa.self_cids.mask ?

IMHO, appy_partition() can clear or rebuild self_cids after the check but before cmask_next_and_set_wrap(). If the intersection is NULL, the helper return CID, and then the value gets passed to needs_immed(), hiwch uses it as an index and do "SCX_DSQ_LOCAL_ON | cid". This will be an invalid CPU.

Maybe do "cid < scx_bpf_nr_cids()" before using it ?

Best regards,
Richard Cheng

Quoted 84 lines
tools/sched_ext/scx_qmap.bpf.c +0 -0
 	/*
 	 * Fault injection: deliberately dispatch one of our own tasks to a cid
 	 * we don't hold. The inserts carry SCX_ENQ_RESCUE and divert to the
tools/sched_ext/scx_qmap.c +10 -3
index 5bb5f687e579c..d5226e0716573 100644
@@ -46,7 +46,7 @@ const char help_fmt[] =
 "See the top-of-file comment in .bpf.c for the design.\n"
 "\n"
 "Usage: %s [-s SLICE_US] [-e COUNT] [-t COUNT] [-T COUNT] [-l COUNT] [-b COUNT]\n"
-"       [-N COUNT] [-P] [-M] [-H] [-c CG_PATH] [-d PID] [-D LEN] [-S] [-p] [-I]\n"
+"       [-N COUNT] [-P] [-M] [-H] [-c CG_PATH] [-d PID] [-D LEN] [-S] [-p] [-I] [-X]\n"
 "       [-F COUNT] [-i SEC] [-R MS] [-J MODE] [-v]\n"
 "\n"
 "  -s SLICE_US   Override slice duration\n"
@@ -65,6 +65,7 @@ const char help_fmt[] =
 "  -S            Suppress qmap-specific debug dump\n"
 "  -p            Switch only tasks on SCHED_EXT policy instead of all\n"
 "  -I            Turn on SCX_OPS_ALWAYS_ENQ_IMMED\n"
+"  -X            Turn on SCX_OPS_ENQ_BLOCKED\n"
 "  -F COUNT      IMMED stress: force every COUNT'th enqueue to a busy local DSQ (use with -I)\n"
 "  -C MODE       cid-override test (shuffle|bad-dup|bad-range|bad-mono)\n"
 "  -i SEC        Stats interval, seconds (default 5)\n"
@@ -107,6 +108,7 @@ struct hier_prev {
 	u64 nr_dsps[MAX_SUB_SCHEDS];
 	u64 nr_reenq_cap;
 	u64 nr_reenq_immed;
+	u64 nr_enq_blocked;
 	u64 nr_inject_attempts;
 	u64 nr_rescue_dsp;
 };
@@ -190,14 +192,16 @@ static void print_hier(struct qmap_arena *qa, struct hier_prev *prev, u64 own_cg
 	}
 
 	format_cid_ranges(qa, CID_SHARED, ranges, sizeof(ranges));
-	printf("hier   : nsub=%llu excl=%u shared=%s rr=%s reenq cap/immed +%llu/+%llu inj=+%llu rescue=+%llu\n",
+	printf("hier   : nsub=%llu excl=%u shared=%s rr=%s reenq cap/immed +%llu/+%llu blocked=+%llu inj=+%llu rescue=+%llu\n",
 	       (unsigned long long)qa->nr_sub_scheds, qa->part.nr_excl, ranges, rr,
 	       (unsigned long long)(qa->nr_reenq_cap - prev->nr_reenq_cap),
 	       (unsigned long long)(qa->nr_reenq_immed - prev->nr_reenq_immed),
+	       (unsigned long long)(qa->nr_enq_blocked - prev->nr_enq_blocked),
 	       (unsigned long long)(qa->nr_inject_attempts - prev->nr_inject_attempts),
 	       (unsigned long long)(qa->nr_rescue_dsp - prev->nr_rescue_dsp));
 	prev->nr_reenq_cap = qa->nr_reenq_cap;
 	prev->nr_reenq_immed = qa->nr_reenq_immed;
+	prev->nr_enq_blocked = qa->nr_enq_blocked;
 	prev->nr_inject_attempts = qa->nr_inject_attempts;
 	prev->nr_rescue_dsp = qa->nr_rescue_dsp;
 
@@ -263,7 +267,7 @@ int main(int argc, char **argv)
 	skel->rodata->max_tasks = 16384;
 
 	while ((opt = getopt(argc, argv,
-			     "s:e:t:T:l:b:N:PMHc:d:D:SpIF:C:i:R:J:B:q:vh")) != -1) {
+			     "s:e:t:T:l:b:N:PMHc:d:D:SpIXF:C:i:R:J:B:q:vh")) != -1) {
 		switch (opt) {
 		case 's':
 			skel->rodata->slice_ns = strtoull(optarg, NULL, 0) * 1000;
@@ -324,6 +328,9 @@ int main(int argc, char **argv)
 		case 'I':
 			skel->struct_ops.qmap_ops->flags |= SCX_OPS_ALWAYS_ENQ_IMMED;
 			break;
+		case 'X':
+			skel->struct_ops.qmap_ops->flags |= SCX_OPS_ENQ_BLOCKED;
+			break;
 		case 'F':
 			skel->rodata->immed_stress_nth = strtoul(optarg, NULL, 0);
 			break;
tools/sched_ext/scx_qmap.h +1 -0
index c78d61806b398..fc10b542fb477 100644
@@ -178,6 +178,7 @@ struct qmap_arena {
 	/* bpf -> userspace: stats */
 	u64 nr_reenq_cap;		/* SCX_TASK_REENQ_CAP bounces */
 	u64 nr_reenq_immed;		/* SCX_TASK_REENQ_IMMED bounces */
+	u64 nr_enq_blocked;		/* SCX_ENQ_BLOCKED dispatches */
 	u64 nr_inject_attempts;		/* fault-injection: dispatches to an unheld cid */
 	u64 nr_rescue_dsp;		/* SCX_ENQ_RESCUE dispatch attempts */
 	u32 inject_mode;		/* fault-injection mode (QMAP_INJ_*) */
Signature · 1 line
2.55.0
author reviewer
#57 lore raw

Re: [PATCH 17/18] sched_ext: scx_qmap: Add proxy execution support

Hi Richard,

Quoted 106 lines · Richard Cheng
Quoted 92 lines · Andrea Righi

Add a -X option to opt scx_qmap into queueing mutex-blocked tasks for proxy execution. Without the option, SCX_OPS_ENQ_BLOCKED remains clear and mutex waiters block normally. With -X, blocked donors are passed to qmap_enqueue() with SCX_ENQ_BLOCKED.

When scx_qmap receives a blocked donor, select a cid allowed by the task and held by qmap, preferring its current cid. Dispatch the donor to that cid's local DSQ with a fresh slice and SCX_ENQ_PREEMPT, adding SCX_ENQ_IMMED when the cid is time-shared. This places the donor at the head of an eligible DSQ and requests an immediate reschedule, allowing the core proxy-exec path to run the mutex owner using the donor's scheduling context as soon as the donor is selected.

The blocked policy is intentionally unfair and can strongly prioritize tasks using contended mutexes, but scx_qmap is a demo scheduler and such aggressive behavior makes proxy-exec support easy to observe. Count all blocked dispatch attempts in nr_enq_blocked and report their per-interval delta.

---
 tools/sched_ext/scx_qmap.bpf.c | 53 +++++++++++++++++++++++++++++++++-
 tools/sched_ext/scx_qmap.c     | 13 +++++++--
 tools/sched_ext/scx_qmap.h     |  1 +
 3 files changed, 63 insertions(+), 4 deletions(-)
tools/sched_ext/scx_qmap.bpf.c +42 -1
index 5bb8b90a275a3..21f8a9be9650c 100644
@@ -439,6 +439,9 @@ void BPF_STRUCT_OPS(qmap_enqueue, struct task_struct *p, u64 enq_flags)
 	 */
 	taskc->core_sched_seq = qa.core_sched_tail_seqs[idx]++;
 
+	if (enq_flags & SCX_ENQ_BLOCKED)
+		__sync_fetch_and_add(&qa.nr_enq_blocked, 1);
+
 	/*
 	 * A task of ours that can run on none of our self cids - the parent
 	 * didn't grant them or we delegated them to children - would starve in
@@ -446,7 +449,8 @@ void BPF_STRUCT_OPS(qmap_enqueue, struct task_struct *p, u64 enq_flags)
 	 *
 	 * Force it onto its first allowed cid's local DSQ. If we hold that cid
 	 * it runs. Otherwise the insert carries SCX_ENQ_RESCUE and the kernel
-	 * diverts the task to its rescue path.
+	 * diverts the task to its rescue path. Do this before the blocked-donor
+	 * fast paths, which also require an eligible self cid to make progress.
 	 */
 	if (!cmask_intersects(&taskc->cpus_allowed, &qa.self_cids.mask)) {
 		s32 c = cmask_next_set_wrap(&taskc->cpus_allowed, 0);
@@ -460,6 +464,53 @@ void BPF_STRUCT_OPS(qmap_enqueue, struct task_struct *p, u64 enq_flags)
 		}
 	}
 
+	/*
+	 * SCX_OPS_ALWAYS_ENQ_IMMED makes the local insertion below implicitly
+	 * carry SCX_ENQ_IMMED. If the CPU can't run the blocked donor immediately,
+	 * the core returns it through ops.enqueue() with SCX_ENQ_REENQ. Inserting
+	 * it into the same local DSQ would repeat the IMMED handback until the
+	 * scheduler is ejected. Move reenqueued blocked donors to the shared DSQ,
+	 * which doesn't carry SCX_ENQ_IMMED, so another CPU can consume them.
+	 */
+	if ((enq_flags & (SCX_ENQ_BLOCKED | SCX_ENQ_REENQ)) ==
+	    (SCX_ENQ_BLOCKED | SCX_ENQ_REENQ)) {
+		taskc->force_local = false;
+		scx_bpf_dsq_insert(p, SHARED_DSQ, 0, enq_flags);
+		cid = cmask_next_and2_set_wrap(&taskc->cpus_allowed,
+					       &qa.idle_cids.mask,
+					       &qa.self_cids.mask, 0);
+		if (cid < scx_bpf_nr_cids())
+			scx_bpf_kick_cid(cid, SCX_KICK_IDLE);
+		return;
+	}
+
+	/*
+	 * Insert a blocked mutex donor at the head of an eligible local DSQ with
+	 * a fresh slice and %SCX_ENQ_PREEMPT, requesting an immediate reschedule.
+	 * The test above guarantees that cpus_allowed intersects self_cids, but
+	 * the donor's current cid may have been delegated to a child. Search the
+	 * intersection starting at the current cid, preserving it when qmap still
+	 * holds it and wrapping to another eligible self cid otherwise.
+	 *
+	 * A self cid may be held exclusively with SCX_CAP_ENQ or time-shared with
+	 * only SCX_CAP_ENQ_IMMED. Add needs_immed() so either kind can accept the
+	 * local insertion instead of rejecting and reenqueuing the donor for a
+	 * capability miss. Once selected, the core proxy-exec path can run the
+	 * mutex owner using the donor's scheduling context.
+	 *
+	 * This policy is intentionally unfair and can strongly prioritize tasks
+	 * using contended mutexes; scx_qmap is a demonstration scheduler and
+	 * this behavior makes proxy-exec support easy to observe.

tools/sched_ext/scx_qmap.bpf.c void BPF_STRUCT_OPS(qmap_enqueue, struct task_struct *p, u64 enq_flags) · line 513

Quoted 10 lines
tools/sched_ext/scx_qmap.bpf.c +10 -0
+	 */
+	if (enq_flags & SCX_ENQ_BLOCKED) {
+		cid = cmask_next_and_set_wrap(&taskc->cpus_allowed,
+					      &qa.self_cids.mask,
+					      scx_bpf_task_cid(p));
+		scx_bpf_dsq_insert(p, SCX_DSQ_LOCAL_ON | cid, slice_ns,
+				   enq_flags | needs_immed(cid) | SCX_ENQ_PREEMPT);
+		return;
+	}
+

Hi Andrea,

Quoted 7 lines

Do we need to do serialization between ops.enqueue() and the paths which rebuild qa.self_cids.mask ?

IMHO, appy_partition() can clear or rebuild self_cids after the check but before cmask_next_and_set_wrap(). If the intersection is NULL, the helper return CID, and then the value gets passed to needs_immed(), hiwch uses it as an index and do "SCX_DSQ_LOCAL_ON | cid". This will be an invalid CPU.

Maybe do "cid < scx_bpf_nr_cids()" before using it ?

I think you're right, the mask can change between the intersection check and cmask_next_and_set_wrap(), so the returned cid needs to be validated before it is passed to needs_immed() or encoded into the local DSQ id.

I don't think serialization is necessary here. If ownership changes after a valid cid has been selected, the core's capability validation should handle the race by rejecting and reenqueuing the task.

I'll add the bounds check and factor out the existing rescue placement.

Thanks,
-Andrea

author reviewer
#58 lore raw

PATCH 18/18 sched: Allow enabling proxy exec with sched_ext

In reply to Andrea Righi

Now that sched_ext can handle proxy donors and BPF schedulers can opt in to blocked-donor enqueueing with SCX_OPS_ENQ_BLOCKED, remove the !SCHED_CLASS_EXT dependency from SCHED_PROXY_EXEC and allow both options to be enabled together.

---
 init/Kconfig | 2 --
 1 file changed, 2 deletions(-)
init/Kconfig +0 -2
index 8583d9f06c522..f3b3bbd14dc41 100644
@@ -938,8 +938,6 @@ config SCHED_PROXY_EXEC
 	bool "Proxy Execution"
 	# Avoid some build failures w/ PREEMPT_RT until it can be fixed
 	depends on !PREEMPT_RT
-	# Need to investigate how to inform sched_ext of split contexts
-	depends on !SCHED_CLASS_EXT
 	# Not particularly useful until we get to multi-rq proxying
 	depends on EXPERT
 	help
Signature · 1 line
2.55.0
reviewer
#59 lore raw
In reply to Andrea Righi

Hello,

Other than some nits, generally looks okay to me from sched_ext POV and it looks ready to merge and iterate in tree. Peter, what do you think?

Thanks.

Signature · 1 line
tejun
maintainer
#60 lore raw
Quoted 4 lines · Tejun Heo

Hello,

Other than some nits, generally looks okay to me from sched_ext POV and it looks ready to merge and iterate in tree. Peter, what do you think?

I'll try and have a look at things this week -- on a very cursory look they do indeed seem fine.