sched: Make proxy execution compatible with sched_ext
Subsystems (4)
Andrea Righi
reviewerPATCH v14 1/1 sched: Make proxy execution compatible with sched_ext
This series enables proxy execution with sched_ext and is based on early work by John Stultz [1].
Background
==========
Proxy execution (proxy-exec) lets a waiting task ("donor") donate its scheduling context to a mutex owner, so the owner can run while the donor stays eligible on the runqueue.
Currently, proxy-exec and sched_ext are mutually exclusive at build time: we can't enable CONFIG_SCHED_PROXY_EXEC=y and CONFIG_SCHED_CLASS_EXT=y in the same kernel.
This restriction can be problematic for Linux distributions and for anyone who wants to ship one kernel and choose features at runtime.
Why are they mutually exclusive?
================================
sched_ext schedulers drive dispatch through their own interfaces. A proxy-exec handoff can run a task that the BPF scheduler never dispatched through that path. sched_ext callbacks then observe a "current" task that does not match what the BPF side considers running, so kfuncs and helper state can see an inconsistent view of the executing task.
sched_ext also tracks runnable work through Dispatch Queues (DSQs) and BPF chosen dispatch rules, while the core scheduler still maintains classic per-CPU runqueues and pick paths. A proxy handoff can therefore switch the CPU to a task that the BPF scheduler never inserted or ordered through its DSQ interface.
DSQ state, vtime, and "who is running" bookkeeping inside the BPF program can then disagree with what the core actually executes, so helpers and kfuncs that assume their dispatched task is current may observe stale or inconsistent state.
Design: supporting proxy-exec with sched_ext
============================================
Provide proxy-exec in sched_ext as optional per-scheduler capability: a BPF scheduler can set the ops flag SCX_OPS_ENQ_BLOCKED to keep mutex-blocked donors runnable and receive them through ops.enqueue() with SCX_ENQ_BLOCKED set.
This flag requires an ops.enqueue() callback and gives BPF control over whether, where, and in what order to dispatch each donor. The core then walks the mutex-owner chain and, when needed, migrates the donor's scheduling context to the owner's rq before executing the owner.
Without the SCX_OPS_ENQ_BLOCKED flag, mutex waiters block normally and do not participate in proxy-exec while owned by that scheduler.
Once BPF dispatches a donor, proxy execution may move its scheduling context to the mutex owner's rq. This move is distinct from BPF placement: wake_cpu continues to identify the donor's pre-proxy CPU, while task_cpu() identifies the proxy CPU. When the donor wakes, the normal wakeup path uses wake_cpu as prev_cpu when returning the task to the BPF scheduler, which can then choose its next placement normally.
The donor-to-owner handoff is modeled like a "function call" from the scheduler's perspective. The donor remains the running scheduling entity selected by BPF: its scheduling context, runtime and slice are consumed while the core temporarily invokes the mutex owner's code to make the critical section progress. It is not a scheduler-visible switch to the owner.
Accordingly, the donor remains the scheduling context presented to sched_ext, while the mutex owner is treated solely as the execution context selected internally by the core scheduler. Scheduling state is accounted against rq->donor where appropriate, while rq->curr identifies the execution context. The internal owner substitution does not generate synthetic sched_ext callbacks for a task that BPF did not dispatch.
Scheduler-facing current-task kfuncs follow the same rule:scx_bpf_task_running(), scx_bpf_cpu_curr() and scx_bpf_cid_curr() report the donor selected by the scheduler rather than the mutex owner whose code happens to execute on its behalf.
The sched_ext callback bookkeeping is adjusted accordingly. A blocked donor enters a tracked ops.running()/ops.stopping() session only after proxy resolution succeeds, and ops.stopping() is only called for tasks that entered such a session. This state is independent of physical rq->curr execution, which may refer to the mutex owner instead of the donor.
Scheduler ownership changes need special handling because a donor may already be blocked when a root scheduler is enabled, when a task moves between root and sub-schedulers, or when a class transition moves it into or out of EXT. Start every scheduling-class or BPF-scheduler ownership change from clean task state:
fully deactivate a retained donor before the transition and let the incoming scheduler apply its own admission policy the next time the task blocks. This conservative rule leaves retaining proxy state across compatible RT/DL PI transitions as a future improvement.
scx_qmap has been modified with a -X option to enable queueing mutex-blocked tasks for proxy-exec. Blocked donors receive a fresh slice and are inserted at the head of an eligible local DSQ selected from the task's allowed cids and qmap's self cids, preferring the current cid. They request immediate preemption, and time-shared targets also carry SCX_ENQ_IMMED. This is an intentionally aggressive policy for making proxy-exec easy to observe.
A new kselftest (enq_blocked) is also introduced to validate proxy-exec with sched_ext. The test creates a priority inversion with a low-priority owner (nice +19), a high-priority donor (nice -20) and one nice 0 contender per available CPU. It exercises a same-CPU topology and a cross-CPU topology, where the donor and owner run on different CPUs. Each configuration runs with SCX_OPS_ENQ_BLOCKED first disabled and then enabled, counts blocked-donor enqueues by CPU and reports mutex hold/wait-time deltas. Access to the mutex is provided by a loadable kernel module built via TEST_GEN_MODS_DIR, with the test responsible for its lifecycle.
Example kselftest run:
===== START =====
TEST: enq_blocked
DESCRIPTION: Verify proxy donor admission under CPU-wide contention OUTPUT:
[topology=same-cpu SCX_OPS_ENQ_BLOCKED=disabled]
proxy_exec=enabled
donor_cpu=0
owner_cpu=0
nr_contenders=16
measured_trials=10
owner_nice=19
donor_nice=-20
contender_nice=0 mutex_hold_avg_ns=263587544 (263.587 ms, samples=10) mutex_wait_avg_ns=263597934 (263.597 ms, samples=10) nr_blocked_enqueues=0 nr_blocked_enqueues_donor_cpu=0 nr_blocked_enqueues_owner_cpu=0 nr_blocked_enqueues_other_cpu=0 nr_blocked_wakeups=0
[topology=same-cpu SCX_OPS_ENQ_BLOCKED=enabled]
proxy_exec=enabled
donor_cpu=0
owner_cpu=0
nr_contenders=16
measured_trials=10
owner_nice=19
donor_nice=-20
contender_nice=0 mutex_hold_avg_ns=251789178 (251.789 ms, samples=10) mutex_wait_avg_ns=209808599 (209.808 ms, samples=10) nr_blocked_enqueues=50 nr_blocked_enqueues_donor_cpu=50 nr_blocked_enqueues_owner_cpu=0 nr_blocked_enqueues_other_cpu=0 nr_blocked_wakeups=0
[topology=same-cpu delta: enabled - disabled]
mutex_hold_delta_ns=-11798366 (-4.48%)
mutex_wait_delta_ns=-53789335 (-20.41%)
[topology=cross-cpu SCX_OPS_ENQ_BLOCKED=disabled]
proxy_exec=enabled
donor_cpu=0
owner_cpu=1
nr_contenders=16
measured_trials=10
owner_nice=19
donor_nice=-20
contender_nice=0 mutex_hold_avg_ns=246794454 (246.794 ms, samples=10) mutex_wait_avg_ns=247298497 (247.298 ms, samples=10) nr_blocked_enqueues=0 nr_blocked_enqueues_donor_cpu=0 nr_blocked_enqueues_owner_cpu=0 nr_blocked_enqueues_other_cpu=0 nr_blocked_wakeups=0
[topology=cross-cpu SCX_OPS_ENQ_BLOCKED=enabled]
proxy_exec=enabled
donor_cpu=0
owner_cpu=1
nr_contenders=16
measured_trials=10
owner_nice=19
donor_nice=-20
contender_nice=0 mutex_hold_avg_ns=215486809 (215.486 ms, samples=10) mutex_wait_avg_ns=215498090 (215.498 ms, samples=10) nr_blocked_enqueues=80 nr_blocked_enqueues_donor_cpu=20 nr_blocked_enqueues_owner_cpu=60 nr_blocked_enqueues_other_cpu=0 nr_blocked_wakeups=0
[topology=cross-cpu delta: enabled - disabled]
mutex_hold_delta_ns=-31307645 (-12.69%)
mutex_wait_delta_ns=-31800407 (-12.86%)
ok 1 enq_blocked #
===== END =====
=============================
RESULTS:
PASSED: 1 SKIPPED: 0 FAILED: 0
References
==========
[1] https://lore.kernel.org/all/[email protected]
Git tree: git://git.kernel.org/pub/scm/linux/kernel/git/arighi/linux.git scx-proxy-exec
Changes in v14:
- Clarify why a waking EXT donor must be dequeued before rq->donor is reset, and remove the unnecessary cached donor-state variable (K Prateek Nayak)
- Track resolved proxy sessions explicitly on NOHZ_FULL CPUs, conservatively re-enable the periodic tick during proxy execution
- Drop the next-class sched_change_begin() argument and move retained-donor cleanup to the sched_ext ownership transitions (Peter Zijlstra)
- Extend the existing post-resolution scx_proxy_reenqueue_retry() hook with NOHZ proxy-state tracking (Peter Zijlstra)
- Rename WF_ON_RQ to WF_TTWU_RQ to identify the exact wakeup path that skipped activation and enqueueing, and split the core marker into a preparatory patch (Peter Zijlstra)
- Gate the blocked-donor admission hook with the scx_enabled() static key and clean up the root-scheduler activation path (Peter Zijlstra)
- Remove impossible lower-priority-donor cases from the curr/donor class interaction description (Peter Zijlstra)
- Route reject-DSQ enqueue outcomes through one cleanup point (Tejun Heo)
- Validate scx_qmap's selected cid after the self-cid mask race and fall back to rescue placement when the intersection disappears (Richard Cheng)
- Drop "sched/core: Avoid false migration warning for proxy donors", now applied upstream as fe3c73d7bc76 ("sched/core: Avoid false migration warning for proxy donors")
- Link to v13: https://lore.kernel.org/all/[email protected]/
Changes in v13:
- Keep re-enqueue reason lifetime out of the generic DSQ unlink primitive; preserve it through ops.enqueue() and clear it before resolving the new placement (Tejun Heo)
- Immediately drain proxy-rejected tasks and request a later retry only when the task is still running or donating, using SCX_RQ_PROXY_RETRY to track the pending retry (Tejun Heo)
- Restructure proxy reject handling, rename the helpers to describe their roles and document why the locked DSQ move deliberately diagnoses ordinary migration-disabled tasks instead of treating them as proxy races (Tejun Heo)
- Introduce WF_ON_RQ to distinguish already-runnable wakeups from full activations, replacing SCX_TASK_ENQ_WAKEUP tracking, and gate the sched_ext reschedule path on proxy execution (Tejun Heo)
- Keep the conservative reset across scheduling-class transitions, retaining proxy state across compatible RT/DL PI transitions remains future work
- Dispatch scx_qmap blocked donors to an eligible self cid, preferring the current cid, carry SCX_ENQ_IMMED for time-shared targets and count reenqueued shared-DSQ dispatches in nr_enq_blocked
- Link to v12: https://lore.kernel.org/all/[email protected]/
Changes in v12:
- Preserve a fresh re-enqueue reason when ops.enqueue() synchronously rejects a task again and clear the consumed reason only after the task leaves the reject DSQ (sashiko)
- Immediately process tasks rejected after becoming migration-disabled during an rq-lock handoff, leaving only active proxy contexts parked (sashiko)
- Use the donor scheduling context for protected-slice admission, rescue preemption, slice save/restore and bypass handling, and charge rescue runtime to the rescued task's scheduler (sashiko)
- Refresh the NOHZ tick dependency after rq->donor changes in __schedule(), preventing a stale constrained FAIR donor from keeping the tick enabled for an incoming RT task (sashiko)
- Clarify that ops.running()/ops.stopping() track the donor's scheduling context including blocked donors after successful proxy resolution (sashiko)
- Restore the change that leaves mutex proxy-lock scopes before calling proxy_resched_idle() (required to avoid scheduling-class callbacks under the extra raw spinlocks)
- Dequeue waking proxy donors before replacing rq->donor references avoiding callbacks under blocked_lock and transient donor re-enqueueing
- Harden scx_qmap blocked-donor fast paths against missing eligible self cids
- Move SCX_TASK_ENQ_WAKEUP to bit 16, leaving room for task-state and re-enqueue-reason expansion
- Link to v11: https://lore.kernel.org/all/[email protected]/
Changes in v11:
- End retained proxy execution before every scheduling-class or BPF-scheduler ownership change, passing the incoming class to sched_change_begin() and removing the prepare_switch() class callback (Tejun Heo)
- Document the conservative class-transition rule and leave retaining proxy state across compatible RT/DL PI transitions for future work (Tejun Heo)
- Split reject-DSQ code movement from its generalization and carry the re-enqueue reason directly in p->scx.flags (Tejun Heo)
- Use one SCX_TASK_REENQ_PROXY reason for all proxy-raced remote transfers, clarify why proxy migration differs from BPF-directed migration and keep tasks parked while affinity migration is pending (Tejun Heo)
- Factor local-DSQ diversion cleanup into a shared helper, clear HEAD, IMMED, PREEMPT and carried-slice state for proxy rejection (Tejun Heo)
- Explain the scx_qmap SCX_OPS_ALWAYS_ENQ_IMMED re-enqueue fallback (Tejun Heo)
- Rename the blocked-donor command-line option to -X, since -B is now taken for the rescue bandwidth setting
- Drop the SCHED_FLAG_KEEP_PARAMS preparatory fix (this is addressed separately by https://lore.kernel.org/all/[email protected])
- Link to v10: https://lore.kernel.org/all/[email protected]/
Changes in v10:
- Check constrained FAIR proxy donors before the RT fast paths in sched_can_stop_tick(), so a throttled RT owner cannot bypass bandwidth enforcement (sashiko)
- Add a preparatory fix that sets DEQUEUE_CLASS only when SCHED_FLAG_KEEP_PARAMS permits the scheduling class to change, preventing class-transition callbacks from running without an actual transition (sashiko)
- Update try_to_block_task() comment to describe the sched_ext blocked-donor admission check (sashiko)
- Link to v9: https://lore.kernel.org/all/[email protected]/
Changes in v9:
- Add an incoming-class prepare_switch() callback and use it to block retained donors before transitions into sched_ext (Tejun Heo)
- Generalize the per-rq reject DSQ and use it for proxy-raced remote transfers instead of bouncing tasks to a global DSQ (Tejun Heo)
- Restrict migration-disabled warning exception to blocked donors whose scheduling context was actually proxy-migrated (sashiko)
- Keep ops.tick() and other donor accounting inside a tracked ops.running()/ops.stopping() session (sashiko)
- Prevent BPF-directed migration of an active proxy donor and use the donor as the current scheduling context during restore (sashiko)
- Update sched_fair_update_stop_tick() to use the number of queued FAIR tasks (sashiko)
- Fold the temporary local-DSQ donor placement into the final BPF-controlled admission change
- Link to v8: https://lore.kernel.org/all/[email protected]/
Changes in v8:
- Rework remote-DSQ migration check so the locked re-check handles only proxy-exec state changes (Tejun Heo)
- Use the current-donor helper when preventing migration of an active donor (sashiko)
- Handle blocked-donor reactivation into scx schedulers without SCX_OPS_ENQ_BLOCKED (John Stultz)
- Document sched_ext callback behavior under proxy execution (sashiko)
- Avoid redispatching a rejected blocked donor to the same local DSQ in scx_qmap (sashiko)
- Add a preparatory fix to avoid false migration warnings when proxy-exec moves a migration-disabled donor
- Skip the enq_blocked kselftest when proxy execution is unavailable
- Link to v7: https://lore.kernel.org/all/[email protected]/
Changes in v7:
- Move the remote CPU check out of the lockless remote-DSQ scan and perform it only after locking the source rq, retain the locked migration recheck immediately before moving the task (sashiko)
- Complete the scheduler-context conversion from rq->curr to rq->donor for capability revocation and preemption checks, make scx_bpf_task_running(), scx_bpf_cpu_curr() and scx_bpf_cid_curr() expose the donor selected by the scheduler (sashiko)
- Handle SCX_ENQ_REENQ before scx_qmap's blocked-donor fast path to avoid an infinite reject/re-enqueue loop (sashiko)
- Add scx_qmap stat for SCX_ENQ_BLOCKED dispatches
- Link to v6: https://lore.kernel.org/all/[email protected]/
Changes in v6:
- Add a scheduler-core fix to make NOHZ CFS bandwidth checks follow the proxy donor instead of the physical execution context (sashiko)
- Block retained donors when entering sched_ext through PI de-boosting or global scheduler activation (sashiko)
- Distinguish ordinary wakeup activations from retained-donor admissions so SCX_ENQ_BLOCKED is not reported for normal wakeups; extend the selftest to detect this regression (sashiko)
- Validate unexpected-CPU blocked-donor enqueues in both same-CPU and cross-CPU test topologies (sashiko)
- Link to v5: https://lore.kernel.org/all/[email protected]/
Changes in v5:
- Split retained-donor deactivation and sched_ext's default rejection into preparatory patches so the scheduler-core changes can be routed separately
- Drop the proxy destination query kfuncs and the preparatory mutex lock-scope change (John Stultz)
- Use p->is_blocked instead of task_is_blocked() to fix a WARN triggered during scx_pair testing (John Stultz)
- Rename SCX_TASK_IS_RUNNING to SCX_TASK_RUN_TRACKED and document that it tracks an ops.running()/stopping() session rather than physical rq->curr execution
- Keep a proxy-migrated blocked donor on the owner's rq until wakeup instead of allowing BPF-directed migration to pull it back to wake_cpu and cause repeated donor migration
- Extend kselftest to cover same-CPU and cross-CPU proxy-exec switches
- Make scx_qmap insert blocked donors at the head of their current cid's local DSQ and request immediate preemption
- Link to v4: https://lore.kernel.org/all/[email protected]/
Changes in v4:
- Harden scx_bpf_task_proxy_cpu() locking and blocked-state validation, return -ENOENT for unrunnable owners and drop unnecessary donor-affinity checks (K Prateek Nayak, sashiko)
- Reschedule remote CPUs after donor deactivation, handle sched_setscheduler() admission and assert scheduler-change locking (sashiko)
- Avoid a potential KCSAN data-race report in the lockless blocked-donor migration check by using rcu_access_pointer() for rq->donor (sashiko)
- Fix tick dependency updates for incoming EXT contexts and keep the tick enabled for blocked donors (sashiko)
- Dump EXT donors in scx_dump_state() (sashiko)
- Reordered the preparatory sched_ext changes so real running-state tracking is established before donor-based accounting, and split the proxy destination query kfuncs into a separate patch
- Add compatibility wrappers for the proxy CPU/cid kfuncs and document their results as scheduling hints
- Link to v3: https://lore.kernel.org/all/[email protected]/
Changes in v3:
- Dropped the core restrictions on proxy-migrating migration-disabled and single-CPU donors: proxy execution moves the scheduling context, not the task's execution context. (Peter Zijlstra, K Prateek Nayak)
- Dropped the sched_ext-specific put_prev_task()/set_next_task() exception and fixed ops.running()/ops.stopping() pairing inside sched_ext instead; track a real running transition even when either callback is absent. (Peter Zijlstra)
- Dropped the kf_tasks[] nesting and nested ops.runnable() patches from v2; extensive proxy-exec testing did not reproduce task-op re-entry, so retain the existing non-nesting invariant.
- Replaced scx_bpf_task_is_blocked() with SCX_ENQ_BLOCKED in ops.enqueue() flags, identifying blocked-donor admission directly at enqueue time.
- Expanded the curr/donor description to cover mixed scheduling classes and fixed local preemption to expire rq->donor's slice. (Aiqun Maria Yu)
- Allow inactive blocked donors to be placed on a remote local DSQ while preventing normal migration of an active rq donor. (Aiqun Maria Yu)
- Deactivate retained donors when ownership changes to a root or sub-scheduler without SCX_OPS_ENQ_BLOCKED, and extend the selftest to cover attaching a scheduler after the donor blocks. (K Prateek Nayak)
- Harden remote DSQ consumption by rejecting active tasks and rechecking migration eligibility after locking the source rq. Fall back to the global DSQ without treating an eligibility change during the lock handoff as a BPF scheduler error.
- scx_qmap has a command line option (-B) to enable blocked-donor queueing
- Link to v2: https://lore.kernel.org/all/[email protected]/
Changes in v2:
- Rebased onto sched_ext/for-7.3 and adapted the series to the split sched_ext implementation and cid-form scheduler interfaces.
- Replaced the global sched_proxy_exec_scx boot-time opt-in with the per-scheduler SCX_OPS_ENQ_BLOCKED capability, allowing BPF to control donor admission and ordering through ops.enqueue().
- Added scx_bpf_task_is_blocked(), scx_bpf_task_proxy_cpu(), and scx_bpf_task_proxy_cid(); enforce CPU/cid API separation for cid-form schedulers.
- Added proxy exec support to scx_qmap, including optional owner-cid steering, affinity validation, and fallback to the donor's current cid.
- Added a kselftest with a kernel mutex test and a three-task priority inversion workload, test is executed with blocked task admission disabled and enabled, validates the behavior, and reports hold/wait-time deltas.
- Link to v1: https://lore.kernel.org/all/[email protected]/
Andrea Righi (16):
sched/core: Drop mutex locks before proxy rescheduling sched/core: Dequeue waking proxy donors before reset sched/core: Mark wakeups completed through ttwu_runnable()
sched: Add helper to block retained proxy donors
sched: Add sched_ext hooks for proxy execution
sched_ext: Block proxy donors before taking control
sched_ext: Fix ops.running/stopping() pairing for proxy-exec donors
sched_ext: Move reject DSQ draining into core
sched_ext: Generalize the reject DSQ reenqueue path
sched_ext: Handle proxy-exec races in remote DSQ transfers
sched_ext: Split curr|donor references properly
sched_ext: Track proxy execution for NOHZ_FULL
sched_ext: Delegate proxy donor admission to BPF schedulers
sched_ext: Add selftest for blocked donor admission
sched_ext: scx_qmap: Add proxy execution support
sched: Allow enabling proxy exec with sched_ext
Documentation/scheduler/sched-ext.rst | 13 + include/linux/sched/ext.h | 3 + init/Kconfig | 2 - kernel/sched/core.c | 88 +- kernel/sched/ext/ext.c | 617 +++++++++++--- kernel/sched/ext/ext.h | 17 + kernel/sched/ext/internal.h | 38 +- kernel/sched/ext/sub.c | 68 +- kernel/sched/ext/sub.h | 13 +- kernel/sched/sched.h | 14 +- kernel/sched/syscalls.c | 2 + tools/sched_ext/include/scx/compat.h | 1 + tools/sched_ext/include/scx/enum_defs.autogen.h | 3 + tools/sched_ext/include/scx/enums.autogen.bpf.h | 3 + tools/sched_ext/include/scx/enums.autogen.h | 1 + tools/sched_ext/include/scx/enums_abi.autogen.h | 2 +- tools/sched_ext/scx_qmap.bpf.c | 79 +- tools/sched_ext/scx_qmap.c | 13 +- tools/sched_ext/scx_qmap.h | 1 + tools/testing/selftests/sched_ext/.gitignore | 4 + tools/testing/selftests/sched_ext/Makefile | 2 + tools/testing/selftests/sched_ext/config | 2 + .../testing/selftests/sched_ext/enq_blocked.bpf.c | 116 +++ tools/testing/selftests/sched_ext/enq_blocked.c | 917 +++++++++++++++++++++ tools/testing/selftests/sched_ext/enq_blocked.h | 28 + .../selftests/sched_ext/test_modules/Makefile | 13 + .../sched_ext/test_modules/scx_enq_blocked_test.c | 195 +++++ 27 files changed, 2051 insertions(+), 204 deletions(-) create mode 100644 tools/testing/selftests/sched_ext/enq_blocked.bpf.c create mode 100644 tools/testing/selftests/sched_ext/enq_blocked.c create mode 100644 tools/testing/selftests/sched_ext/enq_blocked.h create mode 100644 tools/testing/selftests/sched_ext/test_modules/Makefile create mode 100644 tools/testing/selftests/sched_ext/test_modules/scx_enq_blocked_test.c
Andrea Righi
reviewerPATCH 1/16 sched/core: Drop mutex locks before proxy rescheduling
find_proxy_task() can call proxy_resched_idle() from return expressions inside mutex wait_lock and blocked task lock guard scopes. Return expressions are evaluated before scoped guards are released, so proxy_resched_idle() invokes scheduling-class callbacks while both raw spinlocks are held.
Route these cases through a common label outside the guard scopes. The common label is required because a direct return would evaluate proxy_resched_idle() before guard cleanup and run the class callbacks under the proxy-exec locks again.
This is a preparatory change to support proxy execution with sched_ext.
--- kernel/sched/core.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-)
kernel/sched/core.c +6 -4
@@ -6961,7 +6961,7 @@ find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf)if (!READ_ONCE (owner->on_rq) || owner->se.sched_delayed) {/* XXX Don't handle blocked owners/delayed dequeue yet */ if (curr_in_chain) -return proxy_resched_idle (rq); + goto resched_idle;__clear_task_blocked_on (p,NULL ); goto deactivate; }
@@ -6973,7 +6973,7 @@ find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf)* and leave that CPU to sort things out. */ if (curr_in_chain) -return proxy_resched_idle (rq); + goto resched_idle; goto migrate_task; }
@@ -6986,7 +6986,7 @@ find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf)* case we should end up back in find_proxy_task(), this time * hopefully with all relevant tasks already enqueued. */ -return proxy_resched_idle (rq); + goto resched_idle; }/*
@@ -7023,7 +7023,7 @@ find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf)* So schedule rq->idle so that ttwu_runnable() can get the rq * lock and mark owner as running. */ -return proxy_resched_idle (rq); + goto resched_idle; }/* * OK, now we're absolutely sure @owner is on this
@@ -7035,6 +7035,8 @@ find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf)WARN_ON_ONCE (owner && !owner->on_rq);return owner; +resched_idle: +return proxy_resched_idle (rq); deactivate:proxy_deactivate (rq, p);return NULL ;
Signature · 1 line
2.55.0
[tip: sched/core] sched/core: Drop mutex locks before proxy rescheduling
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 313b652837d0541684680a7d07a95c5560441d4f Gitweb: https://git.kernel.org/tip/313b652837d0541684680a7d07a95c5560441d4f Author: Andrea Righi <[email protected]> AuthorDate: Tue, 22 Sep 2026 18:51:40 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Mon, 28 Sep 2026 08:44:38 +02:00
sched/core: Drop mutex locks before proxy rescheduling
find_proxy_task() can call proxy_resched_idle() from return expressions inside mutex wait_lock and blocked task lock guard scopes. Return expressions are evaluated before scoped guards are released, so proxy_resched_idle() invokes scheduling-class callbacks while both raw spinlocks are held.
Route these cases through a common label outside the guard scopes. The common label is required because a direct return would evaluate proxy_resched_idle() before guard cleanup and run the class callbacks under the proxy-exec locks again.
This is a preparatory change to support proxy execution with sched_ext.
--- kernel/sched/core.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-)
kernel/sched/core.c +6 -4
@@ -6980,7 +6980,7 @@ find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf)if (!READ_ONCE (owner->on_rq) || owner->se.sched_delayed) {/* XXX Don't handle blocked owners/delayed dequeue yet */ if (curr_in_chain) -return proxy_resched_idle (rq); + goto resched_idle;__clear_task_blocked_on (p,NULL ); goto deactivate; }
@@ -6992,7 +6992,7 @@ find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf)* and leave that CPU to sort things out. */ if (curr_in_chain) -return proxy_resched_idle (rq); + goto resched_idle; goto migrate_task; }
@@ -7005,7 +7005,7 @@ find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf)* case we should end up back in find_proxy_task(), this time * hopefully with all relevant tasks already enqueued. */ -return proxy_resched_idle (rq); + goto resched_idle; }/*
@@ -7042,7 +7042,7 @@ find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf)* So schedule rq->idle so that ttwu_runnable() can get the rq * lock and mark owner as running. */ -return proxy_resched_idle (rq); + goto resched_idle; }/* * OK, now we're absolutely sure @owner is on this
@@ -7062,6 +7062,8 @@ find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf) }return owner; +resched_idle: +return proxy_resched_idle (rq); deactivate:proxy_deactivate (rq, p);return NULL ;
Andrea Righi
reviewerPATCH 2/16 sched/core: Dequeue waking proxy donors before reset
In reply to Andrea Righiproxy_needs_return() resets an active donor while holding blocked_lock. proxy_reset_donor() invokes scheduling-class callbacks, adding an unnecessary raw-spinlock nesting.
For an EXT donor, resetting rq->donor first calls put_prev_task_scx() while SCX_TASK_QUEUED and is_blocked are still set. That path reenqueues the retained donor through scx_do_enqueue_task(), only for the following block_task() to dequeue it again.
Split block_task() so the waking donor is first dequeued from its scheduling class while it is still rq->donor. This lets dequeue_task_scx() end the donor's running session and clear SCX_TASK_QUEUED before proxy_reset_donor() invokes put_prev_task_scx(). Keep the generic on_rq state set until donor references are replaced, then complete the runqueue removal.
This follows the normal sleep ordering, avoids transiently re-enqueuing the waking donor, and moves the scheduling-class callbacks outside blocked_lock.
This is a preparatory change to support proxy execution with sched_ext.
--- kernel/sched/core.c | 36 ++++++++++++++++++++++++++++-------- 1 file changed, 28 insertions(+), 8 deletions(-)
kernel/sched/core.c +28 -8
@@ -2253,7 +2253,8 @@ void deactivate_task(struct rq *rq, struct task_struct *p, int flags)dequeue_task (rq, p, flags); } -static void block_task (struct rq *rq,struct task_struct *p,unsigned long task_state) +static bool dequeue_block_task (struct rq *rq,struct task_struct *p, +unsigned long task_state) {int flags =DEQUEUE_NOCLOCK ;
@@ -2274,9 +2275,15 @@ static void block_task(struct rq *rq, struct task_struct *p, unsigned long task_* * Where __schedule() and ttwu() have matching control dependencies. * -* After this, schedule() must not care about p->state any more. +* Once the caller invokes __block_task(), schedule() must not care about +* p->state any more. */ -if (dequeue_task (rq, p,DEQUEUE_SLEEP | flags)) +return dequeue_task (rq, p,DEQUEUE_SLEEP | flags); +} + +static void block_task (struct rq *rq,struct task_struct *p,unsigned long task_state) +{ +if (dequeue_block_task (rq, p, task_state))__block_task (rq, p); }
@@ -3775,6 +3782,8 @@ static inline void proxy_reset_donor(struct rq *rq)*/ static inline bool proxy_needs_return (struct rq *rq,struct task_struct *p) { +bool dequeued; +/* * Typically per __set_task_cpu(), task_cpu(p) == p->wake_cpu. *
@@ -3797,12 +3806,23 @@ static inline bool proxy_needs_return(struct rq *rq, struct task_struct *p)/* If already current, don't need to return migrate */ if (task_current (rq, p)) return false; - -/* If we're return migrating the rq->donor, switch it out for idle */ -if (task_current_donor (rq, p)) - proxy_reset_donor(rq); } -block_task (rq, p,TASK_WAKING ); + + dequeued =dequeue_block_task (rq, p,TASK_WAKING ); + +/* +* Dequeue @p from its scheduling class before resetting rq->donor. +* In particular, sched_ext needs to end the donor's running session +* and clear SCX_TASK_QUEUED before put_prev_task_scx() is called by +* proxy_reset_donor(); otherwise it would reenqueue the blocked donor. +* +* Keep on_rq set until all donor references have been replaced. +*/ +if (task_current_donor (rq, p)) +proxy_reset_donor (rq); + +if (dequeued) +__block_task (rq, p);return true; }#else /* !CONFIG_SCHED_PROXY_EXEC */
Signature · 1 line
2.55.0
[tip: sched/core] sched/core: Dequeue waking proxy donors before reset
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 8f8c0417e97382c3473ef56b0cd472badb40e4a0 Gitweb: https://git.kernel.org/tip/8f8c0417e97382c3473ef56b0cd472badb40e4a0 Author: Andrea Righi <[email protected]> AuthorDate: Tue, 22 Sep 2026 18:51:41 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Mon, 28 Sep 2026 08:44:38 +02:00
sched/core: Dequeue waking proxy donors before reset
proxy_needs_return() resets an active donor while holding blocked_lock. proxy_reset_donor() invokes scheduling-class callbacks, adding an unnecessary raw-spinlock nesting.
For an EXT donor, resetting rq->donor first calls put_prev_task_scx() while SCX_TASK_QUEUED and is_blocked are still set. That path reenqueues the retained donor through scx_do_enqueue_task(), only for the following block_task() to dequeue it again.
Split block_task() so the waking donor is first dequeued from its scheduling class while it is still rq->donor. This lets dequeue_task_scx() end the donor's running session and clear SCX_TASK_QUEUED before proxy_reset_donor() invokes put_prev_task_scx(). Keep the generic on_rq state set until donor references are replaced, then complete the runqueue removal.
This follows the normal sleep ordering, avoids transiently re-enqueuing the waking donor, and moves the scheduling-class callbacks outside blocked_lock.
This is a preparatory change to support proxy execution with sched_ext.
--- kernel/sched/core.c | 36 ++++++++++++++++++++++++++++-------- 1 file changed, 28 insertions(+), 8 deletions(-)
kernel/sched/core.c +28 -8
@@ -2252,7 +2252,8 @@ void deactivate_task(struct rq *rq, struct task_struct *p, int flags)dequeue_task (rq, p, flags); } -static void block_task (struct rq *rq,struct task_struct *p,unsigned long task_state) +static bool dequeue_block_task (struct rq *rq,struct task_struct *p, +unsigned long task_state) {int flags =DEQUEUE_NOCLOCK ;
@@ -2273,9 +2274,15 @@ static void block_task(struct rq *rq, struct task_struct *p, unsigned long task_* * Where __schedule() and ttwu() have matching control dependencies. * -* After this, schedule() must not care about p->state any more. +* Once the caller invokes __block_task(), schedule() must not care about +* p->state any more. */ -if (dequeue_task (rq, p,DEQUEUE_SLEEP | flags)) +return dequeue_task (rq, p,DEQUEUE_SLEEP | flags); +} + +static void block_task (struct rq *rq,struct task_struct *p,unsigned long task_state) +{ +if (dequeue_block_task (rq, p, task_state))__block_task (rq, p); }
@@ -3781,6 +3788,8 @@ static inline void proxy_reset_donor(struct rq *rq)*/ static inline bool proxy_needs_return (struct rq *rq,struct task_struct *p) { +bool dequeued; +/* * Typically per __set_task_cpu(), task_cpu(p) == p->wake_cpu. *
@@ -3803,12 +3812,23 @@ static inline bool proxy_needs_return(struct rq *rq, struct task_struct *p)/* If already current, don't need to return migrate */ if (task_current (rq, p)) return false; - -/* If we're return migrating the rq->donor, switch it out for idle */ -if (task_current_donor (rq, p)) - proxy_reset_donor(rq); } -block_task (rq, p,TASK_WAKING ); + + dequeued =dequeue_block_task (rq, p,TASK_WAKING ); + +/* +* Dequeue @p from its scheduling class before resetting rq->donor. +* In particular, sched_ext needs to end the donor's running session +* and clear SCX_TASK_QUEUED before put_prev_task_scx() is called by +* proxy_reset_donor(); otherwise it would reenqueue the blocked donor. +* +* Keep on_rq set until all donor references have been replaced. +*/ +if (task_current_donor (rq, p)) +proxy_reset_donor (rq); + +if (dequeued) +__block_task (rq, p);return true; }#else /* !CONFIG_SCHED_PROXY_EXEC */
Andrea Righi
reviewerPATCH 3/16 sched/core: Mark wakeups completed through ttwu_runnable()
In reply to Andrea Righittwu_runnable() completes a wakeup for a task which remains queued without calling activate_task() or its scheduling-class enqueue callback. Scheduling classes currently cannot distinguish this path from wakeups which performed a full activation.
Add WF_TTWU_RQ to identify wakeups completed through ttwu_runnable(). A later sched_ext change uses it to reconsider retained proxy donors which wake without another ops.enqueue() call.
No functional change.
--- kernel/sched/core.c | 2 +- kernel/sched/sched.h | 1 + 2 files changed, 2 insertions(+), 1 deletion(-)
kernel/sched/core.c +1 -1
@@ -3913,7 +3913,7 @@ static int ttwu_runnable(struct task_struct *p, int wake_flags)* When on_rq && !on_cpu the task is preempted, see if * it should preempt the task that is current now. */ -wakeup_preempt (rq, p, wake_flags); +wakeup_preempt (rq, p, wake_flags |WF_TTWU_RQ ); }ttwu_do_wakeup (p);return 1 ;
kernel/sched/sched.h +1 -0
@@ -2516,6 +2516,7 @@ static inline int task_on_rq_migrating(struct task_struct *p)#define WF_MIGRATED 0x20/* Internal use, task got migrated */ #define WF_CURRENT_CPU 0x40/* Prefer to move the wakee to the current CPU. */ #define WF_RQ_SELECTED 0x80/* ->select_task_rq() was called */ +#define WF_TTWU_RQ 0x100/* Wakeup completed through ttwu_runnable() */ static_assert (WF_EXEC ==SD_BALANCE_EXEC );static_assert (WF_FORK ==SD_BALANCE_FORK );
Signature · 1 line
2.55.0
[tip: sched/core] sched/core: Mark wakeups completed through ttwu_runnable()
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: a49653d0abebee32bf3ecf650fae9f9dd8298135 Gitweb: https://git.kernel.org/tip/a49653d0abebee32bf3ecf650fae9f9dd8298135 Author: Andrea Righi <[email protected]> AuthorDate: Tue, 22 Sep 2026 18:51:42 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Mon, 28 Sep 2026 08:44:39 +02:00
sched/core: Mark wakeups completed through ttwu_runnable()
ttwu_runnable() completes a wakeup for a task which remains queued without calling activate_task() or its scheduling-class enqueue callback. Scheduling classes currently cannot distinguish this path from wakeups which performed a full activation.
Add WF_TTWU_RQ to identify wakeups completed through ttwu_runnable(). A later sched_ext change uses it to reconsider retained proxy donors which wake without another ops.enqueue() call.
No functional change.
--- kernel/sched/core.c | 2 +- kernel/sched/sched.h | 1 + 2 files changed, 2 insertions(+), 1 deletion(-)
kernel/sched/core.c +1 -1
@@ -3919,7 +3919,7 @@ static int ttwu_runnable(struct task_struct *p, int wake_flags)* When on_rq && !on_cpu the task is preempted, see if * it should preempt the task that is current now. */ -wakeup_preempt (rq, p, wake_flags); +wakeup_preempt (rq, p, wake_flags |WF_TTWU_RQ ); }ttwu_do_wakeup (p);return 1 ;
kernel/sched/sched.h +1 -0
@@ -2545,6 +2545,7 @@ static inline int task_on_rq_migrating(struct task_struct *p)#define WF_MIGRATED 0x20/* Internal use, task got migrated */ #define WF_CURRENT_CPU 0x40/* Prefer to move the wakee to the current CPU. */ #define WF_RQ_SELECTED 0x80/* ->select_task_rq() was called */ +#define WF_TTWU_RQ 0x100/* Wakeup completed through ttwu_runnable() */ static_assert (WF_EXEC ==SD_BALANCE_EXEC );static_assert (WF_FORK ==SD_BALANCE_FORK );
Andrea Righi
reviewerPATCH 4/16 sched: Add helper to block retained proxy donors
In reply to Andrea RighiScheduler ownership changes may need to turn a mutex-blocked task retained on the runqueue for proxy execution back into a normally blocked task.
Add sched_proxy_block_task() to perform that transition while holding p->pi_lock and the task's runqueue lock. Reset an active donor to the physical current task before blocking it. If DELAY_DEQUEUE keeps a FAIR donor queued, the immediately following sched_change_begin() completes the dequeue through switching_from_fair() before recording its queued state.
This is a preparatory change to support proxy execution with sched_ext.
--- kernel/sched/core.c | 28 ++++++++++++++++++++++++++++ kernel/sched/sched.h | 6 ++++++ 2 files changed, 34 insertions(+)
kernel/sched/core.c +28 -0
@@ -6822,6 +6822,34 @@ static void proxy_deactivate(struct rq *rq, struct task_struct *donor)block_task (rq, donor, state); } +/* +* Remove a retained proxy donor before changing its scheduler ownership. +* The caller holds p->pi_lock, so p cannot wake and migrate if block_task() +* drops it from the runqueue. If DELAY_DEQUEUE keeps a blocked fair task +* queued, switching_from_fair() completes the dequeue in the immediately +* following sched_change_begin(). +*/ +void sched_proxy_block_task (struct rq *rq,struct task_struct *p) +{ +unsigned long state =READ_ONCE (p->__state); + +lockdep_assert_held (&p->pi_lock); +lockdep_assert_rq_held (rq); + +if (!p->is_blocked || !task_on_rq_queued (p)) +return ; +if (WARN_ON_ONCE (state ==TASK_RUNNING )) +return ; + +if (task_current_donor (rq, p)) +proxy_reset_donor (rq); + +if (!p->se.sched_delayed) +block_task (rq, p, state); + +WARN_ON_ONCE (task_on_rq_queued (p) && !p->se.sched_delayed); +} +static inline void proxy_release_rq_lock (struct rq *rq,struct rq_flags *rf)__releases (__rq_lockp (rq)) {
kernel/sched/sched.h +6 -0
@@ -2492,6 +2492,12 @@ static inline bool task_is_blocked(struct task_struct *p)return !!p->blocked_on; } +#ifdef CONFIG_SCHED_PROXY_EXEC +void sched_proxy_block_task (struct rq *rq,struct task_struct *p); +#else +static inline void sched_proxy_block_task (struct rq *rq,struct task_struct *p) {} +#endif +static inline int task_on_cpu (struct rq *rq,struct task_struct *p) {return p->on_cpu;
Signature · 1 line
2.55.0
[tip: sched/core] sched: Add helper to block retained proxy donors
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 57c75e3ae38c40acb5882a57c36f2d5234e066d0 Gitweb: https://git.kernel.org/tip/57c75e3ae38c40acb5882a57c36f2d5234e066d0 Author: Andrea Righi <[email protected]> AuthorDate: Tue, 22 Sep 2026 18:51:43 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Mon, 28 Sep 2026 08:44:39 +02:00
sched: Add helper to block retained proxy donors
Scheduler ownership changes may need to turn a mutex-blocked task retained on the runqueue for proxy execution back into a normally blocked task.
Add sched_proxy_block_task() to perform that transition while holding p->pi_lock and the task's runqueue lock. Reset an active donor to the physical current task before blocking it. If DELAY_DEQUEUE keeps a FAIR donor queued, the immediately following sched_change_begin() completes the dequeue through switching_from_fair() before recording its queued state.
This is a preparatory change to support proxy execution with sched_ext.
--- kernel/sched/core.c | 28 ++++++++++++++++++++++++++++ kernel/sched/sched.h | 6 ++++++ 2 files changed, 34 insertions(+)
kernel/sched/core.c +28 -0
@@ -6834,6 +6834,34 @@ static void proxy_deactivate(struct rq *rq, struct task_struct *donor)block_task (rq, donor, state); } +/* +* Remove a retained proxy donor before changing its scheduler ownership. +* The caller holds p->pi_lock, so p cannot wake and migrate if block_task() +* drops it from the runqueue. If DELAY_DEQUEUE keeps a blocked fair task +* queued, switching_from_fair() completes the dequeue in the immediately +* following sched_change_begin(). +*/ +void sched_proxy_block_task (struct rq *rq,struct task_struct *p) +{ +unsigned long state =READ_ONCE (p->__state); + +lockdep_assert_held (&p->pi_lock); +lockdep_assert_rq_held (rq); + +if (!p->is_blocked || !task_on_rq_queued (p)) +return ; +if (WARN_ON_ONCE (state ==TASK_RUNNING )) +return ; + +if (task_current_donor (rq, p)) +proxy_reset_donor (rq); + +if (!p->se.sched_delayed) +block_task (rq, p, state); + +WARN_ON_ONCE (task_on_rq_queued (p) && !p->se.sched_delayed); +} +static inline void proxy_release_rq_lock (struct rq *rq,struct rq_flags *rf)__releases (__rq_lockp (rq)) {
kernel/sched/sched.h +6 -0
@@ -2516,6 +2516,12 @@ static inline bool task_is_blocked(struct task_struct *p)return !!p->blocked_on; } +#ifdef CONFIG_SCHED_PROXY_EXEC +void sched_proxy_block_task (struct rq *rq,struct task_struct *p); +#else +static inline void sched_proxy_block_task (struct rq *rq,struct task_struct *p) {} +#endif +static inline int task_on_cpu (struct rq *rq,struct task_struct *p) {return p->on_cpu;
Andrea Righi
reviewerPATCH 5/16 sched: Add sched_ext hooks for proxy execution
In reply to Andrea RighiProxy execution splits the scheduling context (the donor) from the execution context (the lock owner). sched_ext needs to observe that split at three points in __schedule():
- whether a blocked EXT task can be retained on the runqueue as a donor,
- when a donor's scheduling context starts driving a lock owner,
- after proxy resolution, when deferred reenqueue work that was blocked by an active proxy relationship can be retried.
Introduce scx_allow_proxy_exec(), scx_proxy_donor_start() and scx_proxy_reenqueue_retry(), and add their call sites in __schedule(). The implementations are empty here and are filled in by the sched_ext changes that follow, so that all the sched core changes needed by proxy execution stay together in the preparatory patches.
SCHED_PROXY_EXEC still depends on !SCHED_CLASS_EXT, so the new hooks are
inert: they are compiled out with CONFIG_SCHED_CLASS_EXT=n and unreachable otherwise.
This is a preparatory change to support proxy execution with sched_ext. No functional change.
--- kernel/sched/core.c | 12 +++++++----- kernel/sched/ext/ext.c | 13 +++++++++++++ kernel/sched/ext/ext.h | 7 +++++++ 3 files changed, 27 insertions(+), 5 deletions(-)
kernel/sched/core.c +7 -5
@@ -7216,13 +7216,12 @@ static void __sched notrace __schedule(int sched_mode) } }else if (!preempt && prev_state) {/* -* We pass task_is_blocked() as the should_block arg -* in order to keep mutex-blocked tasks on the runqueue -* for slection with proxy-exec (without proxy-exec -* task_is_blocked() will always be false). +* Keep mutex-blocked tasks on the runqueue for proxy execution +* only when their scheduling class allows it. Without proxy +* execution, task_is_blocked() always returns false. */ try_to_block_task (rq, prev, &prev_state, - !task_is_blocked (prev)); + !task_is_blocked (prev) || !scx_allow_proxy_exec (prev)); switch_count = &prev->nvcsw; }
@@ -7243,6 +7242,7 @@ static void __sched notrace __schedule(int sched_mode) }if (next == rq->idle) {zap_balance_callbacks (rq); +scx_proxy_reenqueue_retry (rq, next); goto keep_resched; } }
@@ -7263,6 +7263,8 @@ static void __sched notrace __schedule(int sched_mode) donor->sched_class->put_prev_task (rq, donor, donor); donor->sched_class->set_next_task (rq, donor, true); } +scx_proxy_donor_start (rq); +scx_proxy_reenqueue_retry (rq, next); }else {rq_set_donor (rq, next); }
kernel/sched/ext/ext.c +13 -0
@@ -24,6 +24,11 @@DEFINE_RAW_SPINLOCK (scx_sched_lock); +bool scx_allow_proxy_exec (const struct task_struct *p) +{ +return true; +} +/* * NOTE: sched_ext is in the process of growing multiple scheduler support and * scx_root usage is in a transitional state. Naked dereferences are safe if the
@@ -1110,6 +1115,10 @@ static void schedule_deferred_locked(struct rq *rq)schedule_deferred (rq); } +void scx_proxy_reenqueue_retry (struct rq *rq) +{ +} +void schedule_dsq_reenq (struct scx_sched *sch,struct scx_dispatch_q *dsq,u64 reenq_flags,struct rq *locked_rq) {
@@ -3119,6 +3128,10 @@ static void set_next_task_scx(struct rq *rq, struct task_struct *p, bool first) } } +void scx_proxy_donor_start (struct rq *rq) +{ +} +static enum scx_cpu_preempt_reason preempt_reason_from_class (const struct sched_class *class) {
kernel/sched/ext/ext.h +7 -0
@@ -20,6 +20,9 @@ void scx_rq_deactivate(struct rq *rq);int scx_check_setscheduler (struct task_struct *p,int policy);bool task_should_scx (int policy);bool scx_allow_ttwu_queue (const struct task_struct *p); +bool scx_allow_proxy_exec (const struct task_struct *p); +void scx_proxy_donor_start (struct rq *rq); +void scx_proxy_reenqueue_retry (struct rq *rq,struct task_struct *next);void init_sched_ext_class (void );void __scx_update_idle (struct rq *rq,bool idle,bool do_notify);
@@ -61,6 +64,10 @@ static inline void scx_rq_deactivate(struct rq *rq) {}static inline int scx_check_setscheduler (struct task_struct *p,int policy) {return 0 ; }static inline bool task_on_scx (const struct task_struct *p) {return false; }static inline bool scx_allow_ttwu_queue (const struct task_struct *p) {return true; } +static inline bool scx_allow_proxy_exec (const struct task_struct *p) {return true; } +static inline void scx_proxy_donor_start (struct rq *rq) {} +static inline void scx_proxy_reenqueue_retry (struct rq *rq, +struct task_struct *next) {}static inline void init_sched_ext_class (void ) {}static inline void scx_update_idle (struct rq *rq,bool idle,bool do_notify) {}
Signature · 1 line
2.55.0
Peter Zijlstra
maintainerRe: [PATCH 05/16] sched: Add sched_ext hooks for proxy execution
Quoted 50 lines · Andrea Righi
Proxy execution splits the scheduling context (the donor) from the execution context (the lock owner).
sched_extneeds to observe that split at three points in__schedule():- whether a blocked EXT task can be retained on the runqueue as a donor,
- when a donor's scheduling context starts driving a lock owner,
- after proxy resolution, when deferred reenqueue work that was blocked by an active proxy relationship can be retried.Introduce
scx_allow_proxy_exec(),scx_proxy_donor_start()andscx_proxy_reenqueue_retry(), and add their call sites in__schedule(). The implementations are empty here and are filled in by thesched_extchanges that follow, so that all the sched core changes needed by proxy execution stay together in the preparatory patches.
SCHED_PROXY_EXECstill depends on !SCHED_CLASS_EXT, so the new hooks are
inert: they are compiled out withCONFIG_SCHED_CLASS_EXT=nand unreachable otherwise.This is a preparatory change to support proxy execution with
sched_ext. No functional change.Signed-off-by Andrea Righi <[email protected]>---
kernel/sched/core.c +3
-0@@ -7243,6 +7242,7 @@ static void __sched notrace __schedule(int sched_mode) }if (next == rq->idle) {zap_balance_callbacks (rq); +scx_proxy_reenqueue_retry (rq, next); goto keep_resched; } }@@ -7263,6 +7263,8 @@ static void __sched notrace __schedule(int sched_mode) donor->sched_class->put_prev_task (rq, donor, donor); donor->sched_class->set_next_task (rq, donor, true); } +scx_proxy_donor_start (rq); +scx_proxy_reenqueue_retry (rq, next); }else {rq_set_donor (rq, next); }kernel/sched/ext/ext.c +0
-0
kernel/sched/ext/ext.c static void schedule_deferred_locked(struct rq *rq) · line 1120
Quoted 6 lines
kernel/sched/ext/ext.c +3
-0@@ -1110,6 +1115,10 @@ static void schedule_deferred_locked(struct rq *rq)schedule_deferred (rq); } +void scx_proxy_reenqueue_retry (struct rq *rq) +{ +}
Doesn't match its prototype, also, I'm a little confused about the next argument in the next == rq->idle case.
Hmm, you seem to be using the argument like:
bool proxy = next != rq->donor;
And I suppose that works. But that seems to be about the tick, not putting current back on a dsq.
Quoted 14 lines
kernel/sched/ext/ext.h +3
-0@@ -20,6 +20,9 @@ void scx_rq_deactivate(struct rq *rq);int scx_check_setscheduler (struct task_struct *p,int policy);bool task_should_scx (int policy);bool scx_allow_ttwu_queue (const struct task_struct *p); +bool scx_allow_proxy_exec (const struct task_struct *p); +void scx_proxy_donor_start (struct rq *rq); +void scx_proxy_reenqueue_retry (struct rq *rq,struct task_struct *next);void init_sched_ext_class (void );void __scx_update_idle (struct rq *rq,bool idle,bool do_notify);
Andrea Righi
reviewerRe: [PATCH 05/16] sched: Add sched_ext hooks for proxy execution
Quoted 54 lines · Peter Zijlstra
Quoted 53 lines · Andrea Righi
Proxy execution splits the scheduling context (the donor) from the execution context (the lock owner).
sched_extneeds to observe that split at three points in__schedule():- whether a blocked EXT task can be retained on the runqueue as a donor,
- when a donor's scheduling context starts driving a lock owner,
- after proxy resolution, when deferred reenqueue work that was blocked by an active proxy relationship can be retried.Introduce
scx_allow_proxy_exec(),scx_proxy_donor_start()andscx_proxy_reenqueue_retry(), and add their call sites in__schedule(). The implementations are empty here and are filled in by thesched_extchanges that follow, so that all the sched core changes needed by proxy execution stay together in the preparatory patches.
SCHED_PROXY_EXECstill depends on !SCHED_CLASS_EXT, so the new hooks are
inert: they are compiled out withCONFIG_SCHED_CLASS_EXT=nand unreachable otherwise.This is a preparatory change to support proxy execution with
sched_ext. No functional change.Signed-off-by Andrea Righi <[email protected]>---
kernel/sched/core.c +3
-0@@ -7243,6 +7242,7 @@ static void __sched notrace __schedule(int sched_mode) }if (next == rq->idle) {zap_balance_callbacks (rq); +scx_proxy_reenqueue_retry (rq, next); goto keep_resched; } }@@ -7263,6 +7263,8 @@ static void __sched notrace __schedule(int sched_mode) donor->sched_class->put_prev_task (rq, donor, donor); donor->sched_class->set_next_task (rq, donor, true); } +scx_proxy_donor_start (rq); +scx_proxy_reenqueue_retry (rq, next); }else {rq_set_donor (rq, next); }kernel/sched/ext/ext.c +0
-0@@ -1110,6 +1115,10 @@ static void schedule_deferred_locked(struct rq *rq)schedule_deferred (rq); }
Quoted 6 lines
kernel/sched/ext/ext.c static void schedule_deferred_locked(struct rq *rq) · line 1120
Quoted 3 lines
kernel/sched/ext/ext.c +3
-0+void scx_proxy_reenqueue_retry (struct rq *rq) +{ +}Doesn't match its prototype, also, I'm a little confused about the next argument in the next ==
rq->idlecase.
Ah yes, the empty stub has the wrong signature (fixed now in my scx-proxy-exec-next branch). Thanks for catching it.
Quoted 7 lines
Hmm, you seem to be using the argument like:
bool proxy = next != rq->donor;And I suppose that works. But that seems to be about the tick, not putting current back on a dsq.
Yes, it's used for the NO_HZ_FULL tick bookkeeping.
In the next == rq->idle case, find_proxy_task() has called proxy_resched_idle(), which sets rq->donor to idle too. Passing next through the hook lets the tick bookkeeping see that proxy execution has stopped and clear its proxy tick state.
And the reenqueue retry is separate work in the same hook. I moved both actions in the same hook to avoid adding another call from the sched core, but I agree it looks a bit confusing.
-Andrea
Quoted 15 lines
Quoted 15 lines
kernel/sched/ext/ext.h +3
-0@@ -20,6 +20,9 @@ void scx_rq_deactivate(struct rq *rq);int scx_check_setscheduler (struct task_struct *p,int policy);bool task_should_scx (int policy);bool scx_allow_ttwu_queue (const struct task_struct *p); +bool scx_allow_proxy_exec (const struct task_struct *p); +void scx_proxy_donor_start (struct rq *rq); +void scx_proxy_reenqueue_retry (struct rq *rq,struct task_struct *next);void init_sched_ext_class (void );void __scx_update_idle (struct rq *rq,bool idle,bool do_notify);
[tip: sched/core] sched: Add sched_ext hooks for proxy execution
In reply to Andrea RighiThe following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: be100c77178ef2299a080eb698720832a870248b Gitweb: https://git.kernel.org/tip/be100c77178ef2299a080eb698720832a870248b Author: Andrea Righi <[email protected]> AuthorDate: Tue, 22 Sep 2026 18:51:44 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Mon, 28 Sep 2026 08:44:39 +02:00
sched: Add sched_ext hooks for proxy execution
Proxy execution splits the scheduling context (the donor) from the execution context (the lock owner). sched_ext needs to observe that split at three points in __schedule():
- whether a blocked EXT task can be retained on the runqueue as a donor,
- when a donor's scheduling context starts driving a lock owner,
- after proxy resolution, when deferred reenqueue work that was blocked by an active proxy relationship can be retried.
Introduce scx_allow_proxy_exec(), scx_proxy_donor_start() and scx_proxy_reenqueue_retry(), and add their call sites in __schedule(). The implementations are empty here and are filled in by the sched_ext changes that follow, so that all the sched core changes needed by proxy execution stay together in the preparatory patches.
SCHED_PROXY_EXEC still depends on !SCHED_CLASS_EXT, so the new hooks are
inert: they are compiled out with CONFIG_SCHED_CLASS_EXT=n and unreachable otherwise.
This is a preparatory change to support proxy execution with sched_ext. No functional change.
--- kernel/sched/core.c | 12 +++++++----- kernel/sched/ext/ext.c | 13 +++++++++++++ kernel/sched/ext/ext.h | 7 +++++++ 3 files changed, 27 insertions(+), 5 deletions(-)
kernel/sched/core.c +7 -5
@@ -7243,13 +7243,12 @@ static void __sched notrace __schedule(int sched_mode) } }else if (!preempt && prev_state) {/* -* We pass task_is_blocked() as the should_block arg -* in order to keep mutex-blocked tasks on the runqueue -* for slection with proxy-exec (without proxy-exec -* task_is_blocked() will always be false). +* Keep mutex-blocked tasks on the runqueue for proxy execution +* only when their scheduling class allows it. Without proxy +* execution, task_is_blocked() always returns false. */ try_to_block_task (rq, prev, &prev_state, - !task_is_blocked (prev)); + !task_is_blocked (prev) || !scx_allow_proxy_exec (prev)); switch_count = &prev->nvcsw; }
@@ -7270,6 +7269,7 @@ pick_again: }if (next == rq->idle) {zap_balance_callbacks (rq); +scx_proxy_reenqueue_retry (rq, next); goto keep_resched; } }
@@ -7290,6 +7290,8 @@ pick_again: donor->sched_class->put_prev_task (rq, donor, donor); donor->sched_class->set_next_task (rq, donor,SNT_PICK ); } +scx_proxy_donor_start (rq); +scx_proxy_reenqueue_retry (rq, next); }else {rq_set_donor (rq, next); }
kernel/sched/ext/ext.c +13 -0
@@ -24,6 +24,11 @@DEFINE_RAW_SPINLOCK (scx_sched_lock); +bool scx_allow_proxy_exec (const struct task_struct *p) +{ +return true; +} +/* * NOTE: sched_ext is in the process of growing multiple scheduler support and * scx_root usage is in a transitional state. Naked dereferences are safe if the
@@ -1079,6 +1084,10 @@ static void schedule_deferred_locked(struct rq *rq)schedule_deferred (rq); } +void scx_proxy_reenqueue_retry (struct rq *rq,struct task_struct *next) +{ +} +void schedule_dsq_reenq (struct scx_sched *sch,struct scx_dispatch_q *dsq,u64 reenq_flags,struct rq *locked_rq) {
@@ -3065,6 +3074,10 @@ static void set_next_task_scx(struct rq *rq, struct task_struct *p, enum snt_e t } } +void scx_proxy_donor_start (struct rq *rq) +{ +} +static enum scx_cpu_preempt_reason preempt_reason_from_class (const struct sched_class *class) {
kernel/sched/ext/ext.h +7 -0
@@ -20,6 +20,9 @@ void scx_rq_deactivate(struct rq *rq);int scx_check_setscheduler (struct task_struct *p,int policy);bool task_should_scx (int policy);bool scx_allow_ttwu_queue (const struct task_struct *p); +bool scx_allow_proxy_exec (const struct task_struct *p); +void scx_proxy_donor_start (struct rq *rq); +void scx_proxy_reenqueue_retry (struct rq *rq,struct task_struct *next);void init_sched_ext_class (void );static inline u32 scx_cpuperf_target (s32 cpu)
@@ -54,6 +57,10 @@ static inline void scx_rq_deactivate(struct rq *rq) {}static inline int scx_check_setscheduler (struct task_struct *p,int policy) {return 0 ; }static inline bool task_on_scx (const struct task_struct *p) {return false; }static inline bool scx_allow_ttwu_queue (const struct task_struct *p) {return true; } +static inline bool scx_allow_proxy_exec (const struct task_struct *p) {return true; } +static inline void scx_proxy_donor_start (struct rq *rq) {} +static inline void scx_proxy_reenqueue_retry (struct rq *rq, +struct task_struct *next) {}static inline void init_sched_ext_class (void ) {}#endif /* CONFIG_SCHED_CLASS_EXT */
Andrea Righi
reviewerPATCH 6/16 sched_ext: Block proxy donors before taking control
In reply to Andrea RighiProxy execution retains mutex-blocked donors on the runqueue so their scheduling context can execute a lock owner. sched_ext cannot safely inherit such donors unless the BPF scheduler explicitly participates in their admission and ordering.
Make sched_ext reject retained donors by default. Implement scx_allow_proxy_exec() to force blocked EXT tasks through the regular block path in schedule().
A donor retained by another scheduling class may already be queued when sched_ext is enabled or when a task switches to SCHED_EXT. Fully block the donor before sched_change_begin() records its queued state. This prevents sched_ext from importing a proxy session which the BPF scheduler did not admit, while leaving ordinary FAIR, RT and DL class transitions untouched.
This is a preparatory change to support proxy execution with sched_ext.
--- kernel/sched/ext/ext.c | 37 +++++++++++++++++++++++++++++++++++-- kernel/sched/ext/ext.h | 2 ++ kernel/sched/syscalls.c | 2 ++ 3 files changed, 39 insertions(+), 2 deletions(-)
kernel/sched/ext/ext.c +35 -2
@@ -26,7 +26,22 @@ DEFINE_RAW_SPINLOCK(scx_sched_lock);bool scx_allow_proxy_exec (const struct task_struct *p) { -return true; +return p->sched_class != &ext_sched_class; +} + +/* +* End retained proxy execution before sched_ext takes ownership of @p. +* Called with @p's pi and rq locks held immediately before +* sched_change_begin(). The caller must pass DEQUEUE_NOCLOCK so the rq clock +* is updated only once. +*/ +static void scx_prepare_task_sched_change (struct task_struct *p) +{ +lockdep_assert_held (&p->pi_lock); +lockdep_assert_rq_held (task_rq (p)); + +update_rq_clock (task_rq (p)); +sched_proxy_block_task (task_rq (p), p); }/*
@@ -4370,7 +4385,7 @@ int scx_check_setscheduler(struct task_struct *p, int policy) {lockdep_assert_rq_held (task_rq (p)); -/* if disallow, reject transitioning into SCX */ +/* If disallow, reject transitioning into SCX. */ if (scx_enabled () &&READ_ONCE (p->scx.disallow) && p->policy != policy && policy ==SCHED_EXT )return -EACCES ;
@@ -4378,6 +4393,20 @@ int scx_check_setscheduler(struct task_struct *p, int policy)return 0 ; } +/* +* Don't carry a donor retained by another class into sched_ext. The caller +* has updated the rq clock and invokes this immediately before +* sched_change_begin() records the task's queued state. +*/ +void scx_prepare_setscheduler (struct task_struct *p,int policy) +{ +lockdep_assert_held (&p->pi_lock); +lockdep_assert_rq_held (task_rq (p)); + +if (scx_enabled () && p->policy != policy && policy ==SCHED_EXT ) +sched_proxy_block_task (task_rq (p), p); +} +static void process_ddsp_deferred_locals (struct rq *rq) {struct task_struct *p;
@@ -7892,6 +7921,10 @@ static void scx_root_enable_workfn(struct kthread_work *work)if (old_class != new_class) queue_flags |=DEQUEUE_CLASS ; +if (new_class == &ext_sched_class) { +scx_prepare_task_sched_change (p); + queue_flags |=DEQUEUE_NOCLOCK ; + } scoped_guard (sched_change, p, queue_flags) {scx_set_task_slice (p,READ_ONCE (sch->slice_dfl));
kernel/sched/ext/ext.h +2 -0
@@ -18,6 +18,7 @@ bool scx_can_stop_tick(struct rq *rq);void scx_rq_activate (struct rq *rq);void scx_rq_deactivate (struct rq *rq);int scx_check_setscheduler (struct task_struct *p,int policy); +void scx_prepare_setscheduler (struct task_struct *p,int policy);bool task_should_scx (int policy);bool scx_allow_ttwu_queue (const struct task_struct *p);bool scx_allow_proxy_exec (const struct task_struct *p);
@@ -62,6 +63,7 @@ static inline bool scx_can_stop_tick(struct rq *rq) { return true; }static inline void scx_rq_activate (struct rq *rq) {}static inline void scx_rq_deactivate (struct rq *rq) {}static inline int scx_check_setscheduler (struct task_struct *p,int policy) {return 0 ; } +static inline void scx_prepare_setscheduler (struct task_struct *p,int policy) {}static inline bool task_on_scx (const struct task_struct *p) {return false; }static inline bool scx_allow_ttwu_queue (const struct task_struct *p) {return true; }static inline bool scx_allow_proxy_exec (const struct task_struct *p) {return true; }
kernel/sched/syscalls.c +2 -0
@@ -678,6 +678,8 @@ int __sched_setscheduler(struct task_struct *p,if (prev_class != next_class) queue_flags |=DEQUEUE_CLASS ; +scx_prepare_setscheduler (p, policy); + scoped_guard (sched_change, p, queue_flags) {if (!(attr->sched_flags &SCHED_FLAG_KEEP_PARAMS )) {
Signature · 1 line
2.55.0
Andrea Righi
reviewerPATCH 7/16 sched_ext: Fix ops.running/stopping() pairing for proxy-exec donors
In reply to Andrea RighiWith proxy execution, pick_next_task() can select a blocked task as the scheduling context before find_proxy_task() resolves the execution context.
From the BPF scheduler perspective, that donor is running while its scheduling context drives the lock owner; ops.tick() and other accounting must therefore remain enclosed by a matching ops.running()/ops.stopping() session.
In this scenario, the session boundaries do not always match physical task switches. Keep the "running" session open when the same donor continues on the same CPU. When proxy execution migrates a donor's scheduling context to another CPU, end its running session on the source CPU and start a new session on the destination CPU only after proxy resolution succeeds. This prevents a failed resolution from exposing a provisional ops.running() event.
Track these sessions with a new SCX_TASK_RUN_TRACKED flag. The explicit running-state tracking is also required by later donor-based accounting:
it prevents an EXT owner executing for a non-EXT donor from being treated as the active EXT scheduling context when it is dequeued.
This is a preparatory change for enabling proxy execution together with sched_ext.
--- include/linux/sched/ext.h | 1 + kernel/sched/ext/ext.c | 70 ++++++++++++++----- kernel/sched/ext/internal.h | 6 ++ .../sched_ext/include/scx/enum_defs.autogen.h | 1 + 4 files changed, 62 insertions(+), 16 deletions(-)
include/linux/sched/ext.h +1 -0
@@ -104,6 +104,7 @@ enum scx_ent_flags {SCX_TASK_SUB_INIT =1 <<4 ,/* task being initialized for a sub sched */ SCX_TASK_IMMED =1 <<5 ,/* task is on local DSQ with %SCX_ENQ_IMMED */ SCX_TASK_PROTECTED =1 <<6 ,/* slice and DSQ head position protected */ +SCX_TASK_RUN_TRACKED =1 <<7 ,/* task is in an ops.running()/stopping() session */ /* * Bits 8 to 10 are used to carry task state:
kernel/sched/ext/ext.c +54 -16
109 lines not on this page · raw message
kernel/sched/ext/internal.h +6 -0
@@ -468,6 +468,12 @@ struct sched_ext_ops {* Therefore, always use scx_bpf_task_cpu(@p) to determine the * target CPU the task is going to use. * +* Under proxy execution, the BPF scheduler continues to observe the +* donor as the current scheduling context. A blocked donor enters a +* ->running()/->stopping() session while its scheduling context drives +* the lock owner. The lock owner executing on its behalf is intentionally +* not reported through these callbacks. +* * See ->runnable() for explanation on the task state notifiers. */ void (*running)(struct task_struct *p);
tools/sched_ext/include/scx/enum_defs.autogen.h +1 -0
@@ -106,6 +106,7 @@#define HAVE_SCX_TASK_SUB_INIT #define HAVE_SCX_TASK_IMMED #define HAVE_SCX_TASK_PROTECTED +#define HAVE_SCX_TASK_RUN_TRACKED #define HAVE_SCX_TASK_STATE_SHIFT #define HAVE_SCX_TASK_STATE_BITS #define HAVE_SCX_TASK_STATE_MASK
Signature · 1 line
2.55.0
Andrea Righi
reviewerPATCH 8/16 sched_ext: Move reject DSQ draining into core
In reply to Andrea RighiThe reject DSQ drain will be generalized for placement failures outside sub-scheduler capability rejection. Move the drain into the core sched_ext implementation first so the subsequent behavioral changes can be reviewed separately. No functional change.
--- kernel/sched/ext/ext.c | 49 ++++++++++++++++++++++++++++++++++++++++++ kernel/sched/ext/sub.c | 45 -------------------------------------- kernel/sched/ext/sub.h | 2 -- 3 files changed, 49 insertions(+), 47 deletions(-)
kernel/sched/ext/ext.c +49 -0
@@ -4766,6 +4766,55 @@ static void process_deferred_reenq_users(struct rq *rq) } } +#ifdef CONFIG_EXT_SUB_SCHED +/* +* Drain @rq->scx.reject_dsq, reenqueueing each task so the BPF re-decides +* from p->scx.reenq_reason_*. +* +* A task can be re-rejected repeatedly. The reenqueue is bounded per task in +* scx_do_enqueue_task(), which ejects the owning sub past SCX_REENQ_MAX_REPEAT. +* Rejection can't happen for root. +*/ +static void scx_reenq_reject (struct rq *rq) +{ +LIST_HEAD (tasks); +struct task_struct *p, *n; + +lockdep_assert_rq_held (rq); + +if (!scx_has_subs () ||list_empty (&rq->scx.reject_dsq.list)) +return ; + +/* +* Move to a private list so a task re-rejected by the +* scx_do_enqueue_task() below isn't revisited this round. +*/ +list_for_each_entry_safe (p, n, &rq->scx.reject_dsq.list, scx.dsq_list.node) { +/* migration_pending tasks should have bypassed to local DSQ */ +if (WARN_ON_ONCE (p->migration_pending)) +continue ; + +scx_dispatch_dequeue (rq, p); + +if (WARN_ON_ONCE (p->scx.flags &SCX_TASK_REENQ_REASON_MASK )) + p->scx.flags &= ~SCX_TASK_REENQ_REASON_MASK ; + p->scx.flags |=SCX_TASK_REENQ_CAP ; + +list_add_tail (&p->scx.dsq_list.node, &tasks); + } + +list_for_each_entry_safe (p, n, &tasks, scx.dsq_list.node) { +list_del_init (&p->scx.dsq_list.node); + +scx_do_enqueue_task (rq, p,SCX_ENQ_REENQ ,-1 ); + + p->scx.flags &= ~SCX_TASK_REENQ_REASON_MASK ; + } +} +#else +static void scx_reenq_reject (struct rq *rq) {} +#endif +static void run_deferred (struct rq *rq) {process_ddsp_deferred_locals (rq);
kernel/sched/ext/sub.c +0 -45
51 lines not on this page · raw message
kernel/sched/ext/sub.h +0 -2
@@ -37,7 +37,6 @@ void scx_discard_stale_ecaps_syncs(void);struct scx_dispatch_q *scx_resolve_local_dsq (struct scx_sched *sch,struct rq *rq,struct task_struct *p,u64 *enq_flags);bool scx_task_reenq_on_cap_revoke (struct rq *rq,struct task_struct *p); -void scx_reenq_reject (struct rq *rq);void scx_rescue_charge (struct rq *rq,s64 delta_exec);void scx_rescue_end (struct rq *rq);bool scx_rescue_keep (struct rq *rq,struct task_struct *p);
@@ -93,7 +92,6 @@ static inline void scx_discard_ecaps_to_sync(s32 cpu, struct scx_sched_pcpu *pcpstatic inline void scx_discard_stale_ecaps_syncs (void ) {}static inline struct scx_dispatch_q *scx_resolve_local_dsq (struct scx_sched *sch,struct rq *rq,struct task_struct *p,u64 *enq_flags) {return &rq->scx.local_dsq; }static inline bool scx_task_reenq_on_cap_revoke (struct rq *rq,struct task_struct *p) {return false; } -static inline void scx_reenq_reject (struct rq *rq) {}static inline void scx_rescue_charge (struct rq *rq,s64 delta_exec) {}static inline void scx_rescue_end (struct rq *rq) {}static inline bool scx_rescue_keep (struct rq *rq,struct task_struct *p) {return false; }
Signature · 1 line
2.55.0
Andrea Righi
reviewerPATCH 9/16 sched_ext: Generalize the reject DSQ reenqueue path
In reply to Andrea RighiThe reject DSQ is currently specific to sub-scheduler cap failures. Its storage and initialization depend on CONFIG_EXT_SUB_SCHED, and the drain path assumes every rejected task was rejected for SCX_TASK_REENQ_CAP.
Other transient placement failures need the same ability to park a task on its source rq and return it to the owning BPF scheduler. Make the reject DSQ unconditional and carry the reenqueue reason directly in p->scx.flags from the rejection site.
A reenqueue reason remains valid while ops.enqueue() runs so that BPF can inspect it. Clear it immediately after the callback returns and before resolving a direct dispatch. Paths which bypass ops.enqueue() clear the reason before the kernel-selected placement. A rejection from the new placement can therefore install its reason into a clear field. Route all paths through a common cleanup point to enforce this ordering.
This is a preparatory change to support proxy execution with sched_ext.
--- kernel/sched/ext/ext.c | 91 +++++++++++++++++++++++------------------- kernel/sched/ext/sub.c | 2 + kernel/sched/sched.h | 2 +- 3 files changed, 53 insertions(+), 42 deletions(-)
kernel/sched/ext/ext.c +50 -41
@@ -1669,12 +1669,10 @@ static void scx_dispatch_enqueue(struct scx_sched *sch, struct rq *rq, struct scx_dispatch_q *dsq, struct task_struct *p, u64 slice, u64 vtime, u64 enq_flags) { - bool is_rq_owned = false; + bool is_rq_owned = dsq_is_rq_owned(dsq); -if (dsq->id ==SCX_DSQ_LOCAL ) { +if (dsq->id ==SCX_DSQ_LOCAL ) dsq =scx_resolve_local_dsq (sch, rq, p, &enq_flags); - is_rq_owned = true; - }WARN_ON_ONCE (p->scx.dsq || !list_empty (&p->scx.dsq_list.node)); WARN_ON_ONCE((p->scx.dsq_flags & SCX_TASK_DSQ_ON_PRIQ) ||
@@ -2054,6 +2052,12 @@ bool scx_rq_online(struct rq *rq)void scx_do_enqueue_task (struct rq *rq,struct task_struct *p,u64 enq_flags,int sticky_cpu) { +enum { +ENQ_ACTION_NONE , +ENQ_ACTION_DIRECT , +ENQ_ACTION_LOCAL_NOREFILL , +ENQ_ACTION_ENQUEUE , + } action =ENQ_ACTION_NONE ;struct scx_sched *sch =scx_task_sched (p);struct task_struct **ddsp_taskp;struct scx_dispatch_q *dsq;
@@ -2087,7 +2091,7 @@ void scx_do_enqueue_task(struct rq *rq, struct task_struct *p, u64 enq_flags,__scx_exit (sch,SCX_EXIT_ERROR_REENQ ,0 ,cpu_of (rq),"%s[%d] reenqueued %u times without running" , p->comm, p->pid, p->scx.reenq_cnt); -return ; + goto out; } }
@@ -2152,14 +2156,14 @@ void scx_do_enqueue_task(struct rq *rq, struct task_struct *p, u64 enq_flags,* dequeue may be waiting. The store_release matches their load_acquire. */ atomic_long_set_release (&p->scx.ops_state,SCX_OPSS_QUEUED | qseq); -return ; + goto out; direct: -direct_dispatch (sch, p, enq_flags); -return ; + action =ENQ_ACTION_DIRECT ; + goto out; local_norefill: -scx_dispatch_enqueue (sch, rq, &rq->scx.local_dsq, p,0 ,0 , enq_flags); -return ; + action =ENQ_ACTION_LOCAL_NOREFILL ; + goto out; local: dsq = &rq->scx.local_dsq; goto enqueue;
@@ -2171,9 +2175,27 @@ void scx_do_enqueue_task(struct rq *rq, struct task_struct *p, u64 enq_flags, goto enqueue; enqueue: -refill_task_slice_dfl (sch, p); -clear_direct_dispatch (p); -scx_dispatch_enqueue (sch, rq, dsq, p,0 ,0 , enq_flags); + action =ENQ_ACTION_ENQUEUE ; +out: +/* The reason is input to ops.enqueue(), not to the resulting placement. */ + p->scx.flags &= ~SCX_TASK_REENQ_REASON_MASK ; + +switch (action) { +case ENQ_ACTION_DIRECT : +direct_dispatch (sch, p, enq_flags); +break ; +case ENQ_ACTION_LOCAL_NOREFILL : +scx_dispatch_enqueue (sch, rq, &rq->scx.local_dsq, p,0 ,0 , + enq_flags); +break ; +case ENQ_ACTION_ENQUEUE : +refill_task_slice_dfl (sch, p); +clear_direct_dispatch (p); +scx_dispatch_enqueue (sch, rq, dsq, p,0 ,0 , enq_flags); +break ; +case ENQ_ACTION_NONE : +break ; + } }static bool task_runnable (const struct task_struct *p)
@@ -2393,6 +2415,7 @@ static bool dequeue_task_scx(struct rq *rq, struct task_struct *p, int core_deq_sub_nr_running (rq,1 );scx_dispatch_dequeue (rq, p); + p->scx.flags &= ~SCX_TASK_REENQ_REASON_MASK ;/* see scx_task_slice_ended() for the save/restore exception */ if (!((deq_flags &DEQUEUE_SAVE ) &&task_current (rq, p)))
@@ -3106,6 +3129,7 @@ static void set_next_task_scx(struct rq *rq, struct task_struct *p, bool first)*/ ops_dequeue (rq, p,SCX_DEQ_CORE_SCHED_EXEC );scx_dispatch_dequeue (rq, p); + p->scx.flags &= ~SCX_TASK_REENQ_REASON_MASK ; } p->se.exec_start =rq_clock_task (rq);
@@ -3280,7 +3304,6 @@ static void put_prev_task_scx(struct rq *rq, struct task_struct *p,if (p->scx.flags &SCX_TASK_IMMED ) { p->scx.flags |=SCX_TASK_REENQ_PREEMPTED ;scx_do_enqueue_task (rq, p,SCX_ENQ_REENQ ,-1 ); - p->scx.flags &= ~SCX_TASK_REENQ_REASON_MASK ; }else {u64 enq_flags =0 ;
@@ -4573,8 +4596,7 @@ static u32 reenq_local(struct scx_sched *sch, struct rq *rq, u64 reenq_flags)scx_dispatch_dequeue (rq, p); -if (WARN_ON_ONCE (p->scx.flags &SCX_TASK_REENQ_REASON_MASK )) - p->scx.flags &= ~SCX_TASK_REENQ_REASON_MASK ; +WARN_ON_ONCE (p->scx.flags &SCX_TASK_REENQ_REASON_MASK ); p->scx.flags |= reason;list_add_tail (&p->scx.dsq_list.node, &tasks);
@@ -4585,7 +4607,6 @@ static u32 reenq_local(struct scx_sched *sch, struct rq *rq, u64 reenq_flags)scx_do_enqueue_task (rq, p,SCX_ENQ_REENQ ,-1 ); - p->scx.flags &= ~SCX_TASK_REENQ_REASON_MASK ; nr_enqueued++; }
@@ -4697,14 +4718,11 @@ static void reenq_user(struct rq *rq, struct scx_dispatch_q *dsq, u64 reenq_flagdispatch_dequeue_locked (p, dsq);raw_spin_unlock (&dsq->lock); -if (WARN_ON_ONCE (p->scx.flags &SCX_TASK_REENQ_REASON_MASK )) - p->scx.flags &= ~SCX_TASK_REENQ_REASON_MASK ; +WARN_ON_ONCE (p->scx.flags &SCX_TASK_REENQ_REASON_MASK ); p->scx.flags |= reason;scx_do_enqueue_task (task_rq, p,SCX_ENQ_REENQ ,-1 ); - p->scx.flags &= ~SCX_TASK_REENQ_REASON_MASK ; -if (!(++nr_enqueued %SCX_TASK_ITER_BATCH )) {scx_rq_lock_drop (locked_rq);raw_spin_rq_unlock (locked_rq);
@@ -4766,14 +4784,14 @@ static void process_deferred_reenq_users(struct rq *rq) } } -#ifdef CONFIG_EXT_SUB_SCHED /* -* Drain @rq->scx.reject_dsq, reenqueueing each task so the BPF re-decides -* from p->scx.reenq_reason_*. +* Drain @rq->scx.reject_dsq and reenqueue each task so that its owning BPF +* scheduler chooses placement again. * -* A task can be re-rejected repeatedly. The reenqueue is bounded per task in -* scx_do_enqueue_task(), which ejects the owning sub past SCX_REENQ_MAX_REPEAT. -* Rejection can't happen for root. +* A task can be re-rejected repeatedly. Reenqueues are bounded per task by +* SCX_REENQ_MAX_REPEAT in scx_do_enqueue_task(), which ejects the owning +* scheduler. The private list below prevents a task from being revisited in +* the same round. */ static void scx_reenq_reject (struct rq *rq) {
@@ -4782,24 +4800,20 @@ static void scx_reenq_reject(struct rq *rq)lockdep_assert_rq_held (rq); -if (!scx_has_subs () ||list_empty (&rq->scx.reject_dsq.list)) +if (list_empty (&rq->scx.reject_dsq.list))return ;/* -* Move to a private list so a task re-rejected by the +* Move tasks to a private list so a task re-rejected by * scx_do_enqueue_task() below isn't revisited this round. */ list_for_each_entry_safe (p, n, &rq->scx.reject_dsq.list, scx.dsq_list.node) {/* migration_pending tasks should have bypassed to local DSQ */ -if (WARN_ON_ONCE (p->migration_pending)) -continue ; +WARN_ON_ONCE (p->migration_pending); +WARN_ON_ONCE (!(p->scx.flags &SCX_TASK_REENQ_REASON_MASK ));scx_dispatch_dequeue (rq, p); -if (WARN_ON_ONCE (p->scx.flags &SCX_TASK_REENQ_REASON_MASK )) - p->scx.flags &= ~SCX_TASK_REENQ_REASON_MASK ; - p->scx.flags |=SCX_TASK_REENQ_CAP ; -list_add_tail (&p->scx.dsq_list.node, &tasks); }
@@ -4807,13 +4821,8 @@ static void scx_reenq_reject(struct rq *rq)list_del_init (&p->scx.dsq_list.node);scx_do_enqueue_task (rq, p,SCX_ENQ_REENQ ,-1 ); - - p->scx.flags &= ~SCX_TASK_REENQ_REASON_MASK; } } -#else -static void scx_reenq_reject (struct rq *rq) {} -#endif static void run_deferred (struct rq *rq) {
@@ -8909,8 +8918,8 @@ void __init init_sched_ext_class(void)/* local_dsq's sch will be set during scx_root_enable() */ BUG_ON (scx_init_dsq (&rq->scx.local_dsq,SCX_DSQ_LOCAL ,NULL )); -#ifdef CONFIG_EXT_SUB_SCHED BUG_ON (scx_init_dsq (&rq->scx.reject_dsq,SCX_DSQ_REJECT ,NULL )); +#ifdef CONFIG_EXT_SUB_SCHED scx_rescue_init (rq);#endif
kernel/sched/ext/sub.c +2 -0
@@ -753,6 +753,8 @@ struct scx_dispatch_q *scx_resolve_local_dsq(struct scx_sched *sch, struct rq *r p->scx.reenq_reason_caps = missing; p->scx.reenq_reason_cid = cid; +WARN_ON_ONCE (p->scx.flags &SCX_TASK_REENQ_REASON_MASK ); + p->scx.flags |=SCX_TASK_REENQ_CAP ;return &rq->scx.reject_dsq; }
kernel/sched/sched.h +1 -1
@@ -807,8 +807,8 @@ struct scx_rq_rescue {struct scx_rq {struct scx_dispatch_q local_dsq; +struct scx_dispatch_q reject_dsq;/* staging for rejected tasks */ #ifdef CONFIG_EXT_SUB_SCHED -struct scx_dispatch_q reject_dsq;/* staging for cap-rejected tasks */ struct scx_rq_rescue rescue;#endif struct list_head runnable_list;/* runnable tasks on this rq */
Signature · 1 line
2.55.0
Andrea Righi
reviewerPATCH 10/16 sched_ext: Handle proxy-exec races in remote DSQ transfers
In reply to Andrea RighiWithout proxy execution, the DSQ lock and holding_cpu handshake ensure that a task cannot be dequeued or start running during an rq-lock handoff without clearing holding_cpu.
Proxy execution is an exception: a task can start physically executing as a lock owner while its scheduling context remains on a DSQ; its on-CPU or migration-disabled state can therefore change without clearing holding_cpu.
Recheck these states after acquiring the source rq lock. If the transfer can no longer proceed, park the task on the source rq's reject DSQ and reenqueue it through its owning scheduler. This preserves the BPF scheduler's placement policy and keeps descendant tasks within their sub-scheduler's cap grants.
Queue a deferred drain for every rejection. If a drain finds a proxy-rejected task still running or donating, leave it parked and record a retry request on the rq. scx_proxy_reenqueue_retry() consumes the request and schedules another drain when proxy state changes.
Without this change and proxy execution enabled, stress-ng --pipeherd can trigger this race and migrate an active execution context, leading to sleeping-while-atomic warnings and subsequent lockdep corruption.
This is a preparatory change to support proxy execution with sched_ext.
--- include/linux/sched/ext.h | 2 + kernel/sched/ext/ext.c | 145 +++++++++++++++--- kernel/sched/ext/internal.h | 9 ++ kernel/sched/ext/sub.c | 4 +- kernel/sched/sched.h | 1 + .../sched_ext/include/scx/enum_defs.autogen.h | 1 + 6 files changed, 141 insertions(+), 21 deletions(-)
include/linux/sched/ext.h +2 -0
@@ -137,6 +137,7 @@ enum scx_ent_flags {* IMMED reenqueued due to failed ENQ_IMMED * PREEMPTED preempted while running * CAP sub-sched cap miss, see p->scx.reenq_reason_* +* PROXY proxy state prevented a remote DSQ transfer */ SCX_TASK_REENQ_REASON_SHIFT =12 , SCX_TASK_REENQ_REASON_BITS = 3,
@@ -147,6 +148,7 @@ enum scx_ent_flags {SCX_TASK_REENQ_IMMED =2 <<SCX_TASK_REENQ_REASON_SHIFT ,SCX_TASK_REENQ_PREEMPTED =3 <<SCX_TASK_REENQ_REASON_SHIFT ,SCX_TASK_REENQ_CAP =4 <<SCX_TASK_REENQ_REASON_SHIFT , +SCX_TASK_REENQ_PROXY =5 <<SCX_TASK_REENQ_REASON_SHIFT ,/* iteration cursor, not a task */ SCX_TASK_CURSOR = 1 << 31,
kernel/sched/ext/ext.c +127 -18
218 lines not on this page · raw message
kernel/sched/ext/internal.h +9 -0
@@ -1777,6 +1777,15 @@ enum scx_enq_flags { SCX_ENQ_SLICE_DFL = 1LLU << 62, /* carried slice is a default refill */ }; +/* Strip priority and carried slice state when diverting from a local DSQ. */ +static inline void scx_divert_strip_flags (struct task_struct *p,u64 *enq_flags) +{ + *enq_flags &= ~(SCX_ENQ_IMMED |SCX_ENQ_PREEMPT | +SCX_ENQ_PREEMPT_LAZY |SCX_ENQ_HEAD | +SCX_ENQ_APPLY_SLICE |SCX_ENQ_SLICE_DFL ); + p->scx.flags &= ~SCX_TASK_IMMED ; +} +enum scx_deq_flags {/* expose select DEQUEUE_* flags as enums */ SCX_DEQ_SLEEP =DEQUEUE_SLEEP ,
kernel/sched/ext/sub.c +1 -3
@@ -736,9 +736,7 @@ struct scx_dispatch_q *scx_resolve_local_dsq(struct scx_sched *sch, struct rq *r* or HEAD - a diversion has no priority and IMMED is not allowed on * non-local DSQs. Strip the enq and task flags along with the slice. */ - *enq_flags &= ~(SCX_ENQ_IMMED |SCX_ENQ_PREEMPT |SCX_ENQ_PREEMPT_LAZY | -SCX_ENQ_HEAD |SCX_ENQ_APPLY_SLICE |SCX_ENQ_SLICE_DFL ); - p->scx.flags &= ~SCX_TASK_IMMED ; +scx_divert_strip_flags (p, enq_flags);/* the enqueuer opted for rescue instead of rejection and reenqueue */ if ((*enq_flags &SCX_ENQ_RESCUE ) &&likely (scx_rescue_bw_1024)) {
kernel/sched/sched.h +1 -0
@@ -788,6 +788,7 @@ enum scx_rq_flags {SCX_RQ_BAL_CB_PENDING =1 <<6 ,/* must queue a cb after dispatching */ SCX_RQ_SUB_IDLE_RENOTIFY =1 <<7 ,/* sub-scheds are owed update_idle() */ SCX_RQ_ROOT_IDLE_RENOTIFY =1 <<8 ,/* the root is owed update_idle() */ +SCX_RQ_PROXY_RETRY =1 <<9 ,/* proxy-rejected tasks need retry */ SCX_RQ_IN_WAKEUP =1 <<16 , SCX_RQ_IN_DISPATCH = 1 << 17,
tools/sched_ext/include/scx/enum_defs.autogen.h +1 -0
@@ -124,6 +124,7 @@#define HAVE_SCX_TASK_REENQ_IMMED #define HAVE_SCX_TASK_REENQ_PREEMPTED #define HAVE_SCX_TASK_REENQ_CAP +#define HAVE_SCX_TASK_REENQ_PROXY #define HAVE_SCX_TASK_CURSOR #define HAVE_SCX_ECODE_RSN_HOTPLUG #define HAVE_SCX_ECODE_RSN_CGROUP_OFFLINE
Signature · 1 line
2.55.0
Andrea Righi
reviewerPATCH 11/16 sched_ext: Split curr|donor references properly
In reply to Andrea RighiWith proxy execution, the task selected by the scheduler and the task physically executing can differ. A blocked mutex waiter donates its scheduling context to the lock owner:
D -----------------> M -------------> O ----------------> T [donor] blocked on [mutex] owned by [owner] preempted by [task]
\_________________________________^
donates scheduling context
where:
D = blocked donor
M = mutex
O = mutex owner
T = competing runnable task
During a proxy execution switch, D supplies the scheduling class, priority, and runtime budget, while O supplies the execution context: O is the task whose code physically executes. T is a competing runnable task which may preempt the D/O proxy execution.
Consider FAIR and EXT tasks with sched_ext running in partial mode. FAIR can be replaced with a higher scheduling class such as RT or deadline without changing the class interaction described here.
An active proxy pair can only have a donor in the same scheduling class as the owner or in a higher-priority class. If the owner belongs to a higher-priority class, pick_next_task() selects the owner directly before considering the donor, so find_proxy_task() is not entered for that donor. The blocked-on relationship may still exist, but it does not result in proxy execution.
The relevant combinations are:
1. D is EXT, O is EXT, T is EXT
D can interrupt T according to BPF scheduling policy. O executes with D's EXT priority and runtime budget, while T waits in EXT.
2. D is EXT, O is EXT, T is FAIR
D is visible to the BPF scheduler, but cannot preempt T because EXT is below FAIR. Once T stops, BPF can dispatch D and O executes with D's EXT priority and runtime budget. If T becomes runnable again, it preempts the D/O proxy execution.
3. D is FAIR, O is EXT, T is EXT
D preempts T as the higher-class scheduling context. O executes with D's FAIR priority and runtime budget, while T waits in EXT. D is not visible to the BPF scheduler.
4. D is FAIR, O is EXT, T is FAIR
D competes with T according to its FAIR deadline. When D is selected, O executes with D's FAIR priority and runtime budget. D is not visible to the BPF scheduler.
5. D is FAIR, O is FAIR, T is EXT
The D/O proxy execution has a FAIR scheduling context, so T cannot preempt it from the lower EXT class.
6. D is FAIR, O is FAIR, T is FAIR
O, T, and D all have FAIR scheduling contexts. D remains runnable as a blocked proxy donor. When CFS selects D, O executes using D's FAIR scheduling context. When CFS selects O, O executes using its own FAIR context, and when CFS selects T, T executes normally. D is not visible to the BPF scheduler.
Thus, sched_ext policy and accounting must generally use rq->donor, the scheduler-selected task which supplies the scheduling context, rather than rq->curr, the task whose code physically executes. Without proxy execution they are the same task.
On nohz_full CPUs, keep the tick running whenever the selected donor is blocked. A selected blocked donor is necessarily awaiting proxy execution, so this avoids relying on the transient relationship between rq->curr and rq->donor during context switches. It also leaves ordinary sched_ext context switches and non-nohz_full CPUs untouched. Allowing proxy execution to run tickless remains a future improvement.
Moreover, extend scx_dump_state() to report both contexts. Each CPU record now includes a donor= line. If an EXT donor differs from rq->curr, also emit its detailed task record. The existing '*' marker continues to identify rq->curr, while the donor= line identifies the otherwise unmarked donor record.
Note that at this point in the series, CONFIG_SCHED_PROXY_EXEC still depends on !CONFIG_SCHED_CLASS_EXT, so proxy execution and sched_ext cannot be enabled together. The scheduling changes are therefore preparatory. A later patch removes this restriction.
--- Documentation/scheduler/sched-ext.rst | 13 +++ kernel/sched/ext/ext.c | 133 ++++++++++++++++---------- kernel/sched/ext/sub.c | 5 +- kernel/sched/ext/sub.h | 11 ++- 4 files changed, 107 insertions(+), 55 deletions(-)
Documentation/scheduler/sched-ext.rst +13 -0
@@ -503,6 +503,19 @@ and edge cases, to name a few examples: class, in which case it will exit the tick-dispatch loop even though it is runnable and has a non-zero slice. +* Under proxy execution, sched_ext continues to observe the donor as the + current scheduling context. Accordingly, ``ops.running()`` and + ``ops.stopping()`` report when the donor's scheduling context becomes active + and inactive, even when the donor is blocked and a lock owner executes on its + behalf. The physical execution context is intentionally not reported through + these callbacks. + + A blocked donor enters a running session only after proxy resolution finds + an execution context. The session remains active if only the physical + execution context changes while the donor remains the same. Running sessions + are tracked so that ``ops.running()`` and ``ops.stopping()`` remain paired + and are not emitted recursively. + See the "Scheduling Cycle" section for a more detailed description of how a freshly woken up task gets on a CPU.
kernel/sched/ext/ext.c +85 -48
347 lines not on this page · raw message
kernel/sched/ext/sub.c +3 -2
@@ -285,7 +285,8 @@ void scx_rescue_charge(struct rq *rq, s64 delta_exec) rq->scx.rescue.budget -= delta_exec;/* per-cpu usage average feeds the overload victim pick */ - pcpu =per_cpu_ptr (scx_task_sched (rq->curr)->pcpu,cpu_of (rq)); + pcpu =per_cpu_ptr (scx_task_sched (rq->scx.rescue.curr)->pcpu, +cpu_of (rq)); pcpu->rescue_avg =scx_rescue_decay_avg (pcpu) + delta_exec;if (!scx_rescue_slice_remaining (rq))
@@ -557,7 +558,7 @@ static void scx_rescue_timerfn(struct timer_list *timer)scx_rescue_admit (rq, p, slice);scx_move_local_task_to_local_dsq (scx_task_sched (p), p,SCX_ENQ_IGNORE_CAPS , &rq->scx.rescue.dsq, rq); -if (sched_class_above (&ext_sched_class, rq->curr->sched_class)) +if (sched_class_above (&ext_sched_class, rq->donor->sched_class))resched_curr (rq); }else if (p->scx.dsq && rq->scx.rescue.budget >2 * scx_rescue_quantum_ns) {/*
kernel/sched/ext/sub.h +6 -5
@@ -167,17 +167,18 @@ static inline u64 scx_caps_for_task(struct task_struct *p)return SCX_CAP_ENQ ; } -/* the cap @sch needs to preempt @rq's current task, 0 if none */ -static inline u64 scx_caps_for_preempt (struct scx_sched *sch,struct rq *rq,u64 enq_flags) +/* the cap @sch needs to preempt @rq's current scheduling context, 0 if none */ +static inline u64 scx_caps_for_preempt (struct scx_sched *sch,struct rq *rq, +u64 enq_flags) { -struct task_struct *curr = rq->curr; +struct task_struct *donor = rq->donor;/* a kernel-forced placement preempts regardless of caps */ if (unlikely (enq_flags &SCX_ENQ_IGNORE_CAPS ))return 0 ;/* a non-ext task can't be preempted by ext, own-subtree needs no cap */ -if (curr->sched_class != &ext_sched_class || -scx_is_descendant (scx_task_sched (curr), sch)) +if (donor->sched_class != &ext_sched_class || +scx_is_descendant (scx_task_sched (donor), sch))return 0 ;return SCX_CAP_PREEMPT ; }
Signature · 1 line
2.55.0
Andrea Righi
reviewerPATCH 12/16 sched_ext: Track proxy execution for NOHZ_FULL
In reply to Andrea Righiscx_can_stop_tick() currently treats any blocked rq->donor as an active proxy session. However, rq->donor can temporarily still identify an outgoing blocked task while the scheduler selects an ordinary execution context. A dependency update in that window can keep TICK_DEP_BIT_SCHED set after the context switch.
Track whether proxy resolution actually selected a different execution context. Keep the tick active while that state is set instead of deriving it from a potentially stale donor.
When proxy execution ends, clear the state and queue a balance callback to reevaluate the tick dependency after context_switch() updates rq->curr. This lets sched_can_stop_tick() observe the complete donor and execution selection without adding work outside the existing proxy-execution branch.
Conservatively re-enable the periodic scheduler tick for the duration of proxy execution. Allowing proxy execution itself to run tickless would require making the remote NOHZ scheduler tick aware of the split between rq->curr and rq->donor, which is left as a future improvement.
Perform the tick update from scx_proxy_reenqueue_retry(), which already runs after proxy resolution, so scheduler core needs no additional hook. Compile the additional runqueue state and callback out when CONFIG_NO_HZ_FULL is disabled.
--- kernel/sched/ext/ext.c | 44 +++++++++++++++++++++++++++++++++++++++--- kernel/sched/sched.h | 4 ++++ 2 files changed, 45 insertions(+), 3 deletions(-)
kernel/sched/ext/ext.c +41 -3
@@ -1130,13 +1130,51 @@ static void schedule_deferred_locked(struct rq *rq)schedule_deferred (rq); } +#ifdef CONFIG_NO_HZ_FULL +static void scx_proxy_tick_bal_cb (struct rq *rq) +{ +sched_update_tick_dependency (rq); +} + +static void scx_proxy_update_tick (struct rq *rq,struct task_struct *next) +{ +bool proxy = next != rq->donor; +bool was_proxy = rq->scx.flags &SCX_RQ_PROXY_TICK ; + +if (likely (proxy == was_proxy)) +return ; + +if (proxy) { +/* Keep proxy execution tick-driven for now. */ + rq->scx.flags |=SCX_RQ_PROXY_TICK ; +tick_nohz_dep_set_cpu (cpu_of (rq),TICK_DEP_BIT_SCHED ); + }else { + rq->scx.flags &= ~SCX_RQ_PROXY_TICK ; +/* +* The selected donor is already visible, but rq->curr still +* identifies the outgoing execution context. Reevaluate after +* context_switch() updates rq->curr so sched_can_stop_tick() sees +* the complete selection. +*/ +queue_balance_callback (rq, &rq->scx.proxy_tick_bal_cb, + scx_proxy_tick_bal_cb); + } +} +#endif +/* -* Retry proxy-rejected tasks which couldn't be reenqueued by an earlier drain. +* Complete sched_ext bookkeeping after proxy resolution and retry tasks which +* couldn't be reenqueued by an earlier reject DSQ drain. */ void scx_proxy_reenqueue_retry (struct rq *rq,struct task_struct *next) {lockdep_assert_rq_held (rq); +#ifdef CONFIG_NO_HZ_FULL +if (scx_enabled () &&tick_nohz_full_cpu (cpu_of (rq))) +scx_proxy_update_tick (rq, next); +#endif +if (rq->scx.flags &SCX_RQ_PROXY_RETRY ) { rq->scx.flags &= ~SCX_RQ_PROXY_RETRY ;schedule_deferred_locked (rq);
@@ -4966,8 +5004,8 @@ bool scx_can_stop_tick(struct rq *rq)struct task_struct *p = rq->donor;struct scx_sched *sch =scx_task_sched (p); -/* Keep the tick running while a blocked proxy donor is selected. */ -if (p->is_blocked) +/* Proxy execution is conservatively tick-driven for now. */ +if (rq->scx.flags &SCX_RQ_PROXY_TICK )return false;if (p->sched_class != &ext_sched_class)
kernel/sched/sched.h +4 -0
@@ -789,6 +789,7 @@ enum scx_rq_flags {SCX_RQ_SUB_IDLE_RENOTIFY =1 <<7 ,/* sub-scheds are owed update_idle() */ SCX_RQ_ROOT_IDLE_RENOTIFY =1 <<8 ,/* the root is owed update_idle() */ SCX_RQ_PROXY_RETRY =1 <<9 ,/* proxy-rejected tasks need retry */ +SCX_RQ_PROXY_TICK =1 <<10 ,/* proxy execution requires the tick */ SCX_RQ_IN_WAKEUP =1 <<16 , SCX_RQ_IN_DISPATCH = 1 << 17,
@@ -843,6 +844,9 @@ struct scx_rq {struct list_head deferred_reenq_users;/* user DSQs requesting reenq */ struct balance_callback deferred_bal_cb;struct balance_callback kick_sync_bal_cb; +#ifdef CONFIG_NO_HZ_FULL +struct balance_callback proxy_tick_bal_cb; +#endif struct irq_work deferred_irq_work;struct irq_work kick_cpus_irq_work; };
Signature · 1 line
2.55.0
Andrea Righi
reviewerPATCH 13/16 sched_ext: Delegate proxy donor admission to BPF schedulers
In reply to Andrea RighiProxy execution keeps a mutex-blocked donor runnable so that its scheduling context can execute the mutex owner.
Define SCX_OPS_ENQ_BLOCKED as the admission contract for retained donors. Schedulers without the flag block EXT donors normally. Schedulers with the flag continue to own blocked donors and receive them through ops.enqueue() with SCX_ENQ_BLOCKED, allowing BPF to choose their DSQ, CPU and ordering.
After BPF dispatches a donor, proxy execution may move its context to the rq of the mutex owner. This move is distinct from BPF placement:proxy_set_task_cpu() changes task_cpu() while preserving the donor's pre-proxy CPU in wake_cpu as the starting point for normal wakeup placement. A proxy-migrated donor returns through the full wakeup activation path when the mutex is released.
Do not carry a retained proxy session across BPF scheduler ownership changes. Before root activation, parent-to-child takeover, rehome or punt, fully deactivate a retained donor unconditionally. The incoming scheduler then starts with clean task state and applies its admission policy the next time the task blocks.
A retained donor can wake through ttwu_runnable() without another enqueue_task_scx() call. Use WF_TTWU_RQ to identify this path, where the wakeup skipped activation and enqueueing. wakeup_preempt_scx() requests dispatch reconsideration after is_blocked is cleared. A full wakeup activation has already enqueued the task and does not need another reschedule.
Blocked-donor enqueueing takes precedence over the exiting and migration-disabled local-DSQ fallbacks, so that an opted-in scheduler sees every eligible donor request.
--- kernel/sched/ext/ext.c | 104 ++++++++++++++---- kernel/sched/ext/ext.h | 10 +- kernel/sched/ext/internal.h | 23 +++- kernel/sched/ext/sub.c | 12 +- tools/sched_ext/include/scx/compat.h | 1 + .../sched_ext/include/scx/enum_defs.autogen.h | 1 + .../sched_ext/include/scx/enums.autogen.bpf.h | 3 + tools/sched_ext/include/scx/enums.autogen.h | 1 + .../sched_ext/include/scx/enums_abi.autogen.h | 2 +- 9 files changed, 132 insertions(+), 25 deletions(-)
kernel/sched/ext/ext.c +85 -19
@@ -24,18 +24,24 @@DEFINE_RAW_SPINLOCK (scx_sched_lock); -bool scx_allow_proxy_exec (const struct task_struct *p) +bool __scx_allow_proxy_exec (const struct task_struct *p) { -return p->sched_class != &ext_sched_class; +struct scx_sched *sch; + +if (p->sched_class != &ext_sched_class) +return true; + + sch =scx_task_sched (p); +return !sch || (sch->ops.flags &SCX_OPS_ENQ_BLOCKED ); }/* -* End retained proxy execution before sched_ext takes ownership of @p. +* End retained proxy execution before changing @p's BPF scheduler ownership. * Called with @p's pi and rq locks held immediately before * sched_change_begin(). The caller must pass DEQUEUE_NOCLOCK so the rq clock * is updated only once. */ -static void scx_prepare_task_sched_change (struct task_struct *p) +void scx_prepare_task_sched_change (struct task_struct *p) {lockdep_assert_held (&p->pi_lock);lockdep_assert_rq_held (task_rq (p));
@@ -2167,19 +2173,28 @@ void scx_do_enqueue_task(struct rq *rq, struct task_struct *p, u64 enq_flags,if (p->scx.ddsp_dsq_id !=SCX_DSQ_INVALID ) goto direct; -/* see %SCX_OPS_ENQ_EXITING */ -if (!(sch->ops.flags &SCX_OPS_ENQ_EXITING ) && -unlikely (p->flags &PF_EXITING )) { -__scx_add_event (sch,SCX_EV_ENQ_SKIP_EXITING ,1 ); - enq_flags |=SCX_ENQ_RESCUE ;/* avoid looping on cap rejection */ - goto local; - } +/* +* A full wakeup enqueue happens before is_blocked is cleared. Don't +* report it as another blocked-donor admission. +*/ +if ((sch->ops.flags &SCX_OPS_ENQ_BLOCKED ) && p->is_blocked && + !(enq_flags &SCX_ENQ_WAKEUP )) { + enq_flags |=SCX_ENQ_BLOCKED ; + }else { +/* see %SCX_OPS_ENQ_EXITING */ +if (unlikely (p->flags &PF_EXITING ) && + !(sch->ops.flags &SCX_OPS_ENQ_EXITING )) { +__scx_add_event (sch,SCX_EV_ENQ_SKIP_EXITING ,1 ); + enq_flags |=SCX_ENQ_RESCUE ;/* avoid looping on cap rejection */ + goto local; + } -/* see %SCX_OPS_ENQ_MIGRATION_DISABLED */ -if (!(sch->ops.flags &SCX_OPS_ENQ_MIGRATION_DISABLED ) && -is_migration_disabled (p)) { -__scx_add_event (sch,SCX_EV_ENQ_SKIP_MIGRATION_DISABLED ,1 ); - goto local; +/* see %SCX_OPS_ENQ_MIGRATION_DISABLED */ +if (!(sch->ops.flags &SCX_OPS_ENQ_MIGRATION_DISABLED ) && +is_migration_disabled (p)) { +__scx_add_event (sch,SCX_EV_ENQ_SKIP_MIGRATION_DISABLED ,1 ); + goto local; + } }if (unlikely (!SCX_HAS_OP (sch, enqueue)))
@@ -2515,10 +2530,23 @@ static void wakeup_preempt_scx(struct rq *rq, struct task_struct *p, int wake_fl/* * Preemption between SCX tasks is implemented by resetting the victim * task's slice to 0 and triggering reschedule on the target CPU. -* Nothing to do. -*/ -if (p->sched_class == &ext_sched_class) +* +* A retained proxy donor can wake through ttwu_runnable() without another +* ops.enqueue(). WF_TTWU_RQ identifies this path. Request rescheduling so +* that ops.dispatch() can reconsider the task after ttwu_runnable() clears +* is_blocked. A full wakeup activation has already enqueued the task and +* doesn't need the additional reschedule. +*/ +if (p->sched_class == &ext_sched_class) { +if (sched_proxy_exec () && (wake_flags &WF_TTWU_RQ ) && + p->is_blocked) { +struct scx_sched *sch =scx_task_sched (p); + +if (sch && (sch->ops.flags &SCX_OPS_ENQ_BLOCKED )) +resched_curr (rq); + }return ; + }/* * Getting preempted by a higher-priority class. Reenqueue IMMED tasks.
@@ -2633,6 +2661,21 @@ static bool task_can_run_on_remote_rq(struct scx_sched *sch,WARN_ON_ONCE (task_cpu (p) == cpu); +/* +* proxy_set_task_cpu() preserves wake_cpu when moving a donor's scheduling +* context to its lock owner's rq. For example, if BPF places donor D on +* CPU0 while its lock owner runs on CPU1, proxy execution moves D to CPU1 +* but leaves D->wake_cpu pointing to CPU0. If D is later put on a shared +* DSQ, CPU0 could otherwise consume it and move it back, only for proxy +* execution to return it to CPU1 again. +* +* When task_cpu() == wake_cpu, the donor has not been proxy-migrated and +* BPF may still choose its placement. An actively donating task is +* rejected separately by task_proxy_running_or_donating(). +*/ +if (sched_proxy_exec () && p->is_blocked &&task_cpu (p) != p->wake_cpu) +return false; +/* * If @p has migration disabled, @p->cpus_ptr is updated to contain only * the pinned CPU in migrate_disable_switch() while @p is being switched
@@ -3436,6 +3479,24 @@ static void put_prev_task_scx(struct rq *rq, struct task_struct *p,if (p->scx.flags &SCX_TASK_QUEUED ) {set_task_runnable (rq, p); +/* Delegate retained donor admission to its owning BPF scheduler. */ +if (p->is_blocked) { +/* +* If the donor is the same and only the mutex owner +* changes, avoid triggering another ops.enqueue(): the +* BPF scheduler has already admitted the donor, so it +* can continue running. +*/ +if (next == p) + goto switch_class; + +if (WARN_ON_ONCE (!sch)) + goto switch_class; +WARN_ON_ONCE (!(sch->ops.flags &SCX_OPS_ENQ_BLOCKED )); +scx_do_enqueue_task (rq, p,0 ,-1 ); + goto switch_class; + } +/* * If @p has slice left and is being put, @p is getting * preempted by a higher priority scheduler class or core-sched
@@ -7802,6 +7863,11 @@ int scx_validate_ops(struct scx_sched *sch, const struct sched_ext_ops *ops)return -EINVAL ; } +if ((ops->flags &SCX_OPS_ENQ_BLOCKED ) && !ops->enqueue) { +scx_error (sch,"SCX_OPS_ENQ_BLOCKED requires ops.enqueue() to be implemented" ); +return -EINVAL ; + } +/* * SCX_OPS_TID_TO_TASK is enabled by the root scheduler. A sub-sched * may set it to declare a dependency; reject if the root hasn't
kernel/sched/ext/ext.h +9 -1
@@ -21,12 +21,20 @@ int scx_check_setscheduler(struct task_struct *p, int policy);void scx_prepare_setscheduler (struct task_struct *p,int policy);bool task_should_scx (int policy);bool scx_allow_ttwu_queue (const struct task_struct *p); -bool scx_allow_proxy_exec (const struct task_struct *p); +bool __scx_allow_proxy_exec (const struct task_struct *p);void scx_proxy_donor_start (struct rq *rq);void scx_proxy_reenqueue_retry (struct rq *rq,struct task_struct *next);void init_sched_ext_class (void );void __scx_update_idle (struct rq *rq,bool idle,bool do_notify); +static inline bool scx_allow_proxy_exec (const struct task_struct *p) +{ +if (scx_enabled ()) +return __scx_allow_proxy_exec (p); + +return true; +} +static inline void scx_update_idle (struct rq *rq,bool idle,bool do_notify) {if (scx_enabled ())
kernel/sched/ext/internal.h +22 -1
@@ -230,6 +230,19 @@ enum scx_ops_flags {*/ SCX_OPS_LAZY_RESCHED =1LLU <<9 , +/* +* If set, mutex-blocked tasks remain runnable as proxy donors and are +* passed to ops.enqueue() with %SCX_ENQ_BLOCKED. The BPF scheduler controls +* when donors are dispatched and whether they should preempt other work. +* +* If clear, mutex-blocked tasks are removed from the runqueue normally +* and cannot donate their scheduling context through proxy execution. +* +* For blocked donors, this flag takes precedence over +* %SCX_OPS_ENQ_EXITING and %SCX_OPS_ENQ_MIGRATION_DISABLED. +*/ +SCX_OPS_ENQ_BLOCKED =1LLU <<10 , +SCX_OPS_ALL_FLAGS =SCX_OPS_KEEP_BUILTIN_IDLE |SCX_OPS_ENQ_LAST | SCX_OPS_ENQ_EXITING |
@@ -239,7 +252,8 @@ enum scx_ops_flags {SCX_OPS_BUILTIN_IDLE_PER_NODE |SCX_OPS_ALWAYS_ENQ_IMMED |SCX_OPS_TID_TO_TASK | -SCX_OPS_LAZY_RESCHED , +SCX_OPS_LAZY_RESCHED | +SCX_OPS_ENQ_BLOCKED ,/* high 8 bits are internal, don't include in SCX_OPS_ALL_FLAGS */ __SCX_OPS_INTERNAL_MASK = 0xffLLU << 56,
@@ -1765,6 +1779,12 @@ enum scx_enq_flags {*/ SCX_ENQ_LAST =1LLU <<41 , +/* +* The task is blocked on a mutex and is being kept runnable as a proxy +* donor. Only passed to ops.enqueue() when %SCX_OPS_ENQ_BLOCKED is set. +*/ +SCX_ENQ_BLOCKED =1LLU <<42 , +/* high 8 bits are internal */ __SCX_ENQ_INTERNAL_MASK = 0xffLLU << 56,
@@ -2105,6 +2125,7 @@ struct task_struct *scx_task_iter_next_locked(struct scx_task_iter *iter);bool scx_set_task_slice (struct task_struct *p,u64 slice);void scx_task_slice_ended (struct rq *rq,struct task_struct *p);void scx_task_unlink_from_dsq (struct task_struct *p,struct scx_dispatch_q *dsq); +void scx_prepare_task_sched_change (struct task_struct *p);void scx_dispatch_dequeue (struct rq *rq,struct task_struct *p);void scx_do_enqueue_task (struct rq *rq,struct task_struct *p,u64 enq_flags,int sticky_cpu);
kernel/sched/ext/sub.c +9 -3
@@ -1212,7 +1212,9 @@ static void scx_rehome_task(struct scx_sched *to, struct task_struct *p)lockdep_assert_held (&p->pi_lock);lockdep_assert_rq_held (task_rq (p)); - scoped_guard (sched_change, p, DEQUEUE_SAVE | DEQUEUE_MOVE) { +scx_prepare_task_sched_change (p); +scoped_guard (sched_change, p, + DEQUEUE_SAVE | DEQUEUE_MOVE | DEQUEUE_NOCLOCK) {scx_disable_and_exit_task (scx_task_sched (p), p);scx_set_task_state (p,SCX_TASK_INIT_BEGIN );scx_set_task_state (p,SCX_TASK_INIT );
@@ -1242,7 +1244,9 @@ static void scx_punt_task(struct scx_sched *to, struct task_struct *p)lockdep_assert_rq_held (task_rq (p));WARN_ON_ONCE (!READ_ONCE (to->bypass_depth)); - scoped_guard (sched_change, p, DEQUEUE_SAVE | DEQUEUE_MOVE) { +scx_prepare_task_sched_change (p); +scoped_guard (sched_change, p, + DEQUEUE_SAVE | DEQUEUE_MOVE | DEQUEUE_NOCLOCK) {scx_disable_and_exit_task (scx_task_sched (p), p);scx_set_task_sched (p, to); }
@@ -1898,7 +1902,9 @@ void scx_sub_enable_workfn(struct kthread_work *work)if (!(p->scx.flags &SCX_TASK_SUB_INIT ))continue ; - scoped_guard (sched_change, p, DEQUEUE_SAVE | DEQUEUE_MOVE) { +scx_prepare_task_sched_change (p); +scoped_guard (sched_change, p, + DEQUEUE_SAVE | DEQUEUE_MOVE | DEQUEUE_NOCLOCK) {/* * $p must be either READY or ENABLED. If ENABLED, * __scx_disabled_and_exit_task() first disables and
tools/sched_ext/include/scx/compat.h +1 -0
@@ -214,6 +214,7 @@ static inline bool __COMPAT_struct_has_field(const char *type, const char *field#define SCX_OPS_ALLOW_QUEUED_WAKEUP SCX_OPS_FLAG(SCX_OPS_ALLOW_QUEUED_WAKEUP)#define SCX_OPS_BUILTIN_IDLE_PER_NODE SCX_OPS_FLAG(SCX_OPS_BUILTIN_IDLE_PER_NODE)#define SCX_OPS_ALWAYS_ENQ_IMMED SCX_OPS_FLAG(SCX_OPS_ALWAYS_ENQ_IMMED) +#define SCX_OPS_ENQ_BLOCKED SCX_OPS_FLAG(SCX_OPS_ENQ_BLOCKED)#define SCX_PICK_IDLE_FLAG (name) __COMPAT_ENUM_OR_ZERO("scx_pick_idle_cpu_flags", #name)
tools/sched_ext/include/scx/enum_defs.autogen.h +1 -0
@@ -90,6 +90,7 @@#define HAVE_SCX_ENQ_RESCUE #define HAVE_SCX_ENQ_REENQ #define HAVE_SCX_ENQ_LAST +#define HAVE_SCX_ENQ_BLOCKED #define HAVE___SCX_ENQ_INTERNAL_MASK #define HAVE_SCX_ENQ_CLEAR_OPSS #define HAVE_SCX_ENQ_DSQ_PRIQ
tools/sched_ext/include/scx/enums.autogen.bpf.h +3 -0
@@ -142,6 +142,9 @@ const volatile u64 __SCX_ENQ_REENQ __weak;const volatile u64 __SCX_ENQ_LAST__weak ;#define SCX_ENQ_LAST __SCX_ENQ_LAST +const volatile u64 __SCX_ENQ_BLOCKED__weak ; +#define SCX_ENQ_BLOCKED __SCX_ENQ_BLOCKED +const volatile u64 __SCX_ENQ_CLEAR_OPSS__weak ;#define SCX_ENQ_CLEAR_OPSS __SCX_ENQ_CLEAR_OPSS
tools/sched_ext/include/scx/enums.autogen.h +1 -0
@@ -51,6 +51,7 @@SCX_ENUM_SET (skel, scx_enq_flags,SCX_ENQ_RESCUE ); \SCX_ENUM_SET (skel, scx_enq_flags,SCX_ENQ_REENQ ); \SCX_ENUM_SET (skel, scx_enq_flags,SCX_ENQ_LAST ); \ +SCX_ENUM_SET (skel, scx_enq_flags,SCX_ENQ_BLOCKED ); \SCX_ENUM_SET (skel, scx_enq_flags,SCX_ENQ_CLEAR_OPSS ); \SCX_ENUM_SET (skel, scx_enq_flags,SCX_ENQ_DSQ_PRIQ ); \SCX_ENUM_SET (skel, scx_deq_flags,SCX_DEQ_SCHED_CHANGE ); \
tools/sched_ext/include/scx/enums_abi.autogen.h +1 -1
@@ -180,7 +180,7 @@ static const struct __scx_enum_abi_val __scx_enum_abi_vals[] { "scx_ops_flags", "SCX_OPS_ALWAYS_ENQ_IMMED", 0x80LLU }, { "scx_ops_flags", "SCX_OPS_TID_TO_TASK", 0x100LLU }, { "scx_ops_flags", "SCX_OPS_LAZY_RESCHED", 0x200LLU }, - { "scx_ops_flags", "SCX_OPS_ALL_FLAGS", 0x3ffLLU }, + { "scx_ops_flags", "SCX_OPS_ALL_FLAGS", 0x7ffLLU }, { "scx_ops_flags", "__SCX_OPS_INTERNAL_MASK", 0xff00000000000000LLU }, { "scx_ops_flags", "SCX_OPS_HAS_CPU_PREEMPT", 0x100000000000000LLU }, { "scx_ops_state", "SCX_OPSS_NONE", 0x0LLU },
Signature · 1 line
2.55.0
Andrea Righi
reviewerPATCH 14/16 sched_ext: Add selftest for blocked donor admission
In reply to Andrea RighiSCX_OPS_ENQ_BLOCKED allows BPF schedulers to receive blocked proxy donors through ops.enqueue(). SCX_ENQ_BLOCKED identifies blocked-donor admission requests. Add selftest coverage for this interface.
Exercise a priority inversion using a weighted-vruntime BPF scheduler. A nice +19 owner holds a shared mutex, a nice -20 donor blocks on it, and nice 0 CPU contenders, one per allowed CPU, keep the system busy. Test both a same-CPU topology and a cross-CPU topology with the donor and owner on different CPUs.
Treat blocked donors according to the normal BPF ordering policy and assign the default slice on every enqueue, as for other tasks. Run each CPU placement configuration with SCX_OPS_ENQ_BLOCKED first disabled and then enabled, count blocked-donor enqueues by CPU and report average mutex hold and wait times. Verify that full wakeups are never reported as blocked-donor admissions by checking that SCX_ENQ_WAKEUP and SCX_ENQ_BLOCKED are not set together.
Proxy execution coverage requires CONFIG_SCHED_PROXY_EXEC=y, which the selftest config selects. Access to the kernel mutex is provided via a loadable kernel module, built through TEST_GEN_MODS_DIR and managed by the test.
--- tools/testing/selftests/sched_ext/.gitignore | 4 + tools/testing/selftests/sched_ext/Makefile | 2 + tools/testing/selftests/sched_ext/config | 2 + .../selftests/sched_ext/enq_blocked.bpf.c | 116 +++ .../testing/selftests/sched_ext/enq_blocked.c | 917 ++++++++++++++++++ .../testing/selftests/sched_ext/enq_blocked.h | 28 + .../selftests/sched_ext/test_modules/Makefile | 13 + .../test_modules/scx_enq_blocked_test.c | 195 ++++ 8 files changed, 1277 insertions(+) create mode 100644 tools/testing/selftests/sched_ext/enq_blocked.bpf.c create mode 100644 tools/testing/selftests/sched_ext/enq_blocked.c create mode 100644 tools/testing/selftests/sched_ext/enq_blocked.h create mode 100644 tools/testing/selftests/sched_ext/test_modules/Makefile create mode 100644 tools/testing/selftests/sched_ext/test_modules/scx_enq_blocked_test.c
tools/testing/selftests/sched_ext/.gitignore +4 -0
@@ -4,3 +4,7 @@ !Makefile !.gitignore !config +!test_modules/ +!test_modules/scx_enq_blocked_test.c +!test_modules/Makefile +test_modules/*.mod.c
tools/testing/selftests/sched_ext/Makefile +2 -0
@@ -5,6 +5,7 @@ include ../../../scripts/Makefile.arch include ../../../scripts/Makefile.include TEST_GEN_PROGS := runner +TEST_GEN_MODS_DIR := test_modules # override lib.mk's default rules OVERRIDE_TARGETS := 1
@@ -164,6 +165,7 @@ all_test_bpfprogs := $(foreach prog,$(wildcard *.bpf.c),$(INCLUDE_DIR)/$(patsubs auto-test-targets := \ create_dsq \ dequeue \ + enq_blocked \ enq_last_no_enq_fails \ ddsp_bogus_dsq_fail \ ddsp_vtimelocal_fail \
tools/testing/selftests/sched_ext/config +2 -0
@@ -6,3 +6,5 @@ CONFIG_BPF=y CONFIG_BPF_SYSCALL=y CONFIG_DEBUG_INFO=y CONFIG_DEBUG_INFO_BTF=y +CONFIG_EXPERT=y +CONFIG_SCHED_PROXY_EXEC=y
tools/testing/selftests/sched_ext/enq_blocked.bpf.c new file +116 -0
116 lines not on this page · raw message
tools/testing/selftests/sched_ext/enq_blocked.c new file +917 -0
917 lines not on this page · raw message
tools/testing/selftests/sched_ext/enq_blocked.h new file +28 -0
@@ -0,0 +1,28 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES */ +#ifndef __ENQ_BLOCKED_H +#define __ENQ_BLOCKED_H + +#include <linux/ioctl.h> +#include <linux/types.h> + +struct enq_blocked_stats { +__u64 hold_time_ns; +__u64 wait_time_ns; +__u64 nr_holds; +__u64 nr_waits; +}; + +#define ENQ_BLOCKED_IOCTL_OWNER _IO('s', 1) +#define ENQ_BLOCKED_IOCTL_DONOR _IO('s', 2) +#define ENQ_BLOCKED_IOCTL_RESET_STATS _IO('s', 3) +#define ENQ_BLOCKED_IOCTL_GET_STATS _IOR('s', 4, struct enq_blocked_stats) +#define ENQ_BLOCKED_IOCTL_PREP_ATTACH _IO('s', 5) +#define ENQ_BLOCKED_IOCTL_ATTACH_DONE _IO('s', 6) +#define ENQ_BLOCKED_IOCTL_DONOR_STATE _IO('s', 7) +#define ENQ_BLOCKED_IOCTL_PROXY_SUPPORTED _IO('s', 8) + +#define ENQ_BLOCKED_DONOR_SLEEPING (1U << 0) +#define ENQ_BLOCKED_DONOR_ON_RQ (1U << 1) + +#endif /* __ENQ_BLOCKED_H */
tools/testing/selftests/sched_ext/test_modules/Makefile new file +13 -0
@@ -0,0 +1,13 @@ +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES + +TESTMODS_DIR := $(realpath $(dir $(abspath $(lastword $(MAKEFILE_LIST))))) +KDIR ?= $(if $(O),$(O),$(realpath ../../../../..)) + +obj-m += scx_enq_blocked_test.o + +all: + +$(Q)$(MAKE) -C $(KDIR) M=$(TESTMODS_DIR) modules + +clean: + +$(Q)$(MAKE) -C $(KDIR) M=$(TESTMODS_DIR) clean
tools/testing/selftests/sched_ext/test_modules/scx_enq_blocked_test.c new file +195 -0
195 lines not on this page · raw message
Signature · 1 line
2.55.0
Andrea Righi
reviewerPATCH 15/16 sched_ext: scx_qmap: Add proxy execution support
In reply to Andrea RighiAdd a -X option to opt scx_qmap into queueing mutex-blocked tasks for proxy execution. Without the option, SCX_OPS_ENQ_BLOCKED remains clear and mutex waiters block normally. With -X, blocked donors are passed to qmap_enqueue() with SCX_ENQ_BLOCKED.
When scx_qmap receives a blocked donor, select a cid allowed by the task and held by qmap, preferring its current cid. Dispatch the donor to that cid's local DSQ with a fresh slice and SCX_ENQ_PREEMPT, adding SCX_ENQ_IMMED when the cid is time-shared. This places the donor at the head of an eligible DSQ and requests an immediate reschedule, allowing the core proxy-exec path to run the mutex owner using the donor's scheduling context as soon as the donor is selected.
Partition ownership can change concurrently with enqueue, so the self-cid intersection can disappear between the initial check and cid selection. Validate the selected cid before indexing per-cid state or constructing a local DSQ id, and fall back to rescue placement if the intersection has disappeared.
The blocked policy is intentionally unfair and can strongly prioritize tasks using contended mutexes, but scx_qmap is a demo scheduler and such aggressive behavior makes proxy-exec support easy to observe. Count all blocked dispatch attempts in nr_enq_blocked and report their per-interval delta.
--- tools/sched_ext/scx_qmap.bpf.c | 79 +++++++++++++++++++++++++++++++--- tools/sched_ext/scx_qmap.c | 13 ++++-- tools/sched_ext/scx_qmap.h | 1 + 3 files changed, 83 insertions(+), 10 deletions(-)
tools/sched_ext/scx_qmap.bpf.c +72 -7
@@ -366,6 +366,22 @@ static u64 needs_immed(s32 cid)return qa.cid_shared[cid] ?SCX_ENQ_IMMED :0 ; } +static void dispatch_to_rescue (struct task_struct *p,task_ctx_t *taskc, +u64 enq_flags) +{ +u32 cid =cmask_next_set_wrap (&taskc->cpus_allowed,0 ); + +if (cid >=scx_bpf_nr_cids ()) { +scx_bpf_error ("task %d has no allowed cid" , p->pid); +return ; + } + + taskc->force_local = false; +__sync_fetch_and_add (&qa.nr_rescue_dsp,1 ); +scx_bpf_dsq_insert (p,SCX_DSQ_LOCAL_ON | cid, slice_ns, + enq_flags |needs_immed (cid) |SCX_ENQ_RESCUE ); +} +/* first cid this node does NOT hold for fault injection, -1 if none */ static s32 first_unavail_cid (void ) {
@@ -439,6 +455,9 @@ void BPF_STRUCT_OPS(qmap_enqueue, struct task_struct *p, u64 enq_flags)*/ taskc->core_sched_seq = qa.core_sched_tail_seqs[idx]++; +if (enq_flags &SCX_ENQ_BLOCKED ) +__sync_fetch_and_add (&qa.nr_enq_blocked,1 ); +/* * A task of ours that can run on none of our self cids - the parent * didn't grant them or we delegated them to children - would starve in
@@ -446,18 +465,64 @@ void BPF_STRUCT_OPS(qmap_enqueue, struct task_struct *p, u64 enq_flags)* * Force it onto its first allowed cid's local DSQ. If we hold that cid * it runs. Otherwise the insert carries SCX_ENQ_RESCUE and the kernel -* diverts the task to its rescue path. +* diverts the task to its rescue path. Do this before the blocked-donor +* fast paths, which also require an eligible self cid to make progress. */ if (!cmask_intersects (&taskc->cpus_allowed, &qa.self_cids.mask)) { -s32 c =cmask_next_set_wrap (&taskc->cpus_allowed,0 ); +dispatch_to_rescue (p, taskc, enq_flags); +return ; + } -if (c >=0 && c <scx_bpf_nr_cids ()) { - taskc->force_local = false; -__sync_fetch_and_add (&qa.nr_rescue_dsp,1 ); -scx_bpf_dsq_insert (p,SCX_DSQ_LOCAL_ON | c, slice_ns, - enq_flags |needs_immed (c) |SCX_ENQ_RESCUE ); +/* +* SCX_OPS_ALWAYS_ENQ_IMMED makes the local insertion below implicitly +* carry SCX_ENQ_IMMED. If the CPU can't run the blocked donor immediately, +* the core returns it through ops.enqueue() with SCX_ENQ_REENQ. Inserting +* it into the same local DSQ would repeat the IMMED handback until the +* scheduler is ejected. Move reenqueued blocked donors to the shared DSQ, +* which doesn't carry SCX_ENQ_IMMED, so another CPU can consume them. +*/ +if ((enq_flags & (SCX_ENQ_BLOCKED |SCX_ENQ_REENQ )) == + (SCX_ENQ_BLOCKED |SCX_ENQ_REENQ )) { + taskc->force_local = false; +scx_bpf_dsq_insert (p,SHARED_DSQ ,0 , enq_flags); + cid =cmask_next_and2_set_wrap (&taskc->cpus_allowed, + &qa.idle_cids.mask, + &qa.self_cids.mask,0 ); +if (cid <scx_bpf_nr_cids ()) +scx_bpf_kick_cid (cid,SCX_KICK_IDLE ); +return ; + } + +/* +* Insert a blocked mutex donor at the head of an eligible local DSQ with +* a fresh slice and %SCX_ENQ_PREEMPT, requesting an immediate reschedule. +* The test above guarantees that cpus_allowed intersects self_cids, but +* the donor's current cid may have been delegated to a child. Search the +* intersection starting at the current cid, preserving it when qmap still +* holds it and wrapping to another eligible self cid otherwise. +* +* A self cid may be held exclusively with SCX_CAP_ENQ or time-shared with +* only SCX_CAP_ENQ_IMMED. Add needs_immed() so either kind can accept the +* local insertion instead of rejecting and reenqueuing the donor for a +* capability miss. Once selected, the core proxy-exec path can run the +* mutex owner using the donor's scheduling context. +* +* This policy is intentionally unfair and can strongly prioritize tasks +* using contended mutexes; scx_qmap is a demonstration scheduler and +* this behavior makes proxy-exec support easy to observe. +*/ +if (enq_flags &SCX_ENQ_BLOCKED ) { + cid =cmask_next_and_set_wrap (&taskc->cpus_allowed, + &qa.self_cids.mask, +scx_bpf_task_cid (p)); +if (cid >=scx_bpf_nr_cids ()) { +/* self_cids may have changed since the intersection test */ +dispatch_to_rescue (p, taskc, enq_flags);return ; } +scx_bpf_dsq_insert (p,SCX_DSQ_LOCAL_ON | cid, slice_ns, + enq_flags |needs_immed (cid) |SCX_ENQ_PREEMPT ); +return ; }/*
tools/sched_ext/scx_qmap.c +10 -3
@@ -46,7 +46,7 @@ const char help_fmt[] ="See the top-of-file comment in .bpf.c for the design.\n" "\n" "Usage: %s [-s SLICE_US] [-e COUNT] [-t COUNT] [-T COUNT] [-l COUNT] [-b COUNT]\n" -" [-N COUNT] [-P] [-M] [-H] [-c CG_PATH] [-d PID] [-D LEN] [-S] [-p] [-I]\n" +" [-N COUNT] [-P] [-M] [-H] [-c CG_PATH] [-d PID] [-D LEN] [-S] [-p] [-I] [-X]\n" " [-F COUNT] [-i SEC] [-R MS] [-J MODE] [-v]\n" "\n" " -s SLICE_US Override slice duration\n"
@@ -65,6 +65,7 @@ const char help_fmt[] =" -S Suppress qmap-specific debug dump\n" " -p Switch only tasks on SCHED_EXT policy instead of all\n" " -I Turn on SCX_OPS_ALWAYS_ENQ_IMMED\n" +" -X Turn on SCX_OPS_ENQ_BLOCKED\n" " -F COUNT IMMED stress: force every COUNT'th enqueue to a busy local DSQ (use with -I)\n" " -C MODE cid-override test (shuffle|bad-dup|bad-range|bad-mono)\n" " -i SEC Stats interval, seconds (default 5)\n"
@@ -107,6 +108,7 @@ struct hier_prev {u64 nr_dsps[MAX_SUB_SCHEDS ];u64 nr_reenq_cap;u64 nr_reenq_immed; +u64 nr_enq_blocked;u64 nr_inject_attempts;u64 nr_rescue_dsp; };
@@ -190,14 +192,16 @@ static void print_hier(struct qmap_arena *qa, struct hier_prev *prev, u64 own_cg }format_cid_ranges (qa,CID_SHARED , ranges,sizeof (ranges)); -printf ("hier : nsub=%llu excl=%u shared=%s rr=%s reenq cap/immed +%llu/+%llu inj=+%llu rescue=+%llu\n" , +printf ("hier : nsub=%llu excl=%u shared=%s rr=%s reenq cap/immed +%llu/+%llu blocked=+%llu inj=+%llu rescue=+%llu\n" , (unsigned long long )qa->nr_sub_scheds, qa->part.nr_excl, ranges, rr, (unsigned long long )(qa->nr_reenq_cap - prev->nr_reenq_cap), (unsigned long long )(qa->nr_reenq_immed - prev->nr_reenq_immed), + (unsigned long long )(qa->nr_enq_blocked - prev->nr_enq_blocked), (unsigned long long )(qa->nr_inject_attempts - prev->nr_inject_attempts), (unsigned long long )(qa->nr_rescue_dsp - prev->nr_rescue_dsp)); prev->nr_reenq_cap = qa->nr_reenq_cap; prev->nr_reenq_immed = qa->nr_reenq_immed; + prev->nr_enq_blocked = qa->nr_enq_blocked; prev->nr_inject_attempts = qa->nr_inject_attempts; prev->nr_rescue_dsp = qa->nr_rescue_dsp;
@@ -263,7 +267,7 @@ int main(int argc, char **argv) skel->rodata->max_tasks =16384 ;while ((opt =getopt (argc, argv, -"s:e:t:T:l:b:N:PMHc:d:D:SpIF:C:i:R:J:B:q:vh" )) !=-1 ) { +"s:e:t:T:l:b:N:PMHc:d:D:SpIXF:C:i:R:J:B:q:vh" )) !=-1 ) {switch (opt) {case 's' : skel->rodata->slice_ns =strtoull (optarg,NULL ,0 ) *1000 ;
@@ -324,6 +328,9 @@ int main(int argc, char **argv)case 'I' : skel->struct_ops.qmap_ops->flags |=SCX_OPS_ALWAYS_ENQ_IMMED ;break ; +case 'X' : + skel->struct_ops.qmap_ops->flags |=SCX_OPS_ENQ_BLOCKED ; +break ;case 'F' : skel->rodata->immed_stress_nth =strtoul (optarg,NULL ,0 );break ;
tools/sched_ext/scx_qmap.h +1 -0
@@ -178,6 +178,7 @@ struct qmap_arena {/* bpf -> userspace: stats */ u64 nr_reenq_cap;/* SCX_TASK_REENQ_CAP bounces */ u64 nr_reenq_immed;/* SCX_TASK_REENQ_IMMED bounces */ +u64 nr_enq_blocked;/* SCX_ENQ_BLOCKED dispatches */ u64 nr_inject_attempts;/* fault-injection: dispatches to an unheld cid */ u64 nr_rescue_dsp;/* SCX_ENQ_RESCUE dispatch attempts */ u32 inject_mode;/* fault-injection mode (QMAP_INJ_*) */
Signature · 1 line
2.55.0
Andrea Righi
reviewerPATCH 16/16 sched: Allow enabling proxy exec with sched_ext
In reply to Andrea RighiNow that sched_ext can handle proxy donors and BPF schedulers can opt in to blocked-donor enqueueing with SCX_OPS_ENQ_BLOCKED, remove the !SCHED_CLASS_EXT dependency from SCHED_PROXY_EXEC and allow both options to be enabled together.
--- init/Kconfig | 2 -- 1 file changed, 2 deletions(-)
init/Kconfig +0 -2
@@ -937,8 +937,6 @@ config SCHED_PROXY_EXEC bool "Proxy Execution" # Avoid some build failures w/ PREEMPT_RT until it can be fixed depends on !PREEMPT_RT - # Need to investigate how to inform sched_ext of split contexts - depends on !SCHED_CLASS_EXT # Not particularly useful until we get to multi-rq proxying depends on EXPERT help
Signature · 1 line
2.55.0
Peter Zijlstra
maintainerQuoted 6 lines · Andrea Righi
Andrea Righi (16):
sched/core: Drop mutex locks before proxy rescheduling sched/core: Dequeue waking proxy donors before reset sched/core: Mark wakeups completed throughttwu_runnable()
sched: Add helper to block retained proxy donors
sched: Addsched_exthooks for proxy execution
Right, so these add:
WF_TTWU_RQ:
Used like ENQUEUE_DELAYED; could be fixed by generalizing that to cover all of p->is_blocked.
scx_allow_proxy_exec():
Hook to kill proxy exec for scx
scx_proxy_reenqueue_retry():
Like put_prev_task(), but for current. Ensures current gets put back on a DSQ once its done running.
scx_proxy_donor_start():
Delayed set_next_task(), confirms donor will be used.
sched_proxy_block_task():
Almost like switching_to_scx(), except it needs to change ctx->queued in case of p->is_blocked. Hence a new callback ran before sched_change_begin().
Now, I have:
https://patch.msgid.link/[email protected],
pending, would something like the below on top of both this work?
(although I'm not convinced SC_CONFIRM is actually making it better)
kernel/sched/core.c +9 -8
@@ -1876,7 +1876,7 @@ static inline void uclamp_rq_inc(structif (!uclamp_is_used ())return ; -if (unlikely (!p->sched_class->uclamp_enabled)) +if (unlikely (!(p->sched_class->flags &SC_UCLAMP )))return ;/* Only inc the delayed task which being woken up. */
@@ -1904,7 +1904,7 @@ static inline void uclamp_rq_dec(structif (!uclamp_is_used ())return ; -if (unlikely (!p->sched_class->uclamp_enabled)) +if (unlikely (!(p->sched_class->flags &SC_UCLAMP )))return ;if (p->se.sched_delayed)
@@ -7268,9 +7268,10 @@ static void __sched notrace __schedule(i rq->next_class = next->sched_class;if (sched_proxy_exec ()) {struct task_struct *prev_donor = rq->donor; +struct task_struct *donor = next; -rq_set_donor (rq, next); - next->blocked_donor =NULL ; +rq_set_donor (rq, donor); + donor->blocked_donor =NULL ;if (unlikely (next->is_blocked)) { next =find_proxy_task (rq, next, &rf);if (!next) {
@@ -7283,8 +7284,7 @@ static void __sched notrace __schedule(i goto keep_resched; } } -if (rq->donor == prev_donor && prev != next) { -struct task_struct *donor = rq->donor; +if (donor == prev_donor && prev != next) {/* * When transitioning like: *
@@ -7298,9 +7298,10 @@ static void __sched notrace __schedule(i* on_cpu. */ donor->sched_class->put_prev_task (rq, donor, donor); - donor->sched_class->set_next_task (rq, donor, true); + donor->sched_class->set_next_task (rq, donor,SNT_PICK ); } -scx_proxy_donor_start (rq); +if (donor->sched_class->flags &SC_CONFIRM ) + donor->sched_class->set_next_task (rq, donor,SNT_CONFIRM );scx_proxy_reenqueue_retry (rq, next); }else {rq_set_donor (rq, next);
kernel/sched/ext/ext.c +5 -1
@@ -5069,8 +5069,12 @@ DEFINE_SCHED_CLASS(ext) = { .update_curr = update_curr_scx, + .flags = 0 #ifdef CONFIG_UCLAMP_TASK - .uclamp_enabled = 1, + | SC_UCLAMP +#endif +#ifdef CONFIG_SCHED_PROXY_EXEC + | SC_CONFIRM#endif };
kernel/sched/fair.c +1 -1
@@ -15887,7 +15887,7 @@ DEFINE_SCHED_CLASS(fair) = {#endif #ifdef CONFIG_UCLAMP_TASK - .uclamp_enabled = 1, + .flags = SC_UCLAMP,#endif };
kernel/sched/rt.c +1 -1
@@ -2636,7 +2636,7 @@ DEFINE_SCHED_CLASS(rt) = {#endif #ifdef CONFIG_UCLAMP_TASK - .uclamp_enabled = 1, + .flags = SC_UCLAMP,#endif };
kernel/sched/sched.h +7 -3
@@ -2641,13 +2641,17 @@ enum snt_e { SNT_NORMAL, /* set_next_task() */SNT_PICK ,/* put_prev_set_next_task(): prev != next */ SNT_REPICK ,/* put_prev_set_next_task(): prev == next */ + SNT_CONFIRM, /* confirm pick */ +}; + +enum { +SC_UCLAMP =1 <<0 , +SC_CONFIRM =1 <<1 , };struct sched_class { -#ifdef CONFIG_UCLAMP_TASK -int uclamp_enabled; -#endif +unsigned int flags;/* * move_queued_task/activate_task/enqueue_task: rq->lock
Peter Zijlstra
maintainerQuoted 25 lines · Peter Zijlstra
Quoted 6 lines · Andrea Righi
Andrea Righi (16):
sched/core: Drop mutex locks before proxy rescheduling sched/core: Dequeue waking proxy donors before reset sched/core: Mark wakeups completed throughttwu_runnable()
sched: Add helper to block retained proxy donors
sched: Addsched_exthooks for proxy executionRight, so these add:
WF_TTWU_RQ:Used like
ENQUEUE_DELAYED; could be fixed by generalizing that to cover all ofp->is_blocked.
scx_allow_proxy_exec():Hook to kill proxy exec for scx
scx_proxy_reenqueue_retry():Like
put_prev_task(), but for current. Ensures current gets put back on a DSQ once its done running.
Quoted 8 lines
scx_proxy_donor_start():Delayed
set_next_task(), confirms donor will be used.
sched_proxy_block_task():Almost like
switching_to_scx(), except it needs to changectx->queuedin case ofp->is_blocked. Hence a new callback ran beforesched_change_begin().
And I wonder, would something like so work?
---
kernel/sched/core.c +7 -2
@@ -11215,8 +11215,13 @@ void sched_change_end(struct sched_change_ctx *ctx)*/ WARN_ON_ONCE (p->sched_class != ctx->class && !(ctx->flags &ENQUEUE_CLASS )); -if ((ctx->flags &ENQUEUE_CLASS ) && p->sched_class->switching_to) - p->sched_class->switching_to (rq, p); +if ((ctx->flags &ENQUEUE_CLASS ) && p->sched_class->switching_to) { +bool block = p->sched_class->switching_to (rq, p); +if (block && p->is_blocked) { +block_task (rq, p,READ_ONCE (p->__state)); + ctx->queued = false; + } + }if (ctx->queued)enqueue_task (rq, p, ctx->flags);
kernel/sched/ext/ext.c +4 -2
@@ -4288,12 +4288,12 @@ static void prio_changed_scx(struct rq *rq, struct task_struct *p, u64 oldprio) { } -static void switching_to_scx (struct rq *rq,struct task_struct *p) +static bool switching_to_scx (struct rq *rq,struct task_struct *p) {struct scx_sched *sch =scx_task_sched (p);if (task_dead_and_done (p)) -return ; +return false;scx_enable_task (sch, p);
@@ -4303,6 +4303,8 @@ static void switching_to_scx(struct rq *rq, struct task_struct *p)*/ if (SCX_HAS_OP (sch, set_cpumask))scx_call_op_set_cpumask (sch, rq, p, (struct cpumask *)p->cpus_ptr); + +return !scx_allow_proxy_exec (p); }static void switched_from_scx (struct rq *rq,struct task_struct *p)
kernel/sched/idle.c +1 -1
@@ -543,7 +543,7 @@ static void task_tick_idle(struct rq *rq, struct task_struct *curr, int queued)update_curr_idle (rq); } -static void switching_to_idle (struct rq *rq,struct task_struct *p) +static bool switching_to_idle (struct rq *rq,struct task_struct *p) {BUG (); }
kernel/sched/sched.h +1 -1
@@ -2759,7 +2759,7 @@ struct sched_class {*/ void (*switching_from)(struct rq *this_rq,struct task_struct *task);void (*switched_from) (struct rq *this_rq,struct task_struct *task); -void (*switching_to) (struct rq *this_rq,struct task_struct *task); +bool (*switching_to) (struct rq *this_rq,struct task_struct *task);void (*switched_to) (struct rq *this_rq,struct task_struct *task);u64 (*get_prio) (struct rq *this_rq,struct task_struct *task); void (*prio_changed) (struct rq *this_rq, struct task_struct *task,
kernel/sched/stop_task.c +1 -1
@@ -78,7 +78,7 @@ static void task_tick_stop(struct rq *rq, struct task_struct *curr, int queued) { } -static void switching_to_stop (struct rq *rq,struct task_struct *p) +static bool switching_to_stop (struct rq *rq,struct task_struct *p) {BUG ();/* its impossible to change to this class */ }
K Prateek Nayak
reviewerQuoted 11 lines · Peter Zijlstra
Quoted 2 lines
Almost like
switching_to_scx(), except it needs to changectx->queuedin case ofp->is_blocked. Hence a new callback ran beforesched_change_begin().And I wonder, would something like so work?
---
kernel/sched/core.c +0
-0
kernel/sched/core.c void sched_change_end(struct sched_change_ctx *ctx) · line 11220
Quoted 8 lines
kernel/sched/core.c +3
-2@@ -11215,8 +11215,13 @@ void sched_change_end(struct sched_change_ctx *ctx)*/ WARN_ON_ONCE (p->sched_class != ctx->class && !(ctx->flags &ENQUEUE_CLASS )); -if ((ctx->flags &ENQUEUE_CLASS ) && p->sched_class->switching_to) - p->sched_class->switching_to (rq, p); +if ((ctx->flags &ENQUEUE_CLASS ) && p->sched_class->switching_to) { +bool block = p->sched_class->switching_to (rq, p); +if (block && p->is_blocked) {
Shouldn't this also check for "ctx->queued"?
kernel/sched/core.c void sched_change_end(struct sched_change_ctx *ctx) · line 11221
Quoted 1 line
kernel/sched/core.c +1
-0+block_task (rq, p,READ_ONCE (p->__state));
Otherwise we may block an already off rq task.
Quoted 6 lines
kernel/sched/core.c +3
-0+ ctx->queued = false; + } + }if (ctx->queued)enqueue_task (rq, p, ctx->flags);
Signature · 2 lines
Thanks and Regards, Prateek
Peter Zijlstra
maintainerQuoted 23 lines · K Prateek Nayak
Quoted 11 lines · Peter Zijlstra
Quoted 2 lines
Almost like
switching_to_scx(), except it needs to changectx->queuedin case ofp->is_blocked. Hence a new callback ran beforesched_change_begin().And I wonder, would something like so work?
---
kernel/sched/core.c +0
-0kernel/sched/core.c void sched_change_end(struct sched_change_ctx *ctx) · line 11220
Quoted 8 lines
kernel/sched/core.c +3
-2@@ -11215,8 +11215,13 @@ void sched_change_end(struct sched_change_ctx *ctx)*/ WARN_ON_ONCE (p->sched_class != ctx->class && !(ctx->flags &ENQUEUE_CLASS )); -if ((ctx->flags &ENQUEUE_CLASS ) && p->sched_class->switching_to) - p->sched_class->switching_to (rq, p); +if ((ctx->flags &ENQUEUE_CLASS ) && p->sched_class->switching_to) { +bool block = p->sched_class->switching_to (rq, p); +if (block && p->is_blocked) {Shouldn't this also check for "
ctx->queued"?
Quoted 7 lines
kernel/sched/core.c void sched_change_end(struct sched_change_ctx *ctx) · line 11221
Quoted 1 line
kernel/sched/core.c +1
-0+block_task (rq, p,READ_ONCE (p->__state));Otherwise we may block an already off rq task.
kernel/sched/core.c void sched_change_end(struct sched_change_ctx *ctx) · line 11224
Quoted 3 lines
kernel/sched/core.c +3
-0+ ctx->queued = false; + } + }
Yes.
Andrea Righi
reviewerHi Peter,
Quoted 25 lines · Peter Zijlstra
Quoted 6 lines · Andrea Righi
Andrea Righi (16):
sched/core: Drop mutex locks before proxy rescheduling sched/core: Dequeue waking proxy donors before reset sched/core: Mark wakeups completed throughttwu_runnable()
sched: Add helper to block retained proxy donors
sched: Addsched_exthooks for proxy executionRight, so these add:
WF_TTWU_RQ:Used like
ENQUEUE_DELAYED; could be fixed by generalizing that to cover all ofp->is_blocked.
scx_allow_proxy_exec():Hook to kill proxy exec for scx
scx_proxy_reenqueue_retry():Like
put_prev_task(), but for current. Ensures current gets put back on a DSQ once its done running.
Quoted 8 lines
scx_proxy_donor_start():Delayed
set_next_task(), confirms donor will be used.
sched_proxy_block_task():Almost like
switching_to_scx(), except it needs to changectx->queuedin case ofp->is_blocked. Hence a new callback ran beforesched_change_begin().
Yes, that matches the intent, with one small clarification:scx_proxy_reenqueue_retry() doesn't directly put current back on a DSQ, a task that couldn't be reenqueued remains on the reject DSQ and the hook schedules a deferred retry after proxy resolution.
Quoted 10 lines
Now, I have:
https://patch.msgid.link/[email protected],
pending, would something like the below on top of both this work?
(although I'm not convinced
SC_CONFIRMis actually making it better)
Yes, this works.
I applied Shubhang's v4 hrtick patch as a preliminary commit, then added the SNT_CONFIRM callback and reworked the proxy-exec series to use it. sched_ext now confirms the donor in set_next_task_scx() and scx_proxy_donor_start() is gone.
I tested the updated series and it passed all my scx proxy exec tests. The branch is available here:
git://git.kernel.org/pub/scm/linux/kernel/git/arighi/linux.git scx-proxy-exec-next
It's not an obvious simplification, but if we want to go this way, we can express both the provisional pick and donor confirmation through the sched class callback.
Thanks,
-Andrea
Quoted 118 lines
kernel/sched/core.c +9
-8@@ -1876,7 +1876,7 @@ static inline void uclamp_rq_inc(structif (!uclamp_is_used ())return ; -if (unlikely (!p->sched_class->uclamp_enabled)) +if (unlikely (!(p->sched_class->flags &SC_UCLAMP )))return ;/* Only inc the delayed task which being woken up. */ @@ -1904,7 +1904,7 @@ static inline void uclamp_rq_dec(structif (!uclamp_is_used ())return ; -if (unlikely (!p->sched_class->uclamp_enabled)) +if (unlikely (!(p->sched_class->flags &SC_UCLAMP )))return ;if (p->se.sched_delayed)@@ -7268,9 +7268,10 @@ static void __sched notrace __schedule(i rq->next_class = next->sched_class;if (sched_proxy_exec ()) {struct task_struct *prev_donor = rq->donor; +struct task_struct *donor = next; -rq_set_donor (rq, next); - next->blocked_donor =NULL ; +rq_set_donor (rq, donor); + donor->blocked_donor =NULL ;if (unlikely (next->is_blocked)) { next =find_proxy_task (rq, next, &rf);if (!next) {@@ -7283,8 +7284,7 @@ static void __sched notrace __schedule(i goto keep_resched; } } -if (rq->donor == prev_donor && prev != next) { -struct task_struct *donor = rq->donor; +if (donor == prev_donor && prev != next) {/* * When transitioning like: * @@ -7298,9 +7298,10 @@ static void __sched notrace __schedule(i* on_cpu. */ donor->sched_class->put_prev_task (rq, donor, donor); - donor->sched_class->set_next_task (rq, donor, true); + donor->sched_class->set_next_task (rq, donor,SNT_PICK ); } -scx_proxy_donor_start (rq); +if (donor->sched_class->flags &SC_CONFIRM ) + donor->sched_class->set_next_task (rq, donor,SNT_CONFIRM );scx_proxy_reenqueue_retry (rq, next); }else {rq_set_donor (rq, next);kernel/sched/ext/ext.c +5
-1@@ -5069,8 +5069,12 @@ DEFINE_SCHED_CLASS(ext) = { .update_curr = update_curr_scx, + .flags = 0 #ifdef CONFIG_UCLAMP_TASK - .uclamp_enabled = 1, + | SC_UCLAMP +#endif +#ifdef CONFIG_SCHED_PROXY_EXEC + | SC_CONFIRM#endif };kernel/sched/fair.c +1
-1@@ -15887,7 +15887,7 @@ DEFINE_SCHED_CLASS(fair) = {#endif #ifdef CONFIG_UCLAMP_TASK - .uclamp_enabled = 1, + .flags = SC_UCLAMP,#endif };kernel/sched/rt.c +1
-1@@ -2636,7 +2636,7 @@ DEFINE_SCHED_CLASS(rt) = {#endif #ifdef CONFIG_UCLAMP_TASK - .uclamp_enabled = 1, + .flags = SC_UCLAMP,#endif };kernel/sched/sched.h +7
-3@@ -2641,13 +2641,17 @@ enum snt_e { SNT_NORMAL, /* set_next_task() */SNT_PICK ,/* put_prev_set_next_task(): prev != next */ SNT_REPICK ,/* put_prev_set_next_task(): prev == next */ + SNT_CONFIRM, /* confirm pick */ +}; + +enum { +SC_UCLAMP =1 <<0 , +SC_CONFIRM =1 <<1 , };struct sched_class { -#ifdef CONFIG_UCLAMP_TASK -int uclamp_enabled; -#endif +unsigned int flags;/* * move_queued_task/activate_task/enqueue_task: rq->lock
Peter Zijlstra
maintainerQuoted 59 lines · Andrea Righi
Hi Peter,
Quoted 25 lines · Peter Zijlstra
Quoted 6 lines · Andrea Righi
Andrea Righi (16):
sched/core: Drop mutex locks before proxy rescheduling sched/core: Dequeue waking proxy donors before reset sched/core: Mark wakeups completed throughttwu_runnable()
sched: Add helper to block retained proxy donors
sched: Addsched_exthooks for proxy executionRight, so these add:
WF_TTWU_RQ:Used like
ENQUEUE_DELAYED; could be fixed by generalizing that to cover all ofp->is_blocked.
scx_allow_proxy_exec():Hook to kill proxy exec for scx
scx_proxy_reenqueue_retry():Like
put_prev_task(), but for current. Ensures current gets put back on a DSQ once its done running.Quoted 8 lines
scx_proxy_donor_start():Delayed
set_next_task(), confirms donor will be used.
sched_proxy_block_task():Almost like
switching_to_scx(), except it needs to changectx->queuedin case ofp->is_blocked. Hence a new callback ran beforesched_change_begin().Yes, that matches the intent, with one small clarification:
scx_proxy_reenqueue_retry()doesn't directly put current back on a DSQ, a task that couldn't be reenqueued remains on the reject DSQ and the hook schedules a deferred retry after proxy resolution.Quoted 10 lines
Now, I have:
https://patch.msgid.link/[email protected],
pending, would something like the below on top of both this work?
(although I'm not convinced
SC_CONFIRMis actually making it better)Yes, this works.
I applied Shubhang's v4 hrtick patch as a preliminary commit, then added the
SNT_CONFIRMcallback and reworked the proxy-exec series to use it.sched_extnow confirms the donor inset_next_task_scx()andscx_proxy_donor_start()is gone.
Quoted 8 lines
I tested the updated series and it passed all my scx proxy exec tests. The branch is available here:
git://git.kernel.org/pub/scm/linux/kernel/git/arighi/linux.gitscx-proxy-exec-nextIt's not an obvious simplification, but if we want to go this way, we can express both the provisional pick and donor confirmation through the sched class callback.
Yeah, I'm not convinced either, but I'm also not liking scx specific hooks. I'll apply your patches as is, and we can noodle on the difference later.
TJ, would you like me to take the sched_ext part of this series too, or will you pull in sched/core after these land and then put those patches on top in your tree?
Tejun Heo
reviewerHello, Peter.
Quoted 3 lines · Peter Zijlstra
TJ, would you like me to take the
sched_extpart of this series too, or will you pull in sched/core after these land and then put those patches on top in your tree?
I can pull from sched/core and then apply sched_ext parts on top.
Thanks.
Signature · 1 line
tejun
Tejun Heo
reviewerQuoted 1 line · Andrea Righi
Andrea Righi (16):
...
Quoted 11 lines
sched_ext: Block proxy donors before taking control
sched_ext: Fix ops.running/stopping() pairing for proxy-exec donors
sched_ext: Move reject DSQ draining into core
sched_ext: Generalize the reject DSQ reenqueue path
sched_ext: Handle proxy-exec races in remote DSQ transfers
sched_ext: Split curr|donor references properly
sched_ext: Track proxy execution forNOHZ_FULL
sched_ext: Delegate proxy donor admission to BPF schedulers
sched_ext: Add selftest for blocked donor admission
sched_ext:scx_qmap: Add proxy execution support
sched: Allow enabling proxy exec withsched_ext
Applied 6-16 to sched_ext/for-7.4 on top of tip/sched/core (be100c77178e) with the following conflict resolutions:
- 07: sched/core's c72945693b90 ("sched: Restart fair hrtick after same-task repicks") changed set_next_task_scx() to take enum snt_e. @first is now derived as type == SNT_PICK. SNT_REPICK returns early as before.
- 08: for-7.4's 7a919c7f86de ("sched_ext: Test scx_has_subs() inline before calling sub-sched hooks") split scx_reenq_reject() into an inline wrapper and __scx_reenq_reject(). Moved the combined function into ext.c and dropped both from sub.h.
- 08, 09, 10: Kept scx_reenq_wait_dispatching() from 7de9a6fb44ea ("sched_ext: Wait for SCX_OPSS_DISPATCHING before reenqueueing a task") in reenq_local() and scx_reenq_reject(), after the new proxy retry checks.
- 10: be100c77178e already has the fixed scx_proxy_reenqueue_retry() signature, so only the body was applied. The rq_owned_post_enq() comment update was applied over the context changed by cb86607ada73 ("sched_ext:
Don't run ops.dequeue() with a DSQ lock held").
- 11: Context only, the can_stop_tick declaration in set_next_task_scx() and the rq->donor change in scx_rescue_timerfn().
- 14: Context only, enq_blocked in the selftest Makefile.
- 15: dispatch_to_rescue() inserts without needs_immed() per a0d356696f87 ("sched_ext: scx_qmap: Do not add IMMED to rescue inserts"), and the rescue comment keeps the for-7.4 text.
Can you please verify the result?
Thanks.
--
tejun
Andrea Righi
reviewerHi Tejun,
Quoted 39 lines · Tejun Heo
Quoted 1 line · Andrea Righi
Andrea Righi (16):
...
Quoted 11 lines
sched_ext: Block proxy donors before taking control
sched_ext: Fix ops.running/stopping() pairing for proxy-exec donors
sched_ext: Move reject DSQ draining into core
sched_ext: Generalize the reject DSQ reenqueue path
sched_ext: Handle proxy-exec races in remote DSQ transfers
sched_ext: Split curr|donor references properly
sched_ext: Track proxy execution forNOHZ_FULL
sched_ext: Delegate proxy donor admission to BPF schedulers
sched_ext: Add selftest for blocked donor admission
sched_ext:scx_qmap: Add proxy execution support
sched: Allow enabling proxy exec withsched_extApplied 6-16 to sched_ext/for-7.4 on top of tip/sched/core (
be100c77178e) with the following conflict resolutions:- 07: sched/core's
c72945693b90("sched: Restart fair hrtick after same-task repicks") changedset_next_task_scx()to take enumsnt_e. @first is now derived as type ==SNT_PICK.SNT_REPICKreturns early as before.- 08: for-7.4's
7a919c7f86de("sched_ext: Testscx_has_subs()inline before calling sub-sched hooks") splitscx_reenq_reject()into an inline wrapper and__scx_reenq_reject(). Moved the combined function intoext.cand dropped both fromsub.h.- 08, 09, 10: Kept
scx_reenq_wait_dispatching()from7de9a6fb44ea("sched_ext: Wait forSCX_OPSS_DISPATCHINGbefore reenqueueing a task") inreenq_local()andscx_reenq_reject(), after the new proxy retry checks.- 10:
be100c77178ealready has the fixedscx_proxy_reenqueue_retry()signature, so only the body was applied. Therq_owned_post_enq()comment update was applied over the context changed bycb86607ada73("sched_ext:
Don't runops.dequeue()with a DSQ lock held").- 11: Context only, the
can_stop_tickdeclaration inset_next_task_scx()and therq->donorchange inscx_rescue_timerfn().
Quoted 7 lines
- 14: Context only,
enq_blockedin the selftestMakefile.- 15:
dispatch_to_rescue()inserts withoutneeds_immed()pera0d356696f87("sched_ext:scx_qmap: Do not add IMMED to rescue inserts"), and the rescue comment keeps the for-7.4 text.Can you please verify the result?
I reran all my proxy-exec tests on sched_ext/for-7.4. The conflict resolutions look good. The only issues I saw were unrelated virtio console warnings while testing inside virtme-ng, fixed by:
b144dc5a2414 ("virtio_console: allocate the port_buffer with the caller's gfp")
Otherwise, everything works great!
Thanks,
-Andrea