sched: Make proxy execution compatible with sched_ext
Subsystems (4)
Andrea Righi
reviewerPATCH v8 1/1 sched: Make proxy execution compatible with sched_ext
This series enables using proxy execution with sched_ext and is based on early work by John Stultz [1].
Background
==========
Proxy execution (proxy-exec) lets a waiting task ("donor") donate its scheduling context to a mutex owner, so the owner can run while the donor stays eligible on the runqueue.
Currently, proxy-exec and sched_ext are mutually exclusive at build time: we can't enable CONFIG_SCHED_PROXY_EXEC=y and CONFIG_SCHED_CLASS_EXT=y in the same kernel.
This restriction can be problematic for Linux distributions and for anyone who wants to ship one kernel and choose features at runtime.
Why are they mutually exclusive?
================================
sched_ext schedulers drive dispatch through their own interfaces. A proxy-exec handoff can run a task that the BPF scheduler never dispatched through that path. sched_ext callbacks then observe a "current" task that does not match what the BPF side considers running, so kfuncs and helper state can see an inconsistent view of the executing task.
sched_ext also tracks runnable work through Dispatch Queues (DSQs) and BPF chosen dispatch rules, while the core scheduler still maintains classic per-CPU runqueues and pick paths. A proxy handoff can therefore switch the CPU to a task that the BPF scheduler never inserted or ordered through its DSQ interface.
DSQ state, vtime, and "who is running" bookkeeping inside the BPF program can then disagree with what the core actually executes, so helpers and kfuncs that assume their dispatched task is current may observe stale or inconsistent state.
Design: supporting proxy-exec with sched_ext
============================================
Provide proxy-exec in sched_ext as optional per-scheduler capability: a BPF scheduler can set the ops flag SCX_OPS_ENQ_BLOCKED to keep mutex-blocked donors runnable and receive them through ops.enqueue() with SCX_ENQ_BLOCKED set.
This flag requires an ops.enqueue() callback and gives BPF control over whether, where, and in what order to dispatch each donor. The core then walks the mutex-owner chain and, when needed, migrates the donor's scheduling context to the owner's rq before executing the owner.
Without the SCX_OPS_ENQ_BLOCKED flag, mutex waiters block normally and do not participate in proxy-exec while owned by that scheduler.
The donor-to-owner handoff is modeled like a "function call" from the scheduler's perspective. The donor remains the running scheduling entity selected by BPF: its scheduling context, runtime and slice are consumed while the core temporarily invokes the mutex owner's code to make the critical section progress. It is not a scheduler-visible switch to the owner.
Accordingly, the donor remains the scheduling context presented to sched_ext, while the mutex owner is treated solely as the execution context selected internally by the core scheduler. Scheduling state is accounted against rq->donor where appropriate, while rq->curr identifies the execution context. The internal owner substitution does not generate synthetic sched_ext callbacks for a task that BPF did not dispatch.
Scheduler-facing current-task kfuncs follow the same rule:scx_bpf_task_running(), scx_bpf_cpu_curr() and scx_bpf_cid_curr() report the donor selected by the scheduler rather than the mutex owner whose code happens to execute on its behalf.
The sched_ext callback bookkeeping is adjusted accordingly. Blocked proxy donors do not generate spurious ops.running() callbacks and ops.stopping() is only called for tasks that entered a tracked ops.running()/stopping() session. This state is independent of physical rq->curr execution, which may refer to the mutex owner instead of the donor.
NOHZ CFS bandwidth checks also follow rq->donor rather than rq->curr, so a bandwidth-constrained FAIR donor keeps the tick running while an owner executes on its behalf.
Scheduler ownership changes need special handling because a donor may already be blocked when a root scheduler is enabled, when a task moves between root and sub-schedulers, or when PI de-boosting moves it into EXT. If the incoming scheduler does not set SCX_OPS_ENQ_BLOCKED, the retained donor is removed from the runqueue and stays on the normal mutex wait path. Otherwise it remains eligible for BPF admission.
scx_qmap has been modified with a -B option to enable queueing mutex-blocked tasks for proxy-exec. Blocked donors receive a fresh slice, are inserted at the head of their current cid's local DSQ and request immediate preemption. This is an intentionally aggressive policy for making proxy-exec easy to observe. Capability-rejection re-enqueues bypass this blocked-donor fast path and fall back to the shared DSQ, avoiding repeated rejection on a cid that no longer has the required capability. A new statistic reports blocked-donor dispatches.
A new kselftest (enq_blocked) is also introduced to validate proxy-exec with sched_ext. The test creates a priority inversion with a low-priority owner (nice +19), a high-priority donor (nice -20) and one nice 0 contender per available CPU. It exercises a same-CPU topology and a cross-CPU topology, where the donor and owner run on different CPUs. Each configuration runs with SCX_OPS_ENQ_BLOCKED first disabled and then enabled, counts blocked-donor enqueues by CPU and reports mutex hold/wait-time deltas. Access to the mutex is provided by a loadable kernel module built via TEST_GEN_MODS_DIR, with the test responsible for its lifecycle.
Example kselftest run:
===== START =====
TEST: enq_blocked
DESCRIPTION: Verify proxy donor admission under CPU-wide contention OUTPUT:
[topology=same-cpu SCX_OPS_ENQ_BLOCKED=disabled]
proxy_exec=enabled
donor_cpu=0
owner_cpu=0
nr_contenders=16
measured_trials=10
owner_nice=19
donor_nice=-20
contender_nice=0 mutex_hold_avg_ns=263587544 (263.587 ms, samples=10) mutex_wait_avg_ns=263597934 (263.597 ms, samples=10) nr_blocked_enqueues=0 nr_blocked_enqueues_donor_cpu=0 nr_blocked_enqueues_owner_cpu=0 nr_blocked_enqueues_other_cpu=0 nr_blocked_wakeups=0
[topology=same-cpu SCX_OPS_ENQ_BLOCKED=enabled]
proxy_exec=enabled
donor_cpu=0
owner_cpu=0
nr_contenders=16
measured_trials=10
owner_nice=19
donor_nice=-20
contender_nice=0 mutex_hold_avg_ns=251789178 (251.789 ms, samples=10) mutex_wait_avg_ns=209808599 (209.808 ms, samples=10) nr_blocked_enqueues=60 nr_blocked_enqueues_donor_cpu=60 nr_blocked_enqueues_owner_cpu=0 nr_blocked_enqueues_other_cpu=0 nr_blocked_wakeups=0
[topology=same-cpu delta: enabled - disabled]
mutex_hold_delta_ns=-11798366 (-4.48%)
mutex_wait_delta_ns=-53789335 (-20.41%)
[topology=cross-cpu SCX_OPS_ENQ_BLOCKED=disabled]
proxy_exec=enabled
donor_cpu=0
owner_cpu=1
nr_contenders=16
measured_trials=10
owner_nice=19
donor_nice=-20
contender_nice=0 mutex_hold_avg_ns=246794454 (246.794 ms, samples=10) mutex_wait_avg_ns=247298497 (247.298 ms, samples=10) nr_blocked_enqueues=0 nr_blocked_enqueues_donor_cpu=0 nr_blocked_enqueues_owner_cpu=0 nr_blocked_enqueues_other_cpu=0 nr_blocked_wakeups=0
[topology=cross-cpu SCX_OPS_ENQ_BLOCKED=enabled]
proxy_exec=enabled
donor_cpu=0
owner_cpu=1
nr_contenders=16
measured_trials=10
owner_nice=19
donor_nice=-20
contender_nice=0 mutex_hold_avg_ns=215486809 (215.486 ms, samples=10) mutex_wait_avg_ns=215498090 (215.498 ms, samples=10) nr_blocked_enqueues=80 nr_blocked_enqueues_donor_cpu=20 nr_blocked_enqueues_owner_cpu=60 nr_blocked_enqueues_other_cpu=0 nr_blocked_wakeups=0
[topology=cross-cpu delta: enabled - disabled]
mutex_hold_delta_ns=-31307645 (-12.69%)
mutex_wait_delta_ns=-31800407 (-12.86%)
ok 1 enq_blocked #
===== END =====
=============================
RESULTS:
PASSED: 1 SKIPPED: 0 FAILED: 0
References
==========
[1] https://lore.kernel.org/all/[email protected]
Git tree: git://git.kernel.org/pub/scm/linux/kernel/git/arighi/linux.git scx-proxy-exec
Changes in v8:
- Rework remote-DSQ migration check so the locked re-check handles only proxy-exec state changes (Tejun Heo)
- Use the current-donor helper when preventing migration of an active donor (sashiko)
- Handle blocked-donor reactivation into scx schedulers without SCX_OPS_ENQ_BLOCKED (John Stultz)
- Document sched_ext callback behavior under proxy execution (sashiko)
- Avoid redispatching a rejected blocked donor to the same local DSQ in scx_qmap (sashiko)
- Add a preparatory fix to avoid false migration warnings when proxy-exec moves a migration-disabled donor
- Skip the enq_blocked kselftest when proxy execution is unavailable
- Link to v7: https://lore.kernel.org/all/[email protected]/
Changes in v7:
- Move the remote CPU check out of the lockless remote-DSQ scan and perform it only after locking the source rq, retain the locked migration recheck immediately before moving the task (sashiko)
- Complete the scheduler-context conversion from rq->curr to rq->donor for capability revocation and preemption checks, make scx_bpf_task_running(), scx_bpf_cpu_curr() and scx_bpf_cid_curr() expose the donor selected by the scheduler (sashiko)
- Handle SCX_ENQ_REENQ before scx_qmap's blocked-donor fast path to avoid an infinite reject/re-enqueue loop (sashiko)
- Add scx_qmap stat for SCX_ENQ_BLOCKED dispatches
- Link to v6: https://lore.kernel.org/all/[email protected]/
Changes in v6:
- Add a scheduler-core fix to make NOHZ CFS bandwidth checks follow the proxy donor instead of the physical execution context (sashiko)
- Block retained donors when entering sched_ext through PI de-boosting or global scheduler activation (sashiko)
- Distinguish ordinary wakeup activations from retained-donor admissions so SCX_ENQ_BLOCKED is not reported for normal wakeups; extend the selftest to detect this regression (sashiko)
- Validate unexpected-CPU blocked-donor enqueues in both same-CPU and cross-CPU test topologies (sashiko)
- Link to v5: https://lore.kernel.org/all/[email protected]/
Changes in v5:
- Split retained-donor deactivation and sched_ext's default rejection into preparatory patches so the scheduler-core changes can be routed separately
- Drop the proxy destination query kfuncs and the preparatory mutex lock-scope change (John Stultz)
- Use p->is_blocked instead of task_is_blocked() to fix a WARN triggered during scx_pair testing (John Stultz)
- Rename SCX_TASK_IS_RUNNING to SCX_TASK_RUN_TRACKED and document that it tracks an ops.running()/stopping() session rather than physical rq->curr execution
- Keep a proxy-migrated blocked donor on the owner's rq until wakeup instead of allowing BPF-directed migration to pull it back to wake_cpu and cause repeated donor migration
- Extend kselftest to cover same-CPU and cross-CPU proxy-exec switches
- Make scx_qmap insert blocked donors at the head of their current cid's local DSQ and request immediate preemption
- Link to v4: https://lore.kernel.org/all/[email protected]/
Changes in v4:
- Harden scx_bpf_task_proxy_cpu() locking and blocked-state validation, return -ENOENT for unrunnable owners and drop unnecessary donor-affinity checks (K Prateek Nayak, sashiko)
- Reschedule remote CPUs after donor deactivation, handle sched_setscheduler() admission and assert scheduler-change locking (sashiko)
- Avoid a potential KCSAN data-race report in the lockless blocked-donor migration check by using rcu_access_pointer() for rq->donor (sashiko)
- Fix tick dependency updates for incoming EXT contexts and keep the tick enabled for blocked donors (sashiko)
- Dump EXT donors in scx_dump_state() (sashiko)
- Reordered the preparatory sched_ext changes so real running-state tracking is established before donor-based accounting, and split the proxy destination query kfuncs into a separate patch
- Add compatibility wrappers for the proxy CPU/cid kfuncs and document their results as scheduling hints
- Link to v3: https://lore.kernel.org/all/[email protected]/
Changes in v3:
- Dropped the core restrictions on proxy-migrating migration-disabled and single-CPU donors: proxy execution moves the scheduling context, not the task's execution context. (Peter Zijlstra, K Prateek Nayak)
- Dropped the sched_ext-specific put_prev_task()/set_next_task() exception and fixed ops.running()/ops.stopping() pairing inside sched_ext instead; track a real running transition even when either callback is absent. (Peter Zijlstra)
- Dropped the kf_tasks[] nesting and nested ops.runnable() patches from v2; extensive proxy-exec testing did not reproduce task-op re-entry, so retain the existing non-nesting invariant.
- Replaced scx_bpf_task_is_blocked() with SCX_ENQ_BLOCKED in ops.enqueue() flags, identifying blocked-donor admission directly at enqueue time.
- Expanded the curr/donor description to cover mixed scheduling classes and fixed local preemption to expire rq->donor's slice. (Aiqun Maria Yu)
- Allow inactive blocked donors to be placed on a remote local DSQ while preventing normal migration of an active rq donor. (Aiqun Maria Yu)
- Deactivate retained donors when ownership changes to a root or sub-scheduler without SCX_OPS_ENQ_BLOCKED, and extend the selftest to cover attaching a scheduler after the donor blocks. (K Prateek Nayak)
- Harden remote DSQ consumption by rejecting active tasks and rechecking migration eligibility after locking the source rq. Fall back to the global DSQ without treating an eligibility change during the lock handoff as a BPF scheduler error.
- scx_qmap has a command line option (-B) to enable blocked-donor queueing
- Link to v2: https://lore.kernel.org/all/[email protected]/
Changes in v2:
- Rebased onto sched_ext/for-7.3 and adapted the series to the split sched_ext implementation and cid-form scheduler interfaces.
- Replaced the global sched_proxy_exec_scx boot-time opt-in with the per-scheduler SCX_OPS_ENQ_BLOCKED capability, allowing BPF to control donor admission and ordering through ops.enqueue().
- Added scx_bpf_task_is_blocked(), scx_bpf_task_proxy_cpu(), and scx_bpf_task_proxy_cid(); enforce CPU/cid API separation for cid-form schedulers.
- Added proxy exec support to scx_qmap, including optional owner-cid steering, affinity validation, and fallback to the donor's current cid.
- Added a kselftest with a kernel mutex test and a three-task priority inversion workload, test is executed with blocked task admission disabled and enabled, validates the behavior, and reports hold/wait-time deltas.
- Link to v1: https://lore.kernel.org/all/[email protected]/
Andrea Righi (11):
sched/core: Avoid false migration warning for proxy donors
sched: Make NOHZ CFS bandwidth checks follow proxy donor
sched: Add helper to block retained proxy donors
sched_ext: Block proxy donors across scheduler transitions
sched_ext: Fix ops.running/stopping() pairing for proxy-exec donors
sched_ext: Fix proxy-exec race in consume_remote_task()
sched_ext: Split curr|donor references properly
sched_ext: Delegate proxy donor admission to BPF schedulers
sched_ext: Add selftest for blocked donor admission
sched_ext: scx_qmap: Add proxy execution support
sched: Allow enabling proxy exec with sched_ext
John Stultz (1):
sched_ext: Handle blocked donor migration with proxy execution
Documentation/scheduler/sched-ext.rst | 6 + include/linux/sched/ext.h | 3 + init/Kconfig | 2 - kernel/sched/core.c | 59 +- kernel/sched/ext/ext.c | 374 +++++++-- kernel/sched/ext/ext.h | 6 + kernel/sched/ext/internal.h | 29 +- kernel/sched/ext/sub.c | 8 +- kernel/sched/ext/sub.h | 8 +- kernel/sched/sched.h | 6 + kernel/sched/syscalls.c | 3 + tools/sched_ext/include/scx/compat.h | 1 + tools/sched_ext/include/scx/enum_defs.autogen.h | 1 + tools/sched_ext/include/scx/enums.autogen.bpf.h | 3 + tools/sched_ext/include/scx/enums.autogen.h | 1 + tools/sched_ext/scx_qmap.bpf.c | 34 + tools/sched_ext/scx_qmap.c | 13 +- tools/sched_ext/scx_qmap.h | 1 + tools/testing/selftests/sched_ext/.gitignore | 4 + tools/testing/selftests/sched_ext/Makefile | 2 + tools/testing/selftests/sched_ext/config | 2 + .../testing/selftests/sched_ext/enq_blocked.bpf.c | 116 +++ tools/testing/selftests/sched_ext/enq_blocked.c | 918 +++++++++++++++++++++ tools/testing/selftests/sched_ext/enq_blocked.h | 28 + .../selftests/sched_ext/test_modules/Makefile | 13 + .../sched_ext/test_modules/scx_enq_blocked_test.c | 195 +++++ 26 files changed, 1753 insertions(+), 83 deletions(-) create mode 100644 tools/testing/selftests/sched_ext/enq_blocked.bpf.c create mode 100644 tools/testing/selftests/sched_ext/enq_blocked.c create mode 100644 tools/testing/selftests/sched_ext/enq_blocked.h create mode 100644 tools/testing/selftests/sched_ext/test_modules/Makefile create mode 100644 tools/testing/selftests/sched_ext/test_modules/scx_enq_blocked_test.c
Andrea Righi
reviewerPATCH 1/12 sched/core: Avoid false migration warning for proxy donors
Proxy execution can move a blocked donor's scheduling context to the lock owner's CPU even when the donor is migration-disabled. The donor does not execute there, and its original execution CPU remains recorded in wake_cpu.
set_task_cpu() warns unconditionally for migration-disabled tasks, so a subsequent proxy migration or the wakeup path returning the donor home triggers a false positive. Moving a blocked scheduling context does not violate the migration-disabled execution context.
Exclude blocked proxy donors from the warning. The proxy wakeup path restores an executable placement before clearing the blocked state.
This is a preparatory fix for enabling proxy execution together with sched_ext.
--- kernel/sched/core.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)
kernel/sched/core.c +7 -1
@@ -3376,7 +3376,13 @@ void set_task_cpu(struct task_struct *p, unsigned int new_cpu)*/ WARN_ON_ONCE (!cpu_online (new_cpu)); -WARN_ON_ONCE (is_migration_disabled (p)); +/* +* Proxy execution can move a blocked task's scheduling context to any +* CPU without moving its migration-disabled execution context. The +* wakeup path will return the task to a CPU where it can execute. +*/ +WARN_ON_ONCE (is_migration_disabled (p) && + !(sched_proxy_exec () && p->is_blocked));trace_sched_migrate_task (p, new_cpu);
Signature · 1 line
2.55.0
[email protected]
AI reviewRe: [PATCH 01/12] sched/core: Avoid false migration warning for proxy donors
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] The warning for migrating a migration-disabled task is incorrectly suppressed for all sleeping…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] The warning for migrating a migration-disabled task is incorrectly suppressed for all sleeping tasks when proxy execution is enabled, instead of just for blocked proxy donors. --
--- Patch [1]: [PATCH 01/12] sched/core: Avoid false migration warning for proxy donors --- commit 3b0fd432ea71e1ba27a8e56bb387fb372b053daf
Author: Andrea Righi <[email protected]>
sched/core: Avoid false migration warning for proxy donors
This commit updates set_task_cpu to suppress a false positive warning when migrating a proxy donor. It allows moving a blocked scheduling context to another CPU even when the donor task has migration disabled.
Quoted 3 lines
kernel/sched/core.c +0
-0
[ ... ]
kernel/sched/core.c void set_task_cpu(struct task_struct *p, unsigned int new_cpu) · line 3385
Quoted 12 lines
kernel/sched/core.c +7
-1@@ -3376,7 +3376,13 @@ void set_task_cpu(struct task_struct *p, unsigned int new_cpu)*/ WARN_ON_ONCE (!cpu_online (new_cpu)); -WARN_ON_ONCE (is_migration_disabled (p)); +/* +* Proxy execution can move a blocked task's scheduling context to any +* CPU without moving its migration-disabled execution context. The +* wakeup path will return the task to a CPU where it can execute. +*/ +WARN_ON_ONCE (is_migration_disabled (p) && + !(sched_proxy_exec () && p->is_blocked));
[Severity: Medium]
Does this logic inadvertently suppress the warning for all sleeping tasks rather than just proxy donors?
Because p->is_blocked is set unconditionally for all sleeping tasks when __schedule calls try_to_block_task, this condition in set_task_cpu appears to suppress the warning for any migration-disabled sleeping task whenever proxy execution is globally enabled.
If another scheduler component, such as a load balancer or a BPF sched_ext extension, incorrectly migrates a normal sleeping task that has migration disabled, this overly broad check might hide the invalid migration and mask further issues.
Quoted 2 lines
kernel/sched/core.c +0
-0trace_sched_migrate_task (p, new_cpu);
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Re: [PATCH 01/12] sched/core: Avoid false migration warning for proxy donors
In reply to Andrea RighiQuoted 11 lines · Andrea Righi
Proxy execution can move a blocked donor's scheduling context to the lock owner's CPU even when the donor is migration-disabled. The donor does not execute there, and its original execution CPU remains recorded in
wake_cpu.
set_task_cpu()warns unconditionally for migration-disabled tasks, so a subsequent proxy migration or the wakeup path returning the donor home triggers a false positive. Moving a blocked scheduling context does not violate the migration-disabled execution context.
Quoted 7 lines
Exclude blocked proxy donors from the warning. The proxy wakeup path restores an executable placement before clearing the blocked state.
This is a preparatory fix for enabling proxy execution together with
sched_ext.Signed-off-by Andrea Righi <[email protected]>
Nice catch on this!
I'm wondering why I've not tripped this myself yet. Is migration_disabled set more often in the sched_ext paths? Or was there a specific workload that you hit this with?
thanks
-john
Andrea Righi
reviewerRe: [PATCH 01/12] sched/core: Avoid false migration warning for proxy donors
Quoted 18 lines · John Stultz
Quoted 17 lines · Andrea Righi
Proxy execution can move a blocked donor's scheduling context to the lock owner's CPU even when the donor is migration-disabled. The donor does not execute there, and its original execution CPU remains recorded in
wake_cpu.
set_task_cpu()warns unconditionally for migration-disabled tasks, so a subsequent proxy migration or the wakeup path returning the donor home triggers a false positive. Moving a blocked scheduling context does not violate the migration-disabled execution context.Exclude blocked proxy donors from the warning. The proxy wakeup path restores an executable placement before clearing the blocked state.
This is a preparatory fix for enabling proxy execution together with
sched_ext.
Quoted 7 lines
Quoted 1 line
Signed-off-by Andrea Righi <[email protected]>Nice catch on this!
I'm wondering why I've not tripped this myself yet. Is
migration_disabledset more often in thesched_extpaths? Or was there a specific workload that you hit this with?
I triggered this with a modified scx_cosmos with proxy-exec support (option --proxy-exec): https://github.com/sched-ext/scx/tree/cosmos-proxy-exec
This was the trace, triggered when scx_cosmos was stopped:
[ 12.992201] WARNING: kernel/sched/core.c:3444 at set_task_cpu+0x1d3/0x280, CPU#6: kworker/u65:1/103 [ 12.992387] Modules linked in: [ 12.992593] CPU: 6 UID: 0 PID: 103 Comm: kworker/u65:1 Not tainted 7.1.0-virtme #3 PREEMPT(full) [ 12.993012] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 12.993075] Workqueue: events_unbound bpf_map_free_deferred [ 12.993139] RIP: 0010:set_task_cpu+0x1d3/0x280 [ 12.993285] Code: 14 89 6b 64 c6 83 09 17 00 00 01 5b 41 5e 41 5f 5d c3 cc cc cc cc cc 90 0f 0b 90 66 83 bb 80 05 00 00 00 0f 84 f8 fe ff ff 90 <0f> 0b 90 e9 ef fe ff ff 48 8d 3d ce 12 4c 02 67 48 0f b9 3a e9 03 [ 12.993656] RSP: 0018:ffffc900003f7cf8 EFLAGS: 00010002 [ 12.994642] RAX: 0000000000000000 RBX: ffff888100c6c800 RCX: 0000000000000046 [ 12.994756] RDX: ffffc900003f7d2c RSI: ffff888100c6d338 RDI: ffff888100c454d0 [ 12.994860] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 [ 12.994956] R10: 000000000000003b R11: ffffffff813f5c30 R12: 0000000000000000 [ 12.995055] R13: ffff888100c6d320 R14: ffff888100c44800 R15: 0000000000000206 [ 12.995167] FS: 0000000000000000(0000) GS:ffff8882b380a000(0000) knlGS:0000000000000000 [ 12.995273] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 12.995354] CR2: 00007fe19b902000 CR3: 000000000364c000 CR4: 0000000000750ef0 [ 12.995462] PKRU: 55555554 [ 12.995502] Call Trace: [ 12.995540] <TASK> [ 12.995580] try_to_wake_up+0x360/0x6f0 [ 12.995640] __mutex_unlock_slowpath+0x32f/0x540 [ 12.995716] ? bpf_local_storage_map_free+0x84/0x280 [ 12.995789] ? process_scheduled_works+0x225/0x520 [ 12.995861] bpf_local_storage_map_free+0x257/0x280 [ 12.995934] bpf_map_free+0x84/0x110 [ 12.995992] process_scheduled_works+0x271/0x520 [ 12.996071] worker_thread+0x1e1/0x300 [ 12.996133] ? __pfx_worker_thread+0x10/0x10 [ 12.996206] kthread+0x10c/0x140 [ 12.996266] ? __pfx_kthread+0x10/0x10 [ 12.996319] ret_from_fork+0x189/0x330 [ 12.996378] ? __pfx_kthread+0x10/0x10 [ 12.996435] ret_from_fork_asm+0x1a/0x30 [ 12.996500] </TASK>
Looking at bpf_local_storage_map_free(), I think I triggered this warning, because bpf_map_free() calls migrate_disable() around map->ops->map_free() and the deferred map-free workers can block on mutexes while migration is disabled. Then proxy-exec moves the worker to the mutex owner's CPU and when the mutex is released, ttwu returns the worker to its pinned CPU and trips the set_task_cpu() warning.
Quoted 5 lines
Acked-by John Stultz <[email protected]>thanks
-john
Thanks,
-Andrea
Andrea Righi
reviewerPATCH 2/12 sched: Make NOHZ CFS bandwidth checks follow proxy donor
In reply to Andrea RighiProxy execution separates the scheduling context in rq->donor from the physical execution context in rq->curr. sched_can_stop_tick() checks the latter for CFS bandwidth constraints and only does so when nr_running is one.
A retained proxy donor keeps both the donor and mutex owner queued. The check therefore misses a constrained FAIR donor and may stop the tick while its runtime still needs to be enforced.
Check the selected donor instead and remove the nr_running restriction. The donor being a queued FAIR task is sufficient to require bandwidth accounting regardless of other runnable tasks.
--- kernel/sched/core.c | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-)
kernel/sched/core.c +9 -9
@@ -1409,11 +1409,8 @@ static void nohz_csd_func(void *info)#endif /* CONFIG_NO_HZ_COMMON */ #ifdef CONFIG_NO_HZ_FULL -static inline bool __need_bw_check (struct rq *rq,struct task_struct *p) +static inline bool __need_bw_check (struct task_struct *p) { -if (rq->nr_running !=1 ) -return false; -if (p->sched_class != &fair_sched_class)return false;
@@ -1462,14 +1459,17 @@ bool sched_can_stop_tick(struct rq *rq)return false;/* -* If there is one task and it has CFS runtime bandwidth constraints -* and it's on the cpu now we don't want to stop the tick. +* If the selected scheduling context has CFS runtime bandwidth +* constraints, don't stop the tick. With proxy execution, rq->donor is +* the selected scheduling context while rq->curr is the task physically +* executing on its behalf. +* * This check prevents clearing the bit if a newly enqueued task here is -* dequeued by migrating while the constrained task continues to run. +* dequeued by migrating while the constrained donor continues to run. * E.g. going from 2->1 without going through pick_next_task(). */ -if (__need_bw_check (rq, rq->curr)) { -if (cfs_task_bw_constrained (rq->curr)) +if (__need_bw_check (rq->donor)) { +if (cfs_task_bw_constrained (rq->donor))return false; }
Signature · 1 line
2.55.0
[email protected]
AI reviewRe: [PATCH 02/12] sched: Make NOHZ CFS bandwidth checks follow proxy donor
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Pre-existing issues:
- [High] The patch fails to remove the rq->nr_running != 1 restriction from sched_fair_update_stop_tick(), leaving proxy execution bandwidth enforcement broken during context switches. --
--- Patch [2]: [PATCH 02/12] sched: Make NOHZ CFS bandwidth checks follow proxy donor --- commit 4af18b2dbe3d9116ee4049aa70ca4c8662c0104c
Author: Andrea Righi <[email protected]>
sched: Make NOHZ CFS bandwidth checks follow proxy donor
Proxy execution separates the scheduling context in rq->donor from the physical execution context in rq->curr. sched_can_stop_tick() checks the latter for CFS bandwidth constraints and only does so when nr_running is one. A retained proxy donor keeps both the donor and mutex owner queued, so this patch checks the selected donor instead and removes the nr_running restriction.
Quoted 5 lines
kernel/sched/core.c +0
-0
kernel/sched/core.c static void nohz_csd_func(void *info) · line 1416
Quoted 9 lines
kernel/sched/core.c +1
-4@@ -1409,11 +1409,8 @@ static void nohz_csd_func(void *info)#endif /* CONFIG_NO_HZ_COMMON */ #ifdef CONFIG_NO_HZ_FULL -static inline bool __need_bw_check (struct rq *rq,struct task_struct *p) +static inline bool __need_bw_check (struct task_struct *p) { -if (rq->nr_running !=1 ) -return false; -
[Severity: High]
This is a pre-existing issue, but does removing this restriction here fully resolve the proxy execution bandwidth enforcement bug?
Looking at sched_fair_update_stop_tick() in kernel/sched/fair.c, there is an identical check that prevents tick updates when nr_running != 1:
kernel/sched/fair.c:sched_fair_update_stop_tick() {
if (rq->nr_running != 1)
return;
...
}If a constrained FAIR task donates its scheduling context to a lower-priority task (like an SCX task) via proxy execution, nr_running becomes >= 2 while h_nr_queued remains 1.
During enqueue, sched_can_stop_tick() evaluates against the old rq->donor (e.g., the idle task) and may clear the tick dependency. When the scheduler picks the FAIR donor, sched_fair_update_stop_tick() returns early because rq->nr_running != 1.
Can this result in the tick remaining cleared, allowing a constrained FAIR task to bypass its CFS bandwidth quota?
Should the nr_running restriction in sched_fair_update_stop_tick() also be removed or updated to properly account for proxy execution?
Quoted 3 lines
kernel/sched/core.c +0
-0if (p->sched_class != &fair_sched_class)return false;
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
Andrea Righi
reviewerPATCH 3/12 sched: Add helper to block retained proxy donors
In reply to Andrea RighiScheduler ownership changes may need to turn a mutex-blocked task retained on the runqueue for proxy execution back into a normally blocked task.
Add sched_proxy_block_task() to perform that transition while holding p->pi_lock and the task's runqueue lock. Reset an active donor to the physical current task before blocking it. If DELAY_DEQUEUE keeps a FAIR donor queued, the immediately following sched_change_begin() completes the dequeue through switching_from_fair() before recording its queued state.
This is a preparatory change to support proxy execution with sched_ext.
--- kernel/sched/core.c | 28 ++++++++++++++++++++++++++++ kernel/sched/sched.h | 6 ++++++ 2 files changed, 34 insertions(+)
kernel/sched/core.c +28 -0
@@ -6770,6 +6770,34 @@ static void proxy_deactivate(struct rq *rq, struct task_struct *donor)block_task (rq, donor, state); } +/* +* Remove a retained proxy donor before changing its scheduler ownership. +* The caller holds p->pi_lock, so p cannot wake and migrate if block_task() +* drops it from the runqueue. If DELAY_DEQUEUE keeps a blocked fair task +* queued, switching_from_fair() completes the dequeue in the immediately +* following sched_change_begin(). +*/ +void sched_proxy_block_task (struct rq *rq,struct task_struct *p) +{ +unsigned long state =READ_ONCE (p->__state); + +lockdep_assert_held (&p->pi_lock); +lockdep_assert_rq_held (rq); + +if (!p->is_blocked || !task_on_rq_queued (p)) +return ; +if (WARN_ON_ONCE (state ==TASK_RUNNING )) +return ; + +if (task_current_donor (rq, p)) +proxy_reset_donor (rq); + +if (!p->se.sched_delayed) +block_task (rq, p, state); + +WARN_ON_ONCE (task_on_rq_queued (p) && !p->se.sched_delayed); +} +static inline void proxy_release_rq_lock (struct rq *rq,struct rq_flags *rf)__releases (__rq_lockp (rq)) {
kernel/sched/sched.h +6 -0
@@ -2477,6 +2477,12 @@ static inline bool task_is_blocked(struct task_struct *p)return !!p->blocked_on; } +#ifdef CONFIG_SCHED_PROXY_EXEC +void sched_proxy_block_task (struct rq *rq,struct task_struct *p); +#else +static inline void sched_proxy_block_task (struct rq *rq,struct task_struct *p) {} +#endif +static inline int task_on_cpu (struct rq *rq,struct task_struct *p) {return p->on_cpu;
Signature · 1 line
2.55.0
Re: [PATCH 03/12] sched: Add helper to block retained proxy donors
Quoted 12 lines · Andrea Righi
Scheduler ownership changes may need to turn a mutex-blocked task retained on the runqueue for proxy execution back into a normally blocked task.
Add
sched_proxy_block_task()to perform that transition while holdingp->pi_lockand the task's runqueue lock. Reset an active donor to the physical current task before blocking it. IfDELAY_DEQUEUEkeeps a FAIR donor queued, the immediately followingsched_change_begin()completes the dequeue throughswitching_from_fair()before recording its queued state.
Quoted 3 lines
This is a preparatory change to support proxy execution with
sched_ext.Signed-off-by Andrea Righi <[email protected]>
Andrea Righi
reviewerPATCH 4/12 sched_ext: Block proxy donors across scheduler transitions
In reply to Andrea RighiProxy execution retains mutex-blocked donors on the runqueue so their scheduling context can execute a lock owner. sched_ext cannot safely retain such donors unless the BPF scheduler explicitly participates in their admission and ordering.
Make sched_ext reject retained donors by default. Force blocked EXT tasks through the regular block path in schedule(), and fully deactivate a retained donor before sched_setscheduler(), PI de-boosting, or global activation moves it into the EXT class.
This is a preparatory change to support proxy execution with sched_ext.
--- kernel/sched/core.c | 5 ++++- kernel/sched/ext/ext.c | 44 +++++++++++++++++++++++++++++++++++++++++ kernel/sched/ext/ext.h | 6 ++++++ kernel/sched/syscalls.c | 3 +++ 4 files changed, 57 insertions(+), 1 deletion(-)
kernel/sched/core.c +4 -1
@@ -7168,7 +7168,8 @@ static void __sched notrace __schedule(int sched_mode)* task_is_blocked() will always be false). */ try_to_block_task (rq, prev, &prev_state, - !task_is_blocked (prev)); + !task_is_blocked (prev) || + !scx_allow_proxy_exec (prev)); switch_count = &prev->nvcsw; }
@@ -7721,6 +7722,8 @@ void rt_mutex_setprio(struct task_struct *p, struct task_struct *pi_task)if (prev_class != next_class) queue_flag |=DEQUEUE_CLASS ; +scx_prepare_setscheduler (p, next_class); + scoped_guard (sched_change, p, queue_flag) {/* * Boosting condition are:
kernel/sched/ext/ext.c +44 -0
56 lines not on this page · raw message
kernel/sched/ext/ext.h +6 -0
@@ -18,8 +18,11 @@ bool scx_can_stop_tick(struct rq *rq);void scx_rq_activate (struct rq *rq);void scx_rq_deactivate (struct rq *rq);int scx_check_setscheduler (struct task_struct *p,int policy); +void scx_prepare_setscheduler (struct task_struct *p, +const struct sched_class *next_class);bool task_should_scx (int policy);bool scx_allow_ttwu_queue (const struct task_struct *p); +bool scx_allow_proxy_exec (const struct task_struct *p);void init_sched_ext_class (void );static inline u32 scx_cpuperf_target (s32 cpu)
@@ -52,8 +55,11 @@ static inline bool scx_can_stop_tick(struct rq *rq) { return true; }static inline void scx_rq_activate (struct rq *rq) {}static inline void scx_rq_deactivate (struct rq *rq) {}static inline int scx_check_setscheduler (struct task_struct *p,int policy) {return 0 ; } +static inline void scx_prepare_setscheduler (struct task_struct *p, +const struct sched_class *next_class) {}static inline bool task_on_scx (const struct task_struct *p) {return false; }static inline bool scx_allow_ttwu_queue (const struct task_struct *p) {return true; } +static inline bool scx_allow_proxy_exec (const struct task_struct *p) {return true; }static inline void init_sched_ext_class (void ) {}#endif /* CONFIG_SCHED_CLASS_EXT */
kernel/sched/syscalls.c +3 -0
@@ -678,6 +678,9 @@ int __sched_setscheduler(struct task_struct *p,if (prev_class != next_class) queue_flags |=DEQUEUE_CLASS ; +if (!(attr->sched_flags &SCHED_FLAG_KEEP_PARAMS )) +scx_prepare_setscheduler (p, next_class); + scoped_guard (sched_change, p, queue_flags) {if (!(attr->sched_flags &SCHED_FLAG_KEEP_PARAMS )) {
Signature · 1 line
2.55.0
Tejun Heo
reviewerRe: [PATCH 04/12] sched_ext: Block proxy donors across scheduler transitions
Hello, Andrea.
kernel/sched/core.c static void __sched notrace __schedule(int sched_mode) · line 7172
Quoted 7 lines · Andrea Righi
kernel/sched/core.c +2
-1@@ -7168,7 +7168,8 @@ static void __sched notrace __schedule(int sched_mode)* task_is_blocked() will always be false). */ try_to_block_task (rq, prev, &prev_state, - !task_is_blocked (prev)); + !task_is_blocked (prev) || + !scx_allow_proxy_exec (prev));
The condition fits on one line.
kernel/sched/core.c void rt_mutex_setprio(struct task_struct *p, struct task_struct *pi_task) · line 7727
Quoted 7 lines
kernel/sched/core.c +2
-0@@ -7721,6 +7722,8 @@ void rt_mutex_setprio(struct task_struct *p, struct task_struct *pi_task)if (prev_class != next_class) queue_flag |=DEQUEUE_CLASS ; +scx_prepare_setscheduler (p, next_class); + scoped_guard (sched_change, p, queue_flag) {
Calling directly into sched_ext from rt_mutex_setprio() and __sched_setscheduler() isn't the prettiest. This may want to be a class callback invoked from sched_change_begin() on the incoming class before the queued state is recorded - the existing ones are either too late or on the outgoing class.
Peter, what do you think?
kernel/sched/ext/ext.c line 37
Quoted 2 lines
kernel/sched/ext/ext.c +2
-0+void scx_prepare_setscheduler (struct task_struct *p, +const struct sched_class *next_class)
Fits on one line. Ditto for the declaration in ext.h.
Thanks.
Signature · 1 line
tejun
Andrea Righi
reviewerRe: [PATCH 04/12] sched_ext: Block proxy donors across scheduler transitions
Quoted 12 lines · Tejun Heo
Hello, Andrea.
kernel/sched/core.c static void __sched notrace __schedule(int sched_mode) · line 7172
Quoted 7 lines · Andrea Righi
kernel/sched/core.c +2
-1@@ -7168,7 +7168,8 @@ static void __sched notrace __schedule(int sched_mode)* task_is_blocked() will always be false). */ try_to_block_task (rq, prev, &prev_state, - !task_is_blocked (prev)); + !task_is_blocked (prev) || + !scx_allow_proxy_exec (prev));The condition fits on one line.
Ack.
Quoted 15 lines
kernel/sched/core.c void rt_mutex_setprio(struct task_struct *p, struct task_struct *pi_task) · line 7727
Quoted 8 lines
kernel/sched/core.c +2
-0@@ -7721,6 +7722,8 @@ void rt_mutex_setprio(struct task_struct *p, struct task_struct *pi_task)if (prev_class != next_class) queue_flag |=DEQUEUE_CLASS ; +scx_prepare_setscheduler (p, next_class); + scoped_guard (sched_change, p, queue_flag) {Calling directly into
sched_extfromrt_mutex_setprio()and__sched_setscheduler()isn't the prettiest. This may want to be a class callback invoked fromsched_change_begin()on the incoming class before the queued state is recorded - the existing ones are either too late or on the outgoing class.
Quoted 1 line
Peter, what do you think?
Yeah, makes sense to me, it'd be a lot cleaner. If Peter is ok with that, I can add a preparatory patch to introduce this new class callback, something like prepare_switch()?
Quoted 5 lines
kernel/sched/ext/ext.c line 37
Quoted 3 lines
kernel/sched/ext/ext.c +2
-0+void scx_prepare_setscheduler (struct task_struct *p, +const struct sched_class *next_class)Fits on one line. Ditto for the declaration in
ext.h.
Ack.
Thanks,
-Andrea
Andrea Righi
reviewerPATCH 5/12 sched_ext: Fix ops.running/stopping() pairing for proxy-exec donors
In reply to Andrea RighiWith proxy-exec, pick_next_task() can return a task with p->is_blocked set (a proxy donor). put_prev_set_next_task() then calls set_next_task_scx() on this "ghost" task even though the task only provides scheduling context and never actually runs.
Calling ops.running() for such a donor produces a spurious running event. Simply suppressing ops.running() is not sufficient because the following put_prev_task_scx() would still invoke ops.stopping(), resulting in an unpaired stopping event.
Introduce SCX_TASK_RUN_TRACKED to track whether a task entered an ops.running()/stopping() session. Set and clear the flag independently of ops.running() and ops.stopping(), as the callbacks are independently optional. Invoke ops.running() only for non-blocked tasks and invoke ops.stopping() only after entering a tracked session. This keeps the callbacks paired for proxy donors while preserving stopping notifications for schedulers which only implement ops.stopping().
This is a preparatory change for enabling proxy execution together with sched_ext. The explicit running-state tracking is also required by later donor-based accounting: it prevents an EXT owner executing for a non-EXT donor from being treated as the active EXT scheduling context when it is dequeued.
--- include/linux/sched/ext.h | 2 ++ kernel/sched/ext/ext.c | 33 +++++++++++++++++++++++++-------- 2 files changed, 27 insertions(+), 8 deletions(-)
include/linux/sched/ext.h +2 -0
@@ -103,6 +103,8 @@ enum scx_ent_flags {SCX_TASK_SUB_INIT =1 <<4 ,/* task being initialized for a sub sched */ SCX_TASK_IMMED =1 <<5 ,/* task is on local DSQ with %SCX_ENQ_IMMED */ + SCX_TASK_RUN_TRACKED = 1 << 6, /* task is in an ops.running()/stopping() session */ +/* * Bits 8 to 10 are used to carry task state: *
kernel/sched/ext/ext.c +25 -8
@@ -2217,9 +2217,13 @@ static bool dequeue_task_scx(struct rq *rq, struct task_struct *p, int core_deq_* information meaningful to the BPF scheduler and can be suppressed by * skipping the callbacks if the task is !QUEUED. */ -if (SCX_HAS_OP (sch, stopping) &&task_current (rq, p)) { -update_curr_scx (rq); -SCX_CALL_OP_TASK (sch, stopping, rq, p, false); +if (task_current (rq, p) && + (p->scx.flags &SCX_TASK_RUN_TRACKED )) { +if (SCX_HAS_OP (sch, stopping)) { +update_curr_scx (rq); +SCX_CALL_OP_TASK (sch, stopping, rq, p, false); + } + p->scx.flags &= ~SCX_TASK_RUN_TRACKED ; }if (SCX_HAS_OP (sch, quiescent) && !task_on_rq_migrating (p))
@@ -2921,9 +2925,17 @@ static void set_next_task_scx(struct rq *rq, struct task_struct *p, bool first) p->se.exec_start =rq_clock_task (rq); -/* see dequeue_task_scx() on why we skip when !QUEUED */ -if (SCX_HAS_OP (sch, running) && (p->scx.flags &SCX_TASK_QUEUED )) -SCX_CALL_OP_TASK (sch, running, rq, p); +/* +* See dequeue_task_scx() for why we skip when !QUEUED. A blocked task can +* remain queued here only as a retained proxy donor. Skip it because it +* provides scheduling context but never runs itself. +*/ +if ((p->scx.flags &SCX_TASK_QUEUED ) && !p->is_blocked) { +if (SCX_HAS_OP (sch, running)) +SCX_CALL_OP_TASK (sch, running, rq, p); + + p->scx.flags |=SCX_TASK_RUN_TRACKED ; + }clr_task_runnable (p, true);
@@ -3029,8 +3041,13 @@ static void put_prev_task_scx(struct rq *rq, struct task_struct *p,update_curr_scx (rq);/* see dequeue_task_scx() on why we skip when !QUEUED */ -if (SCX_HAS_OP (sch, stopping) && (p->scx.flags &SCX_TASK_QUEUED )) -SCX_CALL_OP_TASK (sch, stopping, rq, p, true); +if ((p->scx.flags &SCX_TASK_QUEUED ) && + (p->scx.flags &SCX_TASK_RUN_TRACKED )) { +if (SCX_HAS_OP (sch, stopping)) +SCX_CALL_OP_TASK (sch, stopping, rq, p, true); + + p->scx.flags &= ~SCX_TASK_RUN_TRACKED ; + }if (p->scx.flags &SCX_TASK_QUEUED ) {set_task_runnable (rq, p);
Signature · 1 line
2.55.0
Tejun Heo
reviewerRe: [PATCH 05/12] sched_ext: Fix ops.running/stopping() pairing for proxy-exec donors
Hello, Andrea.
include/linux/sched/ext.h enum scx_ent_flags { · line 109
Quoted 7 lines · Andrea Righi
include/linux/sched/ext.h +2
-0SCX_TASK_SUB_INIT =1 <<4 ,/* task being initialized for a sub sched */ SCX_TASK_IMMED =1 <<5 ,/* task is on local DSQ with %SCX_ENQ_IMMED */ + SCX_TASK_RUN_TRACKED = 1 << 6, /* task is in an ops.running()/stopping() session */ +/* * Bits 8 to 10 are used to carry task state:
Please drop the blank line above the new entry.
kernel/sched/ext/ext.c static bool dequeue_task_scx(struct rq *rq, struct task_struct *p, int core_deq_ · line 2221
Quoted 2 lines
kernel/sched/ext/ext.c +2
-0+if (task_current (rq, p) && + (p->scx.flags &SCX_TASK_RUN_TRACKED )) {
The condition fits on one line, ditto the one in put_prev_task_scx().
Thanks.
Signature · 1 line
tejun
Andrea Righi
reviewerPATCH 6/12 sched_ext: Fix proxy-exec race in consume_remote_task()
In reply to Andrea RighiWithout proxy execution, the DSQ lock and holding_cpu handshake ensure that a task cannot be dequeued or start running during the rq-lock handoff without clearing holding_cpu.
Proxy execution is an exception. A task can start physically executing as a lock owner while its own scheduling context remains on the DSQ. Its on-CPU or migration-disabled state can therefore change without clearing holding_cpu.
Keep the existing lockless destination validation unchanged. After switching to the source rq lock, recheck only the on-CPU and migration-disabled states when proxy execution is enabled. If either changed, cancel the transfer and requeue the task on the global DSQ without treating the race as a BPF scheduler error.
A rejected task may be requeued onto the DSQ currently being consumed. Snapshot the DSQ sequence and skip tasks added after the consume attempt started to prevent repeatedly selecting the same task. Keep this retry bounding confined to proxy execution so the non-proxy path remains unchanged.
With proxy execution enabled, stress-ng --pipeherd can trigger this race and migrate an active execution context, leading to sleeping-while-atomic warnings and subsequent lockdep corruption.
--- kernel/sched/ext/ext.c | 79 ++++++++++++++++++++++++++++++++++++------ 1 file changed, 69 insertions(+), 10 deletions(-)
kernel/sched/ext/ext.c +69 -10
@@ -2372,8 +2372,10 @@ static void move_remote_task_to_local_dsq(struct scx_sched *sch,* - The BPF scheduler is bypassed while the rq is offline and we can always say * no to the BPF scheduler initiated migrations while offline. * -* The caller must ensure that @p and @rq are on different CPUs. -* If enforce == true, caller must hold @p's rq lock. +* The caller must ensure that @p and @rq are on different CPUs. If @enforce is +* true, report violations attributable to BPF-directed migrations. The caller +* must hold @p's rq lock to avoid reporting a transient race as a scheduler +* error. */ static bool task_can_run_on_remote_rq(struct scx_sched *sch, struct task_struct *p, struct rq *rq,
@@ -2381,11 +2383,6 @@ static bool task_can_run_on_remote_rq(struct scx_sched *sch, {s32 cpu =cpu_of (rq); -/* -* To prevent races with @p still running on its old CPU while switching -* out, make sure we're holding @p's rq lock so as not to risk -* erroneously killing the BPF scheduler. -*/ if (enforce)lockdep_assert_rq_held (task_rq (p));
@@ -2432,6 +2429,28 @@ static bool task_can_run_on_remote_rq(struct scx_sched *sch,return true; } +/* +* Proxy execution can change @p's execution and migration-disabled state +* without touching its DSQ entry or clearing holding_cpu. Check those states +* with @p's rq locked. Without proxy execution, the holding_cpu handshake is +* sufficient and this must not affect the existing migration path. +*/ +static bool task_can_move_from_locked_rq (struct task_struct *p) +{ +struct rq *src_rq =task_rq (p); + +lockdep_assert_rq_held (src_rq); + +if (!sched_proxy_exec ()) +return true; + +/* @p may be rq->curr under another task's proxy scheduling context. */ +if (task_on_cpu (src_rq, p)) +return false; + +return !is_migration_disabled (p); +} +/** * unlink_dsq_and_switch_rq_lock() - Unlink task and switch to its rq lock * @p: target task
@@ -2489,6 +2508,27 @@ static bool consume_remote_task(struct scx_sched *sch, struct rq *this_rq, struct scx_dispatch_q *dsq, struct rq *src_rq) { if (unlink_dsq_and_switch_rq_lock(p, dsq, this_rq, src_rq)) { +/* +* Proxy execution may have changed @p's running or migration-disabled +* state while switching rq locks without clearing holding_cpu. This is +* a kernel-side race, not an invalid BPF placement request, so don't +* abort the scheduler on failure. Fall back to the global DSQ, where +* normal consumption filters can select a valid destination. +*/ +if (unlikely (!task_can_move_from_locked_rq (p))) { + p->scx.dsq =NULL ; + p->scx.holding_cpu =-1 ; +scx_dispatch_enqueue (sch, src_rq, +find_global_dsq (sch,task_cpu (p)), p, + enq_flags |SCX_ENQ_CLEAR_OPSS | +SCX_ENQ_GDSQ_FALLBACK ); +if (sched_class_above (p->sched_class, + src_rq->donor->sched_class)) +resched_curr (src_rq); +switch_rq_lock (src_rq, this_rq); +return false; + } +move_remote_task_to_local_dsq (sch, p, enq_flags, src_rq, this_rq);return true; } else {
@@ -2527,7 +2567,8 @@ static struct rq *move_task_between_dsqs(struct scx_sched *sch,if (dst_dsq->id ==SCX_DSQ_LOCAL ) { dst_rq = container_of(dst_dsq, struct rq, scx.local_dsq);if (src_rq != dst_rq && -unlikely (!task_can_run_on_remote_rq (sch, p, dst_rq, true))) { +unlikely (!task_can_move_from_locked_rq (p) || + !task_can_run_on_remote_rq (sch, p, dst_rq, true))) { dst_dsq =find_global_dsq (sch,task_cpu (p)); dst_rq = src_rq; enq_flags |=SCX_ENQ_GDSQ_FALLBACK ;
@@ -2570,7 +2611,9 @@ bool scx_consume_dispatch_q(struct scx_sched *sch, struct rq *rq, struct scx_dispatch_q *dsq, u64 enq_flags) {struct task_struct *p; -retry: +bool proxy_exec =sched_proxy_exec (); +u32 scan_seq =0 ; +/* * The caller can't expect to successfully consume a task if the task's * addition to @dsq isn't guaranteed to be visible somehow. Test
@@ -2580,10 +2623,23 @@ bool scx_consume_dispatch_q(struct scx_sched *sch, struct rq *rq,return false;raw_spin_lock (&dsq->lock); +if (proxy_exec) + scan_seq =READ_ONCE (dsq->seq); +retry:nldsq_for_each_task (p, dsq) {struct rq *task_rq =task_rq (p); +/* +* consume_remote_task() may drop @dsq->lock and requeue a +* rejected task onto this same DSQ. Don't reconsider tasks queued +* after this consume attempt started. This bounds the retry loop +* even when the rejected task remains on-CPU. +*/ +if (proxy_exec && +unlikely (u32_before (scan_seq, p->scx.dsq_seq))) +continue ; +/* * This loop can lead to multiple lockup scenarios, e.g. the BPF * scheduler can put an enormous number of affinitized tasks into
@@ -2605,6 +2661,8 @@ bool scx_consume_dispatch_q(struct scx_sched *sch, struct rq *rq,if (task_can_run_on_remote_rq (sch, p, rq, false)) {if (likely (consume_remote_task (sch, rq, p, enq_flags, dsq, task_rq)))return true; +/* consume_remote_task() returns with @dsq unlocked. */ +raw_spin_lock (&dsq->lock); goto retry; } }
@@ -2690,7 +2748,8 @@ static void dispatch_to_local_dsq(struct scx_sched *sch, struct rq *rq, p->scx.holding_cpu =-1 ;scx_dispatch_enqueue (sch, dst_rq, &dst_rq->scx.local_dsq, p, enq_flags); - }else if (unlikely (!task_can_run_on_remote_rq (sch, p, dst_rq, true))) { + }else if (unlikely (!task_can_move_from_locked_rq (p) || + !task_can_run_on_remote_rq (sch, p, dst_rq, true))) { p->scx.holding_cpu =-1 ; fallback = true; scx_dispatch_enqueue(sch, src_rq, find_global_dsq(sch, task_cpu(p)),
Signature · 1 line
2.55.0
Re: [PATCH 06/12] sched_ext: Fix proxy-exec race in consume_remote_task()
Quoted 22 lines · Andrea Righi
Without proxy execution, the DSQ lock and
holding_cpuhandshake ensure that a task cannot be dequeued or start running during the rq-lock handoff without clearingholding_cpu.Proxy execution is an exception. A task can start physically executing as a lock owner while its own scheduling context remains on the DSQ. Its on-CPU or migration-disabled state can therefore change without clearing
holding_cpu.Keep the existing lockless destination validation unchanged. After switching to the source rq lock, recheck only the on-CPU and migration-disabled states when proxy execution is enabled. If either changed, cancel the transfer and requeue the task on the global DSQ without treating the race as a BPF scheduler error.
A rejected task may be requeued onto the DSQ currently being consumed. Snapshot the DSQ sequence and skip tasks added after the consume attempt started to prevent repeatedly selecting the same task. Keep this retry bounding confined to proxy execution so the non-proxy path remains unchanged.
Quoted 5 lines
With proxy execution enabled, stress-ng --pipeherd can trigger this race and migrate an active execution context, leading to sleeping-while-atomic warnings and subsequent lockdep corruption.
Signed-off-by Andrea Righi <[email protected]>
So, I've acked the core scheduler changes, but I'm not sure I know the sched_ext logic well enough for my ack to be very meaningful.
However, as a sign of general positivity the series, if you feel its useful, feel free to add "Acked-by: John Stultz <[email protected]>" to the entire series.
thanks
-john
[email protected]
AI reviewRe: [PATCH 06/12] sched_ext: Fix proxy-exec race in consume_remote_task()
In reply to Andrea RighiThank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] task_can_move_from_locked_rq() fails to check if the task is the active donor on the source…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] task_can_move_from_locked_rq() fails to check if the task is the active donor on the source runqueue under proxy execution. --
--- Patch [6]: [PATCH 06/12] sched_ext: Fix proxy-exec race in consume_remote_task() --- commit 681e47b7800ef6cd6756031dfb833542e391c7df
Author: Andrea Righi <[email protected]>
sched_ext: Fix proxy-exec race in consume_remote_task()
This patch addresses a proxy execution race where a task's state can change without clearing holding_cpu. It introduces task_can_move_from_locked_rq() to recheck the on-CPU and migration-disabled states after switching to the source runqueue lock. If the state changed, the task is requeued on the global DSQ instead of aborting the BPF scheduler.
Quoted 4 lines
kernel/sched/ext/ext.c +0
-0
[ ... ]
Quoted 13 lines
kernel/sched/ext/ext.c +9
-0@@ -2432,6 +2429,28 @@ static bool task_can_run_on_remote_rq(struct scx_sched *sch,return true; } +/* +* Proxy execution can change @p's execution and migration-disabled state +* without touching its DSQ entry or clearing holding_cpu. Check those states +* with @p's rq locked. Without proxy execution, the holding_cpu handshake is +* sufficient and this must not affect the existing migration path. +*/ +static bool task_can_move_from_locked_rq (struct task_struct *p) +{ +struct rq *src_rq =task_rq (p);
kernel/sched/ext/ext.c static bool task_can_run_on_remote_rq(struct scx_sched *sch, · line 2450
Quoted 10 lines
kernel/sched/ext/ext.c +10
-0+ +lockdep_assert_rq_held (src_rq); + +if (!sched_proxy_exec ()) +return true; + +/* @p may be rq->curr under another task's proxy scheduling context. */ +if (task_on_cpu (src_rq, p)) +return false; +
[Severity: High]
Is it possible for the active scheduling context (the donor) to be migrated here?
When a task blocks on a mutex and becomes a proxy donor, it is retained on the runqueue (p->on_rq == 1) without being dequeued. put_prev_task_scx() sees SCX_TASK_QUEUED and enqueues this blocked task onto a DSQ.
When another CPU attempts to consume from this DSQ, consume_remote_task() invokes this new task_can_move_from_locked_rq(). Since the donor is blocked, task_on_cpu(src_rq, p) evaluates to false.
Without a check for task_current_donor(src_rq, p), could this return true and allow move_remote_task_to_local_dsq() to migrate the active donor, corrupting the proxy execution PI chain?
Quoted 2 lines
kernel/sched/ext/ext.c +2
-0+return !is_migration_disabled (p); +}
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=6
Tejun Heo
reviewerRe: [PATCH 06/12] sched_ext: Fix proxy-exec race in consume_remote_task()
In reply to Andrea RighiHello, Andrea.
Quoted 10 lines · Andrea Righi
kernel/sched/ext/ext.c +10
-0+/* +* Proxy execution may have changed @p's running or migration-disabled +* state while switching rq locks without clearing holding_cpu. This is +* a kernel-side race, not an invalid BPF placement request, so don't +* abort the scheduler on failure. Fall back to the global DSQ, where +* normal consumption filters can select a valid destination. +*/ +if (unlikely (!task_can_move_from_locked_rq (p))) { + p->scx.dsq =NULL ; + p->scx.holding_cpu =-1 ;
kernel/sched/ext/ext.c static bool consume_remote_task(struct scx_sched *sch, struct rq *this_rq, · line 2530
Quoted 10 lines
kernel/sched/ext/ext.c +10
-0+scx_dispatch_enqueue (sch, src_rq, +find_global_dsq (sch,task_cpu (p)), p, + enq_flags |SCX_ENQ_CLEAR_OPSS | +SCX_ENQ_GDSQ_FALLBACK ); +if (sched_class_above (p->sched_class, + src_rq->donor->sched_class)) +resched_curr (src_rq); +switch_rq_lock (src_rq, this_rq); +return false; + }
I don't think bouncing the rejected task to the global DSQ is the right thing to do. Unlike the existing GDSQ_FALLBACK cases, this fires on plain mutex contention with proxy execution enabled, so userspace can trigger it routinely, and the bounce bypasses the BPF scheduler's placement. It also breaks sub-scheduler containment - when a host consumes the bypass DSQ for its bypassing descendants, @sch is the host and a descendant's task lands on the host's global DSQ outside its cap grants.
How about parking the rejected task on the reject DSQ and bouncing it back through the reenqueue path so that the owning scheduler re-places it? The two rejection reasons want different timings:
1. A migration-disabled donor can be reenqueued right away. The forced-admit path in scx_local_or_reject_dsq() covers pinned admission and the deferred irq_work wakes up an idle target CPU.
2. An on-CPU proxy owner should stay parked until it's switched out, or the reenqueue can just cycle back to rejection. This needs a callout at proxy resolution - the owner is still rq->curr when balance runs for the pick that switches it out, so draining from balance alone isn't enough.
That'd make the scan_seq tracking, the resched_curr() and the repeated consume attempts against an in-flight proxy section unnecessary. The reject DSQ and scx_reenq_reject() are currently sub-sched only and would need generalizing.
kernel/sched/ext/ext.c bool scx_consume_dispatch_q(struct scx_sched *sch, struct rq *rq, · line 2641
Quoted 9 lines
kernel/sched/ext/ext.c +9
-0+/* +* consume_remote_task() may drop @dsq->lock and requeue a +* rejected task onto this same DSQ. Don't reconsider tasks queued +* after this consume attempt started. This bounds the retry loop +* even when the rejected task remains on-CPU. +*/ +if (proxy_exec && +unlikely (u32_before (scan_seq, p->scx.dsq_seq))) +continue ;
If the requeue approach stays, the comment should mention when a task can get bounced back - proxy execution turning it on-CPU or migration-disabled while its context is still queued. Also, the condition fits on one line, ditto the sched_class_above() one above.
Thanks.
Signature · 1 line
tejun
Andrea Righi
reviewerPATCH 7/12 sched_ext: Split curr|donor references properly
In reply to Andrea RighiWith proxy execution, the task selected by the scheduler and the task physically executing can differ. A blocked mutex waiter donates its scheduling context to the lock owner:
D -----------------> M -------------> O ----------------> T [donor] blocked on [mutex] owned by [owner] preempted by [task]
\_________________________________^
donates scheduling context
where:
D = blocked donor
M = mutex
O = mutex owner
T = competing runnable task
During a proxy execution switch, D supplies the scheduling class, priority, and runtime budget, while O supplies the execution context: O is the task whose code physically executes. T is a competing runnable task which may preempt the D/O proxy execution.
Consider FAIR and EXT tasks with sched_ext running in partial mode. FAIR can be replaced with a higher scheduling class such as RT or deadline without changing the class interaction described here. The possible combinations are:
1. D is EXT, O is EXT, T is EXT
D can interrupt T according to BPF scheduling policy. O executes with D's EXT priority and runtime budget, while T waits in EXT.
2. D is EXT, O is EXT, T is FAIR
D is visible to the BPF scheduler, but cannot preempt T because EXT is below FAIR. Once T stops, BPF can dispatch D and O executes with D's EXT priority and runtime budget. If T becomes runnable again, it preempts the D/O proxy execution.
3. D is EXT, O is FAIR, T is EXT
This cannot represent T preempting O because EXT is below FAIR.
4. D is EXT, O is FAIR, T is FAIR
D cannot boost O above T because EXT is below FAIR. O and T continue competing under FAIR. Once O releases M, D wakes and resumes normal EXT scheduling.
5. D is FAIR, O is EXT, T is EXT
D preempts T as the higher-class scheduling context. O executes with D's FAIR priority and runtime budget, while T waits in EXT. D is not visible to the BPF scheduler.
6. D is FAIR, O is EXT, T is FAIR
D competes with T according to its FAIR deadline. When D is selected, O executes with D's FAIR priority and runtime budget. D is not visible to the BPF scheduler.
7. D is FAIR, O is FAIR, T is EXT
This cannot represent T preempting O because EXT is below FAIR.
8. D is FAIR, O is FAIR, T is FAIR
O, T, and D all have FAIR scheduling contexts. D remains runnable as a blocked proxy donor. When CFS selects D, O executes using D's FAIR scheduling context. When CFS selects O, O executes using its own FAIR context, and when CFS selects T, T executes normally. D is not visible to the BPF scheduler.
Thus, sched_ext policy and accounting must generally use rq->donor, the scheduler-selected task which supplies the scheduling context, rather than rq->curr, the task whose code physically executes. Without proxy execution they are the same task.
On nohz_full CPUs, a blocked proxy donor must retain the scheduler tick even when it has an infinite slice. Otherwise, a full dynticks CPU could stop the tick while rq->curr and rq->donor differ, violating assumptions made by the remote NOHZ tick path.
This is a conservative compromise that keeps the change local to sched_ext, at the cost of a periodic tick while a blocked proxy donor is selected. Allowing blocked proxy donors to run tickless would require making the core scheduler's remote tick handling aware that rq->curr and rq->donor can differ.
Moreover, extend scx_dump_state() to report both contexts. Each CPU record now includes a donor= line. If an EXT donor differs from rq->curr, also emit its detailed task record. The existing '*' marker continues to identify rq->curr, while the donor= line identifies the otherwise unmarked donor record.
Note that at this point in the series, CONFIG_SCHED_PROXY_EXEC still depends on !CONFIG_SCHED_CLASS_EXT, so proxy execution and sched_ext cannot be enabled together. The scheduling changes are therefore preparatory. A later patch removes this restriction.
--- Documentation/scheduler/sched-ext.rst | 6 ++ kernel/sched/ext/ext.c | 107 +++++++++++++++++--------- kernel/sched/ext/internal.h | 6 ++ kernel/sched/ext/sub.h | 8 +- 4 files changed, 87 insertions(+), 40 deletions(-)
Documentation/scheduler/sched-ext.rst +6 -0
@@ -487,6 +487,12 @@ and edge cases, to name a few examples: class, in which case it will exit the tick-dispatch loop even though it is runnable and has a non-zero slice. +* Under proxy execution, sched_ext continues to observe the donor as the current + scheduling context. A blocked donor does not enter an ``ops.running()`` / + ``ops.stopping()`` session because it does not execute itself, and the lock + owner executing on its behalf is intentionally not reported through these + callbacks. + See the "Scheduling Cycle" section for a more detailed description of how a freshly woken up task gets on a CPU.
kernel/sched/ext/ext.c +71 -36
274 lines not on this page · raw message
kernel/sched/ext/internal.h +6 -0
@@ -447,6 +447,12 @@ struct sched_ext_ops {* Therefore, always use scx_bpf_task_cpu(@p) to determine the * target CPU the task is going to use. * +* Under proxy execution, the BPF scheduler continues to observe the +* donor as the current scheduling context. A blocked donor does not +* enter a ->running()/->stopping() session because it does not execute +* itself, and the lock owner executing on its behalf is intentionally +* not reported through these callbacks. +* * See ->runnable() for explanation on the task state notifiers. */ void (*running)(struct task_struct *p);
kernel/sched/ext/sub.h +4 -4
@@ -141,14 +141,14 @@ static inline u64 scx_caps_for_task(struct task_struct *p)return SCX_CAP_ENQ ; } -/* the cap @sch needs to preempt @rq's current task, 0 if none */ +/* the cap @sch needs to preempt @rq's current scheduling context, 0 if none */ static inline u64 scx_caps_for_preempt (struct scx_sched *sch,struct rq *rq) { -struct task_struct *curr = rq->curr; +struct task_struct *donor = rq->donor;/* a non-ext task can't be preempted by ext, own-subtree needs no cap */ -if (curr->sched_class != &ext_sched_class || -scx_is_descendant (scx_task_sched (curr), sch)) +if (donor->sched_class != &ext_sched_class || +scx_is_descendant (scx_task_sched (donor), sch))return 0 ;return SCX_CAP_PREEMPT ; }
Signature · 1 line
2.55.0
[email protected]
AI reviewRe: [PATCH 07/12] sched_ext: Split curr|donor references properly
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [High] task_tick_scx() unconditionally invokes ops.tick() for blocked proxy donors, violating the BPF…
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [High] task_tick_scx() unconditionally invokes ops.tick() for blocked proxy donors, violating the BPF state machine contract.
- [Medium] Missing task_current_donor() check in enqueue_task_scx() violates DSQ invariants for proxy donors. --
--- Patch [7]: [PATCH 07/12] sched_ext: Split curr|donor references properly --- commit 0a1e9539e4e14034ca89a7b3b4988f413e2e416b
Author: Andrea Righi <[email protected]>
sched_ext: Split curr|donor references properly
This commit updates the sched_ext scheduler class to distinguish between the physical execution context and the scheduling context during proxy execution. It ensures that sched_ext policy, accounting, and various BPF kfuncs observe the donor task instead of the physically executing task.
Quoted 4 lines
kernel/sched/ext/ext.c +0
-0
[ ... ]
Quoted 14 lines
kernel/sched/ext/ext.c +2
-2@@ -3566,9 +3577,9 @@ void scx_tick(struct rq *rq)update_other_load_avgs (rq); } -static void task_tick_scx (struct rq *rq,struct task_struct *curr,int queued) +static void task_tick_scx (struct rq *rq,struct task_struct *donor,int queued) { -struct scx_sched *sch =scx_task_sched (curr); +struct scx_sched *sch =scx_task_sched (donor);update_curr_scx (rq);@@ -3577,13 +3588,13 @@ static void task_tick_scx(struct rq *rq, struct task_struct *curr, int queued)* we can't trust the slice management or ops.core_sched_before().
kernel/sched/ext/ext.c static void task_tick_scx(struct rq *rq, struct task_struct *curr, int queued) · line 3595
Quoted 10 lines
kernel/sched/ext/ext.c +3
-3*/ if (scx_bypassing (sch,cpu_of (rq))) { -set_task_slice (curr,0 ); -touch_core_sched (rq, curr); +set_task_slice (donor,0 ); +touch_core_sched (rq, donor); }else if (SCX_HAS_OP (sch, tick)) { -SCX_CALL_OP_TASK (sch, tick, rq, curr); +SCX_CALL_OP_TASK (sch, tick, rq, donor); }
[Severity: High]
Does this code incorrectly call ops.tick() for a blocked donor?
The commit message explicitly states that a blocked donor does not enter an ops.running() or ops.stopping() session. However, this unconditionally invokes ops.tick() for the donor without verifying if it is blocked or tracked as running.
Since BPF schedulers may allocate per-CPU state upon ops.running(), could this crash or leak state when ops.tick() fires for a task the scheduler never tracked as running?
[ ... ]
Quoted 9 lines
kernel/sched/ext/ext.c +3
-2@@ -9172,13 +9195,15 @@ __bpf_kfunc bool scx_bpf_task_set_slice(struct task_struct *p, u64 slice,return false;/* -* Directly write only when we hold the lock of the rq @p is queued or -* running on. See the slice write rules above. +* Directly write only when we hold the lock of the rq @p is queued on or +* provides the current scheduling context for. Under proxy execution, +* rq->donor owns and consumes the slice while rq->curr executes on its
kernel/sched/ext/ext.c __bpf_kfunc bool scx_bpf_task_set_slice(struct task_struct *p, u64 slice, · line 9209
Quoted 10 lines
kernel/sched/ext/ext.c +2
-1+* behalf. See the slice write rules above. */ locked_rq =scx_locked_rq ();if (!locked_rq || (READ_ONCE (p->scx.runnable_cpu) !=cpu_of (locked_rq) && - !task_current (locked_rq, p))) { + !task_current_donor (locked_rq, p))) {set_task_slice_oob (sch, p, slice);return true; }
This patch updates several task_current() checks to task_current_donor(), but it appears one was missed in enqueue_task_scx():
kernel/sched/ext/ext.c:enqueue_task_scx() {
...
/*
* Restoring a running task will be immediately followed by
* set_next_task_scx() which expects the task to not be on the BPF
* scheduler as tasks can only start running through local DSQs. Force
* direct-dispatch into the local DSQ by setting the sticky_cpu. Mark
* IGNORE_CAPS to force entry into the local DSQ.
*/
if (unlikely(enq_flags & ENQUEUE_RESTORE) && task_current(rq, p)) {
sticky_cpu = cpu_of(rq);
enq_flags |= SCX_ENQ_IGNORE_CAPS;
}
...
}[Severity: Medium]
Should this use task_current_donor(rq, p) instead of task_current(rq, p)?
If a blocked proxy donor undergoes priority inheritance, it could be dequeued and re-enqueued with ENQUEUE_RESTORE. Since the donor is not physically rq->curr, task_current(rq, p) evaluates to false.
Would this bypass the local DSQ assignment and erroneously push the active proxy donor into the BPF scheduler instead of the local DSQ?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=7
Andrea Righi
reviewerPATCH 8/12 sched_ext: Handle blocked donor migration with proxy execution
In reply to Andrea RighiFrom: John Stultz <[email protected]>
With proxy execution enabled, mutex-blocked donors stay runnable so their scheduling context can execute the lock owner.
sched_ext may normally relocate a queued blocked donor: set_task_cpu() updates both task_cpu() and wake_cpu, making the destination rq the donor's new callback home. This remains valid if the donor was previously proxy-migrated: the BPF scheduler is intentionally selecting a new return destination, which the next proxy migration will preserve. The usual migration-disabled, affinity, and rq-online checks apply.
An active donor cannot be moved normally because the source rq still references it through rq->donor for scheduling-class operations and runtime accounting. Moving it would associate the task with a different rq while leaving those source-rq references in place. The core proxy migration path avoids this by switching the rq donor to idle before moving the scheduling context.
Check active-donor state in task_can_move_from_locked_rq(), where the source rq is held and the result is authoritative. Keep it out of the preliminary lockless DSQ filter, so a donor transition during the rq-lock handoff is resolved by the locked recheck rather than consuming a dispatch kick and skipping the task.
Allow normal sched_ext migration of blocked donors unless the donor is active on its rq. In that case, defer migration until it is switched out or leave it to the proxy machinery.
Keep blocked donors on the local DSQ so they remain visible to the proxy pick path. This is a temporary solution: in a later commit, proxy donor admission will be delegated to the BPF scheduler and donors will instead be routed through ops.enqueue().
interactive rebase in progress; onto c0414e1387386
--- kernel/sched/ext/ext.c | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+)
kernel/sched/ext/ext.c +23 -0
@@ -2455,6 +2455,10 @@ static bool task_can_move_from_locked_rq(struct task_struct *p)if (task_on_cpu (src_rq, p))return false; +/* Don't move the current scheduling context off its source rq. */ +if (task_current_donor (src_rq, p)) +return false; +return !is_migration_disabled (p); }
@@ -3122,6 +3126,25 @@ static void put_prev_task_scx(struct rq *rq, struct task_struct *p,if (p->scx.flags &SCX_TASK_QUEUED ) {set_task_runnable (rq, p); +/* +* Mutex-blocked donors stay queued on the runqueue under proxy +* execution, but the donor never runs as itself, proxy-exec +* walks the blocked_on chain on the next __schedule() and runs +* the lock owner in its place. +* +* Put the donor on the local DSQ directly so pick_next_task() +* can still see it. find_proxy_task() will either run the chain +* owner or deactivate the donor so the wakeup path can return it +* and let BPF make a new dispatch decision once it is unblocked. +* +* This is preparatory code: a later patch will delegate blocked-donor +* admission to the BPF scheduler. +*/ +if (p->is_blocked) { +scx_dispatch_enqueue (sch, rq, &rq->scx.local_dsq, p,0 ); + goto switch_class; + } +/* * If @p has slice left and is being put, @p is getting * preempted by a higher priority scheduler class or core-sched
Signature · 1 line
2.55.0
[email protected]
AI reviewRe: [PATCH 08/12] sched_ext: Handle blocked donor migration with proxy execution
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [High] The newly added proxy-execution handling for blocked donors in put_prev_task_scx() is unreachable…
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [High] The newly added proxy-execution handling for blocked donors in put_prev_task_scx() is unreachable for sched_ext tasks, completely breaking proxy execution for this class.
- [Medium] Blocked proxy donors are enqueued to the tail of the local DSQ instead of the head, causing priority inversion when other tasks are batched. --
--- Patch [8]: [PATCH 08/12] sched_ext: Handle blocked donor migration with proxy execution --- commit 6380b59c7e35fb635b7bc11e5fa121a8296ed2b7
Author: John Stultz <[email protected]>
sched_ext: Handle blocked donor migration with proxy execution
With proxy execution enabled, mutex-blocked donors stay runnable so their scheduling context can execute the lock owner.
Quoted 4 lines
kernel/sched/ext/ext.c +0
-0
[ ... ]
Quoted 10 lines
kernel/sched/ext/ext.c +6
-0@@ -3122,6 +3126,25 @@ static void put_prev_task_scx(struct rq *rq, struct task_struct *p,if (p->scx.flags &SCX_TASK_QUEUED ) {set_task_runnable (rq, p); +/* +* Mutex-blocked donors stay queued on the runqueue under proxy +* execution, but the donor never runs as itself, proxy-exec +* walks the blocked_on chain on the next __schedule() and runs +* the lock owner in its place. +*
kernel/sched/ext/ext.c static void put_prev_task_scx(struct rq *rq, struct task_struct *p, · line 3144
Quoted 10 lines
kernel/sched/ext/ext.c +10
-0+* Put the donor on the local DSQ directly so pick_next_task() +* can still see it. find_proxy_task() will either run the chain +* owner or deactivate the donor so the wakeup path can return it +* and let BPF make a new dispatch decision once it is unblocked. +* +* This is preparatory code: a later patch will delegate blocked-donor +* admission to the BPF scheduler. +*/ +if (p->is_blocked) { +scx_dispatch_enqueue (sch, rq, &rq->scx.local_dsq, p,0 );
[Severity: High]
Is this newly added proxy execution handling for blocked donors reachable for sched_ext tasks?
When a sched_ext task blocks on a mutex, try_to_block_task() checks !task_is_blocked(prev) || !scx_allow_proxy_exec(prev) to decide if it should be fully dequeued. scx_allow_proxy_exec() unconditionally returns false for ext_sched_class:
kernel/sched/ext/ext.c:scx_allow_proxy_exec() {
return p->sched_class != &ext_sched_class;
}Since should_block evaluates to true, the task is passed to block_task(), which unconditionally dequeues it via dequeue_task_scx(). Dequeuing clears SCX_TASK_QUEUED.
When put_prev_task_scx() is then called by __schedule() during the task switch, the task is no longer queued (p->scx.flags & SCX_TASK_QUEUED is false), meaning this entire block of newly added proxy donor retention logic seems to be bypassed as dead code.
[Severity: Medium]
Will enqueuing the blocked proxy donor to the tail of the local DSQ cause a priority inversion?
If the BPF scheduler has previously dispatched a batch of tasks to the local DSQ, using 0 (tail enqueue) places the blocked donor behind them. pick_next_task() will pop and execute the batched tasks before the donor, delaying the execution of the lock owner.
Should this use SCX_ENQ_HEAD instead, similar to slice-preempted tasks, so the lock owner can execute immediately to release the contended mutex?
Quoted 6 lines
kernel/sched/ext/ext.c +3
-0+ goto switch_class; + } +/* * If @p has slice left and is being put, @p is getting * preempted by a higher priority scheduler class or core-sched
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=8
Andrea Righi
reviewerPATCH 9/12 sched_ext: Delegate proxy donor admission to BPF schedulers
In reply to Andrea RighiProxy execution keeps a blocked donor runnable so its scheduling context can execute the mutex owner. Dispatching sched_ext donors on a local DSQ bypasses the BPF scheduler ordering policy and can give donors more CPU priority than intended to perform the proxy execution handoff.
Add SCX_OPS_ENQ_BLOCKED as an explicit proxy execution capability. Tasks owned by schedulers without the flag block normally. Schedulers with the flag receive blocked donors through ops.enqueue() with SCX_ENQ_BLOCKED set in enq_flags and can apply their own admission policy.
From a high-level perspective, the resulting flow for a BPF scheduler with SCX_OPS_ENQ_BLOCKED is:
D ------ blocked on -----> M ------ owned by -----> O
[donor] [mutex] [owner]
|
| ops.enqueue(D, SCX_ENQ_BLOCKED)
| BPF dispatches D to CPUi
v
+-----------------+
| CPUi local DSQ |
+-------+---------+
|
| pick_next_task() selects D
v
+-----------------+
| proxy exec | move D to O's rq
+-------+---------+
|
| run O using D's scheduling context
v
rq->curr = O
rq->donor = D
|
| O releases M
v
+-----------------+
| proxy exec | return D to CPUi via wakeup
+-----------------+A proxy migration preserves the donor's wake_cpu while moving its scheduling context to the owner's rq, so task_cpu() differs from wake_cpu until the donor wakes. Prevent BPF-directed migrations from pulling a blocked donor off this proxy rq. Otherwise each blocked re-enqueue may move the donor back to its callback rq only for proxy execution to move it to the owner again.
Scheduler ownership can change after a donor has already blocked. Since sched_change preserves queued state, handle both iterator-driven moves to root or sub-schedulers and sched_setscheduler() transitions into EXT. Before entering a scheduler without the flag, deactivate the retained donor. It remains on the mutex wait path and wakes normally when the mutex becomes available.
The global sched_ext enable state can change while __schedule() holds the runqueue lock. Once an EXT task has an assigned scheduler, consult it directly so the task cannot retain a donor during the transition unless the scheduler opted in. Tasks without an assigned scheduler keep the generic proxy execution behavior.
The preparation helpers require and assert that p->pi_lock and p's rq lock are held before updating rq state.
The donor starts associated with its original CPU. A BPF scheduler may dispatch it directly into that CPU local DSQ to let the core resolve the mutex owner and execute it with the donor scheduling context.
A normal sleeper also has p->is_blocked set until after wakeup_preempt(). Exclude full wakeup activations from SCX_ENQ_BLOCKED and carry a transient task flag from enqueue_task_scx() to wakeup_preempt_scx(). Retained donor wakeups skip enqueue_task_scx(), so they remain distinguishable without changing core scheduler wake flags.
Reschedule a retained donor when its mutex wakes it so ops.dispatch() can reconsider the now-unblocked task. Make SCX_OPS_ENQ_BLOCKED override the exiting and migration-disabled enqueue fallbacks so opted-in schedulers receive all eligible donor requests.
--- include/linux/sched/ext.h | 1 + kernel/sched/ext/ext.c | 122 ++++++++++++++---- kernel/sched/ext/internal.h | 23 +++- kernel/sched/ext/sub.c | 8 +- tools/sched_ext/include/scx/compat.h | 1 + .../sched_ext/include/scx/enum_defs.autogen.h | 1 + .../sched_ext/include/scx/enums.autogen.bpf.h | 3 + tools/sched_ext/include/scx/enums.autogen.h | 1 + 8 files changed, 134 insertions(+), 26 deletions(-)
include/linux/sched/ext.h +1 -0
@@ -104,6 +104,7 @@ enum scx_ent_flags {SCX_TASK_IMMED =1 <<5 ,/* task is on local DSQ with %SCX_ENQ_IMMED */ SCX_TASK_RUN_TRACKED =1 <<6 ,/* task is in an ops.running()/stopping() session */ + SCX_TASK_ENQ_WAKEUP = 1 << 7, /* wakeup enqueue awaiting wakeup_preempt() *//* * Bits 8 to 10 are used to carry task state:
kernel/sched/ext/ext.c +99 -23
218 lines not on this page · raw message
kernel/sched/ext/internal.h +22 -1
@@ -213,6 +213,19 @@ enum scx_ops_flags {*/ SCX_OPS_TID_TO_TASK =1LLU <<8 , +/* +* If set, mutex-blocked tasks remain runnable as proxy donors and are +* passed to ops.enqueue() with %SCX_ENQ_BLOCKED. The BPF scheduler controls +* when donors are dispatched and whether they should preempt other work. +* +* If clear, mutex-blocked tasks are removed from the runqueue normally +* and cannot donate their scheduling context through proxy execution. +* +* For blocked donors, this flag takes precedence over +* %SCX_OPS_ENQ_EXITING and %SCX_OPS_ENQ_MIGRATION_DISABLED. +*/ +SCX_OPS_ENQ_BLOCKED =1LLU <<9 , +SCX_OPS_ALL_FLAGS =SCX_OPS_KEEP_BUILTIN_IDLE |SCX_OPS_ENQ_LAST | SCX_OPS_ENQ_EXITING |
@@ -221,7 +234,8 @@ enum scx_ops_flags {SCX_OPS_SWITCH_PARTIAL |SCX_OPS_BUILTIN_IDLE_PER_NODE |SCX_OPS_ALWAYS_ENQ_IMMED | -SCX_OPS_TID_TO_TASK , +SCX_OPS_TID_TO_TASK | +SCX_OPS_ENQ_BLOCKED ,/* high 8 bits are internal, don't include in SCX_OPS_ALL_FLAGS */ __SCX_OPS_INTERNAL_MASK = 0xffLLU << 56,
@@ -1650,6 +1664,12 @@ enum scx_enq_flags {*/ SCX_ENQ_LAST =1LLU <<41 , +/* +* The task is blocked on a mutex and is being kept runnable as a proxy +* donor. Only passed to ops.enqueue() when %SCX_OPS_ENQ_BLOCKED is set. +*/ +SCX_ENQ_BLOCKED =1LLU <<42 , +/* high 8 bits are internal */ __SCX_ENQ_INTERNAL_MASK = 0xffLLU << 56,
@@ -1946,6 +1966,7 @@ void scx_task_iter_start(struct scx_task_iter *iter, struct cgroup *cgrp);void scx_task_iter_unlock (struct scx_task_iter *iter);void scx_task_iter_stop (struct scx_task_iter *iter);struct task_struct *scx_task_iter_next_locked (struct scx_task_iter *iter); +void scx_prepare_task_sched_change (struct task_struct *p,struct scx_sched *sch);void scx_dispatch_dequeue (struct rq *rq,struct task_struct *p);void scx_do_enqueue_task (struct rq *rq,struct task_struct *p,u64 enq_flags,int sticky_cpu);
kernel/sched/ext/sub.c +6 -2
@@ -773,7 +773,9 @@ static void scx_rehome_task(struct scx_sched *to, struct task_struct *p)lockdep_assert_held (&p->pi_lock);lockdep_assert_rq_held (task_rq (p)); - scoped_guard (sched_change, p, DEQUEUE_SAVE | DEQUEUE_MOVE) { +scx_prepare_task_sched_change (p, to); +scoped_guard (sched_change, p,DEQUEUE_SAVE |DEQUEUE_MOVE | + DEQUEUE_NOCLOCK) {scx_disable_and_exit_task (scx_task_sched (p), p);scx_set_task_state (p,SCX_TASK_INIT_BEGIN );scx_set_task_state (p,SCX_TASK_INIT );
@@ -803,7 +805,9 @@ static void scx_punt_task(struct scx_sched *to, struct task_struct *p)lockdep_assert_rq_held (task_rq (p));WARN_ON_ONCE (!READ_ONCE (to->bypass_depth)); - scoped_guard (sched_change, p, DEQUEUE_SAVE | DEQUEUE_MOVE) { +scx_prepare_task_sched_change (p, to); +scoped_guard (sched_change, p,DEQUEUE_SAVE |DEQUEUE_MOVE | + DEQUEUE_NOCLOCK) {scx_disable_and_exit_task (scx_task_sched (p), p);scx_set_task_sched (p, to); }
tools/sched_ext/include/scx/compat.h +1 -0
@@ -117,6 +117,7 @@ static inline bool __COMPAT_struct_has_field(const char *type, const char *field#define SCX_OPS_ALLOW_QUEUED_WAKEUP SCX_OPS_FLAG(SCX_OPS_ALLOW_QUEUED_WAKEUP)#define SCX_OPS_BUILTIN_IDLE_PER_NODE SCX_OPS_FLAG(SCX_OPS_BUILTIN_IDLE_PER_NODE)#define SCX_OPS_ALWAYS_ENQ_IMMED SCX_OPS_FLAG(SCX_OPS_ALWAYS_ENQ_IMMED) +#define SCX_OPS_ENQ_BLOCKED SCX_OPS_FLAG(SCX_OPS_ENQ_BLOCKED)#define SCX_PICK_IDLE_FLAG (name) __COMPAT_ENUM_OR_ZERO("scx_pick_idle_cpu_flags", #name)
tools/sched_ext/include/scx/enum_defs.autogen.h +1 -0
@@ -55,6 +55,7 @@#define HAVE_SCX_ENQ_IMMED #define HAVE_SCX_ENQ_REENQ #define HAVE_SCX_ENQ_LAST +#define HAVE_SCX_ENQ_BLOCKED #define HAVE___SCX_ENQ_INTERNAL_MASK #define HAVE_SCX_ENQ_CLEAR_OPSS #define HAVE_SCX_ENQ_DSQ_PRIQ
tools/sched_ext/include/scx/enums.autogen.bpf.h +3 -0
@@ -130,6 +130,9 @@ const volatile u64 __SCX_ENQ_REENQ __weak;const volatile u64 __SCX_ENQ_LAST__weak ;#define SCX_ENQ_LAST __SCX_ENQ_LAST +const volatile u64 __SCX_ENQ_BLOCKED__weak ; +#define SCX_ENQ_BLOCKED __SCX_ENQ_BLOCKED +const volatile u64 __SCX_ENQ_CLEAR_OPSS__weak ;#define SCX_ENQ_CLEAR_OPSS __SCX_ENQ_CLEAR_OPSS
tools/sched_ext/include/scx/enums.autogen.h +1 -0
@@ -47,6 +47,7 @@SCX_ENUM_SET (skel, scx_enq_flags,SCX_ENQ_IMMED ); \SCX_ENUM_SET (skel, scx_enq_flags,SCX_ENQ_REENQ ); \SCX_ENUM_SET (skel, scx_enq_flags,SCX_ENQ_LAST ); \ +SCX_ENUM_SET (skel, scx_enq_flags,SCX_ENQ_BLOCKED ); \SCX_ENUM_SET (skel, scx_enq_flags,SCX_ENQ_CLEAR_OPSS ); \SCX_ENUM_SET (skel, scx_enq_flags,SCX_ENQ_DSQ_PRIQ ); \SCX_ENUM_SET (skel, scx_deq_flags,SCX_DEQ_SCHED_CHANGE ); \
Signature · 1 line
2.55.0
Tejun Heo
reviewerRe: [PATCH 09/12] sched_ext: Delegate proxy donor admission to BPF schedulers
Hello, Andrea.
Quoted 6 lines · Andrea Righi
A proxy migration preserves the donor's
wake_cpuwhile moving its scheduling context to the owner's rq, sotask_cpu()differs fromwake_cpuuntil the donor wakes. Prevent BPF-directed migrations from pulling a blocked donor off this proxy rq. Otherwise each blocked re-enqueue may move the donor back to its callback rq only for proxy execution to move it to the owner again.
The description reads scattered - from here on, it's a flat list of corner cases, one per hunk, and the design that connects them never gets stated.
Quoted 12 lines
kernel/sched/ext/ext.c +7
-1+/* %SCX_OPS_ENQ_BLOCKED takes precedence over the fallbacks below. */ + enq_blocked = (sch->ops.flags &SCX_OPS_ENQ_BLOCKED ) && + p->is_blocked && !(enq_flags &SCX_ENQ_WAKEUP ); +if (enq_blocked) + enq_flags |=SCX_ENQ_BLOCKED ; +/* see %SCX_OPS_ENQ_EXITING */ -if (!(sch->ops.flags &SCX_OPS_ENQ_EXITING ) && +if (!enq_blocked && !(sch->ops.flags &SCX_OPS_ENQ_EXITING ) &&unlikely (p->flags &PF_EXITING )) {__scx_add_event (sch,SCX_EV_ENQ_SKIP_EXITING ,1 ); goto local;
kernel/sched/ext/ext.c void scx_do_enqueue_task(struct rq *rq, struct task_struct *p, u64 enq_flags,
Quoted 10 lines
kernel/sched/ext/ext.c +2
-1}/* see %SCX_OPS_ENQ_MIGRATION_DISABLED */ -if (!(sch->ops.flags &SCX_OPS_ENQ_MIGRATION_DISABLED ) && +if (!enq_blocked && + !(sch->ops.flags &SCX_OPS_ENQ_MIGRATION_DISABLED ) &&is_migration_disabled (p)) {__scx_add_event (sch,SCX_EV_ENQ_SKIP_MIGRATION_DISABLED ,1 ); goto local; }
Both fallbacks repeating !enq_blocked reads awkward. How about grouping them?
if (enq_blocked) {
enq_flags |= SCX_ENQ_BLOCKED;
} else {
/* see %SCX_OPS_ENQ_EXITING */
if (!(sch->ops.flags & SCX_OPS_ENQ_EXITING) &&
unlikely(p->flags & PF_EXITING)) {
...
}
/* see %SCX_OPS_ENQ_MIGRATION_DISABLED */
if (!(sch->ops.flags & SCX_OPS_ENQ_MIGRATION_DISABLED) &&
is_migration_disabled(p)) {
...
}
}The "takes precedence" comment can then go away as the structure says it.
Thanks.
Signature · 1 line
tejun
Andrea Righi
reviewerPATCH 10/12 sched_ext: Add selftest for blocked donor admission
In reply to Andrea RighiSCX_OPS_ENQ_BLOCKED allows BPF schedulers to receive blocked proxy donors through ops.enqueue(). SCX_ENQ_BLOCKED identifies blocked-donor admission requests. Add selftest coverage for this interface.
Exercise a priority inversion using a weighted-vruntime BPF scheduler. A nice +19 owner holds a shared mutex, a nice -20 donor blocks on it, and nice 0 CPU contenders, one per allowed CPU, keep the system busy. Test both a same-CPU topology and a cross-CPU topology with the donor and owner on different CPUs.
Treat blocked donors according to the normal BPF ordering policy and assign the default slice on every enqueue, as for other tasks. Run each CPU placement configuration with SCX_OPS_ENQ_BLOCKED first disabled and then enabled, count blocked-donor enqueues by CPU and report average mutex hold and wait times. Verify that full wakeups are never reported as blocked-donor admissions by checking that SCX_ENQ_WAKEUP and SCX_ENQ_BLOCKED are not set together.
Proxy execution coverage requires CONFIG_SCHED_PROXY_EXEC=y, which the selftest config selects. Access to the kernel mutex is provided via a loadable kernel module, built through TEST_GEN_MODS_DIR and managed by the test.
--- tools/testing/selftests/sched_ext/.gitignore | 4 + tools/testing/selftests/sched_ext/Makefile | 2 + tools/testing/selftests/sched_ext/config | 2 + .../selftests/sched_ext/enq_blocked.bpf.c | 116 +++ .../testing/selftests/sched_ext/enq_blocked.c | 918 ++++++++++++++++++ .../testing/selftests/sched_ext/enq_blocked.h | 28 + .../selftests/sched_ext/test_modules/Makefile | 13 + .../test_modules/scx_enq_blocked_test.c | 195 ++++ 8 files changed, 1278 insertions(+) create mode 100644 tools/testing/selftests/sched_ext/enq_blocked.bpf.c create mode 100644 tools/testing/selftests/sched_ext/enq_blocked.c create mode 100644 tools/testing/selftests/sched_ext/enq_blocked.h create mode 100644 tools/testing/selftests/sched_ext/test_modules/Makefile create mode 100644 tools/testing/selftests/sched_ext/test_modules/scx_enq_blocked_test.c
tools/testing/selftests/sched_ext/.gitignore +4 -0
@@ -4,3 +4,7 @@ !Makefile !.gitignore !config +!test_modules/ +!test_modules/scx_enq_blocked_test.c +!test_modules/Makefile +test_modules/*.mod.c
tools/testing/selftests/sched_ext/Makefile +2 -0
@@ -5,6 +5,7 @@ include ../../../scripts/Makefile.arch include ../../../scripts/Makefile.include TEST_GEN_PROGS := runner +TEST_GEN_MODS_DIR := test_modules # override lib.mk's default rules OVERRIDE_TARGETS := 1
@@ -164,6 +165,7 @@ all_test_bpfprogs := $(foreach prog,$(wildcard *.bpf.c),$(INCLUDE_DIR)/$(patsubs auto-test-targets := \ create_dsq \ dequeue \ + enq_blocked \ enq_last_no_enq_fails \ ddsp_bogus_dsq_fail \ ddsp_vtimelocal_fail \
tools/testing/selftests/sched_ext/config +2 -0
@@ -6,3 +6,5 @@ CONFIG_BPF=y CONFIG_BPF_SYSCALL=y CONFIG_DEBUG_INFO=y CONFIG_DEBUG_INFO_BTF=y +CONFIG_EXPERT=y +CONFIG_SCHED_PROXY_EXEC=y
tools/testing/selftests/sched_ext/enq_blocked.bpf.c new file +116 -0
116 lines not on this page · raw message
tools/testing/selftests/sched_ext/enq_blocked.c new file +918 -0
918 lines not on this page · raw message
tools/testing/selftests/sched_ext/enq_blocked.h new file +28 -0
@@ -0,0 +1,28 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES */ +#ifndef __ENQ_BLOCKED_H +#define __ENQ_BLOCKED_H + +#include <linux/ioctl.h> +#include <linux/types.h> + +struct enq_blocked_stats { +__u64 hold_time_ns; +__u64 wait_time_ns; +__u64 nr_holds; +__u64 nr_waits; +}; + +#define ENQ_BLOCKED_IOCTL_OWNER _IO('s', 1) +#define ENQ_BLOCKED_IOCTL_DONOR _IO('s', 2) +#define ENQ_BLOCKED_IOCTL_RESET_STATS _IO('s', 3) +#define ENQ_BLOCKED_IOCTL_GET_STATS _IOR('s', 4, struct enq_blocked_stats) +#define ENQ_BLOCKED_IOCTL_PREP_ATTACH _IO('s', 5) +#define ENQ_BLOCKED_IOCTL_ATTACH_DONE _IO('s', 6) +#define ENQ_BLOCKED_IOCTL_DONOR_STATE _IO('s', 7) +#define ENQ_BLOCKED_IOCTL_PROXY_SUPPORTED _IO('s', 8) + +#define ENQ_BLOCKED_DONOR_SLEEPING (1U << 0) +#define ENQ_BLOCKED_DONOR_ON_RQ (1U << 1) + +#endif /* __ENQ_BLOCKED_H */
tools/testing/selftests/sched_ext/test_modules/Makefile new file +13 -0
@@ -0,0 +1,13 @@ +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES + +TESTMODS_DIR := $(realpath $(dir $(abspath $(lastword $(MAKEFILE_LIST))))) +KDIR ?= $(if $(O),$(O),$(realpath ../../../../..)) + +obj-m += scx_enq_blocked_test.o + +all: + +$(Q)$(MAKE) -C $(KDIR) M=$(TESTMODS_DIR) modules + +clean: + +$(Q)$(MAKE) -C $(KDIR) M=$(TESTMODS_DIR) clean
tools/testing/selftests/sched_ext/test_modules/scx_enq_blocked_test.c new file +195 -0
195 lines not on this page · raw message
Signature · 1 line
2.55.0
Andrea Righi
reviewerPATCH 11/12 sched_ext: scx_qmap: Add proxy execution support
In reply to Andrea RighiAdd a -B option to opt scx_qmap into queueing mutex-blocked tasks for proxy execution. Without the option, SCX_OPS_ENQ_BLOCKED remains clear and mutex waiters block normally. With -B, blocked donors are passed to qmap_enqueue() with SCX_ENQ_BLOCKED.
When scx_qmap receives a blocked donor, dispatch it directly to the local DSQ of its current cid with a fresh slice and SCX_ENQ_PREEMPT. This places the donor at the head of the DSQ and requests an immediate reschedule, allowing the core proxy-exec path to run the mutex owner using the donor's scheduling context as soon as the donor is selected.
The blocked policy is intentionally unfair and can strongly prioritize tasks using contended mutexes, but scx_qmap is a demo scheduler and such aggressive behavior makes proxy-exec support easy to observe. Count these dispatch attempts in nr_enq_blocked and report their per-interval delta.
--- tools/sched_ext/scx_qmap.bpf.c | 34 ++++++++++++++++++++++++++++++++++ tools/sched_ext/scx_qmap.c | 13 ++++++++++--- tools/sched_ext/scx_qmap.h | 1 + 3 files changed, 45 insertions(+), 3 deletions(-)
tools/sched_ext/scx_qmap.bpf.c +34 -0
@@ -451,6 +451,40 @@ void BPF_STRUCT_OPS(qmap_enqueue, struct task_struct *p, u64 enq_flags)*/ taskc->core_sched_seq = qa.core_sched_tail_seqs[idx]++; +/* +* A blocked donor rejected by a local DSQ must not be dispatched back +* to the same DSQ immediately. Move it to the global DSQ so another CPU +* can pick it up and kick an idle cid to accelerate its consumption. +*/ +if ((enq_flags & (SCX_ENQ_BLOCKED |SCX_ENQ_REENQ )) == + (SCX_ENQ_BLOCKED |SCX_ENQ_REENQ )) { + taskc->force_local = false; +scx_bpf_dsq_insert (p,SHARED_DSQ ,0 , enq_flags); + cid =cmask_next_and2_set_wrap (&taskc->cpus_allowed, + &qa.idle_cids.mask, + &qa.self_cids.mask,0 ); +if (cid <scx_bpf_nr_cids ()) +scx_bpf_kick_cid (cid,SCX_KICK_IDLE ); +return ; + } + +/* +* Insert a blocked mutex donor at the head of its current cid's local +* DSQ with a fresh slice and %SCX_ENQ_PREEMPT, requesting an immediate +* reschedule. Once selected, the core proxy-exec path can immediately +* run the mutex owner using the donor's scheduling context. +* +* This policy is intentionally unfair and can strongly prioritize tasks +* using contended mutexes; scx_qmap is a demonstration scheduler and +* this behavior makes proxy-exec support easy to observe. +*/ +if (enq_flags &SCX_ENQ_BLOCKED ) { +__sync_fetch_and_add (&qa.nr_enq_blocked,1 ); +scx_bpf_dsq_insert (p,SCX_DSQ_LOCAL_ON |scx_bpf_task_cid (p), + slice_ns, enq_flags |SCX_ENQ_PREEMPT ); +return ; + } +/* * A node with children delegates most cids. A task of ours that can run * on none of our self cids (e.g. a per-NUMA kthread pinned to delegated
tools/sched_ext/scx_qmap.c +10 -3
@@ -46,7 +46,7 @@ const char help_fmt[] ="See the top-of-file comment in .bpf.c for the design.\n" "\n" "Usage: %s [-s SLICE_US] [-e COUNT] [-t COUNT] [-T COUNT] [-l COUNT] [-b COUNT]\n" -" [-N COUNT] [-P] [-M] [-H] [-c CG_PATH] [-d PID] [-D LEN] [-S] [-p] [-I]\n" +" [-N COUNT] [-P] [-M] [-H] [-c CG_PATH] [-d PID] [-D LEN] [-S] [-p] [-I] [-B]\n" " [-F COUNT] [-i SEC] [-R MS] [-J MODE] [-v]\n" "\n" " -s SLICE_US Override slice duration\n"
@@ -65,6 +65,7 @@ const char help_fmt[] =" -S Suppress qmap-specific debug dump\n" " -p Switch only tasks on SCHED_EXT policy instead of all\n" " -I Turn on SCX_OPS_ALWAYS_ENQ_IMMED\n" +" -B Turn on SCX_OPS_ENQ_BLOCKED\n" " -F COUNT IMMED stress: force every COUNT'th enqueue to a busy local DSQ (use with -I)\n" " -C MODE cid-override test (shuffle|bad-dup|bad-range|bad-mono)\n" " -i SEC Stats interval, seconds (default 5)\n"
@@ -105,6 +106,7 @@ struct hier_prev {u64 nr_dsps[MAX_SUB_SCHEDS ];u64 nr_reenq_cap;u64 nr_reenq_immed; +u64 nr_enq_blocked;u64 nr_inject_attempts; };
@@ -187,13 +189,15 @@ static void print_hier(struct qmap_arena *qa, struct hier_prev *prev, u64 own_cg }format_cid_ranges (qa,CID_SHARED , ranges,sizeof (ranges)); -printf ("hier : nsub=%llu excl=%u shared=%s rr=%s reenq cap/immed +%llu/+%llu inj=+%llu\n" , +printf ("hier : nsub=%llu excl=%u shared=%s rr=%s reenq cap/immed +%llu/+%llu blocked=+%llu inj=+%llu\n" , (unsigned long long )qa->nr_sub_scheds, qa->part.nr_excl, ranges, rr, (unsigned long long )(qa->nr_reenq_cap - prev->nr_reenq_cap), (unsigned long long )(qa->nr_reenq_immed - prev->nr_reenq_immed), + (unsigned long long )(qa->nr_enq_blocked - prev->nr_enq_blocked), (unsigned long long )(qa->nr_inject_attempts - prev->nr_inject_attempts)); prev->nr_reenq_cap = qa->nr_reenq_cap; prev->nr_reenq_immed = qa->nr_reenq_immed; + prev->nr_enq_blocked = qa->nr_enq_blocked; prev->nr_inject_attempts = qa->nr_inject_attempts; printf("hier : %-4s %10s %4s %6s %8s %s\n",
@@ -256,7 +260,7 @@ int main(int argc, char **argv) skel->rodata->slice_ns =__COMPAT_ENUM_OR_ZERO ("scx_public_consts" ,"SCX_SLICE_DFL" ); skel->rodata->max_tasks =16384 ; -while ((opt =getopt (argc, argv,"s:e:t:T:l:b:N:PMHc:d:D:SpIF:C:i:R:J:vh" )) !=-1 ) { +while ((opt =getopt (argc, argv,"s:e:t:T:l:b:N:PMHc:d:D:SpIBF:C:i:R:J:vh" )) !=-1 ) {switch (opt) {case 's' : skel->rodata->slice_ns =strtoull (optarg,NULL ,0 ) *1000 ;
@@ -318,6 +322,9 @@ int main(int argc, char **argv) skel->rodata->always_enq_immed = true; skel->struct_ops.qmap_ops->flags |=SCX_OPS_ALWAYS_ENQ_IMMED ;break ; +case 'B' : + skel->struct_ops.qmap_ops->flags |=SCX_OPS_ENQ_BLOCKED ; +break ;case 'F' : skel->rodata->immed_stress_nth =strtoul (optarg,NULL ,0 );break ;
tools/sched_ext/scx_qmap.h +1 -0
@@ -174,6 +174,7 @@ struct qmap_arena {/* bpf -> userspace: stats */ u64 nr_reenq_cap;/* SCX_TASK_REENQ_CAP bounces */ u64 nr_reenq_immed;/* SCX_TASK_REENQ_IMMED bounces */ +u64 nr_enq_blocked;/* SCX_ENQ_BLOCKED dispatches */ u64 nr_inject_attempts;/* fault-injection: dispatches to an unheld cid */ u32 inject_mode;/* fault-injection mode (QMAP_INJ_*) */ };
Signature · 1 line
2.55.0
Andrea Righi
reviewerPATCH 12/12 sched: Allow enabling proxy exec with sched_ext
In reply to Andrea RighiNow that sched_ext can handle proxy donors and BPF schedulers can opt in to blocked-donor enqueueing with SCX_OPS_ENQ_BLOCKED, remove the !SCHED_CLASS_EXT dependency from SCHED_PROXY_EXEC and allow both options to be enabled together.
--- init/Kconfig | 2 -- 1 file changed, 2 deletions(-)
init/Kconfig +0 -2
@@ -934,8 +934,6 @@ config SCHED_PROXY_EXEC bool "Proxy Execution" # Avoid some build failures w/ PREEMPT_RT until it can be fixed depends on !PREEMPT_RT - # Need to investigate how to inform sched_ext of split contexts - depends on !SCHED_CLASS_EXT # Not particularly useful until we get to multi-rq proxying depends on EXPERT help
Signature · 1 line
2.55.0
Re: [PATCH 12/12] sched: Allow enabling proxy exec with sched_ext
Quoted 7 lines · Andrea Righi
Now that
sched_extcan handle proxy donors and BPF schedulers can opt in to blocked-donor enqueueing withSCX_OPS_ENQ_BLOCKED, remove the !SCHED_CLASS_EXT dependency fromSCHED_PROXY_EXECand allow both options to be enabled together.Signed-off-by Andrea Righi <[email protected]>
So far things have held up pretty well in my testing.
I've seen a few cases where the scx scheduler gets disabled due to stalls under stress-ng pressure or locktorture+other workloads, but I suspect that may be separate from proxy-exec (I'll continue to do some testing both with and without to confirm).
Huge thanks for this effort!
-john