sched: Make proxy execution compatible with sched_ext
Subsystems (3)
Andrea Righi
reviewerPATCH v7 1/1 sched: Make proxy execution compatible with sched_ext
This series enables using proxy execution with sched_ext and is based on early work by John Stultz [1].
Background
==========
Proxy execution (proxy-exec) lets a waiting task ("donor") donate its scheduling context to a mutex owner, so the owner can run while the donor stays eligible on the runqueue.
Currently, proxy-exec and sched_ext are mutually exclusive at build time: we can't enable CONFIG_SCHED_PROXY_EXEC=y and CONFIG_SCHED_CLASS_EXT=y in the same kernel.
This restriction can be problematic for Linux distributions and for anyone who wants to ship one kernel and choose features at runtime.
Why are they mutually exclusive?
================================
sched_ext schedulers drive dispatch through their own interfaces. A proxy-exec handoff can run a task that the BPF scheduler never dispatched through that path. sched_ext callbacks then observe a "current" task that does not match what the BPF side considers running, so kfuncs and helper state can see an inconsistent view of the executing task.
sched_ext also tracks runnable work through Dispatch Queues (DSQs) and BPF chosen dispatch rules, while the core scheduler still maintains classic per-CPU runqueues and pick paths. A proxy handoff can therefore switch the CPU to a task that the BPF scheduler never inserted or ordered through its DSQ interface.
DSQ state, vtime, and "who is running" bookkeeping inside the BPF program can then disagree with what the core actually executes, so helpers and kfuncs that assume their dispatched task is current may observe stale or inconsistent state.
Design: supporting proxy-exec with sched_ext
============================================
Provide proxy-exec in sched_ext as optional per-scheduler capability: a BPF scheduler can set the ops flag SCX_OPS_ENQ_BLOCKED to keep mutex-blocked donors runnable and receive them through ops.enqueue() with SCX_ENQ_BLOCKED set.
This flag requires an ops.enqueue() callback and gives BPF control over whether, where, and in what order to dispatch each donor. The core then walks the mutex-owner chain and, when needed, migrates the donor's scheduling context to the owner's rq before executing the owner.
Without the SCX_OPS_ENQ_BLOCKED flag, mutex waiters block normally and do not participate in proxy-exec while owned by that scheduler.
The donor-to-owner handoff is modeled like a "function call" from the scheduler's perspective. The donor remains the running scheduling entity selected by BPF: its scheduling context, runtime and slice are consumed while the core temporarily invokes the mutex owner's code to make the critical section progress. It is not a scheduler-visible switch to the owner.
Accordingly, the donor remains the scheduling context presented to sched_ext, while the mutex owner is treated solely as the execution context selected internally by the core scheduler. Scheduling state is accounted against rq->donor where appropriate, while rq->curr identifies the execution context. The internal owner substitution does not generate synthetic sched_ext callbacks for a task that BPF did not dispatch.
Scheduler-facing current-task kfuncs follow the same rule:scx_bpf_task_running(), scx_bpf_cpu_curr() and scx_bpf_cid_curr() report the donor selected by the scheduler rather than the mutex owner whose code happens to execute on its behalf.
The sched_ext callback bookkeeping is adjusted accordingly. Blocked proxy donors do not generate spurious ops.running() callbacks and ops.stopping() is only called for tasks that entered a tracked ops.running()/stopping() session. This state is independent of physical rq->curr execution, which may refer to the mutex owner instead of the donor.
NOHZ CFS bandwidth checks also follow rq->donor rather than rq->curr, so a bandwidth-constrained FAIR donor keeps the tick running while an owner executes on its behalf.
Scheduler ownership changes need special handling because a donor may already be blocked when a root scheduler is enabled, when a task moves between root and sub-schedulers, or when PI de-boosting moves it into EXT. If the incoming scheduler does not set SCX_OPS_ENQ_BLOCKED, the retained donor is removed from the runqueue and stays on the normal mutex wait path. Otherwise it remains eligible for BPF admission.
scx_qmap has been modified with a -B option to enable queueing mutex-blocked tasks for proxy-exec. Blocked donors receive a fresh slice, are inserted at the head of their current cid's local DSQ and request immediate preemption. This is an intentionally aggressive policy for making proxy-exec easy to observe. Capability-rejection re-enqueues bypass this blocked-donor fast path and fall back to the shared DSQ, avoiding repeated rejection on a cid that no longer has the required capability. A new statistic reports blocked-donor dispatches.
A new kselftest (enq_blocked) is also introduced to validate proxy-exec with sched_ext. The test creates a priority inversion with a low-priority owner (nice +19), a high-priority donor (nice -20) and one nice 0 contender per available CPU. It exercises a same-CPU topology and a cross-CPU topology, where the donor and owner run on different CPUs. Each configuration runs with SCX_OPS_ENQ_BLOCKED first disabled and then enabled, counts blocked-donor enqueues by CPU and reports mutex hold/wait-time deltas. Access to the mutex is provided by a loadable kernel module built via TEST_GEN_MODS_DIR, with the test responsible for its lifecycle.
Example kselftest run:
===== START =====
TEST: enq_blocked
DESCRIPTION: Verify proxy donor admission under CPU-wide contention OUTPUT:
[topology=same-cpu SCX_OPS_ENQ_BLOCKED=disabled]
proxy_exec=enabled
donor_cpu=0
owner_cpu=0
nr_contenders=16
measured_trials=10
owner_nice=19
donor_nice=-20
contender_nice=0 mutex_hold_avg_ns=263587544 (263.587 ms, samples=10) mutex_wait_avg_ns=263597934 (263.597 ms, samples=10) nr_blocked_enqueues=0 nr_blocked_enqueues_donor_cpu=0 nr_blocked_enqueues_owner_cpu=0 nr_blocked_enqueues_other_cpu=0 nr_blocked_wakeups=0
[topology=same-cpu SCX_OPS_ENQ_BLOCKED=enabled]
proxy_exec=enabled
donor_cpu=0
owner_cpu=0
nr_contenders=16
measured_trials=10
owner_nice=19
donor_nice=-20
contender_nice=0 mutex_hold_avg_ns=251789178 (251.789 ms, samples=10) mutex_wait_avg_ns=209808599 (209.808 ms, samples=10) nr_blocked_enqueues=60 nr_blocked_enqueues_donor_cpu=60 nr_blocked_enqueues_owner_cpu=0 nr_blocked_enqueues_other_cpu=0 nr_blocked_wakeups=0
[topology=same-cpu delta: enabled - disabled]
mutex_hold_delta_ns=-11798366 (-4.48%)
mutex_wait_delta_ns=-53789335 (-20.41%)
[topology=cross-cpu SCX_OPS_ENQ_BLOCKED=disabled]
proxy_exec=enabled
donor_cpu=0
owner_cpu=1
nr_contenders=16
measured_trials=10
owner_nice=19
donor_nice=-20
contender_nice=0 mutex_hold_avg_ns=246794454 (246.794 ms, samples=10) mutex_wait_avg_ns=247298497 (247.298 ms, samples=10) nr_blocked_enqueues=0 nr_blocked_enqueues_donor_cpu=0 nr_blocked_enqueues_owner_cpu=0 nr_blocked_enqueues_other_cpu=0 nr_blocked_wakeups=0
[topology=cross-cpu SCX_OPS_ENQ_BLOCKED=enabled]
proxy_exec=enabled
donor_cpu=0
owner_cpu=1
nr_contenders=16
measured_trials=10
owner_nice=19
donor_nice=-20
contender_nice=0 mutex_hold_avg_ns=215486809 (215.486 ms, samples=10) mutex_wait_avg_ns=215498090 (215.498 ms, samples=10) nr_blocked_enqueues=80 nr_blocked_enqueues_donor_cpu=20 nr_blocked_enqueues_owner_cpu=60 nr_blocked_enqueues_other_cpu=0 nr_blocked_wakeups=0
[topology=cross-cpu delta: enabled - disabled]
mutex_hold_delta_ns=-31307645 (-12.69%)
mutex_wait_delta_ns=-31800407 (-12.86%)
ok 1 enq_blocked #
===== END =====
=============================
RESULTS:
PASSED: 1 SKIPPED: 0 FAILED: 0
References
==========
[1] https://lore.kernel.org/all/[email protected]
Git tree: git://git.kernel.org/pub/scm/linux/kernel/git/arighi/linux.git scx-proxy-exec
Changes in v7:
- Move the remote CPU check out of the lockless remote-DSQ scan and perform it only after locking the source rq, retain the locked migration recheck immediately before moving the task (sashiko)
- Complete the scheduler-context conversion from rq->curr to rq->donor for capability revocation and preemption checks, make scx_bpf_task_running(), scx_bpf_cpu_curr() and scx_bpf_cid_curr() expose the donor selected by the scheduler (sashiko)
- Handle SCX_ENQ_REENQ before scx_qmap's blocked-donor fast path to avoid an infinite reject/re-enqueue loop (sashiko)
- Add scx_qmap stat for SCX_ENQ_BLOCKED dispatches
- Link to v6: https://lore.kernel.org/all/[email protected]/
Changes in v6:
- Add a scheduler-core fix to make NOHZ CFS bandwidth checks follow the proxy donor instead of the physical execution context (sashiko)
- Block retained donors when entering sched_ext through PI de-boosting or global scheduler activation (sashiko)
- Distinguish ordinary wakeup activations from retained-donor admissions so SCX_ENQ_BLOCKED is not reported for normal wakeups; extend the selftest to detect this regression (sashiko)
- Validate unexpected-CPU blocked-donor enqueues in both same-CPU and cross-CPU test topologies (sashiko)
- Link to v5: https://lore.kernel.org/all/[email protected]/
Changes in v5:
- Split retained-donor deactivation and sched_ext's default rejection into preparatory patches so the scheduler-core changes can be routed separately
- Drop the proxy destination query kfuncs and the preparatory mutex lock-scope change (John Stultz)
- Use p->is_blocked instead of task_is_blocked() to fix a WARN triggered during scx_pair testing (John Stultz)
- Rename SCX_TASK_IS_RUNNING to SCX_TASK_RUN_TRACKED and document that it tracks an ops.running()/stopping() session rather than physical rq->curr execution
- Keep a proxy-migrated blocked donor on the owner's rq until wakeup instead of allowing BPF-directed migration to pull it back to wake_cpu and cause repeated donor migration
- Extend kselftest to cover same-CPU and cross-CPU proxy-exec switches
- Make scx_qmap insert blocked donors at the head of their current cid's local DSQ and request immediate preemption
- Link to v4: https://lore.kernel.org/all/[email protected]/
Changes in v4:
- Harden scx_bpf_task_proxy_cpu() locking and blocked-state validation, return -ENOENT for unrunnable owners and drop unnecessary donor-affinity checks (K Prateek Nayak, sashiko)
- Reschedule remote CPUs after donor deactivation, handle sched_setscheduler() admission and assert scheduler-change locking (sashiko)
- Avoid a potential KCSAN data-race report in the lockless blocked-donor migration check by using rcu_access_pointer() for rq->donor (sashiko)
- Fix tick dependency updates for incoming EXT contexts and keep the tick enabled for blocked donors (sashiko)
- Dump EXT donors in scx_dump_state() (sashiko)
- Reordered the preparatory sched_ext changes so real running-state tracking is established before donor-based accounting, and split the proxy destination query kfuncs into a separate patch
- Add compatibility wrappers for the proxy CPU/cid kfuncs and document their results as scheduling hints
- Link to v3: https://lore.kernel.org/all/[email protected]/
Changes in v3:
- Dropped the core restrictions on proxy-migrating migration-disabled and single-CPU donors: proxy execution moves the scheduling context, not the task's execution context. (Peter Zijlstra, K Prateek Nayak)
- Dropped the sched_ext-specific put_prev_task()/set_next_task() exception and fixed ops.running()/ops.stopping() pairing inside sched_ext instead; track a real running transition even when either callback is absent. (Peter Zijlstra)
- Dropped the kf_tasks[] nesting and nested ops.runnable() patches from v2; extensive proxy-exec testing did not reproduce task-op re-entry, so retain the existing non-nesting invariant.
- Replaced scx_bpf_task_is_blocked() with SCX_ENQ_BLOCKED in ops.enqueue() flags, identifying blocked-donor admission directly at enqueue time.
- Expanded the curr/donor description to cover mixed scheduling classes and fixed local preemption to expire rq->donor's slice. (Aiqun Maria Yu)
- Allow inactive blocked donors to be placed on a remote local DSQ while preventing normal migration of an active rq donor. (Aiqun Maria Yu)
- Deactivate retained donors when ownership changes to a root or sub-scheduler without SCX_OPS_ENQ_BLOCKED, and extend the selftest to cover attaching a scheduler after the donor blocks. (K Prateek Nayak)
- Harden remote DSQ consumption by rejecting active tasks and rechecking migration eligibility after locking the source rq. Fall back to the global DSQ without treating an eligibility change during the lock handoff as a BPF scheduler error.
- scx_qmap has a command line option (-B) to enable blocked-donor queueing
- Link to v2: https://lore.kernel.org/all/[email protected]/
Changes in v2:
- Rebased onto sched_ext/for-7.3 and adapted the series to the split sched_ext implementation and cid-form scheduler interfaces.
- Replaced the global sched_proxy_exec_scx boot-time opt-in with the per-scheduler SCX_OPS_ENQ_BLOCKED capability, allowing BPF to control donor admission and ordering through ops.enqueue().
- Added scx_bpf_task_is_blocked(), scx_bpf_task_proxy_cpu(), and scx_bpf_task_proxy_cid(); enforce CPU/cid API separation for cid-form schedulers.
- Added proxy exec support to scx_qmap, including optional owner-cid steering, affinity validation, and fallback to the donor's current cid.
- Added a kselftest with a kernel mutex test and a three-task priority inversion workload, test is executed with blocked task admission disabled and enabled, validates the behavior, and reports hold/wait-time deltas.
- Link to v1: https://lore.kernel.org/all/[email protected]/
Andrea Righi (10):
sched: Make NOHZ CFS bandwidth checks follow proxy donor
sched: Add helper to block retained proxy donors
sched_ext: Block proxy donors across scheduler transitions
sched_ext: Fix ops.running/stopping() pairing for proxy-exec donors
sched_ext: Fix TOCTOU race in consume_remote_task()
sched_ext: Split curr|donor references properly
sched_ext: Delegate proxy donor admission to BPF schedulers
sched_ext: Add selftest for blocked donor admission
sched_ext: scx_qmap: Add proxy execution support
sched: Allow enabling proxy exec with sched_ext
John Stultz (1):
sched_ext: Handle blocked donor migration with proxy execution
include/linux/sched/ext.h | 3 + init/Kconfig | 2 - kernel/sched/core.c | 62 +- kernel/sched/ext/ext.c | 327 ++++++-- kernel/sched/ext/ext.h | 6 + kernel/sched/ext/internal.h | 23 +- kernel/sched/ext/sub.c | 12 +- kernel/sched/ext/sub.h | 8 +- kernel/sched/sched.h | 6 + kernel/sched/syscalls.c | 3 + tools/sched_ext/include/scx/compat.h | 1 + tools/sched_ext/include/scx/enum_defs.autogen.h | 1 + tools/sched_ext/include/scx/enums.autogen.bpf.h | 3 + tools/sched_ext/include/scx/enums.autogen.h | 1 + tools/sched_ext/scx_qmap.bpf.c | 52 +- tools/sched_ext/scx_qmap.c | 13 +- tools/sched_ext/scx_qmap.h | 1 + tools/testing/selftests/sched_ext/.gitignore | 4 + tools/testing/selftests/sched_ext/Makefile | 2 + tools/testing/selftests/sched_ext/config | 2 + .../testing/selftests/sched_ext/enq_blocked.bpf.c | 116 +++ tools/testing/selftests/sched_ext/enq_blocked.c | 908 +++++++++++++++++++++ tools/testing/selftests/sched_ext/enq_blocked.h | 27 + .../selftests/sched_ext/test_modules/Makefile | 13 + .../sched_ext/test_modules/scx_enq_blocked_test.c | 193 +++++ 25 files changed, 1692 insertions(+), 97 deletions(-) create mode 100644 tools/testing/selftests/sched_ext/enq_blocked.bpf.c create mode 100644 tools/testing/selftests/sched_ext/enq_blocked.c create mode 100644 tools/testing/selftests/sched_ext/enq_blocked.h create mode 100644 tools/testing/selftests/sched_ext/test_modules/Makefile create mode 100644 tools/testing/selftests/sched_ext/test_modules/scx_enq_blocked_test.c
Andrea Righi
reviewerPATCH 1/11 sched: Make NOHZ CFS bandwidth checks follow proxy donor
Proxy execution separates the scheduling context in rq->donor from the physical execution context in rq->curr. sched_can_stop_tick() checks the latter for CFS bandwidth constraints and only does so when nr_running is one.
A retained proxy donor keeps both the donor and mutex owner queued. The check therefore misses a constrained FAIR donor and may stop the tick while its runtime still needs to be enforced.
Check the selected donor instead and remove the nr_running restriction. The donor being a queued FAIR task is sufficient to require bandwidth accounting regardless of other runnable tasks.
--- kernel/sched/core.c | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-)
kernel/sched/core.c +9 -9
@@ -1409,11 +1409,8 @@ static void nohz_csd_func(void *info)#endif /* CONFIG_NO_HZ_COMMON */ #ifdef CONFIG_NO_HZ_FULL -static inline bool __need_bw_check (struct rq *rq,struct task_struct *p) +static inline bool __need_bw_check (struct task_struct *p) { -if (rq->nr_running !=1 ) -return false; -if (p->sched_class != &fair_sched_class)return false;
@@ -1462,14 +1459,17 @@ bool sched_can_stop_tick(struct rq *rq)return false;/* -* If there is one task and it has CFS runtime bandwidth constraints -* and it's on the cpu now we don't want to stop the tick. +* If the selected scheduling context has CFS runtime bandwidth +* constraints, don't stop the tick. With proxy execution, rq->donor is +* the selected scheduling context while rq->curr is the task physically +* executing on its behalf. +* * This check prevents clearing the bit if a newly enqueued task here is -* dequeued by migrating while the constrained task continues to run. +* dequeued by migrating while the constrained donor continues to run. * E.g. going from 2->1 without going through pick_next_task(). */ -if (__need_bw_check (rq, rq->curr)) { -if (cfs_task_bw_constrained (rq->curr)) +if (__need_bw_check (rq->donor)) { +if (cfs_task_bw_constrained (rq->donor))return false; }
Signature · 1 line
2.55.0
Re: [PATCH 01/11] sched: Make NOHZ CFS bandwidth checks follow proxy donor
Quoted 10 lines · Andrea Righi
Proxy execution separates the scheduling context in
rq->donorfrom the physical execution context inrq->curr.sched_can_stop_tick()checks the latter for CFS bandwidth constraints and only does so whennr_runningis one.A retained proxy donor keeps both the donor and mutex owner queued. The check therefore misses a constrained FAIR donor and may stop the tick while its runtime still needs to be enforced.
Quoted 8 lines
Check the selected donor instead and remove the
nr_runningrestriction. The donor being a queued FAIR task is sufficient to require bandwidth accounting regardless of other runnable tasks.Fixes af0c8b2bf67b ("sched: Split scheduler and execution contexts")Reported-by Sashiko <[email protected]>Signed-off-by Andrea Righi <[email protected]>
Andrea Righi
reviewerPATCH 2/11 sched: Add helper to block retained proxy donors
In reply to Andrea RighiScheduler ownership changes may need to turn a mutex-blocked task retained on the runqueue for proxy execution back into a normally blocked task.
Add sched_proxy_block_task() to perform that transition while holding p->pi_lock and the task's runqueue lock. Move an active donor off the CPU first, and complete any delayed dequeue so a following sched_change cannot preserve and re-enqueue the task.
This is a preparatory change to support proxy execution with sched_ext.
--- kernel/sched/core.c | 39 +++++++++++++++++++++++++++++++++++++++ kernel/sched/sched.h | 6 ++++++ 2 files changed, 45 insertions(+)
kernel/sched/core.c +39 -0
@@ -6764,6 +6764,45 @@ static void proxy_deactivate(struct rq *rq, struct task_struct *donor)block_task (rq, donor, state); } +/* +* Remove a retained proxy donor before changing its scheduler ownership. +* The caller holds p->pi_lock, so p cannot wake and migrate after block_task() +* drops it from the runqueue. +* +* Unlike the regular schedule() path, this must leave @p fully dequeued. +* DELAY_DEQUEUE may keep a blocked fair task queued with sched_delayed set, +* which would let the following sched_change preserve and re-enqueue it under +* the new scheduler. Complete an existing or newly-created delayed dequeue +* before returning. +*/ +void sched_proxy_block_task (struct rq *rq,struct task_struct *p) +{ +unsigned long state =READ_ONCE (p->__state); + +lockdep_assert_held (&p->pi_lock); +lockdep_assert_rq_held (rq); + +if (!p->is_blocked || !task_on_rq_queued (p)) +return ; +if (WARN_ON_ONCE (state ==TASK_RUNNING )) +return ; + +if (task_current_donor (rq, p)) { +proxy_resched_idle (rq); +/* Kick the execution context if @rq is remote. */ +resched_curr (rq); + } + +if (!p->se.sched_delayed) +block_task (rq, p, state); +if (p->se.sched_delayed) +dequeue_task (rq, p,DEQUEUE_SLEEP |DEQUEUE_DELAYED | +DEQUEUE_NOCLOCK ); + +WARN_ON_ONCE (task_on_rq_queued (p)); +WARN_ON_ONCE (p->se.sched_delayed); +} +static inline void proxy_release_rq_lock (struct rq *rq,struct rq_flags *rf)__releases (__rq_lockp (rq)) {
kernel/sched/sched.h +6 -0
@@ -2477,6 +2477,12 @@ static inline bool task_is_blocked(struct task_struct *p)return !!p->blocked_on; } +#ifdef CONFIG_SCHED_PROXY_EXEC +void sched_proxy_block_task (struct rq *rq,struct task_struct *p); +#else +static inline void sched_proxy_block_task (struct rq *rq,struct task_struct *p) {} +#endif +static inline int task_on_cpu (struct rq *rq,struct task_struct *p) {return p->on_cpu;
Signature · 1 line
2.55.0
Andrea Righi
reviewerPATCH 3/11 sched_ext: Block proxy donors across scheduler transitions
In reply to Andrea RighiProxy execution retains mutex-blocked donors on the runqueue so their scheduling context can execute a lock owner. sched_ext cannot safely retain such donors unless the BPF scheduler explicitly participates in their admission and ordering.
Make sched_ext reject retained donors by default. Force blocked EXT tasks through the regular block path in schedule(), and fully deactivate a retained donor before sched_setscheduler(), PI de-boosting, or global activation moves it into the EXT class. These hooks establish the safe default that a later opt-in can relax.
This is a preparatory change to support proxy execution with sched_ext.
--- kernel/sched/core.c | 5 ++++- kernel/sched/ext/ext.c | 44 +++++++++++++++++++++++++++++++++++++++++ kernel/sched/ext/ext.h | 6 ++++++ kernel/sched/syscalls.c | 3 +++ 4 files changed, 57 insertions(+), 1 deletion(-)
kernel/sched/core.c +4 -1
@@ -7173,7 +7173,8 @@ static void __sched notrace __schedule(int sched_mode)* task_is_blocked() will always be false). */ try_to_block_task (rq, prev, &prev_state, - !task_is_blocked (prev)); + !task_is_blocked (prev) || + !scx_allow_proxy_exec (prev)); switch_count = &prev->nvcsw; }
@@ -7726,6 +7727,8 @@ void rt_mutex_setprio(struct task_struct *p, struct task_struct *pi_task)if (prev_class != next_class) queue_flag |=DEQUEUE_CLASS ; +scx_prepare_setscheduler (p, next_class); + scoped_guard (sched_change, p, queue_flag) {/* * Boosting condition are:
kernel/sched/ext/ext.c +44 -0
56 lines not on this page · raw message
kernel/sched/ext/ext.h +6 -0
@@ -18,8 +18,11 @@ bool scx_can_stop_tick(struct rq *rq);void scx_rq_activate (struct rq *rq);void scx_rq_deactivate (struct rq *rq);int scx_check_setscheduler (struct task_struct *p,int policy); +void scx_prepare_setscheduler (struct task_struct *p, +const struct sched_class *next_class);bool task_should_scx (int policy);bool scx_allow_ttwu_queue (const struct task_struct *p); +bool scx_allow_proxy_exec (const struct task_struct *p);void init_sched_ext_class (void );static inline u32 scx_cpuperf_target (s32 cpu)
@@ -52,8 +55,11 @@ static inline bool scx_can_stop_tick(struct rq *rq) { return true; }static inline void scx_rq_activate (struct rq *rq) {}static inline void scx_rq_deactivate (struct rq *rq) {}static inline int scx_check_setscheduler (struct task_struct *p,int policy) {return 0 ; } +static inline void scx_prepare_setscheduler (struct task_struct *p, +const struct sched_class *next_class) {}static inline bool task_on_scx (const struct task_struct *p) {return false; }static inline bool scx_allow_ttwu_queue (const struct task_struct *p) {return true; } +static inline bool scx_allow_proxy_exec (const struct task_struct *p) {return true; }static inline void init_sched_ext_class (void ) {}#endif /* CONFIG_SCHED_CLASS_EXT */
kernel/sched/syscalls.c +3 -0
@@ -678,6 +678,9 @@ int __sched_setscheduler(struct task_struct *p,if (prev_class != next_class) queue_flags |=DEQUEUE_CLASS ; +if (!(attr->sched_flags &SCHED_FLAG_KEEP_PARAMS )) +scx_prepare_setscheduler (p, next_class); + scoped_guard (sched_change, p, queue_flags) {if (!(attr->sched_flags &SCHED_FLAG_KEEP_PARAMS )) {
Signature · 1 line
2.55.0
Re: [PATCH 03/11] sched_ext: Block proxy donors across scheduler transitions
Quoted 27 lines · Andrea Righi
Proxy execution retains mutex-blocked donors on the runqueue so their scheduling context can execute a lock owner.
sched_extcannot safely retain such donors unless the BPF scheduler explicitly participates in their admission and ordering.Make
sched_extreject retained donors by default. Force blocked EXT tasks through the regular block path inschedule(), and fully deactivate a retained donor beforesched_setscheduler(), PI de-boosting, or global activation moves it into the EXT class. These hooks establish the safe default that a later opt-in can relax.This is a preparatory change to support proxy execution with
sched_ext.Signed-off-by Andrea Righi <[email protected]>--- kernel/sched/core.c | 5 ++++- kernel/sched/ext/ext.c | 44 +++++++++++++++++++++++++++++++++++++++++ kernel/sched/ext/ext.h | 6 ++++++ kernel/sched/syscalls.c | 3 +++ 4 files changed, 57 insertions(+), 1 deletion(-)kernel/sched/core.c +0
-0
kernel/sched/core.c static void __sched notrace __schedule(int sched_mode) · line 7179
Quoted 8 lines
kernel/sched/core.c +2
-1@@ -7173,7 +7173,8 @@ static void __sched notrace __schedule(int sched_mode)* task_is_blocked() will always be false). */ try_to_block_task (rq, prev, &prev_state, - !task_is_blocked (prev)); + !task_is_blocked (prev) || + !scx_allow_proxy_exec (prev)); switch_count = &prev->nvcsw; }
This reminds me that since we no longer re-use try_to_block_task() elsewhere, we can probably push the checks down into the function and get rid of teh should_block argument.
But this is a minor cleanup for later, not an issue with this patch.
This looks good to me!
Andrea Righi
reviewerPATCH 4/11 sched_ext: Fix ops.running/stopping() pairing for proxy-exec donors
In reply to Andrea RighiWith proxy-exec, pick_next_task() can return a task with p->is_blocked set (a proxy donor). put_prev_set_next_task() then calls set_next_task_scx() on this "ghost" task even though the task only provides scheduling context and never actually runs.
Calling ops.running() for such a donor produces a spurious running event. Simply suppressing ops.running() is not sufficient because the following put_prev_task_scx() would still invoke ops.stopping(), resulting in an unpaired stopping event.
Introduce SCX_TASK_RUN_TRACKED to track whether a task entered an ops.running()/stopping() session. Set and clear the flag independently of ops.running() and ops.stopping(), as the callbacks are independently optional. Invoke ops.running() only for non-blocked tasks and invoke ops.stopping() only after entering a tracked session. This keeps the callbacks paired for proxy donors while preserving stopping notifications for schedulers which only implement ops.stopping().
This is a preparatory change for enabling proxy execution together with sched_ext. The explicit running-state tracking is also required by later donor-based accounting: it prevents an EXT owner executing for a non-EXT donor from being treated as the active EXT scheduling context when it is dequeued.
--- include/linux/sched/ext.h | 2 ++ kernel/sched/ext/ext.c | 33 +++++++++++++++++++++++++-------- 2 files changed, 27 insertions(+), 8 deletions(-)
include/linux/sched/ext.h +2 -0
@@ -103,6 +103,8 @@ enum scx_ent_flags {SCX_TASK_SUB_INIT =1 <<4 ,/* task being initialized for a sub sched */ SCX_TASK_IMMED =1 <<5 ,/* task is on local DSQ with %SCX_ENQ_IMMED */ + SCX_TASK_RUN_TRACKED = 1 << 6, /* task is in an ops.running()/stopping() session */ +/* * Bits 8 to 10 are used to carry task state: *
kernel/sched/ext/ext.c +25 -8
@@ -2209,9 +2209,13 @@ static bool dequeue_task_scx(struct rq *rq, struct task_struct *p, int core_deq_* information meaningful to the BPF scheduler and can be suppressed by * skipping the callbacks if the task is !QUEUED. */ -if (SCX_HAS_OP (sch, stopping) &&task_current (rq, p)) { -update_curr_scx (rq); -SCX_CALL_OP_TASK (sch, stopping, rq, p, false); +if (task_current (rq, p) && + (p->scx.flags &SCX_TASK_RUN_TRACKED )) { +if (SCX_HAS_OP (sch, stopping)) { +update_curr_scx (rq); +SCX_CALL_OP_TASK (sch, stopping, rq, p, false); + } + p->scx.flags &= ~SCX_TASK_RUN_TRACKED ; }if (SCX_HAS_OP (sch, quiescent) && !task_on_rq_migrating (p))
@@ -2913,9 +2917,17 @@ static void set_next_task_scx(struct rq *rq, struct task_struct *p, bool first) p->se.exec_start =rq_clock_task (rq); -/* see dequeue_task_scx() on why we skip when !QUEUED */ -if (SCX_HAS_OP (sch, running) && (p->scx.flags &SCX_TASK_QUEUED )) -SCX_CALL_OP_TASK (sch, running, rq, p); +/* +* See dequeue_task_scx() for why we skip when !QUEUED. A blocked proxy +* donor is also skipped because it provides scheduling context but never +* runs itself. +*/ +if ((p->scx.flags &SCX_TASK_QUEUED ) && !p->is_blocked) { +if (SCX_HAS_OP (sch, running)) +SCX_CALL_OP_TASK (sch, running, rq, p); + + p->scx.flags |=SCX_TASK_RUN_TRACKED ; + }clr_task_runnable (p, true);
@@ -3021,8 +3033,13 @@ static void put_prev_task_scx(struct rq *rq, struct task_struct *p,update_curr_scx (rq);/* see dequeue_task_scx() on why we skip when !QUEUED */ -if (SCX_HAS_OP (sch, stopping) && (p->scx.flags &SCX_TASK_QUEUED )) -SCX_CALL_OP_TASK (sch, stopping, rq, p, true); +if ((p->scx.flags &SCX_TASK_QUEUED ) && + (p->scx.flags &SCX_TASK_RUN_TRACKED )) { +if (SCX_HAS_OP (sch, stopping)) +SCX_CALL_OP_TASK (sch, stopping, rq, p, true); + + p->scx.flags &= ~SCX_TASK_RUN_TRACKED ; + }if (p->scx.flags &SCX_TASK_QUEUED ) {set_task_runnable (rq, p);
Signature · 1 line
2.55.0
Andrea Righi
reviewerPATCH 5/11 sched_ext: Fix TOCTOU race in consume_remote_task()
In reply to Andrea RighiWhen pulling a task from a non-local DSQ, scx_consume_dispatch_q() filters candidates without holding the task's source rq lock; consume_remote_task() then unlinks the selected task, drops the DSQ and destination rq locks and acquires the source rq lock before migrating it.
Whether the task may be remotely migrated can change across this handoff. In particular, proxy execution allows a task to execute as rq->curr under another task's scheduling context while its own scheduling context remains on a DSQ. The task can therefore become on-CPU or migration disabled without dequeueing its DSQ entry or clearing holding_cpu.
With proxy execution enabled, a mutex-intensive workload such as stress-ng --pipeherd 0 can easily trigger this race when a donor becomes active on its source rq during the unlocked window. Migrating the active execution context can make it resume with IRQ and preemption state from the wrong scheduling path, triggering sleeping-while-atomic warnings and subsequent lockdep corruption.
Fix this by splitting the validation into two helpers:task_can_run_on_remote_rq() remains the preliminary lockless filter used while scanning a DSQ; task_can_move_from_locked_rq() is used at actual migration points, it requires the task's source rq lock, rejects the task if it is still on-CPU and then applies the existing destination checks.
After consume_remote_task() has switched to the source rq lock, run the locked check again immediately before move_remote_task_to_local_dsq(). If the check fails, the task has already been removed from its original DSQ, so cancel the transfer and enqueue it on the global DSQ. Use a non-enforcing check, because the failure results from a kernel-side race, not an invalid BPF placement request.
Deferring this check until the source rq is locked both closes the migration race and ensures the task remains discoverable by an idle CPU.
--- kernel/sched/ext/ext.c | 56 +++++++++++++++++++++++++++++++++++------- 1 file changed, 47 insertions(+), 9 deletions(-)
kernel/sched/ext/ext.c +47 -9
@@ -2364,8 +2364,10 @@ static void move_remote_task_to_local_dsq(struct scx_sched *sch,* - The BPF scheduler is bypassed while the rq is offline and we can always say * no to the BPF scheduler initiated migrations while offline. * -* The caller must ensure that @p and @rq are on different CPUs. -* If enforce == true, caller must hold @p's rq lock. +* The caller must ensure that @p and @rq are on different CPUs. If @enforce is +* true, report violations attributable to BPF-directed migrations. The caller +* must hold @p's rq lock to avoid reporting a transient race as a scheduler +* error. */ static bool task_can_run_on_remote_rq(struct scx_sched *sch, struct task_struct *p, struct rq *rq,
@@ -2373,11 +2375,6 @@ static bool task_can_run_on_remote_rq(struct scx_sched *sch, {s32 cpu =cpu_of (rq); -/* -* To prevent races with @p still running on its old CPU while switching -* out, make sure we're holding @p's rq lock so as not to risk -* erroneously killing the BPF scheduler. -*/ if (enforce)lockdep_assert_rq_held (task_rq (p));
@@ -2424,6 +2421,25 @@ static bool task_can_run_on_remote_rq(struct scx_sched *sch,return true; } +/* +* Final migration check with @p's rq locked. Unlike the lockless DSQ scan, +* this can safely reject transient execution state without losing a kick. +*/ +static bool task_can_move_from_locked_rq (struct scx_sched *sch, +struct task_struct *p,struct rq *dst_rq, +bool enforce) +{ +struct rq *src_rq =task_rq (p); + +lockdep_assert_rq_held (src_rq); + +/* @p may be rq->curr under another task's proxy scheduling context. */ +if (task_on_cpu (src_rq, p)) +return false; + +return task_can_run_on_remote_rq (sch, p, dst_rq, enforce); +} +/** * unlink_dsq_and_switch_rq_lock() - Unlink task and switch to its rq lock * @p: target task
@@ -2481,6 +2497,28 @@ static bool consume_remote_task(struct scx_sched *sch, struct rq *this_rq, struct scx_dispatch_q *dsq, struct rq *src_rq) { if (unlink_dsq_and_switch_rq_lock(p, dsq, this_rq, src_rq)) { +/* +* Whether @p may be migrated to @this_rq may have changed while +* switching rq locks. This is a kernel-side race, not an invalid BPF +* placement request, so don't abort the scheduler on failure. Fall +* back to the global DSQ, where normal consumption filters can select +* a valid destination without forcing the task onto the source local +* DSQ. +*/ +if (unlikely (!task_can_move_from_locked_rq (sch, p, this_rq, false))) { + p->scx.dsq =NULL ; + p->scx.holding_cpu =-1 ; +scx_dispatch_enqueue (sch, src_rq, +find_global_dsq (sch,task_cpu (p)), p, + enq_flags |SCX_ENQ_CLEAR_OPSS | +SCX_ENQ_GDSQ_FALLBACK ); +if (sched_class_above (p->sched_class, + src_rq->donor->sched_class)) +resched_curr (src_rq); +switch_rq_lock (src_rq, this_rq); +return false; + } +move_remote_task_to_local_dsq (sch, p, enq_flags, src_rq, this_rq);return true; } else {
@@ -2519,7 +2557,7 @@ static struct rq *move_task_between_dsqs(struct scx_sched *sch,if (dst_dsq->id ==SCX_DSQ_LOCAL ) { dst_rq = container_of(dst_dsq, struct rq, scx.local_dsq);if (src_rq != dst_rq && -unlikely (!task_can_run_on_remote_rq (sch, p, dst_rq, true))) { +unlikely (!task_can_move_from_locked_rq (sch, p, dst_rq, true))) { dst_dsq =find_global_dsq (sch,task_cpu (p)); dst_rq = src_rq; enq_flags |=SCX_ENQ_GDSQ_FALLBACK ;
@@ -2682,7 +2720,7 @@ static void dispatch_to_local_dsq(struct scx_sched *sch, struct rq *rq, p->scx.holding_cpu =-1 ;scx_dispatch_enqueue (sch, dst_rq, &dst_rq->scx.local_dsq, p, enq_flags); - }else if (unlikely (!task_can_run_on_remote_rq (sch, p, dst_rq, true))) { + }else if (unlikely (!task_can_move_from_locked_rq (sch, p, dst_rq, true))) { p->scx.holding_cpu =-1 ; fallback = true; scx_dispatch_enqueue(sch, src_rq, find_global_dsq(sch, task_cpu(p)),
Signature · 1 line
2.55.0
[email protected]
AI reviewRe: [PATCH 05/11] sched_ext: Fix TOCTOU race in consume_remote_task()
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] Infinite livelock loop in scx_consume_dispatch_q() when falling back to the global DSQ. --
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] Infinite livelock loop in scx_consume_dispatch_q() when falling back to the global DSQ. --
--- Patch [5]: [PATCH 05/11] sched_ext: Fix TOCTOU race in consume_remote_task() --- commit bf5934bbeaad49172aa99cf40eef02a6a7ea4bd6
Author: Andrea Righi <[email protected]>
sched_ext: Fix TOCTOU race in consume_remote_task()
This commit attempts to fix a TOCTOU race in consume_remote_task() when pulling a task from a non-local DSQ by deferring the validation until the task's source rq is locked. It splits the check into lockless and locked variants, and falls back to enqueueing the task to the global DSQ if the locked check fails.
Quoted 4 lines
kernel/sched/ext/ext.c +0
-0
[ ... ]
Quoted 14 lines
kernel/sched/ext/ext.c +10
-0@@ -2480,6 +2496,28 @@ static bool consume_remote_task(struct scx_sched *sch, struct rq *this_rq, struct scx_dispatch_q *dsq, struct rq *src_rq) { if (unlink_dsq_and_switch_rq_lock(p, dsq, this_rq, src_rq)) { +/* +* Whether @p may be migrated to @this_rq may have changed while +* switching rq locks. This is a kernel-side race, not an invalid BPF +* placement request, so don't abort the scheduler on failure. Fall +* back to the global DSQ, where normal consumption filters can select +* a valid destination without forcing the task onto the source local +* DSQ. +*/ +if (unlikely (!task_can_move_from_locked_rq (sch, p, this_rq, false))) { + p->scx.dsq =NULL ;
kernel/sched/ext/ext.c static bool consume_remote_task(struct scx_sched *sch, struct rq *this_rq, · line 2518
Quoted 10 lines
kernel/sched/ext/ext.c +10
-0+ p->scx.holding_cpu =-1 ; +scx_dispatch_enqueue (sch, src_rq, +find_global_dsq (sch,task_cpu (p)), p, + enq_flags |SCX_ENQ_CLEAR_OPSS | +SCX_ENQ_GDSQ_FALLBACK ); +if (sched_class_above (p->sched_class, + src_rq->donor->sched_class)) +resched_curr (src_rq); +switch_rq_lock (src_rq, this_rq); +return false;
[Severity: High]
Can this fallback path cause an infinite livelock if the DSQ being scanned is already the global DSQ?
When scx_consume_dispatch_q() locklessly picks a task that is executing on a remote CPU, task_can_move_from_locked_rq() correctly fails.
Because consume_remote_task() enqueues the task back to find_global_dsq() and returns false, scx_consume_dispatch_q() will react to the false return value by jumping to its retry label.
If scx_consume_dispatch_q() was scanning the global DSQ:
kernel/sched/ext/ext.c:scx_consume_dispatch_q() {
retry:
...
nldsq_for_each_task(p, dsq) {
...
if (task_can_run_on_remote_rq(sch, p, rq, false)) {
if (likely(consume_remote_task(sch, rq, p, enq_flags,
dsq, task_rq)))
return true;
goto retry;
}
}
...
}Does this cause the scanner to jump back to retry, rescan the global DSQ, find the exact same task again, fail the locked check again, and repeat the sequence indefinitely while holding runqueue locks?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=5
Tejun Heo
reviewerRe: [PATCH 05/11] sched_ext: Fix TOCTOU race in consume_remote_task()
In reply to Andrea RighiHello,
Quoted 6 lines · Andrea Righi
When pulling a task from a non-local DSQ,
scx_consume_dispatch_q()filters candidates without holding the task's source rq lock;consume_remote_task()then unlinks the selected task, drops the DSQ and destination rq locks and acquires the source rq lock before migrating it.
Quoted 6 lines
Whether the task may be remotely migrated can change across this handoff. In particular, proxy execution allows a task to execute as
rq->currunder another task's scheduling context while its own scheduling context remains on a DSQ. The task can therefore become on-CPU or migration disabled without dequeueing its DSQ entry or clearingholding_cpu.
I don't think this can happen w/o proxy execution. The task is enqueued and can't be dequeued or start running without grabbing the DSQ lock. Before the DSQ lock is dropped, p->scx.holding_cpu is set and if that hasn't been cleared, we know that nothing else has touched while we were switching locks.
If this no longer holds w/ proxy execution, adding further conditions are fine but let's make it explicitly for and confined to proxy execution enabled cases. The logic there is already subtle and I want to avoid adding misleading checks.
Thanks.
Signature · 1 line
tejun
Tejun Heo
reviewerRe: [PATCH 05/11] sched_ext: Fix TOCTOU race in consume_remote_task()
Quoted 4 lines · Tejun Heo
If this no longer holds w/ proxy execution, adding further conditions are fine but let's make it explicitly for and confined to proxy execution enabled cases. The logic there is already subtle and I want to avoid adding misleading checks.
Thought a bit more about it and the only thing that can change are the running state and migration disabled, right? So, maybe just test those again after the lock switch?
Thanks.
Signature · 1 line
tejun
Andrea Righi
reviewerRe: [PATCH 05/11] sched_ext: Fix TOCTOU race in consume_remote_task()
Hi Tejun,
Quoted 9 lines · Tejun Heo
Quoted 4 lines · Tejun Heo
If this no longer holds w/ proxy execution, adding further conditions are fine but let's make it explicitly for and confined to proxy execution enabled cases. The logic there is already subtle and I want to avoid adding misleading checks.
Thought a bit more about it and the only thing that can change are the running state and migration disabled, right? So, maybe just test those again after the lock switch?
Right, without proxy execution, the DSQ lock and holding_cpu handshake guarantee that with an unchanged holding_cpu the task could not have been dequeued or started running during the rq-lock handoff.
Proxy execution is the exception: a task can start physically executing as a lock owner while its own scheduling context remains on the DSQ, allowing its on-CPU or migration-disabled state to change without clearing holding_cpu.
I'll leave the existing validation unchanged and, only when proxy execution is enabled, recheck the running and migration-disabled state after switching to the source rq lock.
Thanks for taking a look!
-Andrea
Andrea Righi
reviewerPATCH 6/11 sched_ext: Split curr|donor references properly
In reply to Andrea RighiWith proxy execution, the task selected by the scheduler and the task physically executing can differ. A blocked mutex waiter donates its scheduling context to the lock owner:
D -----------------> M -------------> O ----------------> T [donor] blocked on [mutex] owned by [owner] preempted by [task]
\_________________________________^
donates scheduling context
where:
D = blocked donor
M = mutex
O = mutex owner
T = competing runnable task
During a proxy execution switch, D supplies the scheduling class, priority, and runtime budget, while O supplies the execution context: O is the task whose code physically executes. T is a competing runnable task which may preempt the D/O proxy execution.
Consider FAIR and EXT tasks with sched_ext running in partial mode. FAIR can be replaced with a higher scheduling class such as RT or deadline without changing the class interaction described here. The possible combinations are:
1. D is EXT, O is EXT, T is EXT
D can interrupt T according to BPF scheduling policy. O executes with D's EXT priority and runtime budget, while T waits in EXT.
2. D is EXT, O is EXT, T is FAIR
D is visible to the BPF scheduler, but cannot preempt T because EXT is below FAIR. Once T stops, BPF can dispatch D and O executes with D's EXT priority and runtime budget. If T becomes runnable again, it preempts the D/O proxy execution.
3. D is EXT, O is FAIR, T is EXT
This cannot represent T preempting O because EXT is below FAIR.
4. D is EXT, O is FAIR, T is FAIR
D cannot boost O above T because EXT is below FAIR. O and T continue competing under FAIR. Once O releases M, D wakes and resumes normal EXT scheduling.
5. D is FAIR, O is EXT, T is EXT
D preempts T as the higher-class scheduling context. O executes with D's FAIR priority and runtime budget, while T waits in EXT. D is not visible to the BPF scheduler.
6. D is FAIR, O is EXT, T is FAIR
D competes with T according to its FAIR deadline. When D is selected, O executes with D's FAIR priority and runtime budget. D is not visible to the BPF scheduler.
7. D is FAIR, O is FAIR, T is EXT
This cannot represent T preempting O because EXT is below FAIR.
8. D is FAIR, O is FAIR, T is FAIR
O, T, and D all have FAIR scheduling contexts. D remains runnable as a blocked proxy donor. When CFS selects D, O executes using D's FAIR scheduling context. When CFS selects O, O executes using its own FAIR context, and when CFS selects T, T executes normally. D is not visible to the BPF scheduler.
Thus, sched_ext policy and accounting must generally use rq->donor, the scheduler-selected task which supplies the scheduling context, rather than rq->curr, the task whose code physically executes. Without proxy execution they are the same task.
On nohz_full CPUs, a blocked proxy donor must retain the scheduler tick even when it has an infinite slice. Otherwise, a full dynticks CPU could stop the tick while rq->curr and rq->donor differ, violating assumptions made by the remote NOHZ tick path.
This is a conservative compromise that keeps the change local to sched_ext, at the cost of a periodic tick while a blocked proxy donor is selected. Allowing blocked proxy donors to run tickless would require making the core scheduler's remote tick handling aware that rq->curr and rq->donor can differ.
Moreover, extend scx_dump_state() to report both contexts. Each CPU record now includes a donor= line. If an EXT donor differs from rq->curr, also emit its detailed task record. The existing '*' marker continues to identify rq->curr, while the donor= line identifies the otherwise unmarked donor record.
Note that at this point in the series, CONFIG_SCHED_PROXY_EXEC still depends on !CONFIG_SCHED_CLASS_EXT, so proxy execution and sched_ext cannot be enabled together. The scheduling changes are therefore preparatory. A later patch removes this restriction.
--- kernel/sched/ext/ext.c | 93 +++++++++++++++++++++++++++--------------- kernel/sched/ext/sub.h | 8 ++-- 2 files changed, 64 insertions(+), 37 deletions(-)
kernel/sched/ext/ext.c +60 -33
@@ -1333,20 +1333,27 @@ static void apply_task_slice_oob(struct rq *rq, struct task_struct *p)static void update_curr_scx (struct rq *rq) { -struct task_struct *curr = rq->curr; +struct task_struct *donor;s64 delta_exec; +/* +* update_curr_scx() is selected through rq->donor->sched_class, not +* rq->curr->sched_class, so @donor is always an EXT task here. If an EXT +* owner executes for a FAIR donor, FAIR's update_curr() runs instead. +*/ + donor = rq->donor; +/* apply even on 0 delta_exec, callers may still act on the slice */ -apply_task_slice_oob (rq, curr); +apply_task_slice_oob (rq, donor); delta_exec =update_curr_common (rq);if (unlikely (delta_exec <=0 ))return ; -if (curr->scx.slice !=SCX_SLICE_INF ) { - curr->scx.slice -=min_t (u64, curr->scx.slice, delta_exec); -if (!curr->scx.slice) -touch_core_sched (rq, curr); +if (donor->scx.slice !=SCX_SLICE_INF ) { + donor->scx.slice -=min_t (u64, donor->scx.slice, delta_exec); +if (!donor->scx.slice) +touch_core_sched (rq, donor); }dl_server_update (&rq->ext_server, delta_exec);
@@ -1516,9 +1523,9 @@ static void rq_owned_post_enq(struct scx_sched *sch, struct rq *rq,if (rq->scx.flags &SCX_RQ_IN_BALANCE )return ; -if ((enq_flags &SCX_ENQ_PREEMPT ) && p != rq->curr && - rq->curr->sched_class == &ext_sched_class) { -set_task_slice (rq->curr,0 ); +if ((enq_flags &SCX_ENQ_PREEMPT ) && p != rq->donor && + rq->donor->sched_class == &ext_sched_class) { +set_task_slice (rq->donor,0 );resched_curr (rq); } }
@@ -2732,7 +2739,8 @@ static void dispatch_to_local_dsq(struct scx_sched *sch, struct rq *rq, }/* if the destination CPU is idle, wake it up */ -if (!fallback &&sched_class_above (p->sched_class, dst_rq->curr->sched_class)) +if (!fallback &&sched_class_above (p->sched_class, + dst_rq->donor->sched_class))resched_curr (dst_rq); }
@@ -2943,6 +2951,7 @@ static int balance_one(struct rq *rq, struct task_struct *prev)static void set_next_task_scx (struct rq *rq,struct task_struct *p,bool first) {struct scx_sched *sch =scx_task_sched (p); +bool can_stop_tick;if (p->scx.flags &SCX_TASK_QUEUED ) {/*
@@ -2971,6 +2980,7 @@ static void set_next_task_scx(struct rq *rq, struct task_struct *p, bool first)/* apply any pending out-of-band slice request before the tick decision */ apply_task_slice_oob (rq, p); + can_stop_tick = p->scx.slice ==SCX_SLICE_INF && !p->is_blocked;/* * @p is getting newly scheduled or got kicked after someone updated its
@@ -2981,7 +2991,7 @@ static void set_next_task_scx(struct rq *rq, struct task_struct *p, bool first)* nohz. In the future, we might want to add a mechanism to update * load_avgs periodically on tick-stopped CPUs. */ -if (p->scx.slice ==SCX_SLICE_INF ) { +if (can_stop_tick) {if (!(rq->scx.flags &SCX_RQ_CAN_STOP_TICK )) {/* * Bypass mode always assigns finite slices, so @p
@@ -3002,7 +3012,8 @@ static void set_next_task_scx(struct rq *rq, struct task_struct *p, bool first)/* * @rq still references the outgoing scheduling context. A finite -* slice is sufficient by itself to require the tick. +* slice or a blocked proxy donor is sufficient by itself to require +* the tick. */ if (tick_nohz_full_cpu (cpu_of (rq)))tick_nohz_dep_set_cpu (cpu_of (rq),TICK_DEP_BIT_SCHED );
@@ -3177,7 +3188,7 @@ static struct task_struct *first_local_task(struct rq *rq)static struct task_struct *do_pick_task_scx (struct rq *rq,struct rq_flags *rf,bool force_scx) { -struct task_struct *prev = rq->curr; +struct task_struct *prev = rq->donor;bool keep_prev;struct task_struct *p;
@@ -3537,9 +3548,9 @@ void scx_tick(struct rq *rq)update_other_load_avgs (rq); } -static void task_tick_scx (struct rq *rq,struct task_struct *curr,int queued) +static void task_tick_scx (struct rq *rq,struct task_struct *donor,int queued) { -struct scx_sched *sch =scx_task_sched (curr); +struct scx_sched *sch =scx_task_sched (donor);update_curr_scx (rq);
@@ -3548,13 +3559,13 @@ static void task_tick_scx(struct rq *rq, struct task_struct *curr, int queued)* we can't trust the slice management or ops.core_sched_before(). */ if (scx_bypassing (sch,cpu_of (rq))) { -set_task_slice (curr,0 ); -touch_core_sched (rq, curr); +set_task_slice (donor,0 ); +touch_core_sched (rq, donor); }else if (SCX_HAS_OP (sch, tick)) { -SCX_CALL_OP_TASK (sch, tick, rq, curr); +SCX_CALL_OP_TASK (sch, tick, rq, donor); } -if (!curr->scx.slice) +if (!donor->scx.slice)resched_curr (rq); }
@@ -4165,16 +4176,16 @@ static u32 reenq_local(struct scx_sched *sch, struct rq *rq, u64 reenq_flags) }/* -* The revoke that scheduled this scan may have raced the pick: curr +* The revoke that scheduled this scan may have raced the pick: donor * may be a now-capless task, either one that kept running or one * promoted off the local DSQ between the ecaps sync and this scan. * Zero the slice to evict it. The enqueue gate blocks new capless * inserts, so no later pick can slip through after the scan. */ if ((reenq_flags &SCX_REENQ_CAP_REVOKE ) && - rq->curr->sched_class == &ext_sched_class && -scx_task_reenq_on_cap_revoke (rq, rq->curr)) { -set_task_slice (rq->curr,0 ); + rq->donor->sched_class == &ext_sched_class && +scx_task_reenq_on_cap_revoke (rq, rq->donor)) { +set_task_slice (rq->donor,0 );resched_curr (rq); }
@@ -4360,14 +4371,14 @@ static void run_deferred(struct rq *rq)#ifdef CONFIG_NO_HZ_FULL bool scx_can_stop_tick (struct rq *rq) { -struct task_struct *p = rq->curr; +struct task_struct *p = rq->donor;struct scx_sched *sch =scx_task_sched (p);if (p->sched_class != &ext_sched_class)return true;/* -* @rq->curr may still reference an outgoing EXT task after it has been +* @rq->donor may still reference an outgoing EXT task after it has been * dequeued. If no EXT tasks are accounted on @rq, ignore its stale * slice state. If another task is dispatched from a DSQ, * set_next_task_scx() will update the dependency for the incoming task.
@@ -4381,7 +4392,8 @@ bool scx_can_stop_tick(struct rq *rq)/* * @rq can dispatch from different DSQs, so we can't tell whether it * needs the tick or not by looking at nr_running. Allow stopping ticks -* iff the BPF scheduler indicated so. See set_next_task_scx(). +* iff set_next_task_scx() determined that the selected scheduling context +* can run tickless. */ return rq->scx.flags &SCX_RQ_CAN_STOP_TICK ; }
@@ -6480,6 +6492,9 @@ static void scx_dump_cpu(struct scx_sched *sch, struct seq_buf *s,dump_line (&ns," curr=%s[%d] class=%ps" , rq->curr->comm, rq->curr->pid, rq->curr->sched_class); +dump_line (&ns," donor=%s[%d] class=%ps" , + rq->donor->comm, rq->donor->pid, + rq->donor->sched_class);if (!cpumask_empty (pcpu->cpus_to_kick))dump_line (&ns," cpus_to_kick : %*pb" ,cpumask_pr_args (pcpu->cpus_to_kick));
@@ -6523,6 +6538,10 @@ static void scx_dump_cpu(struct scx_sched *sch, struct seq_buf *s,if (rq->curr->sched_class == &ext_sched_class && (dump_all_tasks ||scx_task_on_sched (sch, rq->curr)))scx_dump_task (sch, s, dctx, rq, rq->curr,'*' ); +if (rq->donor != rq->curr && + rq->donor->sched_class == &ext_sched_class && + (dump_all_tasks ||scx_task_on_sched (sch, rq->donor))) +scx_dump_task (sch, s, dctx, rq, rq->donor,' ' );list_for_each_entry (p, &rq->scx.runnable_list, scx.runnable_node)if (dump_all_tasks ||scx_task_on_sched (sch, p))
@@ -8035,7 +8054,7 @@ static bool kick_one_cpu(s32 cpu, struct scx_sched_pcpu *pcpu, struct rq *this_runsigned long flags;raw_spin_rq_lock_irqsave (rq, flags); - cur_class = rq->curr->sched_class; + cur_class = rq->donor->sched_class;/* * During CPU hotplug, a CPU may depend on kicking itself to make
@@ -8052,7 +8071,7 @@ static bool kick_one_cpu(s32 cpu, struct scx_sched_pcpu *pcpu, struct rq *this_rif (cur_class == &ext_sched_class) {if (likely (!scx_missing_caps (pcpu->sch, cpu,scx_caps_for_preempt (pcpu->sch, rq)))) -set_task_slice (rq->curr,0 ); +set_task_slice (rq->donor,0 );else __scx_add_event (pcpu->sch,SCX_EV_SUB_PREEMPT_DENIED ,1 );
@@ -9864,12 +9883,16 @@ __bpf_kfunc void scx_bpf_put_cpumask(const struct cpumask *cpumask) }/** -* scx_bpf_task_running - Is task currently running? +* scx_bpf_task_running - Is task the current scheduling context? * @p: task of interest +* +* Under proxy execution, this reports the donor rather than the task whose +* code is physically executing. ops.running() continues to report when the +* task itself starts executing. */ __bpf_kfunc bool scx_bpf_task_running(const struct task_struct *p) { -return task_rq (p)->curr == p; +return rcu_access_pointer (task_rq (p)->donor) == p; }/**
@@ -9926,10 +9949,14 @@ __bpf_kfunc struct rq *scx_bpf_locked_rq(const struct bpf_prog_aux *aux) }/** -* scx_bpf_cpu_curr - Return remote CPU's curr task +* scx_bpf_cpu_curr - Return remote CPU's current scheduling context * @cpu: CPU of interest * @aux: implicit BPF argument to access bpf_prog_aux hidden from BPF progs * +* Under proxy execution, this returns the donor, which supplies the scheduling +* policy and runtime budget, rather than the task whose code is physically +* executing. ops.running() continues to report physical task execution. +* * Callers must hold RCU read lock (KF_RCU). */ __bpf_kfunc struct task_struct *scx_bpf_cpu_curr(s32 cpu, const struct bpf_prog_aux *aux)
@@ -9945,7 +9972,7 @@ __bpf_kfunc struct task_struct *scx_bpf_cpu_curr(s32 cpu, const struct bpf_prog_if (!scx_cpu_valid (sch, cpu,NULL ))return NULL ; -return rcu_dereference (cpu_rq (cpu)->curr); +return rcu_dereference (cpu_rq (cpu)->donor); }/**
@@ -9969,7 +9996,7 @@ __bpf_kfunc struct task_struct *scx_bpf_cid_curr(s32 cid, const struct bpf_prog_ cpu =scx_cid_to_cpu (sch, cid);if (cpu <0 )return NULL ; -return rcu_dereference (cpu_rq (cpu)->curr); +return rcu_dereference (cpu_rq (cpu)->donor); }/**
kernel/sched/ext/sub.h +4 -4
@@ -141,14 +141,14 @@ static inline u64 scx_caps_for_task(struct task_struct *p)return SCX_CAP_ENQ ; } -/* the cap @sch needs to preempt @rq's current task, 0 if none */ +/* the cap @sch needs to preempt @rq's current scheduling context, 0 if none */ static inline u64 scx_caps_for_preempt (struct scx_sched *sch,struct rq *rq) { -struct task_struct *curr = rq->curr; +struct task_struct *donor = rq->donor;/* a non-ext task can't be preempted by ext, own-subtree needs no cap */ -if (curr->sched_class != &ext_sched_class || -scx_is_descendant (scx_task_sched (curr), sch)) +if (donor->sched_class != &ext_sched_class || +scx_is_descendant (scx_task_sched (donor), sch))return 0 ;return SCX_CAP_PREEMPT ; }
Signature · 1 line
2.55.0
[email protected]
AI reviewRe: [PATCH 06/11] sched_ext: Split curr|donor references properly
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [High] Missing TICK_DEP_BIT_SCHED protection for cross-class proxy execution causes remote NOHZ tick…
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [High] Missing TICK_DEP_BIT_SCHED protection for cross-class proxy execution causes remote NOHZ tick warnings and accounting corruption.
- [Medium] The ops.running() callback is never invoked for the physical lock owner under proxy execution, violating the documented API contract. --
--- Patch [6]: [PATCH 06/11] sched_ext: Split curr|donor references properly --- commit 4630a7edb1e1687e72cb21f49d2599ffe928b4de
Author: Andrea Righi <[email protected]>
sched_ext: Split curr|donor references properly
This commit prepares sched_ext for proxy execution by differentiating between the scheduling context provider (rq->donor) and the physically executing task (rq->curr). It adjusts tick management, task selection, and BPF helper functions to use the donor where appropriate.
Quoted 3 lines
kernel/sched/ext/ext.c +0
-0
[ ... ]
kernel/sched/ext/ext.c static void run_deferred(struct rq *rq) · line 4388
Quoted 10 lines
kernel/sched/ext/ext.c +1
-1@@ -4370,14 +4381,14 @@ static void run_deferred(struct rq *rq)#ifdef CONFIG_NO_HZ_FULL bool scx_can_stop_tick (struct rq *rq) { -struct task_struct *p = rq->curr; +struct task_struct *p = rq->donor;struct scx_sched *sch =scx_task_sched (p);if (p->sched_class != &ext_sched_class)return true;
[Severity: High]
Does this missing check cause remote NOHZ tick warnings during cross-class proxy execution?
If a FAIR task donates to an EXT task, rq->donor is FAIR but rq->curr is EXT. When scx_can_stop_tick() is called, it checks p->sched_class (which is FAIR) and immediately returns true. If the FAIR donor isn't bandwidth constrained, this allows the tick to stop.
However, stopping the tick when rq->curr != rq->donor violates core NOHZ assumptions and triggers a WARN_ON_ONCE in sched_tick_remote().
Will this also corrupt time bandwidth accounting by charging the EXT owner instead of the FAIR donor?
[ ... ]
kernel/sched/ext/ext.c __bpf_kfunc struct rq *scx_bpf_locked_rq(const struct bpf_prog_aux *aux) · line 9965
Quoted 13 lines
kernel/sched/ext/ext.c +5
-1@@ -9932,10 +9955,14 @@ __bpf_kfunc struct rq *scx_bpf_locked_rq(const struct bpf_prog_aux *aux) }/** -* scx_bpf_cpu_curr - Return remote CPU's curr task +* scx_bpf_cpu_curr - Return remote CPU's current scheduling context * @cpu: CPU of interest * @aux: implicit BPF argument to access bpf_prog_aux hidden from BPF progs * +* Under proxy execution, this returns the donor, which supplies the scheduling +* policy and runtime budget, rather than the task whose code is physically +* executing. ops.running() continues to report physical task execution. +*
[Severity: Medium]
Is ops.running() actually invoked for the physical lock owner as documented here?
When a task donates its context, the core scheduler calls set_next_task_scx() for the blocked donor. However, set_next_task_scx() explicitly skips ops.running() for blocked tasks:
set_next_task_scx() {
...
if ((p->scx.flags & SCX_TASK_QUEUED) && !p->is_blocked) {
if (SCX_HAS_OP(sch, running))
SCX_CALL_OP_TASK(sch, running, rq, p);
}
...
}Since the core scheduler context-switches to the physical lock owner without invoking its scheduling class callbacks, does this leave the BPF scheduler completely blind to the physical task execution?
Quoted 3 lines
kernel/sched/ext/ext.c +0
-0* Callers must hold RCU read lock (KF_RCU). */ __bpf_kfunc struct task_struct *scx_bpf_cpu_curr(s32 cpu, const struct bpf_prog_aux *aux)
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=6
Andrea Righi
reviewerPATCH 7/11 sched_ext: Handle blocked donor migration with proxy execution
In reply to Andrea RighiFrom: John Stultz <[email protected]>
With proxy execution enabled, mutex-blocked donors stay runnable so their scheduling context can execute the lock owner.
sched_ext may normally relocate a queued blocked donor. set_task_cpu() updates both task_cpu() and wake_cpu, making the destination rq the donor's new callback home. This remains valid if the donor was previously
proxy-migrated: the BPF scheduler is intentionally selecting a new return destination, which the next proxy migration will preserve. The usual migration-disabled, affinity, and rq-online checks apply.
An active donor cannot be moved normally because the source rq still references it through rq->donor for scheduling-class operations and runtime accounting. Moving it would associate the task with a different rq while leaving those source-rq references in place. The core proxy migration path avoids this by switching the rq donor to idle before moving the scheduling context.
Check active-donor state in task_can_move_from_locked_rq(), where the source rq is held and the result is authoritative. Keep it out of the preliminary lockless DSQ filter so a donor transition during the rq-lock handoff is resolved by the locked recheck rather than consuming a dispatch kick and skipping the task.
Allow normal sched_ext migration of blocked donors unless the donor is active on its rq. In that case, defer migration until it is switched out or leave it to the proxy machinery.
Keep blocked donors on the local DSQ when they are put so they remain visible to the proxy pick path.
--- kernel/sched/ext/ext.c | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+)
kernel/sched/ext/ext.c +20 -0
@@ -2444,6 +2444,10 @@ static bool task_can_move_from_locked_rq(struct scx_sched *sch,if (task_on_cpu (src_rq, p))return false; +/* Don't move a donor while its source rq still references it. */ +if (p->is_blocked &&rcu_access_pointer (src_rq->donor) == p) +return false; +return task_can_run_on_remote_rq (sch, p, dst_rq, enforce); }
@@ -3093,6 +3097,22 @@ static void put_prev_task_scx(struct rq *rq, struct task_struct *p,if (p->scx.flags &SCX_TASK_QUEUED ) {set_task_runnable (rq, p); +/* +* Mutex-blocked donors stay queued on the runqueue under proxy +* execution, but the donor never runs as itself, proxy-exec +* walks the blocked_on chain on the next __schedule() and runs +* the lock owner in its place. +* +* Put the donor on the local DSQ directly so pick_next_task() +* can still see it. find_proxy_task() will either run the chain +* owner or deactivate the donor so the wakeup path can return it +* and let BPF make a new dispatch decision once it is unblocked. +*/ +if (p->is_blocked) { +scx_dispatch_enqueue (sch, rq, &rq->scx.local_dsq, p,0 ); + goto switch_class; + } +/* * If @p has slice left and is being put, @p is getting * preempted by a higher priority scheduler class or core-sched
Signature · 1 line
2.55.0
[email protected]
AI reviewRe: [PATCH 07/11] sched_ext: Handle blocked donor migration with proxy execution
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [High] Woken proxy donors can be migrated while still referenced by `rq->donor`, causing data corruption.…
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [High] Woken proxy donors can be migrated while still referenced by rq->donor, causing data corruption.
- [High] Blocked proxy donors are enqueued to the tail of the local DSQ, defeating priority inheritance and bypassing SCX_ENQ_HEAD for preempted tasks with remaining slice. --
--- Patch [7]: [PATCH 07/11] sched_ext: Handle blocked donor migration with proxy execution --- commit e2d80964c94aa96c6517c0aa914235417a9eb814
Author: John Stultz <[email protected]>
sched_ext: Handle blocked donor migration with proxy execution
With proxy execution enabled, mutex-blocked donors stay runnable so their scheduling context can execute the lock owner. This patch ensures that active donors are not migrated by the BPF scheduler while the source runqueue still references them. It also keeps blocked donors on the local dispatch queue when they are put so they remain visible to the proxy pick path.
kernel/sched/ext/ext.c static bool task_can_move_from_locked_rq(struct scx_sched *sch, · line 2449
Quoted 12 lines
kernel/sched/ext/ext.c +4
-0@@ -2443,6 +2443,10 @@ static bool task_can_move_from_locked_rq(struct scx_sched *sch,if (task_on_cpu (src_rq, p))return false; +/* Don't move a donor while its source rq still references it. */ +if (p->is_blocked &&rcu_access_pointer (src_rq->donor) == p) +return false; +
[Severity: High]
Does checking p->is_blocked here leave a race window where the task can be migrated while still referenced by src_rq->donor?
If the mutex is unlocked, waking up the donor task clears its blocked state via ttwu_do_wakeup(). If the lock owner is still running and src_rq hasn't rescheduled yet, src_rq->donor will still point to the task.
When the BPF scheduler picks the task for migration, this check might erroneously pass because p->is_blocked is now 0. If the task is transferred to dst_rq, a subsequent update_curr_scx() could modify p->scx.slice while holding src_rq's lock, racing with dst_rq which now owns the task.
Should this only check rcu_access_pointer(src_rq->donor) == p to prevent the race?
Quoted 2 lines
kernel/sched/ext/ext.c +0
-0return task_can_run_on_remote_rq (sch, p, dst_rq, enforce); }
[ ... ]
Quoted 10 lines
kernel/sched/ext/ext.c +6
-0@@ -3092,6 +3096,22 @@ static void put_prev_task_scx(struct rq *rq, struct task_struct *p,if (p->scx.flags &SCX_TASK_QUEUED ) {set_task_runnable (rq, p); +/* +* Mutex-blocked donors stay queued on the runqueue under proxy +* execution, but the donor never runs as itself, proxy-exec +* walks the blocked_on chain on the next __schedule() and runs +* the lock owner in its place. +*
kernel/sched/ext/ext.c static void put_prev_task_scx(struct rq *rq, struct task_struct *p, · line 3114
Quoted 10 lines
kernel/sched/ext/ext.c +10
-0+* Put the donor on the local DSQ directly so pick_next_task() +* can still see it. find_proxy_task() will either run the chain +* owner or deactivate the donor so the wakeup path can return it +* and let BPF make a new dispatch decision once it is unblocked. +*/ +if (p->is_blocked) { +scx_dispatch_enqueue (sch, rq, &rq->scx.local_dsq, p,0 ); + goto switch_class; + } +
[Severity: High]
Does passing 0 as the enqueue flag here put the blocked donor at the tail of the local DSQ instead of the head?
When a lock owner running on behalf of a blocked donor is preempted, this code unconditionally uses 0 and skips the remaining slice check via the goto. Enqueuing with flag 0 places the donor at the tail of the built-in FIFO local DSQ instead of the head (SCX_ENQ_HEAD).
Could this cause priority inversion, as any tasks previously dispatched to the local DSQ would run before the lock owner can resume, even if the preempted donor still has slice remaining?
Quoted 2 lines
kernel/sched/ext/ext.c +0
-0/* * If @p has slice left and is being put, @p is getting
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=7
Andrea Righi
reviewerPATCH 8/11 sched_ext: Delegate proxy donor admission to BPF schedulers
In reply to Andrea RighiProxy execution keeps a blocked donor runnable so its scheduling context can execute the mutex owner. Dispatching sched_ext donors on a local DSQ bypasses the BPF scheduler ordering policy and can give donors more CPU priority than intended to perform the proxy execution handoff.
Add SCX_OPS_ENQ_BLOCKED as an explicit proxy execution capability. Tasks owned by schedulers without the flag block normally. Schedulers with the flag receive blocked donors through ops.enqueue() with SCX_ENQ_BLOCKED set in enq_flags and can apply their own admission policy.
From a high-level perspective, the resulting flow for a BPF scheduler with SCX_OPS_ENQ_BLOCKED is:
D ------ blocked on -----> M ------ owned by -----> O
[donor] [mutex] [owner]
|
| ops.enqueue(D, SCX_ENQ_BLOCKED)
| BPF dispatches D to CPUi
v
+-----------------+
| CPUi local DSQ |
+-------+---------+
|
| pick_next_task() selects D
v
+-----------------+
| proxy exec | move D to O's rq
+-------+---------+
|
| run O using D's scheduling context
v
rq->curr = O
rq->donor = D
|
| O releases M
v
+-----------------+
| proxy exec | return D to CPUi via wakeup
+-----------------+A proxy migration preserves the donor's wake_cpu while moving its scheduling context to the owner's rq, so task_cpu() differs from wake_cpu until the donor wakes. Prevent BPF-directed migrations from pulling a blocked donor off this proxy rq. Otherwise each blocked re-enqueue may move the donor back to its callback rq only for proxy execution to move it to the owner again.
Scheduler ownership can change after a donor has already blocked. Since sched_change preserves queued state, handle both iterator-driven moves to root or sub-schedulers and sched_setscheduler() transitions into EXT. Before entering a scheduler without the flag, deactivate the retained donor. It remains on the mutex wait path and wakes normally when the mutex becomes available.
The global sched_ext enable state can change while __schedule() holds the runqueue lock. Once an EXT task has an assigned scheduler, consult it directly so the task cannot retain a donor during the transition unless the scheduler opted in. Tasks without an assigned scheduler keep the generic proxy execution behavior.
The preparation helpers require and assert that p->pi_lock and p's rq lock are held before updating rq state.
The donor starts associated with its original CPU. A BPF scheduler may dispatch it directly into that CPU local DSQ to let the core resolve the mutex owner and execute it with the donor scheduling context.
A normal sleeper also has p->is_blocked set until after wakeup_preempt(). Exclude full wakeup activations from SCX_ENQ_BLOCKED and carry a transient task flag from enqueue_task_scx() to wakeup_preempt_scx(). Retained donor wakeups skip enqueue_task_scx(), so they remain distinguishable without changing core scheduler wake flags.
Reschedule a retained donor when its mutex wakes it so ops.dispatch() can reconsider the now-unblocked task. Make SCX_OPS_ENQ_BLOCKED override the exiting and migration-disabled enqueue fallbacks so opted-in schedulers receive all eligible donor requests.
--- include/linux/sched/ext.h | 1 + kernel/sched/ext/ext.c | 109 ++++++++++++++---- kernel/sched/ext/internal.h | 23 +++- kernel/sched/ext/sub.c | 12 +- tools/sched_ext/include/scx/compat.h | 1 + .../sched_ext/include/scx/enum_defs.autogen.h | 1 + .../sched_ext/include/scx/enums.autogen.bpf.h | 3 + tools/sched_ext/include/scx/enums.autogen.h | 1 + 8 files changed, 127 insertions(+), 24 deletions(-)
include/linux/sched/ext.h +1 -0
@@ -104,6 +104,7 @@ enum scx_ent_flags {SCX_TASK_IMMED =1 <<5 ,/* task is on local DSQ with %SCX_ENQ_IMMED */ SCX_TASK_RUN_TRACKED =1 <<6 ,/* task is in an ops.running()/stopping() session */ + SCX_TASK_ENQ_WAKEUP = 1 << 7, /* wakeup enqueue awaiting wakeup_preempt() *//* * Bits 8 to 10 are used to carry task state:
kernel/sched/ext/ext.c +89 -20
205 lines not on this page · raw message
kernel/sched/ext/internal.h +22 -1
@@ -213,6 +213,19 @@ enum scx_ops_flags {*/ SCX_OPS_TID_TO_TASK =1LLU <<8 , +/* +* If set, mutex-blocked tasks remain runnable as proxy donors and are +* passed to ops.enqueue() with %SCX_ENQ_BLOCKED. The BPF scheduler controls +* when donors are dispatched and whether they should preempt other work. +* +* If clear, mutex-blocked tasks are removed from the runqueue normally +* and cannot donate their scheduling context through proxy execution. +* +* For blocked donors, this flag takes precedence over +* %SCX_OPS_ENQ_EXITING and %SCX_OPS_ENQ_MIGRATION_DISABLED. +*/ +SCX_OPS_ENQ_BLOCKED =1LLU <<9 , +SCX_OPS_ALL_FLAGS =SCX_OPS_KEEP_BUILTIN_IDLE |SCX_OPS_ENQ_LAST | SCX_OPS_ENQ_EXITING |
@@ -221,7 +234,8 @@ enum scx_ops_flags {SCX_OPS_SWITCH_PARTIAL |SCX_OPS_BUILTIN_IDLE_PER_NODE |SCX_OPS_ALWAYS_ENQ_IMMED | -SCX_OPS_TID_TO_TASK , +SCX_OPS_TID_TO_TASK | +SCX_OPS_ENQ_BLOCKED ,/* high 8 bits are internal, don't include in SCX_OPS_ALL_FLAGS */ __SCX_OPS_INTERNAL_MASK = 0xffLLU << 56,
@@ -1614,6 +1628,12 @@ enum scx_enq_flags {*/ SCX_ENQ_LAST =1LLU <<41 , +/* +* The task is blocked on a mutex and is being kept runnable as a proxy +* donor. Only passed to ops.enqueue() when %SCX_OPS_ENQ_BLOCKED is set. +*/ +SCX_ENQ_BLOCKED =1LLU <<42 , +/* high 8 bits are internal */ __SCX_ENQ_INTERNAL_MASK = 0xffLLU << 56,
@@ -1909,6 +1929,7 @@ void scx_task_iter_start(struct scx_task_iter *iter, struct cgroup *cgrp);void scx_task_iter_unlock (struct scx_task_iter *iter);void scx_task_iter_stop (struct scx_task_iter *iter);struct task_struct *scx_task_iter_next_locked (struct scx_task_iter *iter); +void scx_prepare_task_sched_change (struct task_struct *p,struct scx_sched *sch);void scx_dispatch_dequeue (struct rq *rq,struct task_struct *p);void scx_do_enqueue_task (struct rq *rq,struct task_struct *p,u64 enq_flags,int sticky_cpu);
kernel/sched/ext/sub.c +9 -3
@@ -780,7 +780,9 @@ static void scx_fail_parent(struct scx_sched *sch,if (scx_task_on_sched (parent, p))continue ; - scoped_guard (sched_change, p, DEQUEUE_SAVE | DEQUEUE_MOVE) { +scx_prepare_task_sched_change (p, parent); +scoped_guard (sched_change, p,DEQUEUE_SAVE |DEQUEUE_MOVE | + DEQUEUE_NOCLOCK) {scx_disable_and_exit_task (sch, p);scx_set_task_sched (p, parent); }
@@ -872,7 +874,9 @@ void scx_sub_disable(struct scx_sched *sch)continue ; } - scoped_guard (sched_change, p, DEQUEUE_SAVE | DEQUEUE_MOVE) { +scx_prepare_task_sched_change (p, parent); +scoped_guard (sched_change, p,DEQUEUE_SAVE |DEQUEUE_MOVE | + DEQUEUE_NOCLOCK) {/* * $p is initialized for $parent and still attached to * @sch. Disable and exit for @sch, switch over to
@@ -1207,7 +1211,9 @@ void scx_sub_enable_workfn(struct kthread_work *work)if (!(p->scx.flags &SCX_TASK_SUB_INIT ))continue ; - scoped_guard (sched_change, p, DEQUEUE_SAVE | DEQUEUE_MOVE) { +scx_prepare_task_sched_change (p, sch); +scoped_guard (sched_change, p,DEQUEUE_SAVE |DEQUEUE_MOVE | + DEQUEUE_NOCLOCK) {/* * $p must be either READY or ENABLED. If ENABLED, * __scx_disabled_and_exit_task() first disables and
tools/sched_ext/include/scx/compat.h +1 -0
@@ -117,6 +117,7 @@ static inline bool __COMPAT_struct_has_field(const char *type, const char *field#define SCX_OPS_ALLOW_QUEUED_WAKEUP SCX_OPS_FLAG(SCX_OPS_ALLOW_QUEUED_WAKEUP)#define SCX_OPS_BUILTIN_IDLE_PER_NODE SCX_OPS_FLAG(SCX_OPS_BUILTIN_IDLE_PER_NODE)#define SCX_OPS_ALWAYS_ENQ_IMMED SCX_OPS_FLAG(SCX_OPS_ALWAYS_ENQ_IMMED) +#define SCX_OPS_ENQ_BLOCKED SCX_OPS_FLAG(SCX_OPS_ENQ_BLOCKED)#define SCX_PICK_IDLE_FLAG (name) __COMPAT_ENUM_OR_ZERO("scx_pick_idle_cpu_flags", #name)
tools/sched_ext/include/scx/enum_defs.autogen.h +1 -0
@@ -55,6 +55,7 @@#define HAVE_SCX_ENQ_IMMED #define HAVE_SCX_ENQ_REENQ #define HAVE_SCX_ENQ_LAST +#define HAVE_SCX_ENQ_BLOCKED #define HAVE___SCX_ENQ_INTERNAL_MASK #define HAVE_SCX_ENQ_CLEAR_OPSS #define HAVE_SCX_ENQ_DSQ_PRIQ
tools/sched_ext/include/scx/enums.autogen.bpf.h +3 -0
@@ -130,6 +130,9 @@ const volatile u64 __SCX_ENQ_REENQ __weak;const volatile u64 __SCX_ENQ_LAST__weak ;#define SCX_ENQ_LAST __SCX_ENQ_LAST +const volatile u64 __SCX_ENQ_BLOCKED__weak ; +#define SCX_ENQ_BLOCKED __SCX_ENQ_BLOCKED +const volatile u64 __SCX_ENQ_CLEAR_OPSS__weak ;#define SCX_ENQ_CLEAR_OPSS __SCX_ENQ_CLEAR_OPSS
tools/sched_ext/include/scx/enums.autogen.h +1 -0
@@ -47,6 +47,7 @@SCX_ENUM_SET (skel, scx_enq_flags,SCX_ENQ_IMMED ); \SCX_ENUM_SET (skel, scx_enq_flags,SCX_ENQ_REENQ ); \SCX_ENUM_SET (skel, scx_enq_flags,SCX_ENQ_LAST ); \ +SCX_ENUM_SET (skel, scx_enq_flags,SCX_ENQ_BLOCKED ); \SCX_ENUM_SET (skel, scx_enq_flags,SCX_ENQ_CLEAR_OPSS ); \SCX_ENUM_SET (skel, scx_enq_flags,SCX_ENQ_DSQ_PRIQ ); \SCX_ENUM_SET (skel, scx_deq_flags,SCX_DEQ_SCHED_CHANGE ); \
Signature · 1 line
2.55.0
Re: [PATCH 08/11] sched_ext: Delegate proxy donor admission to BPF schedulers
Quoted 16 lines · Andrea Righi
kernel/sched/ext/ext.c +4
-7@@ -3098,18 +3164,16 @@ static void put_prev_task_scx(struct rq *rq, struct task_struct *p,set_task_runnable (rq, p);/* -* Mutex-blocked donors stay queued on the runqueue under proxy -* execution, but the donor never runs as itself, proxy-exec -* walks the blocked_on chain on the next __schedule() and runs -* the lock owner in its place. +* Mutex-blocked donors only stay queued when their BPF scheduler +* enables %SCX_OPS_ENQ_BLOCKED. The rq lock has remained held since +* scx_allow_proxy_exec(), so @p's scheduler association cannot have +* changed and @sch must be non-NULL with the flag set. * -* Put the donor on the local DSQ directly so pick_next_task() -* can still see it. find_proxy_task() will either run the chain -* owner or deactivate the donor so the wakeup path can return it
kernel/sched/ext/ext.c static void put_prev_task_scx(struct rq *rq, struct task_struct *p, · line 3179
Quoted 10 lines
kernel/sched/ext/ext.c +3
-2-* and let BPF make a new dispatch decision once it is unblocked. +* Delegate admission to the BPF scheduler. */ if (p->is_blocked) { -scx_dispatch_enqueue (sch, rq, &rq->scx.local_dsq, p,0 ); +WARN_ON_ONCE (!(sch->ops.flags &SCX_OPS_ENQ_BLOCKED )); +scx_do_enqueue_task (rq, p,0 ,-1 ); goto switch_class; }
So this isn't a blocker for your patches, but just as a heads up: when applying my sleeping owner handling changes (even just patches 4-7 from my v30 submission[1]), I managed to trip the above WARN_ON, when running the test-ww_mutex driver under the scx_pair (usually right as scx_pair loads).
[ 5818.970324] WARNING: kernel/sched/ext/ext.c:3175 at put_prev_task_scx+0x527/0x550, CPU#54: kworker/u261:11/30527 [ 5818.974927] CPU: 54 UID: 0 PID: 30527 Comm: kworker/u261:11 Tainted: G W 7.1.0-13303-gb5ecf3c9f881 #116 PREEMPT(full) [ 5818.979484] Tainted: [W]=WARN [ 5818.980654] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-debian-1.17.0-1 04/01/2014 [ 5818.984225] Workqueue: test-ww_mutex test_cycle_work [ 5818.986199] Sched_ext: pair (enabled+all), task: runnable_at=-369ms [ 5818.986202] RIP: 0010:put_prev_task_scx+0x527/0x550 [ 5818.990473] Code: 48 2b 05 34 f3 21 03 75 3c 48 83 c4 30 48 89 de 48 89 ef b9 ff ff ff ff 5b 31 d2 5d 41 5c 41 5d 41 5e 41 5f e9 fa f0 ff ff 90 <0f> 0b 90 e9 b8 fe ff ff 90 0f 0b 90 e9 35 fe ff ff b8 02 00 00 00 [ 5818.997428] RSP: 0018:ffffc900088dfb28 EFLAGS: 00010046 [ 5818.999426] RAX: 0000000000000036 RBX: ffff888101258000 RCX: ffff888101d64990 [ 5819.002113] RDX: ffff8881b9bae638 RSI: ffff888101d64990 RDI: ffff888101258390 [ 5819.004787] RBP: ffff8881b9badb00 R08: ffff8881b9badb00 R09: 0000000000000000 [ 5819.007967] R10: 0000000000000036 R11: ffff888120a9b030 R12: ffff888100c30000 [ 5819.010697] R13: ffff88810d163800 R14: ffff888101258000 R15: ffff888100c30000 [ 5819.013378] FS: 0000000000000000(0000) GS:ffff8882355c0000(0000) knlGS:0000000000000000 [ 5819.016354] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5819.018524] CR2: 00007ffe7b985348 CR3: 0000000123ec6004 CR4: 0000000000370ef0 [ 5819.021163] Call Trace: [ 5819.022168] <TASK> [ 5819.023010] __schedule+0x15c4/0x2460 [ 5819.024504] ? find_held_lock+0x2b/0x80 [ 5819.025981] ? lock_release+0x191/0x310 [ 5819.027503] schedule+0x3d/0x130 [ 5819.028772] schedule_preempt_disabled+0x18/0x30 [ 5819.030539] __ww_mutex_lock.constprop.0+0xaea/0x19e0 [ 5819.032492] ? schedule_timeout+0xca/0x130 [ 5819.034095] ? test_cycle_work+0x15d/0x340 [ 5819.035686] ? ww_mutex_lock+0x3c/0xb0 [ 5819.037094] ww_mutex_lock+0x3c/0xb0 [ 5819.038494] test_cycle_work+0x15d/0x340 [ 5819.039949] process_one_work+0x20c/0x5e0 [ 5819.041631] ? lock_is_held_type+0xcd/0x130 [ 5819.043277] worker_thread+0x1a4/0x360 [ 5819.044717] ? __pfx_worker_thread+0x10/0x10 [ 5819.046392] kthread+0x103/0x130 [ 5819.047650] ? __pfx_kthread+0x10/0x10 [ 5819.049074] ret_from_fork+0x27c/0x340 [ 5819.050525] ? __pfx_kthread+0x10/0x10 [ 5819.051990] ret_from_fork_asm+0x1a/0x30 [ 5819.053488] </TASK> [ 5819.054371] irq event stamp: 38 [ 5819.055583] hardirqs last enabled at (37): [<ffffffff826a7100>] _raw_spin_unlock_irqrestore+0x50/0x60 [ 5819.059054] hardirqs last disabled at (38): [<ffffffff82699205>] __schedule+0xb95/0x2460 [ 5819.061998] softirqs last enabled at (26): [<ffffffff813244d0>] __irq_exit_rcu+0xe0/0x150 [ 5819.065015] softirqs last disabled at (21): [<ffffffff813244d0>] __irq_exit_rcu+0xe0/0x150 [ 5819.068013] ---[ end trace 0000000000000000 ]---
It seems to be coming from the proxy_resched_idle() call in the "if (!READ_ONCE(owner->on_rq) || owner->se.sched_delayed) {" case in find_proxy_task(), prior to proxy_enqueue_on_owner() calling block_task().
I'll have to dig a bit more next week on this, as I'm not yet seeing whats going wrong here.
Anyway, I'm very eager to see your changes move forward!
thanks
-john
Re: [PATCH 08/11] sched_ext: Delegate proxy donor admission to BPF schedulers
Quoted 94 lines · John Stultz
Quoted 16 lines · Andrea Righi
kernel/sched/ext/ext.c +4
-7@@ -3098,18 +3164,16 @@ static void put_prev_task_scx(struct rq *rq, struct task_struct *p,set_task_runnable (rq, p);/* -* Mutex-blocked donors stay queued on the runqueue under proxy -* execution, but the donor never runs as itself, proxy-exec -* walks the blocked_on chain on the next __schedule() and runs -* the lock owner in its place. +* Mutex-blocked donors only stay queued when their BPF scheduler +* enables %SCX_OPS_ENQ_BLOCKED. The rq lock has remained held since +* scx_allow_proxy_exec(), so @p's scheduler association cannot have +* changed and @sch must be non-NULL with the flag set. * -* Put the donor on the local DSQ directly so pick_next_task() -* can still see it. find_proxy_task() will either run the chain -* owner or deactivate the donor so the wakeup path can return it kernel/sched/ext/ext.c static void put_prev_task_scx(struct rq *rq, struct task_struct *p, · line 3179
Quoted 10 lines
kernel/sched/ext/ext.c +3
-2-* and let BPF make a new dispatch decision once it is unblocked. +* Delegate admission to the BPF scheduler. */ if (p->is_blocked) { -scx_dispatch_enqueue (sch, rq, &rq->scx.local_dsq, p,0 ); +WARN_ON_ONCE (!(sch->ops.flags &SCX_OPS_ENQ_BLOCKED )); +scx_do_enqueue_task (rq, p,0 ,-1 ); goto switch_class; }So this isn't a blocker for your patches, but just as a heads up: when applying my sleeping owner handling changes (even just patches 4-7 from my v30 submission[1]), I managed to trip the above
WARN_ON, when running the test-ww_mutex driver under thescx_pair(usually right asscx_pairloads).[ 5818.970324] WARNING: kernel/sched/ext/ext.c:3175 at put_prev_task_scx+0x527/0x550, CPU#54: kworker/u261:11/30527 [ 5818.974927] CPU: 54 UID: 0 PID: 30527 Comm: kworker/u261:11 Tainted: G W 7.1.0-13303-gb5ecf3c9f881 #116 PREEMPT(full) [ 5818.979484] Tainted: [W]=WARN [ 5818.980654] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-debian-1.17.0-1 04/01/2014 [ 5818.984225] Workqueue: test-ww_mutex test_cycle_work [ 5818.986199] Sched_ext: pair (enabled+all), task: runnable_at=-369ms [ 5818.986202] RIP: 0010:put_prev_task_scx+0x527/0x550 [ 5818.990473] Code: 48 2b 05 34 f3 21 03 75 3c 48 83 c4 30 48 89 de 48 89 ef b9 ff ff ff ff 5b 31 d2 5d 41 5c 41 5d 41 5e 41 5f e9 fa f0 ff ff 90 <0f> 0b 90 e9 b8 fe ff ff 90 0f 0b 90 e9 35 fe ff ff b8 02 00 00 00 [ 5818.997428] RSP: 0018:ffffc900088dfb28 EFLAGS: 00010046 [ 5818.999426] RAX: 0000000000000036 RBX: ffff888101258000 RCX: ffff888101d64990 [ 5819.002113] RDX: ffff8881b9bae638 RSI: ffff888101d64990 RDI: ffff888101258390 [ 5819.004787] RBP: ffff8881b9badb00 R08: ffff8881b9badb00 R09: 0000000000000000 [ 5819.007967] R10: 0000000000000036 R11: ffff888120a9b030 R12: ffff888100c30000 [ 5819.010697] R13: ffff88810d163800 R14: ffff888101258000 R15: ffff888100c30000 [ 5819.013378] FS: 0000000000000000(0000) GS:ffff8882355c0000(0000) knlGS:0000000000000000 [ 5819.016354] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5819.018524] CR2: 00007ffe7b985348 CR3: 0000000123ec6004 CR4: 0000000000370ef0 [ 5819.021163] Call Trace: [ 5819.022168] <TASK> [ 5819.023010] __schedule+0x15c4/0x2460 [ 5819.024504] ? find_held_lock+0x2b/0x80 [ 5819.025981] ? lock_release+0x191/0x310 [ 5819.027503] schedule+0x3d/0x130 [ 5819.028772] schedule_preempt_disabled+0x18/0x30 [ 5819.030539] __ww_mutex_lock.constprop.0+0xaea/0x19e0 [ 5819.032492] ? schedule_timeout+0xca/0x130 [ 5819.034095] ? test_cycle_work+0x15d/0x340 [ 5819.035686] ? ww_mutex_lock+0x3c/0xb0 [ 5819.037094] ww_mutex_lock+0x3c/0xb0 [ 5819.038494] test_cycle_work+0x15d/0x340 [ 5819.039949] process_one_work+0x20c/0x5e0 [ 5819.041631] ? lock_is_held_type+0xcd/0x130 [ 5819.043277] worker_thread+0x1a4/0x360 [ 5819.044717] ? __pfx_worker_thread+0x10/0x10 [ 5819.046392] kthread+0x103/0x130 [ 5819.047650] ? __pfx_kthread+0x10/0x10 [ 5819.049074] ret_from_fork+0x27c/0x340 [ 5819.050525] ? __pfx_kthread+0x10/0x10 [ 5819.051990] ret_from_fork_asm+0x1a/0x30 [ 5819.053488] </TASK> [ 5819.054371] irq event stamp: 38 [ 5819.055583] hardirqs last enabled at (37): [<ffffffff826a7100>] _raw_spin_unlock_irqrestore+0x50/0x60 [ 5819.059054] hardirqs last disabled at (38): [<ffffffff82699205>] __schedule+0xb95/0x2460 [ 5819.061998] softirqs last enabled at (26): [<ffffffff813244d0>] __irq_exit_rcu+0xe0/0x150 [ 5819.065015] softirqs last disabled at (21): [<ffffffff813244d0>] __irq_exit_rcu+0xe0/0x150 [ 5819.068013] ---[ end trace 0000000000000000 ]---
Quoted 7 lines
It seems to be coming from the
proxy_resched_idle()call in the "if (!READ_ONCE(owner->on_rq) ||owner->se.sched_delayed) {" case infind_proxy_task(), prior toproxy_enqueue_on_owner()callingblock_task().I'll have to dig a bit more next week on this, as I'm not yet seeing whats going wrong here.
Ok, my tired theory is something like:
1) You have a task A that that is_blocked waiting on a sleeping owner B. It gets enqueued onto that owner and waits.
2) We start scx_pair, and scx_prepare_task_sched_change() calls sched_proxy_block_task(), which bails because !task_on_rq_queued()
3) B wakes up, and that causes us to activate_blocked_waiters(), which re-adds A (with is_blocked still set) to the rq
- this is problematic because now we have is_blocked tasks on thesched_ext DSQ where its not allowed.
4) A is selected as a donor, and maybe B is sleeping again, so we call put_prev_set_next() and hti the warning because we see A is is_blocked when the sched_ext scheduler doesn't support it.
I'll work to prove this out a bit further next week. I suspect we'll need something somewhere between activate_blocked_waiters() -> scx_do_enqueue_task() to skip enquing of is_blocked tasks when SCX_OPS_ENQ_BLOCKED isn't set.
thanks
-john
Andrea Righi
reviewerRe: [PATCH 08/11] sched_ext: Delegate proxy donor admission to BPF schedulers
Hi John,
Quoted 116 lines · John Stultz
Quoted 94 lines · John Stultz
Quoted 16 lines · Andrea Righi
kernel/sched/ext/ext.c +4
-7@@ -3098,18 +3164,16 @@ static void put_prev_task_scx(struct rq *rq, struct task_struct *p,set_task_runnable (rq, p);/* -* Mutex-blocked donors stay queued on the runqueue under proxy -* execution, but the donor never runs as itself, proxy-exec -* walks the blocked_on chain on the next __schedule() and runs -* the lock owner in its place. +* Mutex-blocked donors only stay queued when their BPF scheduler +* enables %SCX_OPS_ENQ_BLOCKED. The rq lock has remained held since +* scx_allow_proxy_exec(), so @p's scheduler association cannot have +* changed and @sch must be non-NULL with the flag set. * -* Put the donor on the local DSQ directly so pick_next_task() -* can still see it. find_proxy_task() will either run the chain -* owner or deactivate the donor so the wakeup path can return it kernel/sched/ext/ext.c static void put_prev_task_scx(struct rq *rq, struct task_struct *p, · line 3179
Quoted 10 lines
kernel/sched/ext/ext.c +3
-2-* and let BPF make a new dispatch decision once it is unblocked. +* Delegate admission to the BPF scheduler. */ if (p->is_blocked) { -scx_dispatch_enqueue (sch, rq, &rq->scx.local_dsq, p,0 ); +WARN_ON_ONCE (!(sch->ops.flags &SCX_OPS_ENQ_BLOCKED )); +scx_do_enqueue_task (rq, p,0 ,-1 ); goto switch_class; }So this isn't a blocker for your patches, but just as a heads up: when applying my sleeping owner handling changes (even just patches 4-7 from my v30 submission[1]), I managed to trip the above
WARN_ON, when running the test-ww_mutex driver under thescx_pair(usually right asscx_pairloads).[ 5818.970324] WARNING: kernel/sched/ext/ext.c:3175 at put_prev_task_scx+0x527/0x550, CPU#54: kworker/u261:11/30527 [ 5818.974927] CPU: 54 UID: 0 PID: 30527 Comm: kworker/u261:11 Tainted: G W 7.1.0-13303-gb5ecf3c9f881 #116 PREEMPT(full) [ 5818.979484] Tainted: [W]=WARN [ 5818.980654] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-debian-1.17.0-1 04/01/2014 [ 5818.984225] Workqueue: test-ww_mutex test_cycle_work [ 5818.986199] Sched_ext: pair (enabled+all), task: runnable_at=-369ms [ 5818.986202] RIP: 0010:put_prev_task_scx+0x527/0x550 [ 5818.990473] Code: 48 2b 05 34 f3 21 03 75 3c 48 83 c4 30 48 89 de 48 89 ef b9 ff ff ff ff 5b 31 d2 5d 41 5c 41 5d 41 5e 41 5f e9 fa f0 ff ff 90 <0f> 0b 90 e9 b8 fe ff ff 90 0f 0b 90 e9 35 fe ff ff b8 02 00 00 00 [ 5818.997428] RSP: 0018:ffffc900088dfb28 EFLAGS: 00010046 [ 5818.999426] RAX: 0000000000000036 RBX: ffff888101258000 RCX: ffff888101d64990 [ 5819.002113] RDX: ffff8881b9bae638 RSI: ffff888101d64990 RDI: ffff888101258390 [ 5819.004787] RBP: ffff8881b9badb00 R08: ffff8881b9badb00 R09: 0000000000000000 [ 5819.007967] R10: 0000000000000036 R11: ffff888120a9b030 R12: ffff888100c30000 [ 5819.010697] R13: ffff88810d163800 R14: ffff888101258000 R15: ffff888100c30000 [ 5819.013378] FS: 0000000000000000(0000) GS:ffff8882355c0000(0000) knlGS:0000000000000000 [ 5819.016354] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5819.018524] CR2: 00007ffe7b985348 CR3: 0000000123ec6004 CR4: 0000000000370ef0 [ 5819.021163] Call Trace: [ 5819.022168] <TASK> [ 5819.023010] __schedule+0x15c4/0x2460 [ 5819.024504] ? find_held_lock+0x2b/0x80 [ 5819.025981] ? lock_release+0x191/0x310 [ 5819.027503] schedule+0x3d/0x130 [ 5819.028772] schedule_preempt_disabled+0x18/0x30 [ 5819.030539] __ww_mutex_lock.constprop.0+0xaea/0x19e0 [ 5819.032492] ? schedule_timeout+0xca/0x130 [ 5819.034095] ? test_cycle_work+0x15d/0x340 [ 5819.035686] ? ww_mutex_lock+0x3c/0xb0 [ 5819.037094] ww_mutex_lock+0x3c/0xb0 [ 5819.038494] test_cycle_work+0x15d/0x340 [ 5819.039949] process_one_work+0x20c/0x5e0 [ 5819.041631] ? lock_is_held_type+0xcd/0x130 [ 5819.043277] worker_thread+0x1a4/0x360 [ 5819.044717] ? __pfx_worker_thread+0x10/0x10 [ 5819.046392] kthread+0x103/0x130 [ 5819.047650] ? __pfx_kthread+0x10/0x10 [ 5819.049074] ret_from_fork+0x27c/0x340 [ 5819.050525] ? __pfx_kthread+0x10/0x10 [ 5819.051990] ret_from_fork_asm+0x1a/0x30 [ 5819.053488] </TASK> [ 5819.054371] irq event stamp: 38 [ 5819.055583] hardirqs last enabled at (37): [<ffffffff826a7100>] _raw_spin_unlock_irqrestore+0x50/0x60 [ 5819.059054] hardirqs last disabled at (38): [<ffffffff82699205>] __schedule+0xb95/0x2460 [ 5819.061998] softirqs last enabled at (26): [<ffffffff813244d0>] __irq_exit_rcu+0xe0/0x150 [ 5819.065015] softirqs last disabled at (21): [<ffffffff813244d0>] __irq_exit_rcu+0xe0/0x150 [ 5819.068013] ---[ end trace 0000000000000000 ]---Quoted 7 lines
It seems to be coming from the
proxy_resched_idle()call in the "if (!READ_ONCE(owner->on_rq) ||owner->se.sched_delayed) {" case infind_proxy_task(), prior toproxy_enqueue_on_owner()callingblock_task().I'll have to dig a bit more next week on this, as I'm not yet seeing whats going wrong here.
Ok, my tired theory is something like:
1) You have a task A that that
is_blockedwaiting on a sleeping owner B. It gets enqueued onto that owner and waits.2) We start
scx_pair, andscx_prepare_task_sched_change()callssched_proxy_block_task(), which bails because !task_on_rq_queued()3) B wakes up, and that causes us to
activate_blocked_waiters(), which re-adds A (withis_blockedstill set) to the rq
- this is problematic because now we haveis_blockedtasks on thesched_extDSQ where its not allowed.
Quoted 8 lines
4) A is selected as a donor, and maybe B is sleeping again, so we call
put_prev_set_next()and hti the warning because we see A isis_blockedwhen thesched_extscheduler doesn't support it.I'll work to prove this out a bit further next week. I suspect we'll need something somewhere between
activate_blocked_waiters()->scx_do_enqueue_task()to skip enquing ofis_blockedtasks whenSCX_OPS_ENQ_BLOCKEDisn't set.
Your theory looks correct to me. I was also able to reproduce this. I think the problem is that scx_prepare_task_sched_change() calls sched_proxy_block_task(), but the latter has nothing to do when the donor is already off the runqueue behind a sleeping owner.
When the owner subsequently wakes, activate_blocked_waiters() unconditionally reactivates the donor while is_blocked is still set. Since this activation carries ENQUEUE_WAKEUP, sched_ext treats it as a normal wakeup rather than a blocked-donor admission. The donor can therefore enter an scx scheduler that doesn't set SCX_OPS_ENQ_BLOCKED (scx_pair in this case), eventually triggering the warning in put_prev_task_scx(): unexpected proxy donor without SCX_OPS_ENQ_BLOCKED set.
I think we can fix this by checking scx_allow_proxy_exec() in do_activate_blocked_waiter() before reactivating the donor. If the scx scheduler doesn't support proxy donors (SCX_OPS_ENQ_BLOCKED not set), the task should remain blocked and will be activated normally by the mutex wakeup.
While looking at this, I noticed another issue in the same path:activate_blocked_waiters() currently passes ENQUEUE_WAKEUP because the generic scheduling classes need wakeup-style enqueue accounting. However, this is not a real mutex wakeup, the donor remains blocked and is only being made runnable so that it can donate its scheduling context.
As a result, even a scheduler that sets SCX_OPS_ENQ_BLOCKED currently receives this sleeping-owner activation as SCX_ENQ_WAKEUP rather than SCX_ENQ_BLOCKED. That's because the existing sched_ext test:
p->is_blocked && !(enq_flags & SCX_ENQ_WAKEUP)
doesn't distinguish a genuine mutex wakeup from this proxy activation.
Maybe we can address this by adding an internal ENQUEUE_PROXY flag to the sleeping-owner activation. The generic classes will continue to see ENQUEUE_WAKEUP, preserving their accounting behavior, while sched_ext will use ENQUEUE_PROXY to expose the event to BPF as SCX_ENQ_BLOCKED without SCX_ENQ_WAKEUP. And a genuine mutex wakeup will continue to be reported as SCX_ENQ_WAKEUP. What do you think?
Thanks,
-Andrea
Andrea Righi
reviewerRe: [PATCH 08/11] sched_ext: Delegate proxy donor admission to BPF schedulers
Hi John,
Quoted 158 lines · Andrea Righi
Hi John,
Quoted 116 lines · John Stultz
Quoted 94 lines · John Stultz
Quoted 16 lines · Andrea Righi
kernel/sched/ext/ext.c +4
-7@@ -3098,18 +3164,16 @@ static void put_prev_task_scx(struct rq *rq, struct task_struct *p,set_task_runnable (rq, p);/* -* Mutex-blocked donors stay queued on the runqueue under proxy -* execution, but the donor never runs as itself, proxy-exec -* walks the blocked_on chain on the next __schedule() and runs -* the lock owner in its place. +* Mutex-blocked donors only stay queued when their BPF scheduler +* enables %SCX_OPS_ENQ_BLOCKED. The rq lock has remained held since +* scx_allow_proxy_exec(), so @p's scheduler association cannot have +* changed and @sch must be non-NULL with the flag set. * -* Put the donor on the local DSQ directly so pick_next_task() -* can still see it. find_proxy_task() will either run the chain -* owner or deactivate the donor so the wakeup path can return it kernel/sched/ext/ext.c static void put_prev_task_scx(struct rq *rq, struct task_struct *p, · line 3179
Quoted 10 lines
kernel/sched/ext/ext.c +3
-2-* and let BPF make a new dispatch decision once it is unblocked. +* Delegate admission to the BPF scheduler. */ if (p->is_blocked) { -scx_dispatch_enqueue (sch, rq, &rq->scx.local_dsq, p,0 ); +WARN_ON_ONCE (!(sch->ops.flags &SCX_OPS_ENQ_BLOCKED )); +scx_do_enqueue_task (rq, p,0 ,-1 ); goto switch_class; }So this isn't a blocker for your patches, but just as a heads up: when applying my sleeping owner handling changes (even just patches 4-7 from my v30 submission[1]), I managed to trip the above
WARN_ON, when running the test-ww_mutex driver under thescx_pair(usually right asscx_pairloads).[ 5818.970324] WARNING: kernel/sched/ext/ext.c:3175 at put_prev_task_scx+0x527/0x550, CPU#54: kworker/u261:11/30527 [ 5818.974927] CPU: 54 UID: 0 PID: 30527 Comm: kworker/u261:11 Tainted: G W 7.1.0-13303-gb5ecf3c9f881 #116 PREEMPT(full) [ 5818.979484] Tainted: [W]=WARN [ 5818.980654] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-debian-1.17.0-1 04/01/2014 [ 5818.984225] Workqueue: test-ww_mutex test_cycle_work [ 5818.986199] Sched_ext: pair (enabled+all), task: runnable_at=-369ms [ 5818.986202] RIP: 0010:put_prev_task_scx+0x527/0x550 [ 5818.990473] Code: 48 2b 05 34 f3 21 03 75 3c 48 83 c4 30 48 89 de 48 89 ef b9 ff ff ff ff 5b 31 d2 5d 41 5c 41 5d 41 5e 41 5f e9 fa f0 ff ff 90 <0f> 0b 90 e9 b8 fe ff ff 90 0f 0b 90 e9 35 fe ff ff b8 02 00 00 00 [ 5818.997428] RSP: 0018:ffffc900088dfb28 EFLAGS: 00010046 [ 5818.999426] RAX: 0000000000000036 RBX: ffff888101258000 RCX: ffff888101d64990 [ 5819.002113] RDX: ffff8881b9bae638 RSI: ffff888101d64990 RDI: ffff888101258390 [ 5819.004787] RBP: ffff8881b9badb00 R08: ffff8881b9badb00 R09: 0000000000000000 [ 5819.007967] R10: 0000000000000036 R11: ffff888120a9b030 R12: ffff888100c30000 [ 5819.010697] R13: ffff88810d163800 R14: ffff888101258000 R15: ffff888100c30000 [ 5819.013378] FS: 0000000000000000(0000) GS:ffff8882355c0000(0000) knlGS:0000000000000000 [ 5819.016354] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5819.018524] CR2: 00007ffe7b985348 CR3: 0000000123ec6004 CR4: 0000000000370ef0 [ 5819.021163] Call Trace: [ 5819.022168] <TASK> [ 5819.023010] __schedule+0x15c4/0x2460 [ 5819.024504] ? find_held_lock+0x2b/0x80 [ 5819.025981] ? lock_release+0x191/0x310 [ 5819.027503] schedule+0x3d/0x130 [ 5819.028772] schedule_preempt_disabled+0x18/0x30 [ 5819.030539] __ww_mutex_lock.constprop.0+0xaea/0x19e0 [ 5819.032492] ? schedule_timeout+0xca/0x130 [ 5819.034095] ? test_cycle_work+0x15d/0x340 [ 5819.035686] ? ww_mutex_lock+0x3c/0xb0 [ 5819.037094] ww_mutex_lock+0x3c/0xb0 [ 5819.038494] test_cycle_work+0x15d/0x340 [ 5819.039949] process_one_work+0x20c/0x5e0 [ 5819.041631] ? lock_is_held_type+0xcd/0x130 [ 5819.043277] worker_thread+0x1a4/0x360 [ 5819.044717] ? __pfx_worker_thread+0x10/0x10 [ 5819.046392] kthread+0x103/0x130 [ 5819.047650] ? __pfx_kthread+0x10/0x10 [ 5819.049074] ret_from_fork+0x27c/0x340 [ 5819.050525] ? __pfx_kthread+0x10/0x10 [ 5819.051990] ret_from_fork_asm+0x1a/0x30 [ 5819.053488] </TASK> [ 5819.054371] irq event stamp: 38 [ 5819.055583] hardirqs last enabled at (37): [<ffffffff826a7100>] _raw_spin_unlock_irqrestore+0x50/0x60 [ 5819.059054] hardirqs last disabled at (38): [<ffffffff82699205>] __schedule+0xb95/0x2460 [ 5819.061998] softirqs last enabled at (26): [<ffffffff813244d0>] __irq_exit_rcu+0xe0/0x150 [ 5819.065015] softirqs last disabled at (21): [<ffffffff813244d0>] __irq_exit_rcu+0xe0/0x150 [ 5819.068013] ---[ end trace 0000000000000000 ]---Quoted 7 lines
It seems to be coming from the
proxy_resched_idle()call in the "if (!READ_ONCE(owner->on_rq) ||owner->se.sched_delayed) {" case infind_proxy_task(), prior toproxy_enqueue_on_owner()callingblock_task().I'll have to dig a bit more next week on this, as I'm not yet seeing whats going wrong here.
Ok, my tired theory is something like:
1) You have a task A that that
is_blockedwaiting on a sleeping owner B. It gets enqueued onto that owner and waits.2) We start
scx_pair, andscx_prepare_task_sched_change()callssched_proxy_block_task(), which bails because !task_on_rq_queued()3) B wakes up, and that causes us to
activate_blocked_waiters(), which re-adds A (withis_blockedstill set) to the rq
- this is problematic because now we haveis_blockedtasks on thesched_extDSQ where its not allowed.Quoted 8 lines
4) A is selected as a donor, and maybe B is sleeping again, so we call
put_prev_set_next()and hti the warning because we see A isis_blockedwhen thesched_extscheduler doesn't support it.I'll work to prove this out a bit further next week. I suspect we'll need something somewhere between
activate_blocked_waiters()->scx_do_enqueue_task()to skip enquing ofis_blockedtasks whenSCX_OPS_ENQ_BLOCKEDisn't set.Your theory looks correct to me. I was also able to reproduce this. I think the problem is that
scx_prepare_task_sched_change()callssched_proxy_block_task(), but the latter has nothing to do when the donor is already off the runqueue behind a sleeping owner.When the owner subsequently wakes,
activate_blocked_waiters()unconditionally reactivates the donor whileis_blockedis still set. Since this activation carriesENQUEUE_WAKEUP,sched_exttreats it as a normal wakeup rather than a blocked-donor admission. The donor can therefore enter an scx scheduler that doesn't setSCX_OPS_ENQ_BLOCKED(scx_pairin this case), eventually triggering the warning input_prev_task_scx(): unexpected proxy donor withoutSCX_OPS_ENQ_BLOCKEDset.I think we can fix this by checking
scx_allow_proxy_exec()indo_activate_blocked_waiter()before reactivating the donor. If the scx scheduler doesn't support proxy donors (SCX_OPS_ENQ_BLOCKEDnot set), the task should remain blocked and will be activated normally by the mutex wakeup.While looking at this, I noticed another issue in the same path:
activate_blocked_waiters()currently passesENQUEUE_WAKEUPbecause the generic scheduling classes need wakeup-style enqueue accounting. However, this is not a real mutex wakeup, the donor remains blocked and is only being made runnable so that it can donate its scheduling context.As a result, even a scheduler that sets
SCX_OPS_ENQ_BLOCKEDcurrently receives this sleeping-owner activation asSCX_ENQ_WAKEUPrather thanSCX_ENQ_BLOCKED. That's because the existingsched_exttest:p->is_blocked && !(enq_flags & SCX_ENQ_WAKEUP)
Quoted 8 lines
doesn't distinguish a genuine mutex wakeup from this proxy activation.
Maybe we can address this by adding an internal
ENQUEUE_PROXYflag to the sleeping-owner activation. The generic classes will continue to seeENQUEUE_WAKEUP, preserving their accounting behavior, whilesched_extwill useENQUEUE_PROXYto expose the event to BPF asSCX_ENQ_BLOCKEDwithoutSCX_ENQ_WAKEUP. And a genuine mutex wakeup will continue to be reported asSCX_ENQ_WAKEUP. What do you think?
FYI, I've applied your sleeping owner changes + the scx-proxy-exec patch series with the changes mentioned above here:
git://git.kernel.org/pub/scm/linux/kernel/git/arighi/linux.git scx-proxy-exec-next
Everything looks good on my side so far.
Thanks,
-Andrea
Re: [PATCH 08/11] sched_ext: Delegate proxy donor admission to BPF schedulers
Quoted 46 lines · Andrea Righi
Quoted 36 lines · Andrea Righi
Quoted 4 lines · John Stultz
I'll work to prove this out a bit further next week. I suspect we'll need something somewhere between
activate_blocked_waiters()->scx_do_enqueue_task()to skip enquing ofis_blockedtasks whenSCX_OPS_ENQ_BLOCKEDisn't set.Your theory looks correct to me. I was also able to reproduce this. I think the problem is that
scx_prepare_task_sched_change()callssched_proxy_block_task(), but the latter has nothing to do when the donor is already off the runqueue behind a sleeping owner.When the owner subsequently wakes,
activate_blocked_waiters()unconditionally reactivates the donor whileis_blockedis still set. Since this activation carriesENQUEUE_WAKEUP,sched_exttreats it as a normal wakeup rather than a blocked-donor admission. The donor can therefore enter an scx scheduler that doesn't setSCX_OPS_ENQ_BLOCKED(scx_pairin this case), eventually triggering the warning input_prev_task_scx(): unexpected proxy donor withoutSCX_OPS_ENQ_BLOCKEDset.I think we can fix this by checking
scx_allow_proxy_exec()indo_activate_blocked_waiter()before reactivating the donor. If the scx scheduler doesn't support proxy donors (SCX_OPS_ENQ_BLOCKEDnot set), the task should remain blocked and will be activated normally by the mutex wakeup.While looking at this, I noticed another issue in the same path:
activate_blocked_waiters()currently passesENQUEUE_WAKEUPbecause the generic scheduling classes need wakeup-style enqueue accounting. However, this is not a real mutex wakeup, the donor remains blocked and is only being made runnable so that it can donate its scheduling context.As a result, even a scheduler that sets
SCX_OPS_ENQ_BLOCKEDcurrently receives this sleeping-owner activation asSCX_ENQ_WAKEUPrather thanSCX_ENQ_BLOCKED. That's because the existingsched_exttest:p->is_blocked && !(enq_flags & SCX_ENQ_WAKEUP)Quoted 8 lines
doesn't distinguish a genuine mutex wakeup from this proxy activation.
Maybe we can address this by adding an internal
ENQUEUE_PROXYflag to the sleeping-owner activation. The generic classes will continue to seeENQUEUE_WAKEUP, preserving their accounting behavior, whilesched_extwill useENQUEUE_PROXYto expose the event to BPF asSCX_ENQ_BLOCKEDwithoutSCX_ENQ_WAKEUP. And a genuine mutex wakeup will continue to be reported asSCX_ENQ_WAKEUP. What do you think?
Quoted 6 lines
FYI, I've applied your sleeping owner changes + the scx-proxy-exec patch series with the changes mentioned above here:
git://git.kernel.org/pub/scm/linux/kernel/git/arighi/linux.gitscx-proxy-exec-nextEverything looks good on my side so far.
That's awesome! Thank you for your efforts here! I'll start testing and playing around with this branch!
Though, looking at your patch stack, I'd rather not make my patches under review a dependency for your patches. Peter has not yet put his laser-eyes upon my series so there's no telling how much rework they may need (and it does have some subtle parts that I'm sure he'll have feedback on). So just to allow your current series to be able to move foward, would it make sense if I added your "sched: Distinguish proxy activations from wakeups" patch to the end of my series? It looks like there's not much interlock on the rest of your changes (though I've not carefully reviewed the diff from your scx-proxy-exec branch as they are on different bases), so maybe that would just work and the rest of your series could move forward?
thanks
-john
Andrea Righi
reviewerRe: [PATCH 08/11] sched_ext: Delegate proxy donor admission to BPF schedulers
Quoted 57 lines · John Stultz
Quoted 46 lines · Andrea Righi
Quoted 36 lines · Andrea Righi
Quoted 4 lines · John Stultz
I'll work to prove this out a bit further next week. I suspect we'll need something somewhere between
activate_blocked_waiters()->scx_do_enqueue_task()to skip enquing ofis_blockedtasks whenSCX_OPS_ENQ_BLOCKEDisn't set.Your theory looks correct to me. I was also able to reproduce this. I think the problem is that
scx_prepare_task_sched_change()callssched_proxy_block_task(), but the latter has nothing to do when the donor is already off the runqueue behind a sleeping owner.When the owner subsequently wakes,
activate_blocked_waiters()unconditionally reactivates the donor whileis_blockedis still set. Since this activation carriesENQUEUE_WAKEUP,sched_exttreats it as a normal wakeup rather than a blocked-donor admission. The donor can therefore enter an scx scheduler that doesn't setSCX_OPS_ENQ_BLOCKED(scx_pairin this case), eventually triggering the warning input_prev_task_scx(): unexpected proxy donor withoutSCX_OPS_ENQ_BLOCKEDset.I think we can fix this by checking
scx_allow_proxy_exec()indo_activate_blocked_waiter()before reactivating the donor. If the scx scheduler doesn't support proxy donors (SCX_OPS_ENQ_BLOCKEDnot set), the task should remain blocked and will be activated normally by the mutex wakeup.While looking at this, I noticed another issue in the same path:
activate_blocked_waiters()currently passesENQUEUE_WAKEUPbecause the generic scheduling classes need wakeup-style enqueue accounting. However, this is not a real mutex wakeup, the donor remains blocked and is only being made runnable so that it can donate its scheduling context.As a result, even a scheduler that sets
SCX_OPS_ENQ_BLOCKEDcurrently receives this sleeping-owner activation asSCX_ENQ_WAKEUPrather thanSCX_ENQ_BLOCKED. That's because the existingsched_exttest:p->is_blocked && !(enq_flags & SCX_ENQ_WAKEUP)Quoted 8 lines
doesn't distinguish a genuine mutex wakeup from this proxy activation.
Maybe we can address this by adding an internal
ENQUEUE_PROXYflag to the sleeping-owner activation. The generic classes will continue to seeENQUEUE_WAKEUP, preserving their accounting behavior, whilesched_extwill useENQUEUE_PROXYto expose the event to BPF asSCX_ENQ_BLOCKEDwithoutSCX_ENQ_WAKEUP. And a genuine mutex wakeup will continue to be reported asSCX_ENQ_WAKEUP. What do you think?Quoted 6 lines
FYI, I've applied your sleeping owner changes + the scx-proxy-exec patch series with the changes mentioned above here:
git://git.kernel.org/pub/scm/linux/kernel/git/arighi/linux.gitscx-proxy-exec-nextEverything looks good on my side so far.
That's awesome! Thank you for your efforts here! I'll start testing and playing around with this branch!
Quoted 11 lines
Though, looking at your patch stack, I'd rather not make my patches under review a dependency for your patches. Peter has not yet put his laser-eyes upon my series so there's no telling how much rework they may need (and it does have some subtle parts that I'm sure he'll have feedback on). So just to allow your current series to be able to move foward, would it make sense if I added your "sched: Distinguish proxy activations from wakeups" patch to the end of my series? It looks like there's not much interlock on the rest of your changes (though I've not carefully reviewed the diff from your scx-proxy-exec branch as they are on different bases), so maybe that would just work and the rest of your series could move forward?
Sounds like a good plan! It'd be helpful if you could include "sched: Distinguish proxy activations from wakeups" at the end of your sleeping-owner series.
I'll move the combined version to scx-proxy-exec-sleeping-owner for testing and rebase the scx-proxy-exec series without the sleeping-owner changes, so that it can move forward independently for now.
Thanks!
-Andrea
Re: [PATCH 08/11] sched_ext: Delegate proxy donor admission to BPF schedulers
Quoted 13 lines · Andrea Righi
Quoted 11 lines · John Stultz
Though, looking at your patch stack, I'd rather not make my patches under review a dependency for your patches. Peter has not yet put his laser-eyes upon my series so there's no telling how much rework they may need (and it does have some subtle parts that I'm sure he'll have feedback on). So just to allow your current series to be able to move foward, would it make sense if I added your "sched: Distinguish proxy activations from wakeups" patch to the end of my series? It looks like there's not much interlock on the rest of your changes (though I've not carefully reviewed the diff from your scx-proxy-exec branch as they are on different bases), so maybe that would just work and the rest of your series could move forward?
Quoted 3 lines
Sounds like a good plan! It'd be helpful if you could include "sched: Distinguish proxy activations from wakeups" at the end of your sleeping-owner series.
Will do! I've got a few more things to cleanup and get through testing, but I'm hoping to get v31 out to the list soon.
Quoted 3 lines
I'll move the combined version to scx-proxy-exec-sleeping-owner for testing and rebase the scx-proxy-exec series without the sleeping-owner changes, so that it can move forward independently for now.
Sounds good!
Thanks so much again for all your work here! Its really appreciated! -john
Andrea Righi
reviewerPATCH 9/11 sched_ext: Add selftest for blocked donor admission
In reply to Andrea RighiSCX_OPS_ENQ_BLOCKED allows BPF schedulers to receive blocked proxy donors through ops.enqueue(). SCX_ENQ_BLOCKED identifies blocked-donor admission requests. Add selftest coverage for this interface.
Exercise a priority inversion using a weighted-vruntime BPF scheduler. A nice +19 owner holds a shared mutex, a nice -20 donor blocks on it, and nice 0 CPU contenders, one per allowed CPU, keep the system busy. Test both a same-CPU topology and a cross-CPU topology with the donor and owner on different CPUs.
Treat blocked donors according to the normal BPF ordering policy and assign the default slice on every enqueue, as for other tasks. Run each CPU placement configuration with SCX_OPS_ENQ_BLOCKED first disabled and then enabled, count blocked-donor enqueues by CPU and report average mutex hold and wait times. Reject enqueues carrying both SCX_ENQ_WAKEUP and SCX_ENQ_BLOCKED, as full wakeups are not retained donor admissions.
Proxy execution coverage requires CONFIG_SCHED_PROXY_EXEC=y, which the selftest config selects. Access to the kernel mutex is provided via a loadable kernel module, built through TEST_GEN_MODS_DIR and managed by the test.
--- tools/testing/selftests/sched_ext/.gitignore | 4 + tools/testing/selftests/sched_ext/Makefile | 2 + tools/testing/selftests/sched_ext/config | 2 + .../selftests/sched_ext/enq_blocked.bpf.c | 116 +++ .../testing/selftests/sched_ext/enq_blocked.c | 908 ++++++++++++++++++ .../testing/selftests/sched_ext/enq_blocked.h | 27 + .../selftests/sched_ext/test_modules/Makefile | 13 + .../test_modules/scx_enq_blocked_test.c | 193 ++++ 8 files changed, 1265 insertions(+) create mode 100644 tools/testing/selftests/sched_ext/enq_blocked.bpf.c create mode 100644 tools/testing/selftests/sched_ext/enq_blocked.c create mode 100644 tools/testing/selftests/sched_ext/enq_blocked.h create mode 100644 tools/testing/selftests/sched_ext/test_modules/Makefile create mode 100644 tools/testing/selftests/sched_ext/test_modules/scx_enq_blocked_test.c
tools/testing/selftests/sched_ext/.gitignore +4 -0
@@ -4,3 +4,7 @@ !Makefile !.gitignore !config +!test_modules/ +!test_modules/scx_enq_blocked_test.c +!test_modules/Makefile +test_modules/*.mod.c
tools/testing/selftests/sched_ext/Makefile +2 -0
@@ -5,6 +5,7 @@ include ../../../scripts/Makefile.arch include ../../../scripts/Makefile.include TEST_GEN_PROGS := runner +TEST_GEN_MODS_DIR := test_modules # override lib.mk's default rules OVERRIDE_TARGETS := 1
@@ -164,6 +165,7 @@ all_test_bpfprogs := $(foreach prog,$(wildcard *.bpf.c),$(INCLUDE_DIR)/$(patsubs auto-test-targets := \ create_dsq \ dequeue \ + enq_blocked \ enq_last_no_enq_fails \ ddsp_bogus_dsq_fail \ ddsp_vtimelocal_fail \
tools/testing/selftests/sched_ext/config +2 -0
@@ -6,3 +6,5 @@ CONFIG_BPF=y CONFIG_BPF_SYSCALL=y CONFIG_DEBUG_INFO=y CONFIG_DEBUG_INFO_BTF=y +CONFIG_EXPERT=y +CONFIG_SCHED_PROXY_EXEC=y
tools/testing/selftests/sched_ext/enq_blocked.bpf.c new file +116 -0
116 lines not on this page · raw message
tools/testing/selftests/sched_ext/enq_blocked.c new file +908 -0
908 lines not on this page · raw message
tools/testing/selftests/sched_ext/enq_blocked.h new file +27 -0
@@ -0,0 +1,27 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES */ +#ifndef __ENQ_BLOCKED_H +#define __ENQ_BLOCKED_H + +#include <linux/ioctl.h> +#include <linux/types.h> + +struct enq_blocked_stats { +__u64 hold_time_ns; +__u64 wait_time_ns; +__u64 nr_holds; +__u64 nr_waits; +}; + +#define ENQ_BLOCKED_IOCTL_OWNER _IO('s', 1) +#define ENQ_BLOCKED_IOCTL_DONOR _IO('s', 2) +#define ENQ_BLOCKED_IOCTL_RESET_STATS _IO('s', 3) +#define ENQ_BLOCKED_IOCTL_GET_STATS _IOR('s', 4, struct enq_blocked_stats) +#define ENQ_BLOCKED_IOCTL_PREP_ATTACH _IO('s', 5) +#define ENQ_BLOCKED_IOCTL_ATTACH_DONE _IO('s', 6) +#define ENQ_BLOCKED_IOCTL_DONOR_STATE _IO('s', 7) + +#define ENQ_BLOCKED_DONOR_SLEEPING (1U << 0) +#define ENQ_BLOCKED_DONOR_ON_RQ (1U << 1) + +#endif /* __ENQ_BLOCKED_H */
tools/testing/selftests/sched_ext/test_modules/Makefile new file +13 -0
@@ -0,0 +1,13 @@ +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES + +TESTMODS_DIR := $(realpath $(dir $(abspath $(lastword $(MAKEFILE_LIST))))) +KDIR ?= $(if $(O),$(O),$(realpath ../../../../..)) + +obj-m += scx_enq_blocked_test.o + +all: + +$(Q)$(MAKE) -C $(KDIR) M=$(TESTMODS_DIR) modules + +clean: + +$(Q)$(MAKE) -C $(KDIR) M=$(TESTMODS_DIR) clean
tools/testing/selftests/sched_ext/test_modules/scx_enq_blocked_test.c new file +193 -0
193 lines not on this page · raw message
Signature · 1 line
2.55.0
Andrea Righi
reviewerPATCH 10/11 sched_ext: scx_qmap: Add proxy execution support
In reply to Andrea RighiAdd a -B option to opt scx_qmap into queueing mutex-blocked tasks for proxy execution. Without the option, SCX_OPS_ENQ_BLOCKED remains clear and mutex waiters block normally. With -B, blocked donors are passed to qmap_enqueue() with SCX_ENQ_BLOCKED.
When scx_qmap receives a blocked donor, dispatch it directly to the local DSQ of its current cid with a fresh slice and SCX_ENQ_PREEMPT. This places the donor at the head of the DSQ and requests an immediate reschedule, allowing the core proxy-exec path to run the mutex owner using the donor's scheduling context as soon as the donor is selected.
The blocked policy is intentionally unfair and can strongly prioritize tasks using contended mutexes, but scx_qmap is a demo scheduler and such aggressive behavior makes proxy-exec support easy to observe. Count these dispatch attempts in nr_enq_blocked and report their per-interval delta.
--- tools/sched_ext/scx_qmap.bpf.c | 52 ++++++++++++++++++++++------------ tools/sched_ext/scx_qmap.c | 13 +++++++-- tools/sched_ext/scx_qmap.h | 1 + 3 files changed, 45 insertions(+), 21 deletions(-)
tools/sched_ext/scx_qmap.bpf.c +34 -18
@@ -451,6 +451,40 @@ void BPF_STRUCT_OPS(qmap_enqueue, struct task_struct *p, u64 enq_flags)*/ taskc->core_sched_seq = qa.core_sched_tail_seqs[idx]++; +/* +* If the task was re-enqueued due to the CPU being preempted by a +* higher priority scheduling class, just re-enqueue the task directly +* on the global DSQ. As we want another CPU to pick it up, find and +* kick an idle cid. +*/ +if (enq_flags &SCX_ENQ_REENQ ) { + taskc->force_local = false; +scx_bpf_dsq_insert (p,SHARED_DSQ ,0 , enq_flags); + cid =cmask_next_and2_set_wrap (&taskc->cpus_allowed, + &qa.idle_cids.mask, + &qa.self_cids.mask,0 ); +if (cid <scx_bpf_nr_cids ()) +scx_bpf_kick_cid (cid,SCX_KICK_IDLE ); +return ; + } + +/* +* Insert a blocked mutex donor at the head of its current cid's local +* DSQ with a fresh slice and %SCX_ENQ_PREEMPT, requesting an immediate +* reschedule. Once selected, the core proxy-exec path can immediately +* run the mutex owner using the donor's scheduling context. +* +* This policy is intentionally unfair and can strongly prioritize tasks +* using contended mutexes; scx_qmap is a demonstration scheduler and +* this behavior makes proxy-exec support easy to observe. +*/ +if (enq_flags &SCX_ENQ_BLOCKED ) { +__sync_fetch_and_add (&qa.nr_enq_blocked,1 ); +scx_bpf_dsq_insert (p,SCX_DSQ_LOCAL_ON |scx_bpf_task_cid (p), + slice_ns, enq_flags |SCX_ENQ_PREEMPT ); +return ; + } +/* * A node with children delegates most cids. A task of ours that can run * on none of our self cids (e.g. a per-NUMA kthread pinned to delegated
@@ -537,24 +571,6 @@ void BPF_STRUCT_OPS(qmap_enqueue, struct task_struct *p, u64 enq_flags)return ; } -/* -* If the task was re-enqueued due to the CPU being preempted by a -* higher priority scheduling class, just re-enqueue the task directly -* on the global DSQ. As we want another CPU to pick it up, find and -* kick an idle cid. -*/ -if (enq_flags &SCX_ENQ_REENQ ) { -s32 cid; - -scx_bpf_dsq_insert (p,SHARED_DSQ ,0 , enq_flags); - cid =cmask_next_and2_set_wrap (&taskc->cpus_allowed, - &qa.idle_cids.mask, - &qa.self_cids.mask,0 ); -if (cid <scx_bpf_nr_cids ()) -scx_bpf_kick_cid (cid,SCX_KICK_IDLE ); -return ; - } -/* Queue on the selected FIFO. */ qmap_fifo_enqueue (&qa.fifos[idx], taskc);
tools/sched_ext/scx_qmap.c +10 -3
@@ -51,7 +51,7 @@ const char help_fmt[] ="See the top-of-file comment in .bpf.c for the design.\n" "\n" "Usage: %s [-s SLICE_US] [-e COUNT] [-t COUNT] [-T COUNT] [-l COUNT] [-b COUNT]\n" -" [-N COUNT] [-P] [-M] [-H] [-c CG_PATH] [-d PID] [-D LEN] [-S] [-p] [-I]\n" +" [-N COUNT] [-P] [-M] [-H] [-c CG_PATH] [-d PID] [-D LEN] [-S] [-p] [-I] [-B]\n" " [-F COUNT] [-i SEC] [-R MS] [-J MODE] [-v]\n" "\n" " -s SLICE_US Override slice duration\n"
@@ -70,6 +70,7 @@ const char help_fmt[] =" -S Suppress qmap-specific debug dump\n" " -p Switch only tasks on SCHED_EXT policy instead of all\n" " -I Turn on SCX_OPS_ALWAYS_ENQ_IMMED\n" +" -B Turn on SCX_OPS_ENQ_BLOCKED\n" " -F COUNT IMMED stress: force every COUNT'th enqueue to a busy local DSQ (use with -I)\n" " -C MODE cid-override test (shuffle|bad-dup|bad-range|bad-mono)\n" " -i SEC Stats and weight-refresh interval, seconds (default 5)\n"
@@ -185,6 +186,7 @@ struct hier_prev {u64 nr_dsps[MAX_SUB_SCHEDS ];u64 nr_reenq_cap;u64 nr_reenq_immed; +u64 nr_enq_blocked;u64 nr_inject_attempts; };
@@ -267,13 +269,15 @@ static void print_hier(struct qmap_arena *qa, struct hier_prev *prev, u64 own_cg }format_cid_ranges (qa,CID_SHARED , ranges,sizeof (ranges)); -printf ("hier : nsub=%llu excl=%u shared=%s rr=%s reenq cap/immed +%llu/+%llu inj=+%llu\n" , +printf ("hier : nsub=%llu excl=%u shared=%s rr=%s reenq cap/immed +%llu/+%llu blocked=+%llu inj=+%llu\n" , (unsigned long long )qa->nr_sub_scheds, qa->part.nr_excl, ranges, rr, (unsigned long long )(qa->nr_reenq_cap - prev->nr_reenq_cap), (unsigned long long )(qa->nr_reenq_immed - prev->nr_reenq_immed), + (unsigned long long )(qa->nr_enq_blocked - prev->nr_enq_blocked), (unsigned long long )(qa->nr_inject_attempts - prev->nr_inject_attempts)); prev->nr_reenq_cap = qa->nr_reenq_cap; prev->nr_reenq_immed = qa->nr_reenq_immed; + prev->nr_enq_blocked = qa->nr_enq_blocked; prev->nr_inject_attempts = qa->nr_inject_attempts; printf("hier : %-4s %10s %4s %6s %8s %s\n",
@@ -336,7 +340,7 @@ int main(int argc, char **argv) skel->rodata->slice_ns =__COMPAT_ENUM_OR_ZERO ("scx_public_consts" ,"SCX_SLICE_DFL" ); skel->rodata->max_tasks =16384 ; -while ((opt =getopt (argc, argv,"s:e:t:T:l:b:N:PMHc:d:D:SpIF:C:i:R:J:vh" )) !=-1 ) { +while ((opt =getopt (argc, argv,"s:e:t:T:l:b:N:PMHc:d:D:SpIBF:C:i:R:J:vh" )) !=-1 ) {switch (opt) {case 's' : skel->rodata->slice_ns =strtoull (optarg,NULL ,0 ) *1000 ;
@@ -398,6 +402,9 @@ int main(int argc, char **argv) skel->rodata->always_enq_immed = true; skel->struct_ops.qmap_ops->flags |=SCX_OPS_ALWAYS_ENQ_IMMED ;break ; +case 'B' : + skel->struct_ops.qmap_ops->flags |=SCX_OPS_ENQ_BLOCKED ; +break ;case 'F' : skel->rodata->immed_stress_nth =strtoul (optarg,NULL ,0 );break ;
tools/sched_ext/scx_qmap.h +1 -0
@@ -173,6 +173,7 @@ struct qmap_arena {/* bpf -> userspace: stats */ u64 nr_reenq_cap;/* SCX_TASK_REENQ_CAP bounces */ u64 nr_reenq_immed;/* SCX_TASK_REENQ_IMMED bounces */ +u64 nr_enq_blocked;/* SCX_ENQ_BLOCKED dispatches */ u64 nr_inject_attempts;/* fault-injection: dispatches to an unheld cid */ u32 inject_mode;/* fault-injection mode (QMAP_INJ_*) */ };
Signature · 1 line
2.55.0
[email protected]
AI reviewRe: [PATCH 10/11] sched_ext: scx_qmap: Add proxy execution support
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Low] Redundant !(enq_flags & SCX_ENQ_REENQ) checks left behind after moving the early return handler. --
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] Redundant !(enq_flags & SCX_ENQ_REENQ) checks left behind after moving the early return handler. --
--- Patch [10]: [PATCH 10/11] sched_ext: scx_qmap: Add proxy execution support --- commit fb0f1d9957f7124711aadbe323f38da6fa6e3857
Author: Andrea Righi <[email protected]>
sched_ext: scx_qmap: Add proxy execution support
This commit adds a -B option to scx_qmap to opt into queueing mutex-blocked tasks for proxy execution. It dispatches blocked donors directly to the local DSQ of their current cid with a fresh slice and SCX_ENQ_PREEMPT.
Quoted 14 lines
tools/sched_ext/scx_qmap.bpf.c +6
-0@@ -451,6 +451,40 @@ void BPF_STRUCT_OPS(qmap_enqueue, struct task_struct *p, u64 enq_flags)*/ taskc->core_sched_seq = qa.core_sched_tail_seqs[idx]++; +/* +* If the task was re-enqueued due to the CPU being preempted by a +* higher priority scheduling class, just re-enqueue the task directly +* on the global DSQ. As we want another CPU to pick it up, find and +* kick an idle cid. +*/
tools/sched_ext/scx_qmap.bpf.c void BPF_STRUCT_OPS(qmap_enqueue, struct task_struct *p, u64 enq_flags) · line 469
Quoted 10 lines
tools/sched_ext/scx_qmap.bpf.c +10
-0+if (enq_flags &SCX_ENQ_REENQ ) { + taskc->force_local = false; +scx_bpf_dsq_insert (p,SHARED_DSQ ,0 , enq_flags); + cid =cmask_next_and2_set_wrap (&taskc->cpus_allowed, + &qa.idle_cids.mask, + &qa.self_cids.mask,0 ); +if (cid <scx_bpf_nr_cids ()) +scx_bpf_kick_cid (cid,SCX_KICK_IDLE ); +return ; + }
[Severity: Low]
Since the early return block for SCX_ENQ_REENQ was moved higher up in qmap_enqueue(), are the later checks for !(enq_flags & SCX_ENQ_REENQ) now redundant?
For example, further down in the same function:
if (qa.nr_sub_scheds && !(enq_flags & SCX_ENQ_REENQ) &&
!cmask_intersects(&taskc->cpus_allowed, &qa.self_cids.mask)) {It seems SCX_ENQ_REENQ will never be set when reaching this code now, making the condition safely removable.
[ ... ]
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=10
Re: [PATCH 10/11] sched_ext: scx_qmap: Add proxy execution support
In reply to Andrea RighiQuoted 11 lines · Andrea Righi
Add a -B option to opt
scx_qmapinto queueing mutex-blocked tasks for proxy execution. Without the option,SCX_OPS_ENQ_BLOCKEDremains clear and mutex waiters block normally. With -B, blocked donors are passed toqmap_enqueue()withSCX_ENQ_BLOCKED.When
scx_qmapreceives a blocked donor, dispatch it directly to the local DSQ of its current cid with a fresh slice andSCX_ENQ_PREEMPT. This places the donor at the head of the DSQ and requests an immediate reschedule, allowing the core proxy-exec path to run the mutex owner using the donor's scheduling context as soon as the donor is selected.
Hey Andrea!
Thanks again for your great work on all this, I'm working to test your changes with my full proxy-exec tree, but before that I caught one issue here.
I was excited to try this out, but when building and running this from your kernel tree in a VM, (who's kernel is running from the same kernel tree, built on the host), I get the following error (with or without -B). The other scx_ schedulers in tools/sched_ext/ seem to load and run ok. Maybe is there something I'm forgetting to do here?
thanks
-john
# ./build/bin/scx_qmap -B
libbpf: prog 'qmap_dequeue': BPF program load failed: -EACCES
libbpf: prog 'qmap_dequeue': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0
; void BPF_STRUCT_OPS(qmap_dequeue, struct task_struct *p, u64
deq_flags) @ scx_qmap.bpf.c:584
0: (79) r2 = *(u64 *)(r1 +0)
func 'dequeue' arg0 has btf_id 142670 type STRUCT 'task_struct'
1: R1=ctx() R2=trusted_ptr_task_struct()
1: (79) r4 = *(u64 *)(r1 +8) ; R1=ctx() R4=scalar()
; __sync_fetch_and_add(&qa.nr_dequeued, 1); @ scx_qmap.bpf.c:588
2: (18) r1 = 0x10000fff7000 ; R1=scalar()
4: (18) r3 = 0x10000fff7000 ; R3=scalar()
6: (07) r3 += 32 ; R3=scalar()
7: (bf) r5 = addr_space_cast(r3, 0, 1) ; R3=scalar() R5=arena
8: (b7) r3 = 1 ; R3=1
9: (b7) r0 = 1 ; R0=1
10: (db) lock *(u64 *)(r5 +0) += r0 ; R0=1 R5=arena
11: (18) r5 = 0x100000000 ; R5=0x100000000
; if (deq_flags & SCX_DEQ_CORE_SCHED_EXEC) @ scx_qmap.bpf.c:589
13: (5f) r4 &= r5 ;
R4=scalar(smin=smin32=0,smax=umax=0x100000000,smax32=umax32=0,var_off=(0x0;
0x100000000))
14: (15) if r4 == 0x0 goto pc+3 ; R4=0x100000000
; __sync_fetch_and_add(&qa.nr_core_sched_execed, 1); @ scx_qmap.bpf.c:590
15: (07) r1 += 48 ; R1=scalar()
16: (bf) r1 = addr_space_cast(r1, 0, 1) ; R1=arena
17: (db) lock *(u64 *)(r1 +0) += r3 ;
; QMAP_TOUCH_ARENA(); @ scx_qmap.bpf.c:268
18: (18) r1 = 0xffff888104842c00 ; R1=map_ptr(map=arena,ks=0,vs=0)
; v = bpf_task_storage_get(&task_ctx_stor, p, 0, 0); @ scx_qmap.bpf.c:270
20: (18) r1 = 0xffff88810c333600 ; R1=map_ptr(map=task_ctx_stor,ks=4,vs=8)
22: (b7) r3 = 0 ; R3=0
23: (b7) r4 = 0 ; R4=0
24: (85) call bpf_task_storage_get#156 ;
R0=map_value_or_null(id=1,map=task_ctx_stor,ks=4,vs=8)
; if (!v || !v->taskc) @ scx_qmap.bpf.c:271
25: (15) if r0 == 0x0 goto pc+76 ;
R0=map_value(id=1,map=task_ctx_stor,ks=4,vs=8)
26: (79) r7 = *(u64 *)(r0 +0) ;
R0=map_value(id=1,map=task_ctx_stor,ks=4,vs=8) R7=scalar()
27: (15) if r7 == 0x0 goto pc+74 ; R7=scalar(umin=1)
28: (bf) r9 = addr_space_cast(r7, 0, 1) ; R7=scalar(umin=1) R9=arena
; if (taskc && taskc->fifo) { @ scx_qmap.bpf.c:593
29: (79) r8 = *(u64 *)(r9 +24) ; R8=scalar() R9=arena
30: (15) if r8 == 0x0 goto pc+71 ; R8=scalar(umin=1)
; if (taskc->highpri) @ scx_qmap.bpf.c:594
31: (71) r1 = *(u8 *)(r9 +45) ;
R1=scalar(smin=smin32=0,smax=umax=smax32=umax32=255,var_off=(0x0;
0xff)) R9=arena
32: (56) if w1 != 0x1 goto pc+9 ; R1=1
33: (b7) r1 = 1 ; R1=1
; __sync_fetch_and_sub(&qa.nr_highpri_queued, 1); @ scx_qmap.bpf.c:595
34: (87) r1 = -r1 ; R1=-1
35: (18) r2 = 0x10000fff7000 ; R2=scalar()
37: (07) r2 += 88 ; R2=scalar()
38: (bf) r2 = addr_space_cast(r2, 0, 1) ; R2=arena
39: (db) lock *(u64 *)(r2 +0) += r1 ; R1=-1 R2=arena
; struct qmap_fifo __arena *fifo = taskc->fifo; @ scx_qmap.bpf.c:320
40: (79) r8 = *(u64 *)(r9 +24) ; R8=scalar() R9=arena
; if (!fifo) @ scx_qmap.bpf.c:323
41: (15) if r8 == 0x0 goto pc+60 ; R8=scalar(umin=1)
; lock = qa_q_lock(fifo->idx); @ scx_qmap.bpf.c:326
42: (bf) r2 = addr_space_cast(r8, 0, 1) ; R2=arena R8=scalar(umin=1)
43: (61) r1 = *(u32 *)(r2 +16) ;
R1=scalar(smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff))
R2=arena
; switch (qid) { @ scx_qmap.bpf.c:129
44: (66) if w1 s> 0x1 goto pc+8 ;
R1=scalar(smin=0,smax=umax=0xffffffff,smax32=1,var_off=(0x0;
0xffffffff))
45: (18) r6 = 0xffff88810c3339f8 ;
R6=map_value(map=.data.qa_q_lock,ks=4,vs=4)
47: (16) if w1 == 0x0 goto pc+17 ;
R1=scalar(smin=0,smax=umax=0xffffffff,smax32=1,var_off=(0x0;
0xffffffff))
48: (16) if w1 == 0x1 goto pc+1 50: R2=arena R7=scalar(umin=1)
R8=scalar(umin=1) R9=arena R10=fp0
; switch (qid) { @ scx_qmap.bpf.c:129
50: (18) r6 = 0xffff88810c333bf8 ;
R6=map_value(map=.data.qa_q_lock,ks=4,vs=4)
52: (05) goto pc+12
65: (7b) *(u64 *)(r10 -16) = r2 ; R2=arena R10=fp0 fp-16=arena
; if (bpf_res_spin_lock(lock)) { @ scx_qmap.bpf.c:212
66: (bf) r1 = r6 ;
R1=map_value(map=.data.qa_q_lock,ks=4,vs=4)
R6=map_value(map=.data.qa_q_lock,ks=4,vs=4)
67: (85) call bpf_res_spin_lock#85478 ; R0=0
68: (16) if w0 == 0x0 goto pc+9 ; R0=0
; if (taskc->fifo != fifo) { @ scx_qmap.bpf.c:331
78: (79) r1 = *(u64 *)(r9 +24) ; R1=scalar() R9=arena
79: (1d) if r1 == r8 goto pc+1 81:
R6=map_value(map=.data.qa_q_lock,ks=4,vs=4) R7=scalar(umin=1)
R8=scalar(umin=1) R9=arena R10=fp0 fp-16=arena
; bpf_res_spin_unlock(lock); @ scx_qmap.bpf.c:332
81: (79) r2 = *(u64 *)(r9 +16) ; R2=scalar() R9=arena
; if (taskc->q_next) @ scx_qmap.bpf.c:336
82: (79) r1 = *(u64 *)(r9 +8) ; R1=scalar() R9=arena
83: (15) if r1 == 0x0 goto pc+4 ; R1=scalar(umin=1)
84: (bf) r3 = addr_space_cast(r1, 0, 1) ; R1=scalar(umin=1) R3=arena
; taskc->q_next->q_prev = taskc->q_prev; @ scx_qmap.bpf.c:337
85: (7b) *(u64 *)(r3 +16) = r2 ; R2=scalar() R3=arena
; if (taskc->q_prev) @ scx_qmap.bpf.c:340
86: (79) r2 = *(u64 *)(r9 +16) ; R2=scalar() R9=arena
; taskc->q_next->q_prev = taskc->q_prev; @ scx_qmap.bpf.c:337
87: (05) goto pc+2
; fifo->tail = taskc->q_prev; @ scx_qmap.bpf.c:339
90: (07) r7 += 8 ; R7=scalar()
; if (taskc->q_prev) @ scx_qmap.bpf.c:340
91: (15) if r2 == 0x0 goto pc+2 ; R2=scalar(umin=1)
92: (07) r2 += 8 ; R2=scalar()
93: (bf) r8 = r2 ; R8=scalar(id=2)
94: (bf) r2 = addr_space_cast(r8, 0, 1) ; R2=arena R8=scalar(id=2)
95: (7b) *(u64 *)(r2 +0) = r1 ; R1=scalar(umin=1) R2=arena
96: (b7) r1 = 0 ; R1=0
; taskc->q_prev = NULL; @ scx_qmap.bpf.c:345
97: (7b) *(u64 *)(r7 +16) = r1
R7 invalid mem access 'scalar'
processed 71 insns (limit 1000000) max_states_per_insn 1 total_states
5 peak_states 5 mark_read 0
-- END PROG LOAD LOG --
libbpf: prog 'qmap_dequeue': failed to load: -EACCES
libbpf: failed to load object 'scx_qmap'
libbpf: failed to load BPF skeleton 'scx_qmap': -EACCES
[SCX_BUG] scx_qmap.c:497 (Permission denied)
Failed to load skelAndrea Righi
reviewerRe: [PATCH 10/11] sched_ext: scx_qmap: Add proxy execution support
Hi John,
Quoted 138 lines · John Stultz
Quoted 11 lines · Andrea Righi
Add a -B option to opt
scx_qmapinto queueing mutex-blocked tasks for proxy execution. Without the option,SCX_OPS_ENQ_BLOCKEDremains clear and mutex waiters block normally. With -B, blocked donors are passed toqmap_enqueue()withSCX_ENQ_BLOCKED.When
scx_qmapreceives a blocked donor, dispatch it directly to the local DSQ of its current cid with a fresh slice andSCX_ENQ_PREEMPT. This places the donor at the head of the DSQ and requests an immediate reschedule, allowing the core proxy-exec path to run the mutex owner using the donor's scheduling context as soon as the donor is selected.Hey Andrea!
Thanks again for your great work on all this, I'm working to test your changes with my full proxy-exec tree, but before that I caught one issue here.
I was excited to try this out, but when building and running this from your kernel tree in a VM, (who's kernel is running from the same kernel tree, built on the host), I get the following error (with or without -B). The other
scx_schedulers intools/sched_ext/seem to load and run ok. Maybe is there something I'm forgetting to do here?thanks
-john# ./build/bin/scx_qmap -B libbpf: prog 'qmap_dequeue': BPF program load failed: -EACCES libbpf: prog 'qmap_dequeue': -- BEGIN PROG LOAD LOG -- 0: R1=ctx() R10=fp0 ; void BPF_STRUCT_OPS(qmap_dequeue, struct task_struct *p, u64 deq_flags) @ scx_qmap.bpf.c:584 0: (79) r2 = *(u64 *)(r1 +0) func 'dequeue' arg0 has btf_id 142670 type STRUCT 'task_struct' 1: R1=ctx() R2=trusted_ptr_task_struct() 1: (79) r4 = *(u64 *)(r1 +8) ; R1=ctx() R4=scalar() ; __sync_fetch_and_add(&qa.nr_dequeued, 1); @ scx_qmap.bpf.c:588 2: (18) r1 = 0x10000fff7000 ; R1=scalar() 4: (18) r3 = 0x10000fff7000 ; R3=scalar() 6: (07) r3 += 32 ; R3=scalar() 7: (bf) r5 = addr_space_cast(r3, 0, 1) ; R3=scalar() R5=arena 8: (b7) r3 = 1 ; R3=1 9: (b7) r0 = 1 ; R0=1 10: (db) lock *(u64 *)(r5 +0) += r0 ; R0=1 R5=arena 11: (18) r5 = 0x100000000 ; R5=0x100000000 ; if (deq_flags & SCX_DEQ_CORE_SCHED_EXEC) @ scx_qmap.bpf.c:589 13: (5f) r4 &= r5 ; R4=scalar(smin=smin32=0,smax=umax=0x100000000,smax32=umax32=0,var_off=(0x0; 0x100000000)) 14: (15) if r4 == 0x0 goto pc+3 ; R4=0x100000000 ; __sync_fetch_and_add(&qa.nr_core_sched_execed, 1); @ scx_qmap.bpf.c:590 15: (07) r1 += 48 ; R1=scalar() 16: (bf) r1 = addr_space_cast(r1, 0, 1) ; R1=arena 17: (db) lock *(u64 *)(r1 +0) += r3 ; ; QMAP_TOUCH_ARENA(); @ scx_qmap.bpf.c:268 18: (18) r1 = 0xffff888104842c00 ; R1=map_ptr(map=arena,ks=0,vs=0) ; v = bpf_task_storage_get(&task_ctx_stor, p, 0, 0); @ scx_qmap.bpf.c:270 20: (18) r1 = 0xffff88810c333600 ; R1=map_ptr(map=task_ctx_stor,ks=4,vs=8) 22: (b7) r3 = 0 ; R3=0 23: (b7) r4 = 0 ; R4=0 24: (85) call bpf_task_storage_get#156 ; R0=map_value_or_null(id=1,map=task_ctx_stor,ks=4,vs=8) ; if (!v || !v->taskc) @ scx_qmap.bpf.c:271 25: (15) if r0 == 0x0 goto pc+76 ; R0=map_value(id=1,map=task_ctx_stor,ks=4,vs=8) 26: (79) r7 = *(u64 *)(r0 +0) ; R0=map_value(id=1,map=task_ctx_stor,ks=4,vs=8) R7=scalar() 27: (15) if r7 == 0x0 goto pc+74 ; R7=scalar(umin=1) 28: (bf) r9 = addr_space_cast(r7, 0, 1) ; R7=scalar(umin=1) R9=arena ; if (taskc && taskc->fifo) { @ scx_qmap.bpf.c:593 29: (79) r8 = *(u64 *)(r9 +24) ; R8=scalar() R9=arena 30: (15) if r8 == 0x0 goto pc+71 ; R8=scalar(umin=1) ; if (taskc->highpri) @ scx_qmap.bpf.c:594 31: (71) r1 = *(u8 *)(r9 +45) ; R1=scalar(smin=smin32=0,smax=umax=smax32=umax32=255,var_off=(0x0; 0xff)) R9=arena 32: (56) if w1 != 0x1 goto pc+9 ; R1=1 33: (b7) r1 = 1 ; R1=1 ; __sync_fetch_and_sub(&qa.nr_highpri_queued, 1); @ scx_qmap.bpf.c:595 34: (87) r1 = -r1 ; R1=-1 35: (18) r2 = 0x10000fff7000 ; R2=scalar() 37: (07) r2 += 88 ; R2=scalar() 38: (bf) r2 = addr_space_cast(r2, 0, 1) ; R2=arena 39: (db) lock *(u64 *)(r2 +0) += r1 ; R1=-1 R2=arena ; struct qmap_fifo __arena *fifo = taskc->fifo; @ scx_qmap.bpf.c:320 40: (79) r8 = *(u64 *)(r9 +24) ; R8=scalar() R9=arena ; if (!fifo) @ scx_qmap.bpf.c:323 41: (15) if r8 == 0x0 goto pc+60 ; R8=scalar(umin=1) ; lock = qa_q_lock(fifo->idx); @ scx_qmap.bpf.c:326 42: (bf) r2 = addr_space_cast(r8, 0, 1) ; R2=arena R8=scalar(umin=1) 43: (61) r1 = *(u32 *)(r2 +16) ; R1=scalar(smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff)) R2=arena ; switch (qid) { @ scx_qmap.bpf.c:129 44: (66) if w1 s> 0x1 goto pc+8 ; R1=scalar(smin=0,smax=umax=0xffffffff,smax32=1,var_off=(0x0; 0xffffffff)) 45: (18) r6 = 0xffff88810c3339f8 ; R6=map_value(map=.data.qa_q_lock,ks=4,vs=4) 47: (16) if w1 == 0x0 goto pc+17 ; R1=scalar(smin=0,smax=umax=0xffffffff,smax32=1,var_off=(0x0; 0xffffffff)) 48: (16) if w1 == 0x1 goto pc+1 50: R2=arena R7=scalar(umin=1) R8=scalar(umin=1) R9=arena R10=fp0 ; switch (qid) { @ scx_qmap.bpf.c:129 50: (18) r6 = 0xffff88810c333bf8 ; R6=map_value(map=.data.qa_q_lock,ks=4,vs=4) 52: (05) goto pc+12 65: (7b) *(u64 *)(r10 -16) = r2 ; R2=arena R10=fp0 fp-16=arena ; if (bpf_res_spin_lock(lock)) { @ scx_qmap.bpf.c:212 66: (bf) r1 = r6 ; R1=map_value(map=.data.qa_q_lock,ks=4,vs=4) R6=map_value(map=.data.qa_q_lock,ks=4,vs=4) 67: (85) call bpf_res_spin_lock#85478 ; R0=0 68: (16) if w0 == 0x0 goto pc+9 ; R0=0 ; if (taskc->fifo != fifo) { @ scx_qmap.bpf.c:331 78: (79) r1 = *(u64 *)(r9 +24) ; R1=scalar() R9=arena 79: (1d) if r1 == r8 goto pc+1 81: R6=map_value(map=.data.qa_q_lock,ks=4,vs=4) R7=scalar(umin=1) R8=scalar(umin=1) R9=arena R10=fp0 fp-16=arena ; bpf_res_spin_unlock(lock); @ scx_qmap.bpf.c:332 81: (79) r2 = *(u64 *)(r9 +16) ; R2=scalar() R9=arena ; if (taskc->q_next) @ scx_qmap.bpf.c:336 82: (79) r1 = *(u64 *)(r9 +8) ; R1=scalar() R9=arena 83: (15) if r1 == 0x0 goto pc+4 ; R1=scalar(umin=1) 84: (bf) r3 = addr_space_cast(r1, 0, 1) ; R1=scalar(umin=1) R3=arena ; taskc->q_next->q_prev = taskc->q_prev; @ scx_qmap.bpf.c:337 85: (7b) *(u64 *)(r3 +16) = r2 ; R2=scalar() R3=arena ; if (taskc->q_prev) @ scx_qmap.bpf.c:340 86: (79) r2 = *(u64 *)(r9 +16) ; R2=scalar() R9=arena ; taskc->q_next->q_prev = taskc->q_prev; @ scx_qmap.bpf.c:337 87: (05) goto pc+2 ; fifo->tail = taskc->q_prev; @ scx_qmap.bpf.c:339 90: (07) r7 += 8 ; R7=scalar() ; if (taskc->q_prev) @ scx_qmap.bpf.c:340 91: (15) if r2 == 0x0 goto pc+2 ; R2=scalar(umin=1)
Quoted 8 lines
92: (07) r2 += 8 ; R2=scalar() 93: (bf) r8 = r2 ; R8=scalar(id=2) 94: (bf) r2 = addr_space_cast(r8, 0, 1) ; R2=arena R8=scalar(id=2) 95: (7b) *(u64 *)(r2 +0) = r1 ; R1=scalar(umin=1) R2=arena 96: (b7) r1 = 0 ; R1=0 ; taskc->q_prev = NULL; @ scx_qmap.bpf.c:345 97: (7b) *(u64 *)(r7 +16) = r1 R7 invalid mem access 'scalar'
This looks like something related to BPF arena pointers and incorrectly generated BPF code (the other schedulers don't use BPF arenas, so that'd explain why you only see this with scx_qmap).
Which version of clang are you using?
Thanks,
-Andrea
Re: [PATCH 10/11] sched_ext: scx_qmap: Add proxy execution support
Quoted 5 lines · Andrea Righi
This looks like something related to BPF arena pointers and incorrectly generated BPF code (the other schedulers don't use BPF arenas, so that'd explain why you only see this with
scx_qmap).Which version of clang are you using?
Debian clang version 19.1.7 (3+b1)
Running in a debian 13 vm. Though I'm using gcc normally for my
builds (but I see the tools/sched_ext/ build does use clang explicitly for some things).
thanks
-john
Andrea Righi
reviewerRe: [PATCH 10/11] sched_ext: scx_qmap: Add proxy execution support
Quoted 5 lines · John Stultz
Quoted 4 lines · Andrea Righi
This looks like something related to BPF arena pointers and incorrectly generated BPF code (the other schedulers don't use BPF arenas, so that'd explain why you only see this with
scx_qmap).
Quoted 7 lines
Quoted 1 line
Which version of clang are you using?
Debian clang version 19.1.7 (3+b1)
Running in a debian 13 vm. Though I'm using gcc normally for my
builds (but I see thetools/sched_ext/build does use clang explicitly for some things).
Yeah, we explicitly use clang to build BPF code. I'm on CachyOS, which ships clang 22.1.8. In theory clang-19 should be enough... if you can mess up the VM maybe give it a try with clang 20? From the dump it really looks like the compiler failed to generate proper BPF code for arenas.
Thanks,
-Andrea
Re: [PATCH 10/11] sched_ext: scx_qmap: Add proxy execution support
Quoted 10 lines · Andrea Righi
Quoted 9 lines · John Stultz
Quoted 5 lines · Andrea Righi
This looks like something related to BPF arena pointers and incorrectly generated BPF code (the other schedulers don't use BPF arenas, so that'd explain why you only see this with
scx_qmap).Which version of clang are you using?
Debian clang version 19.1.7 (3+b1)
Quoted 8 lines
Quoted 3 lines
Running in a debian 13 vm. Though I'm using gcc normally for my
builds (but I see thetools/sched_ext/build does use clang explicitly for some things).Yeah, we explicitly use clang to build BPF code. I'm on CachyOS, which ships clang 22.1.8. In theory clang-19 should be enough... if you can mess up the VM maybe give it a try with clang 20? From the dump it really looks like the compiler failed to generate proper BPF code for arenas.
Looks like you're right. Using a 23.0.0git checkout it seems like its working better there.
Thanks for walking me through this :)
-john
Andrea Righi
reviewerPATCH 11/11 sched: Allow enabling proxy exec with sched_ext
In reply to Andrea RighiNow that sched_ext can handle proxy donors and BPF schedulers can opt in to blocked-donor enqueueing with SCX_OPS_ENQ_BLOCKED, remove the !SCHED_CLASS_EXT dependency from SCHED_PROXY_EXEC and allow both options to be enabled together.
--- init/Kconfig | 2 -- 1 file changed, 2 deletions(-)
init/Kconfig +0 -2
@@ -934,8 +934,6 @@ config SCHED_PROXY_EXEC bool "Proxy Execution" # Avoid some build failures w/ PREEMPT_RT until it can be fixed depends on !PREEMPT_RT - # Need to investigate how to inform sched_ext of split contexts - depends on !SCHED_CLASS_EXT # Not particularly useful until we get to multi-rq proxying depends on EXPERT help
Signature · 1 line
2.55.0