sched_ext: Capability-based CPU delegation for sub-schedulers
Subsystems (2)
Tejun Heo
reviewerPATCH v4 1/1 sched_ext: Capability-based CPU delegation for sub-schedulers
Hello,
Changes in v4:
- New patch (01): record an error on errno-only sub-enable failures, so the disable path runs and a half-initialized sub-scheduler is torn down instead of left linked. Pre-existing bug, split out early with a Fixes:
tag. (sashiko AI)
- New patch (20): stop scx_prog_sched() from resolving a disabled scheduler. A program that outlives disable (a timer it armed, a tracing program it loaded) could still operate on a torn-down generation; gating one choke point closes a cross-generation pshard[] out-of-bounds. (sashiko AI)
- Flush the pending kick irq_work in free_kick_syncs() before clearing the per-cpu kick state, so a late kick unlinks cleanly at teardown (10). (sashiko AI)
- New prep patch (30): track the cpu a task is runnable on (p->scx.runnable_cpu), stamped under the rq lock. The slice write path uses it to test rq ownership (31), closing a remote-wakeup TOCTOU where a cached task_rq() could stale-match a migrated sleeping task. (sashiko AI)
- scx_qmap demo (39): track every cid's idle state regardless of delegation and mask it with the node's own cids at the dispatch pick, instead of reseeding from a separate idle set. Fixes an already-idle reclaimed cid being missed and drops a reseed race. The three-mask cmask iterator it uses is split into its own patch (38). (sashiko AI)
- Introduce pshard[] in its final build-then-publish form in the patch that adds it rather than rewriting it a few patches later (15, 21).
- Document the remaining limitations for full sub-scheduling support below. (Andrea)
v3: https://lore.kernel.org/r/[email protected]
v2: https://lore.kernel.org/r/[email protected]
v1: https://lore.kernel.org/r/[email protected]
The existing sub-scheduler support only covers a part of scheduling. An scx_sched can attach to a cgroup subtree under another scx_sched and detach, and the dispatch path has programmatic delegation: dispatch runs as top-down recursion via scx_bpf_sub_dispatch(), so a parent decides when a child gets to pick tasks for the local cpu by calling into it.
The enqueue path is not implemented yet. Every scheduler in the hierarchy can insert into every local DSQ, so a parent has no way to partition CPUs among its children or protect its own share from them.
This patchset adds that control. Most importantly it enables sub-sched support on the enqueue path, and it extends the same control to the other paths that touch cpu access: occupancy, kicks and preemption, and idle reporting.
The enqueue path cannot use the dispatch path's programmatic model. Dispatch is top-down and the delegation is implicit in the call chain. Enqueue starts at the other end: a leaf picks a cid for a task, and which cids it may use is the cumulative result of its ancestors' delegation decisions. Resolving that programmatically on every enqueue would mean a cross-sched round-trip call chain, possibly retrying when a request cannot be granted as-is. Delegation is therefore state, not calls:
- Ownership is tracked per (scheduler, cid) as a set of capabilities:
- SCX_CAP_ENQ_IMMED: insert an IMMED task onto the cid's local DSQ. This is the baseline cap (SCX_CAP_BASE) required to make any use of a cpu.
- SCX_CAP_ENQ: insert any task onto the cid's local DSQ.
- SCX_CAP_PREEMPT: preempt a task outside the scheduler's own subtree.
Higher caps imply the lower ones for the holder's own use.
- The root owns every cap on every cid. A parent delegates caps to a child with scx_bpf_sub_grant() and takes them back with scx_bpf_sub_revoke(). A child's cap set is always a subset of its parent's.
- The cap state is sharded. The cid space is split into topology-aligned shards, and each scheduler tracks its caps in per-shard cmasks with a per-shard lock. Operations are broken up on shard boundaries and different shards never contend. Shards are expected to serve as the locality unit when cids are handed out to schedulers, so lock granularity scales naturally with the allocation pattern.
- Hot paths check a per-cpu effective-caps copy with a single read. Grant/revoke rings a per-cpu doorbell and the owning cpu folds the new config in at the top of balance(). Cross-sched communication happens only when the delegation set changes.
Enforcement:
- An insert that lacks the required cap is diverted to a kernel-internal per-rq reject DSQ and handed back to the BPF scheduler to re-decide, tagged with SCX_TASK_REENQ_CAP and the missing caps.
- A revoke evicts already-queued and running tasks through the same reenqueue path.
- CPU occupancy is tied to the task slice. Extending a slice requires baseline access on the cpu.
- Kicks are enforced at delivery. Any kick needs baseline access on the target cid, and SCX_KICK_PREEMPT degrades to a plain reschedule if the victim is outside the kicker's subtree and the kicker lacks SCX_CAP_PREEMPT.
Notifications:
- ops.sub_caps_updated() reports config-level changes per shard with coalescing.
- ops.sub_ecaps_updated() reports per-cpu effective changes when they take effect.
- ops.update_idle() is routed to the cid's owner, with a re-notification when ownership changes while the cpu sits idle.
A parent can evict a misbehaving child with scx_bpf_sub_kill().
To make it concrete, consider a root scheduler R with a child A, which in turn has a child A1:
- On enable, R owns every cap on every cid. A child starts with no caps and cannot use any cpu on its own, so a parent normally makes an initial grant when a child attaches.
- R grants cids 8-15 to A by calling scx_bpf_sub_grant() with A's cgroup id and a cmask. A grant is all-or-nothing per cid and can only hand down caps R holds literally.
- A learns of the grant through ops.sub_caps_updated(). As each cpu folds the change into its effective caps, A gets ops.sub_ecaps_updated() for that cid, and if the cpu is sitting idle, an ops.update_idle() re-notification so A can use it right away.
- Nesting chains through the same notifier: from its ops.sub_caps_updated(), A calls scx_bpf_sub_grant() to pass a subset, say cids 12-15, down to A1, whose own ops.sub_caps_updated() then fires. A1's cap set is a subset of A's, which is a subset of R's. Caps A holds only through implication cannot be re-delegated.
Using and then losing a cpu, from A's point of view, looks like this:
- Holding SCX_CAP_ENQ on cid 8, A inserts its tasks into cid 8's local DSQ from its enqueue and dispatch paths, extends their slices, and kicks the cpu as usual.
- R revokes cid 8. The revoke clears it across A's whole subtree, so A1 loses whatever it held on the cid too.
- Once the revoke reaches the cpu's effective caps, A's tasks queued on cid 8 are handed back to A's ops.enqueue() tagged with SCX_TASK_REENQ_CAP and the missing caps, a running task is evicted by zeroing its slice, and new inserts are rejected and handed back the same way.
- A places the bounced tasks somewhere it still holds, and the notifiers tell it that its holdings shrank.
The final three patches expand the scx_qmap hierarchical demo: a qmap instance splits the cpus it fully owns among itself and child qmaps in proportion to cpu.weight, time-shares the rounding leftovers through a round-robin pool, and can fault-inject dispatches to unheld cids to demonstrate the kernel-side enforcement.
Known limitations:
After this series, sub-scheduling works as long as tasks stay within their cgroup boundary and aren't affined away from the cpus their scheduler can use. Two pieces are still missing for full support:
- cgroup migration doesn't move a task across sub-schedulers. A task moved between cgroups served by different sub-scheds keeps its original scheduler.
- there is no fallback when a sub-scheduler holds no cpu a task is allowed to run on. Such a task, e.g. one affined only to cids its scheduler was never granted or has had revoked, has nowhere to go. The likely direction is reserving a slice off the root scheduler to run tasks without cpu access.
Based on sched_ext/for-7.3 (daf8e166ba59).
This patchset contains the following 40 patches.
0001 sched_ext: Record an error on errno-only sub-enable failure
0002 sched_ext: Fix premature ops->priv publication in scx_alloc_and_add_sched() 0003 tools/sched_ext: scx - Fix cmask_subset(), cmask_equal() and cmask_weight() 0004 sched_ext: Use READ_ONCE/WRITE_ONCE in cmask word ops and drop _RACY variants 0005 sched_ext: Add SCX_CALL_CID_OP_TASK() for cid-form op dispatch 0006 sched_ext: Rename extra_enq_flags to remote_activate_enq_flags 0007 tools/sched_ext: scx_qmap - Use bare u64/u32/s32 integer types 0008 sched_ext: Reject direct slice and dsq_vtime writes for cid-form schedulers 0009 sched_ext: Make scx_bpf_kick_cid() return void
0010 sched_ext: Make the kick machinery per-sched
0011 sched_ext: Add ops.init_cids() to finalize the cid layout before init 0012 sched_ext: Add CID sharding
0013 sched_ext: Add shard boundaries to scx_bpf_cid_override()
0014 sched_ext: Defer scx_sched kobj sysfs add into the enable workfns 0015 sched_ext: Add per-shard scx_sched storage scaffolding
0016 sched_ext: Add scx_cmask_ref for validated arena cmask access 0017 sched_ext: Build the set_cmask scratch from trusted geometry 0018 sched_ext: RCU-protect the sub-sched tree's children/sibling lists 0019 sched_ext: Add scx_skip_subtree_pre()
0020 sched_ext: Stop resolving a disabled scheduler's programs
0021 sched_ext: Add per-shard cap delegation for sub-schedulers
0022 sched_ext: Add coalescing sub_caps_updated() notifier for sub-schedulers 0023 sched_ext: Maintain per-cpu effective cap copies for single-read checks 0024 sched_ext: Add sub_ecaps_updated() effective-cap change notifier 0025 sched_ext: Generalize local-DSQ handling to rq-owned DSQs
0026 sched_ext: Add reject DSQ for cap-rejected dispatches
0027 sched_ext: Add the SCX_CAP_ENQ_IMMED cap
0028 sched_ext: Assign a unique id to each scheduler instance
0029 sched_ext: Route task slice writes through set_task_slice()
0030 sched_ext: Track the cpu a task is runnable on
0031 sched_ext: Tie cpu occupancy to SCX_CAP_BASE through the task slice 0032 sched_ext: Add the SCX_CAP_ENQ cap
0033 sched_ext: Gate kicks on SCX_CAP_BASE and preemption on SCX_CAP_PREEMPT 0034 sched_ext: Authorize remote-move inserts against the placing scheduler 0035 sched_ext: Route ops.update_idle() to sub-schedulers and re-notify owed scheds 0036 sched_ext: Replay ecaps notifications suppressed by bypass
0037 sched_ext: Add scx_bpf_sub_kill() to evict a child sub-scheduler 0038 tools/sched_ext: Add three-mask cmask intersection iterator
0039 tools/sched_ext: scx_qmap - Expand hierarchical sub-scheduling 0040 tools/sched_ext: scx_qmap - Add sub-sched cap fault injection
The patches are organized as follows:
- 01-08: fixes and small prep.
- 09-20: plumbing. Per-sched kick machinery (09-10), ops.init_cids() (11), cid sharding (12-13), sysfs and per-shard storage prep (14-15), validated arena cmask access (16) and the trusted-geometry set_cmask build (17), RCU-safe tree walking (18-19), and gating a disabled scheduler's still-resolvable programs (20).
- 21-24: cap delegation. Per-shard caps with grant/revoke (21), the coalescing sub_caps_updated() notifier (22), per-cpu effective caps
(23) and the sub_ecaps_updated() notifier (24).
- 25-34: enforcement. The reject DSQ (25-26), SCX_CAP_ENQ_IMMED (27), slice-based occupancy (28-31), SCX_CAP_ENQ (32), preemption and kick gating (33) and remote-move insert authorization (34).
- 35-37: idle routing and re-notification (35), bypass replay (36) and scx_bpf_sub_kill() (37).
- 38-40: scx_qmap hierarchical demo.
The patchset is also available in the following git branch:
git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext.git scx-sub-caps-v4
diffstat follows. Thanks.
include/linux/sched/ext.h | 33 +- init/init_task.c | 1 + kernel/sched/ext/cid.c | 563 ++++++++++++--- kernel/sched/ext/cid.h | 19 +- kernel/sched/ext/ext.c | 915 +++++++++++++++++++----- kernel/sched/ext/idle.c | 86 ++- kernel/sched/ext/internal.h | 340 ++++++++- kernel/sched/ext/sub.c | 1126 +++++++++++++++++++++++++++++- kernel/sched/ext/sub.h | 118 +++- kernel/sched/ext/types.h | 74 +- kernel/sched/idle.c | 8 +- kernel/sched/sched.h | 18 +- tools/sched_ext/include/scx/cid.bpf.h | 165 ++++- tools/sched_ext/include/scx/common.bpf.h | 26 +- tools/sched_ext/include/scx/compat.bpf.h | 11 +- tools/sched_ext/scx_qmap.bpf.c | 840 ++++++++++++++++++++-- tools/sched_ext/scx_qmap.c | 375 +++++++++- tools/sched_ext/scx_qmap.h | 147 +++- 18 files changed, 4365 insertions(+), 500 deletions(-)
--
tejun
Tejun Heo
reviewerPATCH v4 1/40 sched_ext: Record an error on errno-only sub-enable failure
scx_sub_enable_workfn() has several failure paths that only return an errno (e.g. -ENOMEM from an allocation) and jump to err_disable without calling scx_error(). scx_flush_disable_work() runs the disable, and thus ops.exit(), only when an error has been recorded, so an errno-only failure leaves the half-initialized sub-scheduler linked.
Record an error at the err_disable sink so every errno-only failure runs the disable path.
--- kernel/sched/ext/sub.c | 6 ++++++ 1 file changed, 6 insertions(+)
kernel/sched/ext/sub.c +6 -0
@@ -570,6 +570,12 @@ void scx_sub_enable_workfn(struct kthread_work *work)percpu_up_write (&scx_fork_rwsem); err_disable:mutex_unlock (&scx_enable_mutex); +/* +* Some enable failures only return an errno (e.g. -ENOMEM from an +* allocation) without calling scx_error(). Record it so +* scx_flush_disable_work() runs the disable and ops.exit() fires. +*/ +scx_error (sch,"scx_sub_enable() failed (%d)" , ret);scx_flush_disable_work (sch); cmd->ret =0 ; }
Signature · 1 line
2.54.0
Re: [PATCH v4 sched_ext/for-7.3 01/40] sched_ext: Record an error on errno-only sub-enable failure
Quoted 11 lines · Tejun Heo
scx_sub_enable_workfn()has several failure paths that only return an errno (e.g.-ENOMEMfrom an allocation) and jump toerr_disablewithout callingscx_error().scx_flush_disable_work()runs the disable, and thusops.exit(), only when an error has been recorded, so an errno-only failure leaves the half-initialized sub-scheduler linked.Record an error at the
err_disablesink so every errno-only failure runs the disable path.Fixes ebeca1f930ea ("sched_ext: Introduce cgroup sub-sched support")Signed-off-by Tejun Heo <[email protected]>
Thanks,
-Andrea
Quoted 24 lines
--- kernel/sched/ext/sub.c | 6 ++++++ 1 file changed, 6 insertions(+)kernel/sched/ext/sub.c +6
-0@@ -570,6 +570,12 @@ void scx_sub_enable_workfn(struct kthread_work *work)percpu_up_write (&scx_fork_rwsem); err_disable:mutex_unlock (&scx_enable_mutex); +/* +* Some enable failures only return an errno (e.g. -ENOMEM from an +* allocation) without calling scx_error(). Record it so +* scx_flush_disable_work() runs the disable and ops.exit() fires. +*/ +scx_error (sch,"scx_sub_enable() failed (%d)" , ret);scx_flush_disable_work (sch); cmd->ret =0 ; }Signature · 1 line
2.54.0
Tejun Heo
reviewerPATCH v4 2/40 sched_ext: Fix premature ops->priv publication in scx_alloc_and_add_sched()
In reply to Tejun Heoscx_alloc_and_add_sched() publishes @sch through ops->priv before allocating the cgroup path. If that allocation fails, the unwind path clears ops->priv and frees @sch immediately. scx_prog_sched() callers can dereference ops->priv from RCU context the moment it is set, so freeing without a grace period can use-after-free a concurrent kfunc caller.
Move the publication below the cgroup path allocation so that every failure path after publication frees @sch through kobject_put(), whose release path defers the freeing by a grace period.
--- kernel/sched/ext/ext.c | 18 ++++++++++++------ 1 file changed, 12 insertions(+), 6 deletions(-)
kernel/sched/ext/ext.c +12 -6
@@ -6363,11 +6363,6 @@ struct scx_sched *scx_alloc_and_add_sched(struct scx_enable_cmd *cmd, sch->ops = *cmd->ops; } -rcu_assign_pointer (ops->priv, sch); - - sch->kobj.kset = scx_kset; -INIT_LIST_HEAD (&sch->all); -#ifdef CONFIG_EXT_SUB_SCHED char *buf =kzalloc (PATH_MAX ,GFP_KERNEL );if (!buf) {
@@ -6385,7 +6380,19 @@ struct scx_sched *scx_alloc_and_add_sched(struct scx_enable_cmd *cmd, sch->cgrp = cgrp;INIT_LIST_HEAD (&sch->children);INIT_LIST_HEAD (&sch->sibling); +#endif /* CONFIG_EXT_SUB_SCHED */ +/* +* Publishing makes @sch visible to scx_prog_sched() readers. Failure +* paths after this point must free @sch through kobject_put() whose +* release path defers the actual freeing by an RCU grace period. +*/ +rcu_assign_pointer (ops->priv, sch); + + sch->kobj.kset = scx_kset; +INIT_LIST_HEAD (&sch->all); + +#ifdef CONFIG_EXT_SUB_SCHED if (parent) {/* * Pin @parent for @sch's lifetime. The kobject hierarchy pins
@@ -6440,7 +6447,6 @@ struct scx_sched *scx_alloc_and_add_sched(struct scx_enable_cmd *cmd,#ifdef CONFIG_EXT_SUB_SCHED err_free_lb_resched: -RCU_INIT_POINTER (ops->priv,NULL );free_cpumask_var (sch->stall_cpus);#endif err_free_lb_resched_cpumask:
Signature · 1 line
2.54.0
Re: [PATCH v4 sched_ext/for-7.3 02/40] sched_ext: Fix premature ops->priv publication in scx_alloc_and_add_sched()
Quoted 6 lines · Tejun Heo
scx_alloc_and_add_sched()publishes @sch throughops->privbefore allocating the cgroup path. If that allocation fails, the unwind path clearsops->privand frees @sch immediately.scx_prog_sched()callers can dereferenceops->privfrom RCU context the moment it is set, so freeing without a grace period can use-after-free a concurrent kfunc caller.
Quoted 6 lines
Move the publication below the cgroup path allocation so that every failure path after publication frees @sch through
kobject_put(), whose release path defers the freeing by a grace period.Fixes 105dcd005be2 ("sched_ext: Introduce scx_prog_sched()")Signed-off-by Tejun Heo <[email protected]>
Thanks,
-Andrea
Quoted 51 lines
--- kernel/sched/ext/ext.c | 18 ++++++++++++------ 1 file changed, 12 insertions(+), 6 deletions(-)kernel/sched/ext/ext.c +12
-6@@ -6363,11 +6363,6 @@ struct scx_sched *scx_alloc_and_add_sched(struct scx_enable_cmd *cmd, sch->ops = *cmd->ops; } -rcu_assign_pointer (ops->priv, sch); - - sch->kobj.kset = scx_kset; -INIT_LIST_HEAD (&sch->all); -#ifdef CONFIG_EXT_SUB_SCHED char *buf =kzalloc (PATH_MAX ,GFP_KERNEL );if (!buf) {@@ -6385,7 +6380,19 @@ struct scx_sched *scx_alloc_and_add_sched(struct scx_enable_cmd *cmd, sch->cgrp = cgrp;INIT_LIST_HEAD (&sch->children);INIT_LIST_HEAD (&sch->sibling); +#endif /* CONFIG_EXT_SUB_SCHED */ +/* +* Publishing makes @sch visible to scx_prog_sched() readers. Failure +* paths after this point must free @sch through kobject_put() whose +* release path defers the actual freeing by an RCU grace period. +*/ +rcu_assign_pointer (ops->priv, sch); + + sch->kobj.kset = scx_kset; +INIT_LIST_HEAD (&sch->all); + +#ifdef CONFIG_EXT_SUB_SCHED if (parent) {/* * Pin @parent for @sch's lifetime. The kobject hierarchy pins @@ -6440,7 +6447,6 @@ struct scx_sched *scx_alloc_and_add_sched(struct scx_enable_cmd *cmd,#ifdef CONFIG_EXT_SUB_SCHED err_free_lb_resched: -RCU_INIT_POINTER (ops->priv,NULL );free_cpumask_var (sch->stall_cpus);#endif err_free_lb_resched_cpumask:Signature · 1 line
2.54.0
Tejun Heo
reviewerPATCH v4 3/40 tools/sched_ext: scx - Fix cmask_subset(), cmask_equal() and cmask_weight()
In reply to Tejun Heocmask_equal(), cmask_weight() and cmask_subset() bounded their word walks with CMASK_NR_WORDS(nr_cids), which pads by one word and can't tell the last word in use without @base. The walks could thus cover a slack word past the active range, which cmask_reframe() leaves non-zero: a stale bit there gave cmask_equal() a spurious mismatch, cmask_weight() an inflated count, and cmask_subset() a spurious violation. cmask_subset() could also read @b->bits[] one word past its allocation (within the arena's fault-recovered range, so harmless), and deviated from the kernel scx_cmask_subset() by failing any @a range that doesn't nest inside @b's even when the overhanging bits are all clear.
Bound the cmask_equal() and cmask_weight() walks by the words the range actually spans, with early returns for empty ranges. Rewrite cmask_subset() to match the kernel semantics: scan @a's overhangs for set bits with cmask_next_set() and walk the words of the range intersection. cmask_subset() moves below cmask_next_set(), which it now uses. Padding bits don't need masking as every cmask helper keeps them clear.
--- tools/sched_ext/include/scx/cid.bpf.h | 88 +++++++++++++++++---------- 1 file changed, 55 insertions(+), 33 deletions(-)
tools/sched_ext/include/scx/cid.bpf.h +55 -33
@@ -391,7 +391,9 @@ static __always_inline bool cmask_equal(const struct scx_cmask __arena *a,if (a->base != b->base || a->nr_cids != b->nr_cids)return false; - nr_words =CMASK_NR_WORDS (a->nr_cids); +if (a->nr_cids ==0 ) +return true; + nr_words = (a->base + a->nr_cids -1 ) /64 - a->base /64 +1 ; bpf_for(i, 0, CMASK_MAX_WORDS) {if (i >= nr_words)
@@ -402,36 +404,6 @@ static __always_inline bool cmask_equal(const struct scx_cmask __arena *a,return true; } -/* -* True iff every bit set in @a is also set in @b over the intersection of -* their ranges. Bits of @a outside @b's range fail the test. -*/ -static __always_inline bool cmask_subset(const struct scx_cmask __arena *a, - const struct scx_cmask __arena *b) -{ -u32 a_end = a->base + a->nr_cids; -u32 b_end = b->base + b->nr_cids; -u32 a_wbase = a->base /64 ; -u32 b_wbase = b->base /64 ; -u32 nr_words, i; - -/* any bit of @a outside @b's range is a subset violation */ -if (a->base < b->base || a_end > b_end) -return false; - - nr_words =CMASK_NR_WORDS (a->nr_cids); - bpf_for(i, 0, CMASK_MAX_WORDS) { -u32 wi_b; - -if (i >= nr_words) -break ; - wi_b = a_wbase + i - b_wbase; -if (a->bits[i] & ~b->bits[wi_b]) -return false; - } -return true; -} -/** * cmask_next_set - find the first set bit at or after @cid * @m: cmask to search
@@ -488,16 +460,66 @@ static __always_inline u32 cmask_first_set(const struct scx_cmask __arena *m) (cid) < (m)->base + (m)->nr_cids; \ (cid) = cmask_next_set((m), (cid) + 1)) +/* +* True iff every bit set in @a is also set in @b. Matches the kernel-side +* scx_cmask_subset(): ranges don't need to nest, and set bits of @a outside +* @b's range fail the test. +*/ +static __always_inline bool cmask_subset(const struct scx_cmask __arena *a, + const struct scx_cmask __arena *b) +{ +u32 a_end = a->base + a->nr_cids; +u32 b_end = b->base + b->nr_cids; +u32 a_wbase = a->base /64 ; +u32 b_wbase = b->base /64 ; +u32 lo = a->base > b->base ? a->base : b->base; +u32 hi = a_end < b_end ? a_end : b_end; +u32 lo_word, hi_word, i; + +/* set bits of @a outside @b's range can't be in @b */ +if (a->base < b->base && +cmask_next_set (a, a->base) < (b->base < a_end ? b->base : a_end)) +return false; +if (a_end > b_end && +cmask_next_set (a, a->base > b_end ? a->base : b_end) < a_end) +return false; + +if (lo >= hi) +return true; + +/* +* Walk the words the range intersection spans. Plain word tests +* suffice: the scans above guarantee @a has no set bit outside @b's +* range and padding bits are kept clear by all cmask helpers. +*/ + lo_word = lo /64 ; + hi_word = (hi -1 ) /64 ; + + bpf_for(i, 0, CMASK_MAX_WORDS) { +u32 w = lo_word + i; + +if (w > hi_word) +break ; +if (a->bits[w - a_wbase] & ~b->bits[w - b_wbase]) +return false; + } +return true; +} +/* * Population count over [base, base + nr_cids). Padding bits in the head/tail * words are guaranteed zero by the mutating helpers, so a flat popcount over -* all words is correct. +* the words the range spans is correct. */ static __always_inline u32 cmask_weight(const struct scx_cmask __arena *m) { -u32 nr_words =CMASK_NR_WORDS (m->nr_cids), i; +u32 nr_words, i;u32 count =0 ; +if (!m->nr_cids) +return 0 ; + nr_words = (m->base + m->nr_cids -1 ) /64 - m->base /64 +1 ; + bpf_for(i, 0, CMASK_MAX_WORDS) {if (i >= nr_words)break ;
Signature · 1 line
2.54.0
Re: [PATCH v4 sched_ext/for-7.3 03/40] tools/sched_ext: scx - Fix cmask_subset(), cmask_equal() and cmask_weight()
Quoted 11 lines · Tejun Heo
cmask_equal(),cmask_weight()andcmask_subset()bounded their word walks withCMASK_NR_WORDS(nr_cids), which pads by one word and can't tell the last word in use without @base. The walks could thus cover a slack word past the active range, whichcmask_reframe()leaves non-zero: a stale bit there gavecmask_equal()a spurious mismatch,cmask_weight()an inflated count, andcmask_subset()a spurious violation.cmask_subset()could also read @b->bits[] one word past its allocation (within the arena's fault-recovered range, so harmless), and deviated from the kernelscx_cmask_subset()by failing any @a range that doesn't nest inside @b's even when the overhanging bits are all clear.
Quoted 8 lines
Bound the
cmask_equal()andcmask_weight()walks by the words the range actually spans, with early returns for empty ranges. Rewritecmask_subset()to match the kernel semantics: scan @a's overhangs for set bits withcmask_next_set()and walk the words of the range intersection.cmask_subset()moves belowcmask_next_set(), which it now uses. Padding bits don't need masking as every cmask helper keeps them clear.Signed-off-by Tejun Heo <[email protected]>
Thanks,
-Andrea
Quoted 128 lines
--- tools/sched_ext/include/scx/cid.bpf.h | 88 +++++++++++++++++---------- 1 file changed, 55 insertions(+), 33 deletions(-)tools/sched_ext/include/scx/cid.bpf.h +55
-33@@ -391,7 +391,9 @@ static __always_inline bool cmask_equal(const struct scx_cmask __arena *a,if (a->base != b->base || a->nr_cids != b->nr_cids)return false; - nr_words =CMASK_NR_WORDS (a->nr_cids); +if (a->nr_cids ==0 ) +return true; + nr_words = (a->base + a->nr_cids -1 ) /64 - a->base /64 +1 ; bpf_for(i, 0, CMASK_MAX_WORDS) {if (i >= nr_words)@@ -402,36 +404,6 @@ static __always_inline bool cmask_equal(const struct scx_cmask __arena *a,return true; } -/* -* True iff every bit set in @a is also set in @b over the intersection of -* their ranges. Bits of @a outside @b's range fail the test. -*/ -static __always_inline bool cmask_subset(const struct scx_cmask __arena *a, - const struct scx_cmask __arena *b) -{ -u32 a_end = a->base + a->nr_cids; -u32 b_end = b->base + b->nr_cids; -u32 a_wbase = a->base /64 ; -u32 b_wbase = b->base /64 ; -u32 nr_words, i; - -/* any bit of @a outside @b's range is a subset violation */ -if (a->base < b->base || a_end > b_end) -return false; - - nr_words =CMASK_NR_WORDS (a->nr_cids); - bpf_for(i, 0, CMASK_MAX_WORDS) { -u32 wi_b; - -if (i >= nr_words) -break ; - wi_b = a_wbase + i - b_wbase; -if (a->bits[i] & ~b->bits[wi_b]) -return false; - } -return true; -} -/** * cmask_next_set - find the first set bit at or after @cid * @m: cmask to search @@ -488,16 +460,66 @@ static __always_inline u32 cmask_first_set(const struct scx_cmask __arena *m) (cid) < (m)->base + (m)->nr_cids; \ (cid) = cmask_next_set((m), (cid) + 1)) +/* +* True iff every bit set in @a is also set in @b. Matches the kernel-side +* scx_cmask_subset(): ranges don't need to nest, and set bits of @a outside +* @b's range fail the test. +*/ +static __always_inline bool cmask_subset(const struct scx_cmask __arena *a, + const struct scx_cmask __arena *b) +{ +u32 a_end = a->base + a->nr_cids; +u32 b_end = b->base + b->nr_cids; +u32 a_wbase = a->base /64 ; +u32 b_wbase = b->base /64 ; +u32 lo = a->base > b->base ? a->base : b->base; +u32 hi = a_end < b_end ? a_end : b_end; +u32 lo_word, hi_word, i; + +/* set bits of @a outside @b's range can't be in @b */ +if (a->base < b->base && +cmask_next_set (a, a->base) < (b->base < a_end ? b->base : a_end)) +return false; +if (a_end > b_end && +cmask_next_set (a, a->base > b_end ? a->base : b_end) < a_end) +return false; + +if (lo >= hi) +return true; + +/* +* Walk the words the range intersection spans. Plain word tests +* suffice: the scans above guarantee @a has no set bit outside @b's +* range and padding bits are kept clear by all cmask helpers. +*/ + lo_word = lo /64 ; + hi_word = (hi -1 ) /64 ; + + bpf_for(i, 0, CMASK_MAX_WORDS) { +u32 w = lo_word + i; + +if (w > hi_word) +break ; +if (a->bits[w - a_wbase] & ~b->bits[w - b_wbase]) +return false; + } +return true; +} +/* * Population count over [base, base + nr_cids). Padding bits in the head/tail * words are guaranteed zero by the mutating helpers, so a flat popcount over -* all words is correct. +* the words the range spans is correct. */ static __always_inline u32 cmask_weight(const struct scx_cmask __arena *m) { -u32 nr_words =CMASK_NR_WORDS (m->nr_cids), i; +u32 nr_words, i;u32 count =0 ; +if (!m->nr_cids) +return 0 ; + nr_words = (m->base + m->nr_cids -1 ) /64 - m->base /64 +1 ; + bpf_for(i, 0, CMASK_MAX_WORDS) {if (i >= nr_words)break ;Signature · 1 line
2.54.0
Tejun Heo
reviewerPATCH v4 4/40 sched_ext: Use READ_ONCE/WRITE_ONCE in cmask word ops and drop _RACY variants
In reply to Tejun HeoThe cmask ops can operate on BPF-arena cmasks which BPF programs may read and write concurrently. The _RACY op variants existed to make such lockless reads explicit but this turned out to be too restrictive. Mark the word accesses in all the two-cmask ops with READ_ONCE/WRITE_ONCE instead and drop the _RACY variants.
v2: READ_ONCE the cmask_word_op1() ANY_SET read. (sashiko AI)
--- kernel/sched/ext/cid.c | 55 ++++++++---------------------------------- kernel/sched/ext/cid.h | 2 -- 2 files changed, 10 insertions(+), 47 deletions(-)
kernel/sched/ext/cid.c +10 -45
@@ -395,17 +395,15 @@ __bpf_kfunc s32 scx_bpf_cpu_to_cid(s32 cpu, const struct bpf_prog_aux *aux)* bits outside stay untouched. In particular, scx_cmask_copy() does NOT zero * @dst bits that lie outside @src's range. * -* The _RACY variants are otherwise identical to their non-racy counterpart but -* read @src word-by-word via data_race(). Memory ordering with concurrent -* writers is the caller's responsibility. +* Word accesses use READ_ONCE/WRITE_ONCE so a caller may read @src +* locklessly. Memory ordering against concurrent writers is the caller's +* responsibility. */ enum cmask_op2 {/* mutating */ CMASK_OP2_AND ,CMASK_OP2_OR , -CMASK_OP2_OR_RACY ,CMASK_OP2_COPY , -CMASK_OP2_COPY_RACY ,CMASK_OP2_ANDNOT ,/* predicates - short-circuit when the per-word result is true */ CMASK_OP2_SUBSET ,
@@ -422,28 +420,22 @@ static __always_inline bool cmask_word_op2(u64 *av, const u64 *bp, u64 mask, {switch (op) {case CMASK_OP2_AND : - *av &= ~mask | *bp; +WRITE_ONCE (*av, *av & (~mask |READ_ONCE (*bp)));return false;case CMASK_OP2_OR : - *av |= *bp & mask; -return false; -case CMASK_OP2_OR_RACY : - *av |=data_race (*bp) & mask; +WRITE_ONCE (*av, *av | (READ_ONCE (*bp) & mask));return false;case CMASK_OP2_COPY : - *av = (*av & ~mask) | (*bp & mask); -return false; -case CMASK_OP2_COPY_RACY : - *av = (*av & ~mask) | (data_race (*bp) & mask); +WRITE_ONCE (*av, (*av & ~mask) | (READ_ONCE (*bp) & mask));return false;case CMASK_OP2_ANDNOT : - *av &= ~(*bp & mask); +WRITE_ONCE (*av, *av & ~(READ_ONCE (*bp) & mask));return false;case CMASK_OP2_SUBSET :/* stop on the first bit in @sub not set in @super */ -return (*bp & ~*av) & mask; +return (READ_ONCE (*bp) & ~READ_ONCE (*av)) & mask;case CMASK_OP2_INTERSECTS : -return (*av & *bp) & mask; +return (READ_ONCE (*av) &READ_ONCE (*bp)) & mask; }unreachable (); }
@@ -504,7 +496,7 @@ static __always_inline bool cmask_word_op1(const u64 *ap, u64 mask, {switch (op) {case CMASK_OP1_ANY_SET : -return *ap & mask; +return READ_ONCE (*ap) & mask; }unreachable (); }
@@ -556,39 +548,12 @@ void scx_cmask_or(struct scx_cmask *dst, const struct scx_cmask *src) src->bits, src->base, src->nr_cids, CMASK_OP2_OR); } -/** -* scx_cmask_or_racy - OR @src into @dst, reading @src without locking -* -* @src is read word-by-word through data_race(). Same per-bit independence -* rationale as scx_cmask_copy_racy(). Memory ordering with writers is the -* caller's responsibility. -*/ -void scx_cmask_or_racy (struct scx_cmask *dst,const struct scx_cmask *src) -{ -cmask_walk_op2 (dst->bits, dst->base, dst->nr_cids, - src->bits, src->base, src->nr_cids,CMASK_OP2_OR_RACY ); -} -void scx_cmask_copy (struct scx_cmask *dst,const struct scx_cmask *src) {cmask_walk_op2 (dst->bits, dst->base, dst->nr_cids, src->bits, src->base, src->nr_cids,CMASK_OP2_COPY ); } -/** -* scx_cmask_copy_racy - Snapshot @src into @dst without locking -* -* @src is read word-by-word through data_race(). Head/tail masking matches -* scx_cmask_copy(). Each bit in a cmask is independent, so partial updates -* just leave some bits fresher than others. Memory ordering with writers is -* the caller's responsibility. -*/ -void scx_cmask_copy_racy (struct scx_cmask *dst,const struct scx_cmask *src) -{ -cmask_walk_op2 (dst->bits, dst->base, dst->nr_cids, - src->bits, src->base, src->nr_cids,CMASK_OP2_COPY_RACY ); -} -void scx_cmask_andnot (struct scx_cmask *dst,const struct scx_cmask *src) { cmask_walk_op2(dst->bits, dst->base, dst->nr_cids,
kernel/sched/ext/cid.h +0 -2
@@ -57,9 +57,7 @@ void scx_cmask_clear(struct scx_cmask *m);void scx_cmask_fill (struct scx_cmask *m);void scx_cmask_and (struct scx_cmask *dst,const struct scx_cmask *src);void scx_cmask_or (struct scx_cmask *dst,const struct scx_cmask *src); -void scx_cmask_or_racy (struct scx_cmask *dst,const struct scx_cmask *src);void scx_cmask_copy (struct scx_cmask *dst,const struct scx_cmask *src); -void scx_cmask_copy_racy (struct scx_cmask *dst,const struct scx_cmask *src);void scx_cmask_andnot (struct scx_cmask *dst,const struct scx_cmask *src);bool scx_cmask_subset (const struct scx_cmask *sub,const struct scx_cmask *super);bool scx_cmask_intersects (const struct scx_cmask *a,const struct scx_cmask *b);
Signature · 1 line
2.54.0
Re: [PATCH v4 sched_ext/for-7.3 04/40] sched_ext: Use READ_ONCE/WRITE_ONCE in cmask word ops and drop _RACY variants
Quoted 9 lines · Tejun Heo
The cmask ops can operate on BPF-arena cmasks which BPF programs may read and write concurrently. The
_RACYop variants existed to make such lockless reads explicit but this turned out to be too restrictive. Mark the word accesses in all the two-cmask ops with READ_ONCE/WRITE_ONCE instead and drop the_RACYvariants.v2:
READ_ONCEthecmask_word_op1()ANY_SETread. (sashiko AI)Signed-off-by Tejun Heo <[email protected]>
Thanks,
-Andrea
Quoted 131 lines
--- kernel/sched/ext/cid.c | 55 ++++++++---------------------------------- kernel/sched/ext/cid.h | 2 -- 2 files changed, 10 insertions(+), 47 deletions(-)kernel/sched/ext/cid.c +10
-45@@ -395,17 +395,15 @@ __bpf_kfunc s32 scx_bpf_cpu_to_cid(s32 cpu, const struct bpf_prog_aux *aux)* bits outside stay untouched. In particular, scx_cmask_copy() does NOT zero * @dst bits that lie outside @src's range. * -* The _RACY variants are otherwise identical to their non-racy counterpart but -* read @src word-by-word via data_race(). Memory ordering with concurrent -* writers is the caller's responsibility. +* Word accesses use READ_ONCE/WRITE_ONCE so a caller may read @src +* locklessly. Memory ordering against concurrent writers is the caller's +* responsibility. */ enum cmask_op2 {/* mutating */ CMASK_OP2_AND ,CMASK_OP2_OR , -CMASK_OP2_OR_RACY ,CMASK_OP2_COPY , -CMASK_OP2_COPY_RACY ,CMASK_OP2_ANDNOT ,/* predicates - short-circuit when the per-word result is true */ CMASK_OP2_SUBSET ,@@ -422,28 +420,22 @@ static __always_inline bool cmask_word_op2(u64 *av, const u64 *bp, u64 mask, {switch (op) {case CMASK_OP2_AND : - *av &= ~mask | *bp; +WRITE_ONCE (*av, *av & (~mask |READ_ONCE (*bp)));return false;case CMASK_OP2_OR : - *av |= *bp & mask; -return false; -case CMASK_OP2_OR_RACY : - *av |=data_race (*bp) & mask; +WRITE_ONCE (*av, *av | (READ_ONCE (*bp) & mask));return false;case CMASK_OP2_COPY : - *av = (*av & ~mask) | (*bp & mask); -return false; -case CMASK_OP2_COPY_RACY : - *av = (*av & ~mask) | (data_race (*bp) & mask); +WRITE_ONCE (*av, (*av & ~mask) | (READ_ONCE (*bp) & mask));return false;case CMASK_OP2_ANDNOT : - *av &= ~(*bp & mask); +WRITE_ONCE (*av, *av & ~(READ_ONCE (*bp) & mask));return false;case CMASK_OP2_SUBSET :/* stop on the first bit in @sub not set in @super */ -return (*bp & ~*av) & mask; +return (READ_ONCE (*bp) & ~READ_ONCE (*av)) & mask;case CMASK_OP2_INTERSECTS : -return (*av & *bp) & mask; +return (READ_ONCE (*av) &READ_ONCE (*bp)) & mask; }unreachable (); }@@ -504,7 +496,7 @@ static __always_inline bool cmask_word_op1(const u64 *ap, u64 mask, {switch (op) {case CMASK_OP1_ANY_SET : -return *ap & mask; +return READ_ONCE (*ap) & mask; }unreachable (); }@@ -556,39 +548,12 @@ void scx_cmask_or(struct scx_cmask *dst, const struct scx_cmask *src) src->bits, src->base, src->nr_cids, CMASK_OP2_OR); } -/** -* scx_cmask_or_racy - OR @src into @dst, reading @src without locking -* -* @src is read word-by-word through data_race(). Same per-bit independence -* rationale as scx_cmask_copy_racy(). Memory ordering with writers is the -* caller's responsibility. -*/ -void scx_cmask_or_racy (struct scx_cmask *dst,const struct scx_cmask *src) -{ -cmask_walk_op2 (dst->bits, dst->base, dst->nr_cids, - src->bits, src->base, src->nr_cids,CMASK_OP2_OR_RACY ); -} -void scx_cmask_copy (struct scx_cmask *dst,const struct scx_cmask *src) {cmask_walk_op2 (dst->bits, dst->base, dst->nr_cids, src->bits, src->base, src->nr_cids,CMASK_OP2_COPY ); } -/** -* scx_cmask_copy_racy - Snapshot @src into @dst without locking -* -* @src is read word-by-word through data_race(). Head/tail masking matches -* scx_cmask_copy(). Each bit in a cmask is independent, so partial updates -* just leave some bits fresher than others. Memory ordering with writers is -* the caller's responsibility. -*/ -void scx_cmask_copy_racy (struct scx_cmask *dst,const struct scx_cmask *src) -{ -cmask_walk_op2 (dst->bits, dst->base, dst->nr_cids, - src->bits, src->base, src->nr_cids,CMASK_OP2_COPY_RACY ); -} -void scx_cmask_andnot (struct scx_cmask *dst,const struct scx_cmask *src) { cmask_walk_op2(dst->bits, dst->base, dst->nr_cids,kernel/sched/ext/cid.h +0
-2@@ -57,9 +57,7 @@ void scx_cmask_clear(struct scx_cmask *m);void scx_cmask_fill (struct scx_cmask *m);void scx_cmask_and (struct scx_cmask *dst,const struct scx_cmask *src);void scx_cmask_or (struct scx_cmask *dst,const struct scx_cmask *src); -void scx_cmask_or_racy (struct scx_cmask *dst,const struct scx_cmask *src);void scx_cmask_copy (struct scx_cmask *dst,const struct scx_cmask *src); -void scx_cmask_copy_racy (struct scx_cmask *dst,const struct scx_cmask *src);void scx_cmask_andnot (struct scx_cmask *dst,const struct scx_cmask *src);bool scx_cmask_subset (const struct scx_cmask *sub,const struct scx_cmask *super);bool scx_cmask_intersects (const struct scx_cmask *a,const struct scx_cmask *b);Signature · 1 line
2.54.0
Tejun Heo
reviewerPATCH v4 5/40 sched_ext: Add SCX_CALL_CID_OP_TASK() for cid-form op dispatch
In reply to Tejun HeoThe cid-form ops overlay their cpu-form siblings at the same struct slot. Ops whose signature matches the sibling are invoked through the cpu-form call sites unchanged, but set_cmask() takes an arena cmask address rather than a cpumask, so scx_call_op_set_cpumask() calls ops_cid.set_cmask() directly and hand-rolled the kf_tasks[] and locked_rq bracket that SCX_CALL_OP_TASK() provides. The hand-rolled bracket reset locked_rq to NULL on exit instead of restoring the saved value, so a nested call would clobber the outer op's locked-rq tracking.
Parameterize the dispatch macros by the ops-table member and add SCX_CALL_CID_OP_TASK(), which routes through sch->ops_cid. Convert scx_call_op_set_cpumask() to it and drop the hand-rolled bracket. The only behavioral change is that locked_rq is now saved and restored like every other op call site.
--- kernel/sched/ext/ext.c | 14 +++----------- kernel/sched/ext/internal.h | 23 ++++++++++++++++++++--- 2 files changed, 23 insertions(+), 14 deletions(-)
kernel/sched/ext/ext.c +3 -11
@@ -410,11 +410,6 @@ static inline void scx_call_op_set_cpumask(struct scx_sched *sch, struct rq *rq,struct task_struct *task, const struct cpumask *cpumask) { - WARN_ON_ONCE(current->scx.kf_tasks[0]); - current->scx.kf_tasks[0 ] = task; -if (rq) -update_locked_rq (rq); - if (scx_is_cid_type()) { struct scx_cmask *kern_va = *this_cpu_ptr(sch->set_cmask_scratch);/*
@@ -423,14 +418,11 @@ static inline void scx_call_op_set_cpumask(struct scx_sched *sch, struct rq *rq,* the sole user of the scratch area. */ scx_cpumask_to_cmask (cpumask, kern_va); - sch->ops_cid.set_cmask (task,scx_kaddr_to_arena (sch, kern_va)); +SCX_CALL_CID_OP_TASK (sch, set_cmask, rq, task, +scx_kaddr_to_arena (sch, kern_va)); }else { - sch->ops.set_cpumask (task, cpumask); +SCX_CALL_OP_TASK (sch, set_cpumask, rq, task, cpumask); } - -if (rq) -update_locked_rq (NULL ); - current->scx.kf_tasks[0 ] =NULL ; } enum scx_dsq_iter_flags {
kernel/sched/ext/internal.h +20 -3
@@ -1751,8 +1751,11 @@ static inline void update_locked_rq(struct rq *rq)/* * SCX ops can recurse via scx_bpf_sub_dispatch() - the inner call must not * clobber the outer's scx_locked_rq_state. Save it on entry, restore on exit. +* +* @ops is the ops table to dispatch through: ops for the cpu form, ops_cid +* for the cid form. */ -#define SCX_CALL_OP(sch, op, locked_rq, args...) \ +#define __SCX_CALL_OP(sch, ops, op, locked_rq, args...) \ do { \ struct rq *__prev_locked_rq; \ \
@@ -1765,6 +1768,9 @@ do { \update_locked_rq (__prev_locked_rq); \ } while (0) +#define SCX_CALL_OP(sch, op, locked_rq, args...) \ + __SCX_CALL_OP(sch, ops, op, locked_rq, ##args) + #define SCX_CALL_OP_RET(sch, op, locked_rq, args...) \ ({ \ struct rq *__prev_locked_rq; \
@@ -1796,14 +1802,25 @@ do { \* WARN_ON_ONCE() in each macro catches a re-entry of any of the three variants * while a previous one is still in progress. */ -#define SCX_CALL_OP_TASK(sch, op, locked_rq, task, args...) \ +#define __SCX_CALL_OP_TASK(sch, ops, op, locked_rq, task, args...) \ do { \ WARN_ON_ONCE(current->scx.kf_tasks[0]); \ current->scx.kf_tasks[0] = task; \ - SCX_CALL_OP((sch), op, locked_rq, task, ##args); \ + __SCX_CALL_OP((sch), ops, op, locked_rq, task, ##args); \ current->scx.kf_tasks[0] = NULL; \ } while (0) +#define SCX_CALL_OP_TASK(sch, op, locked_rq, task, args...) \ + __SCX_CALL_OP_TASK(sch, ops, op, locked_rq, task, ##args) + +/* +* Dispatch a task op through the cid-form ops_cid table. Only set_cmask() needs +* this: it takes an arena cmask address instead of a cpumask, so it cannot be +* invoked via its cpu-form set_cpumask() slot. +*/ +#define SCX_CALL_CID_OP_TASK(sch, op, locked_rq, task, args...) \ + __SCX_CALL_OP_TASK(sch, ops_cid, op, locked_rq, task, ##args) + #define SCX_CALL_OP_TASK_RET(sch, op, locked_rq, task, args...) \ ({ \ __typeof__((sch)->ops.op(task, ##args)) __ret; \
Signature · 1 line
2.54.0
Re: [PATCH v4 sched_ext/for-7.3 05/40] sched_ext: Add SCX_CALL_CID_OP_TASK() for cid-form op dispatch
Quoted 9 lines · Tejun Heo
The cid-form ops overlay their cpu-form siblings at the same struct slot. Ops whose signature matches the sibling are invoked through the cpu-form call sites unchanged, but
set_cmask()takes an arena cmask address rather than a cpumask, soscx_call_op_set_cpumask()callsops_cid.set_cmask()directly and hand-rolled the kf_tasks[] andlocked_rqbracket thatSCX_CALL_OP_TASK()provides. The hand-rolled bracket resetlocked_rqto NULL on exit instead of restoring the saved value, so a nested call would clobber the outer op's locked-rq tracking.
Quoted 7 lines
Parameterize the dispatch macros by the ops-table member and add
SCX_CALL_CID_OP_TASK(), which routes throughsch->ops_cid. Convertscx_call_op_set_cpumask()to it and drop the hand-rolled bracket. The only behavioral change is thatlocked_rqis now saved and restored like every other op call site.Signed-off-by Tejun Heo <[email protected]>
Thanks,
-Andrea
Quoted 97 lines
--- kernel/sched/ext/ext.c | 14 +++----------- kernel/sched/ext/internal.h | 23 ++++++++++++++++++++--- 2 files changed, 23 insertions(+), 14 deletions(-)kernel/sched/ext/ext.c +3
-11@@ -410,11 +410,6 @@ static inline void scx_call_op_set_cpumask(struct scx_sched *sch, struct rq *rq,struct task_struct *task, const struct cpumask *cpumask) { - WARN_ON_ONCE(current->scx.kf_tasks[0]); - current->scx.kf_tasks[0 ] = task; -if (rq) -update_locked_rq (rq); - if (scx_is_cid_type()) { struct scx_cmask *kern_va = *this_cpu_ptr(sch->set_cmask_scratch);/* @@ -423,14 +418,11 @@ static inline void scx_call_op_set_cpumask(struct scx_sched *sch, struct rq *rq,* the sole user of the scratch area. */ scx_cpumask_to_cmask (cpumask, kern_va); - sch->ops_cid.set_cmask (task,scx_kaddr_to_arena (sch, kern_va)); +SCX_CALL_CID_OP_TASK (sch, set_cmask, rq, task, +scx_kaddr_to_arena (sch, kern_va)); }else { - sch->ops.set_cpumask (task, cpumask); +SCX_CALL_OP_TASK (sch, set_cpumask, rq, task, cpumask); } - -if (rq) -update_locked_rq (NULL ); - current->scx.kf_tasks[0 ] =NULL ; } enum scx_dsq_iter_flags {kernel/sched/ext/internal.h +20
-3@@ -1751,8 +1751,11 @@ static inline void update_locked_rq(struct rq *rq)/* * SCX ops can recurse via scx_bpf_sub_dispatch() - the inner call must not * clobber the outer's scx_locked_rq_state. Save it on entry, restore on exit. +* +* @ops is the ops table to dispatch through: ops for the cpu form, ops_cid +* for the cid form. */ -#define SCX_CALL_OP(sch, op, locked_rq, args...) \ +#define __SCX_CALL_OP(sch, ops, op, locked_rq, args...) \ do { \ struct rq *__prev_locked_rq; \ \@@ -1765,6 +1768,9 @@ do { \update_locked_rq (__prev_locked_rq); \ } while (0) +#define SCX_CALL_OP(sch, op, locked_rq, args...) \ + __SCX_CALL_OP(sch, ops, op, locked_rq, ##args) + #define SCX_CALL_OP_RET(sch, op, locked_rq, args...) \ ({ \ struct rq *__prev_locked_rq; \@@ -1796,14 +1802,25 @@ do { \* WARN_ON_ONCE() in each macro catches a re-entry of any of the three variants * while a previous one is still in progress. */ -#define SCX_CALL_OP_TASK(sch, op, locked_rq, task, args...) \ +#define __SCX_CALL_OP_TASK(sch, ops, op, locked_rq, task, args...) \ do { \ WARN_ON_ONCE(current->scx.kf_tasks[0]); \ current->scx.kf_tasks[0] = task; \ - SCX_CALL_OP((sch), op, locked_rq, task, ##args); \ + __SCX_CALL_OP((sch), ops, op, locked_rq, task, ##args); \ current->scx.kf_tasks[0] = NULL; \ } while (0) +#define SCX_CALL_OP_TASK(sch, op, locked_rq, task, args...) \ + __SCX_CALL_OP_TASK(sch, ops, op, locked_rq, task, ##args) + +/* +* Dispatch a task op through the cid-form ops_cid table. Only set_cmask() needs +* this: it takes an arena cmask address instead of a cpumask, so it cannot be +* invoked via its cpu-form set_cpumask() slot. +*/ +#define SCX_CALL_CID_OP_TASK(sch, op, locked_rq, task, args...) \ + __SCX_CALL_OP_TASK(sch, ops_cid, op, locked_rq, task, ##args) + #define SCX_CALL_OP_TASK_RET(sch, op, locked_rq, task, args...) \ ({ \ __typeof__((sch)->ops.op(task, ##args)) __ret; \Signature · 1 line
2.54.0
Tejun Heo
reviewerPATCH v4 6/40 sched_ext: Rename extra_enq_flags to remote_activate_enq_flags
In reply to Tejun Heoscx_rq.extra_enq_flags carries scx-specific enqueue flags across the activate_task() boundary during a cross-rq task move in move_remote_task_to_local_dsq(). Rename it to remote_activate_enq_flags to name that role, and fix the stale comment that referenced the old move_task_to_local_dsq() name.
No functional change.
--- kernel/sched/ext/ext.c | 10 +++++----- kernel/sched/sched.h | 3 ++- 2 files changed, 7 insertions(+), 6 deletions(-)
kernel/sched/ext/ext.c +5 -5
@@ -1824,7 +1824,7 @@ static void enqueue_task_scx(struct rq *rq, struct task_struct *p, int core_enq_ {struct scx_sched *sch =scx_task_sched (p);int sticky_cpu = p->scx.sticky_cpu; -u64 enq_flags = core_enq_flags | rq->scx.extra_enq_flags; +u64 enq_flags = core_enq_flags | rq->scx.remote_activate_enq_flags;if (enq_flags &ENQUEUE_WAKEUP ) rq->scx.flags |=SCX_RQ_IN_WAKEUP ;
@@ -2097,13 +2097,13 @@ static void move_remote_task_to_local_dsq(struct task_struct *p, u64 enq_flags,/* * We want to pass scx-specific enq_flags but activate_task() will * truncate the upper 32 bit. As we own @rq, we can pass them through -* @rq->scx.extra_enq_flags instead. +* @rq->scx.remote_activate_enq_flags instead. */ WARN_ON_ONCE (!cpumask_test_cpu (cpu_of (dst_rq), p->cpus_ptr)); -WARN_ON_ONCE (dst_rq->scx.extra_enq_flags); - dst_rq->scx.extra_enq_flags = enq_flags; +WARN_ON_ONCE (dst_rq->scx.remote_activate_enq_flags); + dst_rq->scx.remote_activate_enq_flags = enq_flags;activate_task (dst_rq, p,0 ); - dst_rq->scx.extra_enq_flags =0 ; + dst_rq->scx.remote_activate_enq_flags =0 ; }/*
kernel/sched/sched.h +2 -1
@@ -797,7 +797,8 @@ struct scx_rq {struct list_head runnable_list;/* runnable tasks on this rq */ struct list_head ddsp_deferred_locals;/* deferred ddsps from enq */ unsigned long ops_qseq; -u64 extra_enq_flags;/* see move_task_to_local_dsq() */ +/* see move_remote_task_to_local_dsq() */ +u64 remote_activate_enq_flags;u32 nr_running;u32 cpuperf_target;/* [0, SCHED_CAPACITY_SCALE] */ bool in_select_cpu;
Signature · 1 line
2.54.0
Re: [PATCH v4 sched_ext/for-7.3 06/40] sched_ext: Rename extra_enq_flags to remote_activate_enq_flags
Quoted 9 lines · Tejun Heo
scx_rq.extra_enq_flags carries scx-specific enqueue flags across the
activate_task()boundary during a cross-rq task move inmove_remote_task_to_local_dsq(). Rename it toremote_activate_enq_flagsto name that role, and fix the stale comment that referenced the oldmove_task_to_local_dsq()name.No functional change.
Signed-off-by Tejun Heo <[email protected]>
Thanks,
-Andrea
Quoted 53 lines
--- kernel/sched/ext/ext.c | 10 +++++----- kernel/sched/sched.h | 3 ++- 2 files changed, 7 insertions(+), 6 deletions(-)kernel/sched/ext/ext.c +5
-5@@ -1824,7 +1824,7 @@ static void enqueue_task_scx(struct rq *rq, struct task_struct *p, int core_enq_ {struct scx_sched *sch =scx_task_sched (p);int sticky_cpu = p->scx.sticky_cpu; -u64 enq_flags = core_enq_flags | rq->scx.extra_enq_flags; +u64 enq_flags = core_enq_flags | rq->scx.remote_activate_enq_flags;if (enq_flags &ENQUEUE_WAKEUP ) rq->scx.flags |=SCX_RQ_IN_WAKEUP ;@@ -2097,13 +2097,13 @@ static void move_remote_task_to_local_dsq(struct task_struct *p, u64 enq_flags,/* * We want to pass scx-specific enq_flags but activate_task() will * truncate the upper 32 bit. As we own @rq, we can pass them through -* @rq->scx.extra_enq_flags instead. +* @rq->scx.remote_activate_enq_flags instead. */ WARN_ON_ONCE (!cpumask_test_cpu (cpu_of (dst_rq), p->cpus_ptr)); -WARN_ON_ONCE (dst_rq->scx.extra_enq_flags); - dst_rq->scx.extra_enq_flags = enq_flags; +WARN_ON_ONCE (dst_rq->scx.remote_activate_enq_flags); + dst_rq->scx.remote_activate_enq_flags = enq_flags;activate_task (dst_rq, p,0 ); - dst_rq->scx.extra_enq_flags =0 ; + dst_rq->scx.remote_activate_enq_flags =0 ; }/* kernel/sched/sched.h +2
-1@@ -797,7 +797,8 @@ struct scx_rq {struct list_head runnable_list;/* runnable tasks on this rq */ struct list_head ddsp_deferred_locals;/* deferred ddsps from enq */ unsigned long ops_qseq; -u64 extra_enq_flags;/* see move_task_to_local_dsq() */ +/* see move_remote_task_to_local_dsq() */ +u64 remote_activate_enq_flags;u32 nr_running;u32 cpuperf_target;/* [0, SCHED_CAPACITY_SCALE] */ bool in_select_cpu;Signature · 1 line
2.54.0
Tejun Heo
reviewerPATCH v4 7/40 tools/sched_ext: scx_qmap - Use bare u64/u32/s32 integer types
In reply to Tejun Heoscx_qmap.c and the shared scx_qmap.h mixed __u64/__u32/__s32 with the bare typedefs that scx/common.h provides. Convert the remaining __-prefixed integer types to the bare forms for consistency. The struct fields become bare u64 (uint64_t), so the stats printfs that fed them to %llu now cast to unsigned long long. No functional change.
--- tools/sched_ext/scx_qmap.c | 21 +++++++++++---------- tools/sched_ext/scx_qmap.h | 34 +++++++++++++++++----------------- 2 files changed, 28 insertions(+), 27 deletions(-)
tools/sched_ext/scx_qmap.c +11 -10
@@ -67,8 +67,8 @@ int main(int argc, char **argv)struct scx_qmap *skel;struct bpf_link *link;struct qmap_arena *qa; -__u32 test_error_cnt =0 ; -__u64 ecode; +u32 test_error_cnt =0 ; +u64 ecode;int opt;libbpf_set_print (libbpf_print_fn);
@@ -202,15 +202,16 @@ int main(int argc, char **argv)printf ("stats : enq=%lu dsp=%lu delta=%ld reenq/cid0=%llu/%llu deq=%llu core=%llu enq_ddsp=%llu\n" , nr_enqueued, nr_dispatched, nr_enqueued - nr_dispatched, - qa->nr_reenqueued, qa->nr_reenqueued_cid0, - qa->nr_dequeued, - qa->nr_core_sched_execed, - qa->nr_ddsp_from_enq); + (unsigned long long )qa->nr_reenqueued, + (unsigned long long )qa->nr_reenqueued_cid0, + (unsigned long long )qa->nr_dequeued, + (unsigned long long )qa->nr_core_sched_execed, + (unsigned long long )qa->nr_ddsp_from_enq);printf (" exp_local=%llu exp_remote=%llu exp_timer=%llu exp_lost=%llu\n" , - qa->nr_expedited_local, - qa->nr_expedited_remote, - qa->nr_expedited_from_timer, - qa->nr_expedited_lost); + (unsigned long long )qa->nr_expedited_local, + (unsigned long long )qa->nr_expedited_remote, + (unsigned long long )qa->nr_expedited_from_timer, + (unsigned long long )qa->nr_expedited_lost);if (__COMPAT_has_ksym ("scx_bpf_cidperf_cur" )) printf("cpuperf: cur min/avg/max=%u/%u/%u target min/avg/max=%u/%u/%u\n", qa->cpuperf_min,
tools/sched_ext/scx_qmap.h +17 -17
@@ -28,10 +28,10 @@#define SCX_QMAP_MAX_CPUS 1024struct cpu_ctx { -__u64 dsp_idx;/* dispatch index */ -__u64 dsp_cnt;/* remaining count */ -__u32 avg_weight; -__u32 cpuperf_target; +u64 dsp_idx;/* dispatch index */ +u64 dsp_cnt;/* remaining count */ +u32 avg_weight; +u32 cpuperf_target; };/* Opaque to userspace; defined in scx_qmap.bpf.c. */
@@ -40,25 +40,25 @@ struct task_ctx;struct qmap_fifo { struct task_ctx __arena *head; struct task_ctx __arena *tail; -__s32 idx; +s32 idx; };struct qmap_arena {/* userspace-visible stats */ -__u64 nr_enqueued, nr_dispatched, nr_reenqueued, nr_reenqueued_cid0; -__u64 nr_dequeued, nr_ddsp_from_enq; -__u64 nr_core_sched_execed; -__u64 nr_expedited_local, nr_expedited_remote; -__u64 nr_expedited_lost, nr_expedited_from_timer; -__u64 nr_highpri_queued; -__u32 test_error_cnt; -__u32 cpuperf_min, cpuperf_avg, cpuperf_max; -__u32 cpuperf_target_min, cpuperf_target_avg, cpuperf_target_max; +u64 nr_enqueued, nr_dispatched, nr_reenqueued, nr_reenqueued_cid0; +u64 nr_dequeued, nr_ddsp_from_enq; +u64 nr_core_sched_execed; +u64 nr_expedited_local, nr_expedited_remote; +u64 nr_expedited_lost, nr_expedited_from_timer; +u64 nr_highpri_queued; +u32 test_error_cnt; +u32 cpuperf_min, cpuperf_avg, cpuperf_max; +u32 cpuperf_target_min, cpuperf_target_avg, cpuperf_target_max;/* kernel-side runtime state */ -__u64 sub_sched_cgroup_ids[MAX_SUB_SCHEDS ]; -__u64 core_sched_head_seqs[5 ]; -__u64 core_sched_tail_seqs[5 ]; +u64 sub_sched_cgroup_ids[MAX_SUB_SCHEDS ]; +u64 core_sched_head_seqs[5 ]; +u64 core_sched_tail_seqs[5 ];struct cpu_ctx cpu_ctxs[SCX_QMAP_MAX_CPUS ];
Signature · 1 line
2.54.0
Re: [PATCH v4 sched_ext/for-7.3 07/40] tools/sched_ext: scx_qmap - Use bare u64/u32/s32 integer types
Quoted 7 lines · Tejun Heo
scx_qmap.cand the sharedscx_qmap.hmixed __u64/__u32/__s32 with the bare typedefs thatscx/common.hprovides. Convert the remaining __-prefixed integer types to the bare forms for consistency. The struct fields become bare u64 (uint64_t), so the stats printfs that fed them to %llu now cast to unsigned long long. No functional change.Signed-off-by Tejun Heo <[email protected]>
Thanks,
-Andrea
Quoted 106 lines
--- tools/sched_ext/scx_qmap.c | 21 +++++++++++---------- tools/sched_ext/scx_qmap.h | 34 +++++++++++++++++----------------- 2 files changed, 28 insertions(+), 27 deletions(-)tools/sched_ext/scx_qmap.c +11
-10@@ -67,8 +67,8 @@ int main(int argc, char **argv)struct scx_qmap *skel;struct bpf_link *link;struct qmap_arena *qa; -__u32 test_error_cnt =0 ; -__u64 ecode; +u32 test_error_cnt =0 ; +u64 ecode;int opt;libbpf_set_print (libbpf_print_fn);@@ -202,15 +202,16 @@ int main(int argc, char **argv)printf ("stats : enq=%lu dsp=%lu delta=%ld reenq/cid0=%llu/%llu deq=%llu core=%llu enq_ddsp=%llu\n" , nr_enqueued, nr_dispatched, nr_enqueued - nr_dispatched, - qa->nr_reenqueued, qa->nr_reenqueued_cid0, - qa->nr_dequeued, - qa->nr_core_sched_execed, - qa->nr_ddsp_from_enq); + (unsigned long long )qa->nr_reenqueued, + (unsigned long long )qa->nr_reenqueued_cid0, + (unsigned long long )qa->nr_dequeued, + (unsigned long long )qa->nr_core_sched_execed, + (unsigned long long )qa->nr_ddsp_from_enq);printf (" exp_local=%llu exp_remote=%llu exp_timer=%llu exp_lost=%llu\n" , - qa->nr_expedited_local, - qa->nr_expedited_remote, - qa->nr_expedited_from_timer, - qa->nr_expedited_lost); + (unsigned long long )qa->nr_expedited_local, + (unsigned long long )qa->nr_expedited_remote, + (unsigned long long )qa->nr_expedited_from_timer, + (unsigned long long )qa->nr_expedited_lost);if (__COMPAT_has_ksym ("scx_bpf_cidperf_cur" )) printf("cpuperf: cur min/avg/max=%u/%u/%u target min/avg/max=%u/%u/%u\n", qa->cpuperf_min,tools/sched_ext/scx_qmap.h +17
-17@@ -28,10 +28,10 @@#define SCX_QMAP_MAX_CPUS 1024struct cpu_ctx { -__u64 dsp_idx;/* dispatch index */ -__u64 dsp_cnt;/* remaining count */ -__u32 avg_weight; -__u32 cpuperf_target; +u64 dsp_idx;/* dispatch index */ +u64 dsp_cnt;/* remaining count */ +u32 avg_weight; +u32 cpuperf_target; };/* Opaque to userspace; defined in scx_qmap.bpf.c. */ @@ -40,25 +40,25 @@ struct task_ctx;struct qmap_fifo { struct task_ctx __arena *head; struct task_ctx __arena *tail; -__s32 idx; +s32 idx; };struct qmap_arena {/* userspace-visible stats */ -__u64 nr_enqueued, nr_dispatched, nr_reenqueued, nr_reenqueued_cid0; -__u64 nr_dequeued, nr_ddsp_from_enq; -__u64 nr_core_sched_execed; -__u64 nr_expedited_local, nr_expedited_remote; -__u64 nr_expedited_lost, nr_expedited_from_timer; -__u64 nr_highpri_queued; -__u32 test_error_cnt; -__u32 cpuperf_min, cpuperf_avg, cpuperf_max; -__u32 cpuperf_target_min, cpuperf_target_avg, cpuperf_target_max; +u64 nr_enqueued, nr_dispatched, nr_reenqueued, nr_reenqueued_cid0; +u64 nr_dequeued, nr_ddsp_from_enq; +u64 nr_core_sched_execed; +u64 nr_expedited_local, nr_expedited_remote; +u64 nr_expedited_lost, nr_expedited_from_timer; +u64 nr_highpri_queued; +u32 test_error_cnt; +u32 cpuperf_min, cpuperf_avg, cpuperf_max; +u32 cpuperf_target_min, cpuperf_target_avg, cpuperf_target_max;/* kernel-side runtime state */ -__u64 sub_sched_cgroup_ids[MAX_SUB_SCHEDS ]; -__u64 core_sched_head_seqs[5 ]; -__u64 core_sched_tail_seqs[5 ]; +u64 sub_sched_cgroup_ids[MAX_SUB_SCHEDS ]; +u64 core_sched_head_seqs[5 ]; +u64 core_sched_tail_seqs[5 ];struct cpu_ctx cpu_ctxs[SCX_QMAP_MAX_CPUS ];Signature · 1 line
2.54.0
Tejun Heo
reviewerPATCH v4 8/40 sched_ext: Reject direct slice and dsq_vtime writes for cid-form schedulers
In reply to Tejun HeoDirect writes to p->scx.slice and p->scx.dsq_vtime bypass scx_bpf_task_set_slice/dsq_vtime() and the authority checks they carry. Those checks exist for sub-schedulers, which attach only through the cid-form struct_ops, so the direct writes only need to be closed there.
Give sched_ext_ops_cid its own verifier ops that reject the two fields. cid-form is a new interface with no legacy users, so there is no compatibility to keep. The cpu-form keeps direct writes, and the deprecation warning they carried is dropped.
--- kernel/sched/ext/ext.c | 44 ++++++++++++++++++++++++++++++------------ 1 file changed, 32 insertions(+), 12 deletions(-)
kernel/sched/ext/ext.c +32 -12
@@ -6995,6 +6995,21 @@ static bool bpf_scx_is_valid_access(int off, int size,return btf_ctx_access (off, size, type, prog, info); } +/* common to both forms: only scx.disallow is writable */ +static int bpf_scx_btf_struct_access_common (const struct bpf_reg_state *reg, +int off,int size) +{ +const struct btf_type *t; + + t =btf_type_by_id (reg->btf, reg->btf_id); +if (t == task_struct_type && + off >= offsetof(struct task_struct, scx.disallow) && + off + size <= offsetofend(struct task_struct, scx.disallow)) +return SCALAR_VALUE ; + +return -EACCES ; +} +static int bpf_scx_btf_struct_access (struct bpf_verifier_log *log,const struct bpf_reg_state *reg,int off,int size)
@@ -7003,23 +7018,22 @@ static int bpf_scx_btf_struct_access(struct bpf_verifier_log *log, t =btf_type_by_id (reg->btf, reg->btf_id);if (t == task_struct_type) { -/* -* COMPAT: Will be removed in v6.23. -*/ if ((off >= offsetof(struct task_struct, scx.slice) && off + size <= offsetofend(struct task_struct, scx.slice)) || (off >= offsetof(struct task_struct, scx.dsq_vtime) && - off + size <= offsetofend(struct task_struct, scx.dsq_vtime))) { -pr_warn_ratelimited ("sched_ext: Writing directly to p->scx.slice/dsq_vtime is deprecated, use scx_bpf_task_set_slice/dsq_vtime()\n" ); -return SCALAR_VALUE ; - } - - if (off >= offsetof(struct task_struct, scx.disallow) && - off + size <= offsetofend(struct task_struct, scx.disallow)) + off + size <= offsetofend(struct task_struct, scx.dsq_vtime)))return SCALAR_VALUE ; } -return -EACCES ; +return bpf_scx_btf_struct_access_common (reg, off, size); +} + +/* cid-form rejects direct slice and dsq_vtime writes in favor of the kfuncs */ +static int bpf_scx_cid_btf_struct_access (struct bpf_verifier_log *log, +const struct bpf_reg_state *reg,int off, +int size) +{ +return bpf_scx_btf_struct_access_common (reg, off, size); }static const struct bpf_verifier_ops bpf_scx_verifier_ops = {
@@ -7028,6 +7042,12 @@ static const struct bpf_verifier_ops bpf_scx_verifier_ops = { .btf_struct_access = bpf_scx_btf_struct_access, }; +static const struct bpf_verifier_ops bpf_scx_cid_verifier_ops = { + .get_func_proto = bpf_base_func_proto, + .is_valid_access = bpf_scx_is_valid_access, + .btf_struct_access = bpf_scx_cid_btf_struct_access, +}; +static int bpf_scx_init_member (const struct btf_type *t,const struct btf_member *member,void *kdata,const void *udata)
@@ -7368,7 +7388,7 @@ static struct sched_ext_ops_cid __bpf_ops_sched_ext_ops_cid = {* verified to match by the BUILD_BUG_ON checks in scx_init(). */ static struct bpf_struct_ops bpf_sched_ext_ops_cid = { - .verifier_ops = &bpf_scx_verifier_ops, + .verifier_ops = &bpf_scx_cid_verifier_ops, .reg = bpf_scx_reg_cid, .unreg = bpf_scx_unreg, .check_member = bpf_scx_check_member,
Signature · 1 line
2.54.0
[email protected]
AI reviewRe: [PATCH v4 sched_ext/for-7.3 08/40] sched_ext: Reject direct slice and dsq_vtime writes for cid-form schedulers
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Pre-existing issues:
- [Critical] bpf_scx_btf_struct_access_common permits BPF_WRITE to PTR_TO_BTF_ID | PTR_UNTRUSTED pointers, allowing arbitrary kernel memory writes. --
--- Patch [8]: [PATCH v4 sched_ext/for-7.3 08/40] sched_ext: Reject direct slice and dsq_vtime writes for cid-form schedulers --- commit b5bf42bcb4f4546f3eb981870b33d4470cf01c02
Author: Tejun Heo <[email protected]>
sched_ext: Reject direct slice and dsq_vtime writes for cid-form schedulers
This commit introduces a separate verifier ops struct for cid-form schedulers to reject direct writes to p->scx.slice and p->scx.dsq_vtime. The cpu-form schedulers retain direct writes and drop the previous deprecation warning.
Quoted 9 lines
kernel/sched/ext/ext.c +1
-0@@ -6995,6 +6995,21 @@ static bool bpf_scx_is_valid_access(int off, int size,return btf_ctx_access (off, size, type, prog, info); } +/* common to both forms: only scx.disallow is writable */
kernel/sched/ext/ext.c static bool bpf_scx_is_valid_access(int off, int size, · line 7008
Quoted 10 lines
kernel/sched/ext/ext.c +10
-0+static int bpf_scx_btf_struct_access_common (const struct bpf_reg_state *reg, +int off,int size) +{ +const struct btf_type *t; + + t =btf_type_by_id (reg->btf, reg->btf_id); +if (t == task_struct_type && + off >= offsetof(struct task_struct, scx.disallow) && + off + size <= offsetofend(struct task_struct, scx.disallow)) +return SCALAR_VALUE ;
[Severity: Critical]
This is a pre-existing issue, but does this function permit arbitrary kernel memory writes if the register has the PTR_UNTRUSTED flag set?
If an attacker loads a BPF STRUCT_OPS program and forges an arbitrary address into a PTR_TO_BTF_ID | PTR_UNTRUSTED pointer, they can attempt to write to the scx.disallow field of this pointer. The BPF verifier's check_ptr_to_btf_access unconditionally calls this btf_struct_access callback for BPF_WRITE if the pointer is not program-allocated.
Since this callback returns SCALAR_VALUE without verifying the PTR_UNTRUSTED flag on reg->type, the write is approved. The JIT will then emit a direct BPF_MEM store instruction that silently overwrites the target address because fixups skip BPF_PROBE_MEM conversion for BPF_WRITE on untrusted pointers.
Should we explicitly check for the PTR_UNTRUSTED flag in reg->type here?
if (type_flag(reg->type) & PTR_UNTRUSTED) return -EACCES;
Quoted 7 lines
kernel/sched/ext/ext.c +4
-0+ +return -EACCES ; +} +static int bpf_scx_btf_struct_access (struct bpf_verifier_log *log,const struct bpf_reg_state *reg,int off,int size)
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=8
Tejun Heo
reviewerRe: [PATCH v4 sched_ext/for-7.3 08/40] sched_ext: Reject direct slice and dsq_vtime writes for cid-form schedulers
Quoted 3 lines · On Wed, 08 Jul 2026 21:41:20 +0000, [email protected] wrote:
- [Critical]
bpf_scx_btf_struct_access_commonpermitsBPF_WRITEtoPTR_TO_BTF_ID|PTR_UNTRUSTEDpointers, allowing arbitrary kernel memory writes.
This follows the same btf_struct_access contract as the other implementations (tcp_ca, qdisc, hid): the callback validates the type and confines the write to the one whitelisted field. Whatever the register's trust flags, the store can only land on scx.disallow, a bool which only gates SCHED_EXT admission - not a useful write primitive. No change.
Thanks.
Signature · 1 line
tejun
Re: [PATCH v4 sched_ext/for-7.3 08/40] sched_ext: Reject direct slice and dsq_vtime writes for cid-form schedulers
In reply to Tejun HeoQuoted 11 lines · Tejun Heo
Direct writes to
p->scx.sliceandp->scx.dsq_vtimebypass scx_bpf_task_set_slice/dsq_vtime() and the authority checks they carry. Those checks exist for sub-schedulers, which attach only through the cid-formstruct_ops, so the direct writes only need to be closed there.Give
sched_ext_ops_cidits own verifier ops that reject the two fields. cid-form is a new interface with no legacy users, so there is no compatibility to keep. The cpu-form keeps direct writes, and the deprecation warning they carried is dropped.Signed-off-by Tejun Heo <[email protected]>
The thing reported by sashiko seems more like a BPF issue. From an scx perspective this one looks good to me.
Thanks,
-Andrea
Quoted 89 lines
--- kernel/sched/ext/ext.c | 44 ++++++++++++++++++++++++++++++------------ 1 file changed, 32 insertions(+), 12 deletions(-)kernel/sched/ext/ext.c +32
-12@@ -6995,6 +6995,21 @@ static bool bpf_scx_is_valid_access(int off, int size,return btf_ctx_access (off, size, type, prog, info); } +/* common to both forms: only scx.disallow is writable */ +static int bpf_scx_btf_struct_access_common (const struct bpf_reg_state *reg, +int off,int size) +{ +const struct btf_type *t; + + t =btf_type_by_id (reg->btf, reg->btf_id); +if (t == task_struct_type && + off >= offsetof(struct task_struct, scx.disallow) && + off + size <= offsetofend(struct task_struct, scx.disallow)) +return SCALAR_VALUE ; + +return -EACCES ; +} +static int bpf_scx_btf_struct_access (struct bpf_verifier_log *log,const struct bpf_reg_state *reg,int off,int size)@@ -7003,23 +7018,22 @@ static int bpf_scx_btf_struct_access(struct bpf_verifier_log *log, t =btf_type_by_id (reg->btf, reg->btf_id);if (t == task_struct_type) { -/* -* COMPAT: Will be removed in v6.23. -*/ if ((off >= offsetof(struct task_struct, scx.slice) && off + size <= offsetofend(struct task_struct, scx.slice)) || (off >= offsetof(struct task_struct, scx.dsq_vtime) && - off + size <= offsetofend(struct task_struct, scx.dsq_vtime))) { -pr_warn_ratelimited ("sched_ext: Writing directly to p->scx.slice/dsq_vtime is deprecated, use scx_bpf_task_set_slice/dsq_vtime()\n" ); -return SCALAR_VALUE ; - } - - if (off >= offsetof(struct task_struct, scx.disallow) && - off + size <= offsetofend(struct task_struct, scx.disallow)) + off + size <= offsetofend(struct task_struct, scx.dsq_vtime)))return SCALAR_VALUE ; } -return -EACCES ; +return bpf_scx_btf_struct_access_common (reg, off, size); +} + +/* cid-form rejects direct slice and dsq_vtime writes in favor of the kfuncs */ +static int bpf_scx_cid_btf_struct_access (struct bpf_verifier_log *log, +const struct bpf_reg_state *reg,int off, +int size) +{ +return bpf_scx_btf_struct_access_common (reg, off, size); }static const struct bpf_verifier_ops bpf_scx_verifier_ops = {@@ -7028,6 +7042,12 @@ static const struct bpf_verifier_ops bpf_scx_verifier_ops = { .btf_struct_access = bpf_scx_btf_struct_access, }; +static const struct bpf_verifier_ops bpf_scx_cid_verifier_ops = { + .get_func_proto = bpf_base_func_proto, + .is_valid_access = bpf_scx_is_valid_access, + .btf_struct_access = bpf_scx_cid_btf_struct_access, +}; +static int bpf_scx_init_member (const struct btf_type *t,const struct btf_member *member,void *kdata,const void *udata)@@ -7368,7 +7388,7 @@ static struct sched_ext_ops_cid __bpf_ops_sched_ext_ops_cid = {* verified to match by the BUILD_BUG_ON checks in scx_init(). */ static struct bpf_struct_ops bpf_sched_ext_ops_cid = { - .verifier_ops = &bpf_scx_verifier_ops, + .verifier_ops = &bpf_scx_cid_verifier_ops, .reg = bpf_scx_reg_cid, .unreg = bpf_scx_unreg, .check_member = bpf_scx_check_member,Signature · 1 line
2.54.0
Tejun Heo
reviewerPATCH v4 9/40 sched_ext: Make scx_bpf_kick_cid() return void
In reply to Tejun Heoscx_bpf_kick_cid() returned an error code, but the value conveys nothing actionable and no caller consumes it. The kick is asynchronous, so a successful return only means it was queued. An invalid @cid is already reported through scx_error() by scx_cid_to_cpu(), and a missing scheduler leaves nothing to kick.
Make scx_bpf_kick_cid() return void to match scx_bpf_kick_cpu(). The cid-form kfuncs are not in practical use yet, so the ABI change is safe.
--- kernel/sched/ext/ext.c | 11 +++++------ tools/sched_ext/include/scx/common.bpf.h | 2 +- 2 files changed, 6 insertions(+), 7 deletions(-)
kernel/sched/ext/ext.c +5 -6
@@ -8533,10 +8533,10 @@ __bpf_kfunc void scx_bpf_kick_cpu(s32 cpu, u64 flags, const struct bpf_prog_aux* @flags: %SCX_KICK_* flags * @aux: implicit BPF argument to access bpf_prog_aux hidden from BPF progs * -* cid-addressed equivalent of scx_bpf_kick_cpu(). Return 0 on success, -* -errno otherwise. +* cid-addressed equivalent of scx_bpf_kick_cpu(). An invalid @cid aborts the +* scheduler via scx_cid_to_cpu(). */ -__bpf_kfunc s32 scx_bpf_kick_cid(s32 cid, u64 flags, const struct bpf_prog_aux *aux) +__bpf_kfunc void scx_bpf_kick_cid(s32 cid, u64 flags, const struct bpf_prog_aux *aux) {struct scx_sched *sch;s32 cpu;
@@ -8544,12 +8544,11 @@ __bpf_kfunc s32 scx_bpf_kick_cid(s32 cid, u64 flags, const struct bpf_prog_aux *guard (rcu)(); sch =scx_prog_sched (aux);if (unlikely (!sch)) -return -ENODEV ; +return ; cpu =scx_cid_to_cpu (sch, cid);if (cpu <0 ) -return cpu; +return ;scx_kick_cpu (sch, cpu, flags); -return 0 ; }/**
tools/sched_ext/include/scx/common.bpf.h +1 -1
@@ -104,7 +104,7 @@ void scx_bpf_events(struct scx_event_stats *events, size_t events__sz) __ksym __ s32 scx_bpf_cpu_to_cid(s32 cpu) __ksym __weak; s32 scx_bpf_cid_to_cpu(s32 cid) __ksym __weak; void scx_bpf_cid_topo(s32 cid, struct scx_cid_topo *out) __ksym __weak; -s32 scx_bpf_kick_cid(s32 cid, u64 flags) __ksym __weak; +void scx_bpf_kick_cid(s32 cid, u64 flags) __ksym __weak; s32 scx_bpf_task_cid(const struct task_struct *p) __ksym __weak; s32 scx_bpf_this_cid(void) __ksym __weak;struct task_struct *scx_bpf_cid_curr (s32 cid) __ksym__weak ;
Signature · 1 line
2.54.0
Re: [PATCH v4 sched_ext/for-7.3 09/40] sched_ext: Make scx_bpf_kick_cid() return void
Quoted 10 lines · Tejun Heo
scx_bpf_kick_cid()returned an error code, but the value conveys nothing actionable and no caller consumes it. The kick is asynchronous, so a successful return only means it was queued. An invalid @cid is already reported throughscx_error()byscx_cid_to_cpu(), and a missing scheduler leaves nothing to kick.Make
scx_bpf_kick_cid()return void to matchscx_bpf_kick_cpu(). The cid-form kfuncs are not in practical use yet, so the ABI change is safe.Signed-off-by Tejun Heo <[email protected]>
Thanks,
-Andrea
Quoted 54 lines
--- kernel/sched/ext/ext.c | 11 +++++------ tools/sched_ext/include/scx/common.bpf.h | 2 +- 2 files changed, 6 insertions(+), 7 deletions(-)kernel/sched/ext/ext.c +5
-6@@ -8533,10 +8533,10 @@ __bpf_kfunc void scx_bpf_kick_cpu(s32 cpu, u64 flags, const struct bpf_prog_aux* @flags: %SCX_KICK_* flags * @aux: implicit BPF argument to access bpf_prog_aux hidden from BPF progs * -* cid-addressed equivalent of scx_bpf_kick_cpu(). Return 0 on success, -* -errno otherwise. +* cid-addressed equivalent of scx_bpf_kick_cpu(). An invalid @cid aborts the +* scheduler via scx_cid_to_cpu(). */ -__bpf_kfunc s32 scx_bpf_kick_cid(s32 cid, u64 flags, const struct bpf_prog_aux *aux) +__bpf_kfunc void scx_bpf_kick_cid(s32 cid, u64 flags, const struct bpf_prog_aux *aux) {struct scx_sched *sch;s32 cpu;@@ -8544,12 +8544,11 @@ __bpf_kfunc s32 scx_bpf_kick_cid(s32 cid, u64 flags, const struct bpf_prog_aux *guard (rcu)(); sch =scx_prog_sched (aux);if (unlikely (!sch)) -return -ENODEV ; +return ; cpu =scx_cid_to_cpu (sch, cid);if (cpu <0 ) -return cpu; +return ;scx_kick_cpu (sch, cpu, flags); -return 0 ; }/** tools/sched_ext/include/scx/common.bpf.h +1
-1@@ -104,7 +104,7 @@ void scx_bpf_events(struct scx_event_stats *events, size_t events__sz) __ksym __ s32 scx_bpf_cpu_to_cid(s32 cpu) __ksym __weak; s32 scx_bpf_cid_to_cpu(s32 cid) __ksym __weak; void scx_bpf_cid_topo(s32 cid, struct scx_cid_topo *out) __ksym __weak; -s32 scx_bpf_kick_cid(s32 cid, u64 flags) __ksym __weak; +void scx_bpf_kick_cid(s32 cid, u64 flags) __ksym __weak; s32 scx_bpf_task_cid(const struct task_struct *p) __ksym __weak; s32 scx_bpf_this_cid(void) __ksym __weak;struct task_struct *scx_bpf_cid_curr (s32 cid) __ksym__weak ;Signature · 1 line
2.54.0
Tejun Heo
reviewerPATCH v4 10/40 sched_ext: Make the kick machinery per-sched
In reply to Tejun HeoThe kick machinery kept its targets in rq->scx shared by every sched on the cpu. A preempt kick carried no record of which scheduler requested it.
A later patch needs preempt kicks scoped to the requesting scheduler so a sub-scheduler can preempt only tasks in its own subtree. Move the kick masks into the per-sched per-cpu scx_sched_pcpu and have scx_kick_cpu() link the sched onto a per-cpu list (rq->scx.sched_pcpus_to_kick). The cpu's single kick irq_work walks that list and kicks each sched's targets on its behalf, so a kick stays attributed to its scheduler.
The SCX_KICK_WAIT sync set (cpus_to_sync, the kick_sync snapshot and the balance-callback trigger) stays in rq->scx: the waiter is the cpu, not the scheduler, and its only writers, the kick irq_work and the wait balance callback, are cpu-local.
On disable, free_kick_syncs() flushes each cpu's pending kick irq_work before clearing @ksyncs, so a late kick unlinks its to_kick_node instead of early-returning on a NULL @ksyncs and leaving the node linked at free.
v3: Flush the kick irq_work in free_kick_syncs() before clearing @ksyncs. (sashiko AI)
v2: Warn once per sched on scx_bpf_kick_cpu() from NMI.
--- kernel/sched/ext/ext.c | 121 +++++++++++++++++++++++++++--------- kernel/sched/ext/internal.h | 14 +++++ kernel/sched/sched.h | 6 +- 3 files changed, 106 insertions(+), 35 deletions(-)
kernel/sched/ext/ext.c +90 -31
@@ -4624,6 +4624,17 @@ static void scx_sched_free_rcu_work(struct work_struct *work)*/ WARN_ON_ONCE (!list_empty (&pcpu->deferred_reenq_local.node)); +/* +* Bypass blocks new kicks. Flush the kick irq_work so this +* pcpu's to_kick_node is off the list before it is freed. +*/ +irq_work_sync (&cpu_rq (cpu)->scx.kick_cpus_irq_work); +WARN_ON_ONCE (!list_empty (&pcpu->to_kick_node)); +free_cpumask_var (pcpu->cpus_to_kick); +free_cpumask_var (pcpu->cpus_to_kick_if_idle); +free_cpumask_var (pcpu->cpus_to_preempt); +free_cpumask_var (pcpu->cpus_to_wait); +exit_dsq (scx_bypass_dsq (sch, cpu)); }
@@ -5416,6 +5427,8 @@ static void free_kick_syncs(void)struct scx_kick_syncs __rcu **ksyncs =per_cpu_ptr (&scx_kick_syncs, cpu);struct scx_kick_syncs *to_free; +/* flush the pending kick before freeing @ksyncs */ +irq_work_sync (&cpu_rq (cpu)->scx.kick_cpus_irq_work); to_free =rcu_replace_pointer (*ksyncs,NULL , true);if (to_free)kvfree_rcu (to_free, rcu);
@@ -5967,6 +5980,7 @@ static void scx_dump_cpu(struct scx_sched *sch, struct seq_buf *s, bool dump_all_tasks) {struct rq *rq =cpu_rq (cpu); +struct scx_sched_pcpu *pcpu =per_cpu_ptr (sch->pcpu, cpu);struct rq_flags rf;struct task_struct *p;struct seq_buf ns;
@@ -5999,18 +6013,18 @@ static void scx_dump_cpu(struct scx_sched *sch, struct seq_buf *s,dump_line (&ns," curr=%s[%d] class=%ps" , rq->curr->comm, rq->curr->pid, rq->curr->sched_class); -if (!cpumask_empty (rq->scx.cpus_to_kick)) +if (!cpumask_empty (pcpu->cpus_to_kick))dump_line (&ns," cpus_to_kick : %*pb" , -cpumask_pr_args (rq->scx.cpus_to_kick)); -if (!cpumask_empty (rq->scx.cpus_to_kick_if_idle)) +cpumask_pr_args (pcpu->cpus_to_kick)); +if (!cpumask_empty (pcpu->cpus_to_kick_if_idle))dump_line (&ns," idle_to_kick : %*pb" , -cpumask_pr_args (rq->scx.cpus_to_kick_if_idle)); -if (!cpumask_empty (rq->scx.cpus_to_preempt)) +cpumask_pr_args (pcpu->cpus_to_kick_if_idle)); +if (!cpumask_empty (pcpu->cpus_to_preempt))dump_line (&ns," cpus_to_preempt: %*pb" , -cpumask_pr_args (rq->scx.cpus_to_preempt)); -if (!cpumask_empty (rq->scx.cpus_to_wait)) +cpumask_pr_args (pcpu->cpus_to_preempt)); +if (!cpumask_empty (pcpu->cpus_to_wait))dump_line (&ns," cpus_to_wait : %*pb" , -cpumask_pr_args (rq->scx.cpus_to_wait)); +cpumask_pr_args (pcpu->cpus_to_wait));if (!cpumask_empty (rq->scx.cpus_to_sync))dump_line (&ns," cpus_to_sync : %*pb" ,cpumask_pr_args (rq->scx.cpus_to_sync));
@@ -6302,8 +6316,17 @@ struct scx_sched *scx_alloc_and_add_sched(struct scx_enable_cmd *cmd,for_each_possible_cpu (cpu) {struct scx_sched_pcpu *pcpu =per_cpu_ptr (sch->pcpu, cpu); + node =cpu_to_node (cpu); pcpu->sch = sch;INIT_LIST_HEAD (&pcpu->deferred_reenq_local.node); +INIT_LIST_HEAD (&pcpu->to_kick_node); +if (!zalloc_cpumask_var_node (&pcpu->cpus_to_kick,GFP_KERNEL , node) || + !zalloc_cpumask_var_node (&pcpu->cpus_to_kick_if_idle,GFP_KERNEL , node) || + !zalloc_cpumask_var_node (&pcpu->cpus_to_preempt,GFP_KERNEL , node) || + !zalloc_cpumask_var_node (&pcpu->cpus_to_wait,GFP_KERNEL , node)) { + ret = -ENOMEM ; + goto err_free_pcpu; + } } sch->helper =kthread_run_worker (0 ,"sched_ext_helper" );
@@ -6448,6 +6471,14 @@ struct scx_sched *scx_alloc_and_add_sched(struct scx_enable_cmd *cmd, err_stop_helper:kthread_destroy_worker (sch->helper); err_free_pcpu: +for_each_possible_cpu (cpu) { +struct scx_sched_pcpu *pcpu =per_cpu_ptr (sch->pcpu, cpu); + +free_cpumask_var (pcpu->cpus_to_kick); +free_cpumask_var (pcpu->cpus_to_kick_if_idle); +free_cpumask_var (pcpu->cpus_to_preempt); +free_cpumask_var (pcpu->cpus_to_wait); + }for_each_possible_cpu (cpu) {if (cpu == bypass_fail_cpu)break ;
@@ -7461,7 +7492,8 @@ static bool can_skip_idle_kick(struct rq *rq)return !is_idle_task (rq->curr) && !(rq->scx.flags &SCX_RQ_IN_BALANCE ); } -static bool kick_one_cpu (s32 cpu,struct rq *this_rq,unsigned long *ksyncs) +static bool kick_one_cpu (s32 cpu,struct scx_sched_pcpu *pcpu,struct rq *this_rq, +unsigned long *ksyncs) {struct rq *rq =cpu_rq (cpu);struct scx_rq *this_scx = &this_rq->scx;
@@ -7480,25 +7512,25 @@ static bool kick_one_cpu(s32 cpu, struct rq *this_rq, unsigned long *ksyncs)*/ if ((cpu_online (cpu) || cpu ==cpu_of (this_rq)) && !sched_class_above (cur_class, &ext_sched_class)) { -if (cpumask_test_cpu (cpu, this_scx->cpus_to_preempt)) { +if (cpumask_test_cpu (cpu, pcpu->cpus_to_preempt)) {if (cur_class == &ext_sched_class) rq->curr->scx.slice =0 ; -cpumask_clear_cpu (cpu, this_scx->cpus_to_preempt); +cpumask_clear_cpu (cpu, pcpu->cpus_to_preempt); } -if (cpumask_test_cpu (cpu, this_scx->cpus_to_wait)) { +if (cpumask_test_cpu (cpu, pcpu->cpus_to_wait)) {if (cur_class == &ext_sched_class) {cpumask_set_cpu (cpu, this_scx->cpus_to_sync); ksyncs[cpu] = rq->scx.kick_sync; should_wait = true; } -cpumask_clear_cpu (cpu, this_scx->cpus_to_wait); +cpumask_clear_cpu (cpu, pcpu->cpus_to_wait); }resched_curr (rq); }else { -cpumask_clear_cpu (cpu, this_scx->cpus_to_preempt); -cpumask_clear_cpu (cpu, this_scx->cpus_to_wait); +cpumask_clear_cpu (cpu, pcpu->cpus_to_preempt); +cpumask_clear_cpu (cpu, pcpu->cpus_to_wait); }raw_spin_rq_unlock_irqrestore (rq, flags);
@@ -7525,6 +7557,7 @@ static void kick_cpus_irq_workfn(struct irq_work *irq_work)struct rq *this_rq =this_rq ();struct scx_rq *this_scx = &this_rq->scx;struct scx_kick_syncs __rcu *ksyncs_pcpu =__this_cpu_read (scx_kick_syncs); +struct scx_sched_pcpu *pcpu, *tmp;bool should_wait = false;unsigned long *ksyncs;s32 cpu;
@@ -7535,15 +7568,24 @@ static void kick_cpus_irq_workfn(struct irq_work *irq_work) ksyncs =rcu_dereference_bh (ksyncs_pcpu)->syncs; -for_each_cpu (cpu, this_scx->cpus_to_kick) { - should_wait |=kick_one_cpu (cpu, this_rq, ksyncs); -cpumask_clear_cpu (cpu, this_scx->cpus_to_kick); -cpumask_clear_cpu (cpu, this_scx->cpus_to_kick_if_idle); - } +/* +* Walk scheds with pending kicks on this cpu. scx_kick_cpu() adds to +* the list under local_irq_save() and only this irq_work consumes it. +* A plain list without locking is sufficient. +*/ +list_for_each_entry_safe (pcpu, tmp, &this_scx->sched_pcpus_to_kick, to_kick_node) { +list_del_init (&pcpu->to_kick_node); + +for_each_cpu (cpu, pcpu->cpus_to_kick) { + should_wait |=kick_one_cpu (cpu, pcpu, this_rq, ksyncs); +cpumask_clear_cpu (cpu, pcpu->cpus_to_kick); +cpumask_clear_cpu (cpu, pcpu->cpus_to_kick_if_idle); + } -for_each_cpu (cpu, this_scx->cpus_to_kick_if_idle) { -kick_one_cpu_if_idle (cpu, this_rq); -cpumask_clear_cpu (cpu, this_scx->cpus_to_kick_if_idle); +for_each_cpu (cpu, pcpu->cpus_to_kick_if_idle) { +kick_one_cpu_if_idle (cpu, this_rq); +cpumask_clear_cpu (cpu, pcpu->cpus_to_kick_if_idle); + } }/*
@@ -7668,11 +7710,8 @@ void __init init_sched_ext_class(void)INIT_LIST_HEAD (&rq->scx.runnable_list);INIT_LIST_HEAD (&rq->scx.ddsp_deferred_locals); -BUG_ON (!zalloc_cpumask_var_node (&rq->scx.cpus_to_kick,GFP_KERNEL , n)); -BUG_ON (!zalloc_cpumask_var_node (&rq->scx.cpus_to_kick_if_idle,GFP_KERNEL , n)); -BUG_ON (!zalloc_cpumask_var_node (&rq->scx.cpus_to_preempt,GFP_KERNEL , n)); -BUG_ON (!zalloc_cpumask_var_node (&rq->scx.cpus_to_wait,GFP_KERNEL , n));BUG_ON (!zalloc_cpumask_var_node (&rq->scx.cpus_to_sync,GFP_KERNEL , n)); +INIT_LIST_HEAD (&rq->scx.sched_pcpus_to_kick);raw_spin_lock_init (&rq->scx.deferred_reenq_lock);INIT_LIST_HEAD (&rq->scx.deferred_reenq_locals);INIT_LIST_HEAD (&rq->scx.deferred_reenq_users);
@@ -8458,12 +8497,27 @@ __bpf_kfunc bool scx_bpf_task_set_dsq_vtime(struct task_struct *p, u64 vtime,void scx_kick_cpu (struct scx_sched *sch,s32 cpu,u64 flags) { +struct scx_sched_pcpu *pcpu;struct rq *this_rq;unsigned long irq_flags; +/* +* The per-cpu kick list is guarded only by local_irq_save(), which does +* not mask NMIs, so kicking from NMI could corrupt it and is unsupported. +*/ +if (unlikely (in_nmi ())) { +if (!sch->warned_nmi_kick) { + sch->warned_nmi_kick = true; +pr_warn ("sched_ext: %s: scx_bpf_kick_cpu() from NMI ignored\n" , + sch->ops.name); + } +return ; + } +local_irq_save (irq_flags); this_rq =this_rq (); + pcpu =this_cpu_ptr (sch->pcpu);/* * While bypassing for PM ops, IRQ handling may not be online which can
@@ -8477,6 +8531,9 @@ void scx_kick_cpu(struct scx_sched *sch, s32 cpu, u64 flags)* Actual kicking is bounced to kick_cpus_irq_workfn() to avoid nesting * rq locks. We can probably be smarter and avoid bouncing if called * from ops which don't hold a rq lock. +* +* The kick masks are owned by @sch->pcpu, so that a preempt kick can be +* attributed to @sch. */ if (flags &SCX_KICK_IDLE ) {struct rq *target_rq =cpu_rq (cpu);
@@ -8491,16 +8548,18 @@ void scx_kick_cpu(struct scx_sched *sch, s32 cpu, u64 flags) }raw_spin_rq_unlock (target_rq); } -cpumask_set_cpu (cpu, this_rq->scx.cpus_to_kick_if_idle); +cpumask_set_cpu (cpu, pcpu->cpus_to_kick_if_idle); }else { -cpumask_set_cpu (cpu, this_rq->scx.cpus_to_kick); +cpumask_set_cpu (cpu, pcpu->cpus_to_kick);if (flags &SCX_KICK_PREEMPT ) -cpumask_set_cpu (cpu, this_rq->scx.cpus_to_preempt); +cpumask_set_cpu (cpu, pcpu->cpus_to_preempt);if (flags &SCX_KICK_WAIT ) -cpumask_set_cpu (cpu, this_rq->scx.cpus_to_wait); +cpumask_set_cpu (cpu, pcpu->cpus_to_wait); } +if (list_empty (&pcpu->to_kick_node)) +list_add_tail (&pcpu->to_kick_node, &this_rq->scx.sched_pcpus_to_kick);irq_work_queue (&this_rq->scx.kick_cpus_irq_work); out:local_irq_restore (irq_flags);
kernel/sched/ext/internal.h +14 -0
@@ -1126,6 +1126,19 @@ struct scx_sched_pcpu {struct scx_sched *sch;u64 flags;/* protected by rq lock */ +/* +* Kick state owned by this cpu for this sched. scx_kick_cpu() records +* targets here and links @to_kick_node onto the cpu's +* rq->scx.sched_pcpus_to_kick. The cpu's single kick irq_work walks +* that list and kicks each sched's targets on its behalf. Per-sched so +* a kick stays attributed to its scheduler. +*/ +cpumask_var_t cpus_to_kick; +cpumask_var_t cpus_to_kick_if_idle; +cpumask_var_t cpus_to_preempt; +cpumask_var_t cpus_to_wait; +struct list_head to_kick_node; +/* * The event counters are in a per-CPU variable to minimize the * accounting overhead. A system-wide view on the event counter is
@@ -1215,6 +1228,7 @@ struct scx_sched {*/ bool warned_zero_slice:1; bool warned_unassoc_progs:1; + bool warned_nmi_kick:1;struct list_head all;
kernel/sched/sched.h +2 -4
@@ -806,14 +806,12 @@ struct scx_rq {u32 flags;u32 nr_immed;/* ENQ_IMMED tasks on local_dsq */ u64 clock;/* current per-rq clock -- see scx_bpf_now() */ -cpumask_var_t cpus_to_kick; -cpumask_var_t cpus_to_kick_if_idle; -cpumask_var_t cpus_to_preempt; -cpumask_var_t cpus_to_wait;cpumask_var_t cpus_to_sync;bool kick_sync_pending;unsigned long kick_sync; +struct list_head sched_pcpus_to_kick;/* see kick_cpus_irq_workfn() */ +struct task_struct *sub_dispatch_prev;raw_spinlock_t deferred_reenq_lock;
Signature · 1 line
2.54.0
Tejun Heo
reviewerPATCH v4 11/40 sched_ext: Add ops.init_cids() to finalize the cid layout before init
In reply to Tejun HeoA cid-form scheduler that calls scx_bpf_cid_override() to install a custom cid layout can only do so from ops.init(). Enable-path setup that depends on the cid layout thus has to run after ops.init(), and ops.init() itself can't use anything derived from the final layout, which turned out to be too restrictive.
Add an ops.init_cids() callback dedicated to finalizing the cid layout. It runs before the rest of the enable-path setup, so the final layout is in effect for everything that follows including ops.init(), which now runs after the arena pool and cmask scratch allocations.
scx_bpf_cid_override() is restricted to ops.init_cids() at load time. It sits in a kfunc set gated by SCX_KF_ALLOW_INIT_CIDS, a flag set only on the init_cids op, so the verifier rejects a call from any other context. The runtime root-only check is dropped as ops.init_cids() only runs during root enable.
The qmap demo moves its override into a dedicated qmap_init_cids() and, while at it, introduces an enum for the cid override modes instead of hard-coded integers.
--- kernel/sched/ext/cid.c | 25 ++++++++----------- kernel/sched/ext/cid.h | 2 +- kernel/sched/ext/ext.c | 41 +++++++++++++++++++++++-------- kernel/sched/ext/internal.h | 16 ++++++++++--- tools/sched_ext/scx_qmap.bpf.c | 44 +++++++++++++++++++++------------- tools/sched_ext/scx_qmap.c | 12 +++++----- tools/sched_ext/scx_qmap.h | 8 +++++++ 7 files changed, 96 insertions(+), 52 deletions(-)
kernel/sched/ext/cid.c +10 -15
@@ -13,10 +13,10 @@/* * cid tables. * -* Pointers are published once on first enable and never revoked. The default -* mapping is populated before ops.init() runs; scx_bpf_cid_override() commits -* before it returns. As long as the BPF scheduler only uses the tables from -* those points onward, it sees a consistent view. +* Pointers are allocated on first enable and never freed. During root enable, +* the default mapping is populated and then ops.init_cids() is called which can +* use scx_bpf_cid_override() to change the mapping. The mapping stays stable +* until the root is disabled. */ s16 *scx_cid_to_cpu_tbl;s16 *scx_cpu_to_cid_tbl;
@@ -282,7 +282,7 @@ __bpf_kfunc_start_defs();* @cpu_to_cid__sz: must be nr_cpu_ids * sizeof(s32) bytes * @aux: implicit BPF argument to access bpf_prog_aux hidden from BPF progs * -* May only be called from ops.init() of the root scheduler. Replace the +* May only be called from ops.init_cids() of the root scheduler. Replace the * topology-probed cid mapping with the caller-provided one. Each possible cpu * must map to a unique cid in [0, num_possible_cpus()). Topo info is cleared. * On invalid input, trigger scx_error() to abort the scheduler.
@@ -309,11 +309,6 @@ __bpf_kfunc void scx_bpf_cid_override(const s32 *cpu_to_cid, u32 cpu_to_cid__sz,return ; } -if (scx_parent (sch)) { -scx_error (sch,"scx_bpf_cid_override() only allowed from root sched" ); -return ; - } -if (cpu_to_cid__sz != nr_cpu_ids *sizeof (s32)) {scx_error (sch,"scx_bpf_cid_override: expected %zu bytes, got %u" , nr_cpu_ids *sizeof (s32), cpu_to_cid__sz);
@@ -645,13 +640,13 @@ __bpf_kfunc void scx_bpf_cid_topo(s32 cid, struct scx_cid_topo *out__uninit,__bpf_kfunc_end_defs (); -BTF_KFUNCS_START(scx_kfunc_ids_init) +BTF_KFUNCS_START(scx_kfunc_ids_init_cids) BTF_ID_FLAGS(func, scx_bpf_cid_override, KF_IMPLICIT_ARGS | KF_SLEEPABLE) -BTF_KFUNCS_END(scx_kfunc_ids_init) +BTF_KFUNCS_END(scx_kfunc_ids_init_cids) -static const struct btf_kfunc_id_set scx_kfunc_set_init = { +static const struct btf_kfunc_id_set scx_kfunc_set_init_cids = { .owner = THIS_MODULE, - .set = &scx_kfunc_ids_init, + .set = &scx_kfunc_ids_init_cids, .filter = scx_kfunc_context_filter, };
@@ -668,7 +663,7 @@ static const struct btf_kfunc_id_set scx_kfunc_set_cid = {int scx_cid_kfunc_init (void ) { -return register_btf_kfunc_id_set (BPF_PROG_TYPE_STRUCT_OPS , &scx_kfunc_set_init) ?: +return register_btf_kfunc_id_set (BPF_PROG_TYPE_STRUCT_OPS , &scx_kfunc_set_init_cids) ?:register_btf_kfunc_id_set (BPF_PROG_TYPE_STRUCT_OPS , &scx_kfunc_set_cid) ?:register_btf_kfunc_id_set (BPF_PROG_TYPE_TRACING , &scx_kfunc_set_cid) ?:register_btf_kfunc_id_set (BPF_PROG_TYPE_SYSCALL , &scx_kfunc_set_cid);
kernel/sched/ext/cid.h +1 -1
@@ -51,7 +51,7 @@ struct scx_sched;extern s16 *scx_cid_to_cpu_tbl;extern s16 *scx_cpu_to_cid_tbl;extern struct scx_cid_topo *scx_cid_topo; -extern struct btf_id_set8 scx_kfunc_ids_init; +extern struct btf_id_set8 scx_kfunc_ids_init_cids;void scx_cmask_clear (struct scx_cmask *m);void scx_cmask_fill (struct scx_cmask *m);
kernel/sched/ext/ext.c +31 -10
117 lines not on this page · raw message
kernel/sched/ext/internal.h +13 -3
@@ -780,9 +780,18 @@ struct sched_ext_ops {void (*cpu_offline)(s32 cpu);/* -* All CPU hotplug ops must come before ops.init(). +* All CPU hotplug ops must come before ops.init_cids(). */ +/** +* @init_cids: Finalize the cid layout (cid-form only) +* +* Runs after the default cid layout is built, before ops.init(). A +* cid-form scheduler may call scx_bpf_cid_override() here for a custom +* layout. Ignored for cpu-form schedulers. +*/ +s32 (*init_cids)(void); +/** * @init: Initialize the BPF scheduler */
@@ -958,6 +967,7 @@ struct sched_ext_ops_cid {void (*sub_detach)(struct scx_sub_detach_args *args);void (*cid_online)(s32 cid);void (*cid_offline)(s32 cid); +s32 (*init_cids)(void);s32 (*init)(void);void (*exit)(struct scx_exit_info *info);
@@ -982,8 +992,8 @@ enum scx_opi {SCX_OPI_NORMAL_BEGIN =0 ,SCX_OPI_NORMAL_END =SCX_OP_IDX (cpu_online),SCX_OPI_CPU_HOTPLUG_BEGIN =SCX_OP_IDX (cpu_online), -SCX_OPI_CPU_HOTPLUG_END =SCX_OP_IDX (init), - SCX_OPI_END = SCX_OP_IDX(init), +SCX_OPI_CPU_HOTPLUG_END =SCX_OP_IDX (init_cids), + SCX_OPI_END = SCX_OP_IDX(init_cids), };/*
tools/sched_ext/scx_qmap.bpf.c +27 -17
68 lines not on this page · raw message
tools/sched_ext/scx_qmap.c +6 -6
@@ -157,11 +157,11 @@ int main(int argc, char **argv)u32 mode, i;if (!strcmp (optarg,"shuffle" )) - mode =1 ; + mode =QMAP_CID_OVR_SHUFFLE ;else if (!strcmp (optarg,"bad-dup" )) - mode =2 ; + mode =QMAP_CID_OVR_BAD_DUP ;else if (!strcmp (optarg,"bad-range" )) - mode =3 ; + mode =QMAP_CID_OVR_BAD_RANGE ;else {fprintf (stderr,"unknown cid-override mode '%s'\n" , optarg);return 1 ;
@@ -170,14 +170,14 @@ int main(int argc, char **argv)/* shuffle: reversed cpu_to_cid, bad-dup: dup cid 0, bad-range: identity */ for (i =0 ; i < nr_cpus; i++) { -if (mode ==1 ) +if (mode ==QMAP_CID_OVR_SHUFFLE ) skel->bss->cid_override_cpu_to_cid[i] = nr_cpus -1 - i;else skel->bss->cid_override_cpu_to_cid[i] = i; } -if (mode ==2 && nr_cpus >=2 ) +if (mode ==QMAP_CID_OVR_BAD_DUP && nr_cpus >=2 ) skel->bss->cid_override_cpu_to_cid[1 ] =0 ; -if (mode ==3 ) +if (mode ==QMAP_CID_OVR_BAD_RANGE ) skel->bss->cid_override_cpu_to_cid[0 ] = (s32 )nr_cpus;break ; }
tools/sched_ext/scx_qmap.h +8 -0
@@ -27,6 +27,14 @@*/ #define SCX_QMAP_MAX_CPUS 1024 +/* -C cid-override test modes. Selects cid_override_mode in scx_qmap.bpf.c. */ +enum qmap_cid_override { +QMAP_CID_OVR_OFF =0 ,/* disabled */ +QMAP_CID_OVR_SHUFFLE =1 ,/* valid reversed cpu->cid mapping */ +QMAP_CID_OVR_BAD_DUP =2 ,/* invalid: duplicate cid assignment */ +QMAP_CID_OVR_BAD_RANGE =3 ,/* invalid: out-of-range cid */ +}; +struct cpu_ctx {u64 dsp_idx;/* dispatch index */ u64 dsp_cnt;/* remaining count */
Signature · 1 line
2.54.0
[email protected]
AI reviewRe: [PATCH v4 sched_ext/for-7.3 11/40] sched_ext: Add ops.init_cids() to finalize the cid layout before init
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Pre-existing issues:
- [High] scx_bpf_cid_override() partially mutates global CID topology mapping tables on failure, leaving them in a corrupted state. --
--- Patch [11]: [PATCH v4 sched_ext/for-7.3 11/40] sched_ext: Add ops.init_cids() to finalize the cid layout before init --- commit 0ba160b3b7201c0388f756d1be42ba4c9fc9a45c
Author: Tejun Heo <[email protected]>
sched_ext: Add ops.init_cids() to finalize the cid layout before init
This commit adds a dedicated ops.init_cids() callback for cid-form schedulers to finalize their cid layout using scx_bpf_cid_override(). This callback runs before the main ops.init() setup to ensure a stable layout for subsequent initialization steps.
Quoted 4 lines
kernel/sched/ext/cid.c +0
-0
[ ... ]
kernel/sched/ext/cid.c __bpf_kfunc void scx_bpf_cid_override(const s32 *cpu_to_cid, u32 cpu_to_cid__sz, · line 312
Quoted 10 lines
kernel/sched/ext/cid.c +0
-5@@ -309,11 +309,6 @@ __bpf_kfunc void scx_bpf_cid_override(const s32 *cpu_to_cid, u32 cpu_to_cid__sz,return ; } -if (scx_parent (sch)) { -scx_error (sch,"scx_bpf_cid_override() only allowed from root sched" ); -return ; - } -if (cpu_to_cid__sz != nr_cpu_ids *sizeof (s32)) {
[Severity: High]
This is a pre-existing issue, but can this leave the global CID topology mapping tables in a corrupted state on failure?
Looking at scx_bpf_cid_override() just below this diff, the loop updates the global tables before fully validating the input:
kernel/sched/ext/cid.c:scx_bpf_cid_override() {
...
for_each_possible_cpu(cpu) {
s32 c = cpu_to_cid[cpu];
if (!cid_valid(sch, c))
return;
if (cpumask_test_and_set_cpu(c, seen)) {
scx_error(sch, "cid %d assigned to multiple cpus", c);
return;
}
scx_cpu_to_cid_tbl[cpu] = c;
scx_cid_to_cpu_tbl[c] = cpu;
}
...
}If it hits an invalid or duplicate CID mid-way, it calls scx_error() and returns void. The calling BPF program cannot detect this failure and returns 0 (success), allowing the initialization sequence to proceed using the partially corrupted CID mapping.
Since the asynchronous teardown work scheduled by scx_error() is blocked behind scx_enable_mutex during initialization, does this allow the system to fully transition tasks to the BPF scheduler using the corrupted state before it can be disabled?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=11
Tejun Heo
reviewerRe: [PATCH v4 sched_ext/for-7.3 11/40] sched_ext: Add ops.init_cids() to finalize the cid layout before init
Quoted 2 lines · On Wed, 08 Jul 2026 21:54:31 +0000, [email protected] wrote:
- [High]
scx_bpf_cid_override()partially mutates global CID topology mapping tables on failure, leaving them in a corrupted state.
Not in the current form: the kfunc validates the entire snapshot first so that invalid input leaves the globals untouched, and no abort path exists once the write loop begins, so a half-written mapping can't occur. It's also only callable from init_cids, before any scheduling reader exists. No change.
Thanks.
Signature · 1 line
tejun
Tejun Heo
reviewerPATCH v4 12/40 sched_ext: Add CID sharding
In reply to Tejun HeoSub-sched operations need a scalable locking / work domain smaller than the whole cid space. Carve the cid space into topology-respecting shards: each shard is a contiguous cid range that stays within one LLC, and LLCs larger than the per-shard cap (default 24 cids, configurable via ops.cid_shard_size) split into enough shards to fit. A hard cap of SCX_CID_SHARD_MAX_CPUS prevents pathological sizes under custom configurations.
No-topo cids pack into their own shards so every cid has a shard assignment.
Also build scx_cid_shard_ranges[] for O(1) shard-to-cid-range lookup and scx_shard_node[] so callers can size or place work by NUMA without walking cids. Auto-built shards inherit their LLC's node. No-topo shards carry NUMA_NO_NODE.
--- kernel/sched/ext/cid.c | 132 ++++++++++++++++++++++++++++++++++-- kernel/sched/ext/cid.h | 4 ++ kernel/sched/ext/ext.c | 4 ++ kernel/sched/ext/internal.h | 13 ++++ kernel/sched/ext/types.h | 35 ++++++++-- 5 files changed, 176 insertions(+), 12 deletions(-)
kernel/sched/ext/cid.c +125 -7
@@ -18,13 +18,17 @@* use scx_bpf_cid_override() to change the mapping. The mapping stays stable * until the root is disabled. */ +u32 scx_nr_cid_shards;s16 *scx_cid_to_cpu_tbl;s16 *scx_cpu_to_cid_tbl; +s32 *scx_cid_to_shard; +s32 *scx_shard_node; +struct scx_cid_shard *scx_cid_shard_ranges;struct scx_cid_topo *scx_cid_topo;#define SCX_CID_TOPO_NEG (struct scx_cid_topo) { \ .core_cid = -1, .core_idx = -1, .llc_cid = -1, .llc_idx = -1, \ - .node_cid = -1, .node_idx = -1, \ + .node_cid = -1, .node_idx = -1, .shard_cid = -1, .shard_idx = -1, \ }/*
@@ -43,11 +47,40 @@ static const struct cpumask *cpu_llc_mask(int cpu, struct cpumask *fallbacks)return &ci->info_list[ci->num_leaves -1 ].shared_cpu_map; } +/* +* Compute per-LLC shard layout. Each shard holds at most @shard_size cids, and +* in any case no more than SCX_CID_SHARD_MAX_CPUS. Cores are spread as evenly +* as possible across shards so cpu count is balanced: the first *@nr_large_p +* shards get (*@cores_per_shard_p + 1) cores, the rest get *@cores_per_shard_p. +*/ +static void calc_shard_layout (const struct cpumask *llc_cpus,u32 shard_size, +u32 *cores_per_shard_p,u32 *nr_large_p) +{ +u32 nr_cores =0 , nr_cpus =0 , nr_shards; +int cpu; + +for_each_cpu (cpu, llc_cpus) { + nr_cpus++; +if (cpumask_first (topology_sibling_cpumask (cpu)) == cpu) + nr_cores++; + } + + nr_shards =max_t (u32,1 ,DIV_ROUND_UP (nr_cpus, shard_size)); + nr_shards =max_t (u32, nr_shards, +DIV_ROUND_UP (nr_cpus,SCX_CID_SHARD_MAX_CPUS )); + + *cores_per_shard_p = nr_cores / nr_shards; + *nr_large_p = nr_cores % nr_shards; +} +/* Allocate the cid tables once on first enable; never freed. */ static s32 scx_cid_arrays_alloc (void ) {u32 npossible =num_possible_cpus ();s16 *cid_to_cpu, *cpu_to_cid; +s32 *cid_to_shard; +s32 *shard_node; +struct scx_cid_shard *cid_shard_ranges;struct scx_cid_topo *cid_topo;if (scx_cid_to_cpu_tbl)
@@ -55,17 +88,27 @@ static s32 scx_cid_arrays_alloc(void) cid_to_cpu =kzalloc_objs (*scx_cid_to_cpu_tbl, npossible,GFP_KERNEL ); cpu_to_cid =kzalloc_objs (*scx_cpu_to_cid_tbl, nr_cpu_ids,GFP_KERNEL ); + cid_to_shard =kzalloc_objs (*scx_cid_to_shard, npossible,GFP_KERNEL ); + shard_node =kmalloc_objs (*scx_shard_node, npossible,GFP_KERNEL ); + cid_shard_ranges =kzalloc_objs (*scx_cid_shard_ranges, npossible,GFP_KERNEL ); cid_topo =kmalloc_objs (*scx_cid_topo, npossible,GFP_KERNEL ); -if (!cid_to_cpu || !cpu_to_cid || !cid_topo) { +if (!cid_to_cpu || !cpu_to_cid || !cid_to_shard || !shard_node || + !cid_shard_ranges || !cid_topo) {kfree (cid_to_cpu);kfree (cpu_to_cid); +kfree (cid_to_shard); +kfree (shard_node); +kfree (cid_shard_ranges);kfree (cid_topo);return -ENOMEM ; }WRITE_ONCE (scx_cid_to_cpu_tbl, cid_to_cpu);WRITE_ONCE (scx_cpu_to_cid_tbl, cpu_to_cid); +WRITE_ONCE (scx_cid_to_shard, cid_to_shard); +WRITE_ONCE (scx_shard_node, shard_node); +WRITE_ONCE (scx_cid_shard_ranges, cid_shard_ranges);WRITE_ONCE (scx_cid_topo, cid_topo);return 0 ; }
@@ -90,17 +133,29 @@ s32 scx_cid_init(struct scx_sched *sch) cpumask_var_t online_no_topo __free(free_cpumask_var) = CPUMASK_VAR_NULL;u32 next_cid =0 ;s32 next_node_idx =0 , next_llc_idx =0 , next_core_idx =0 ; -s32 cpu, ret; +s32 next_shard_idx =0 ; +u32 shard_size, max_cids; +u32 notopo_in_shard; +s32 notopo_shard_cid, notopo_shard_idx; +s32 cpu, cid, si, ret;/* CMASK_MAX_WORDS in cid.bpf.h covers NR_CPUS up to 8192 */ BUILD_BUG_ON (NR_CPUS >8192 );lockdep_assert_cpus_held (); + shard_size = sch->ops.cid_shard_size ?:SCX_CID_SHARD_SIZE_DFL ; + max_cids =min_t (u32, shard_size,SCX_CID_SHARD_MAX_CPUS ); + ret =scx_cid_arrays_alloc ();if (ret)return ret; +/* clear shard ranges and reset shard_node for repopulate */ +memset (scx_cid_shard_ranges,0 ,num_possible_cpus () *sizeof (*scx_cid_shard_ranges)); +for (si =0 ; si <num_possible_cpus (); si++) + scx_shard_node[si] =NUMA_NO_NODE ; + if (!zalloc_cpumask_var(&to_walk, GFP_KERNEL) || !zalloc_cpumask_var(&node_scratch, GFP_KERNEL) || !zalloc_cpumask_var(&llc_scratch, GFP_KERNEL) ||
@@ -142,29 +197,60 @@ s32 scx_cid_init(struct scx_sched *sch)const struct cpumask *llc_mask =cpu_llc_mask (ncpu, llc_fallback);s32 llc_cid = next_cid;s32 llc_idx = next_llc_idx++; +u32 cores_per_shard, nr_large; +u32 shard_local =0 , cores_in_shard =0 , cids_in_shard =0 ; +s32 shard_cid, shard_idx;/* llc_scratch = node_scratch & this llc */ cpumask_and (llc_scratch, node_scratch, llc_mask);if (WARN_ON_ONCE (!cpumask_test_cpu (ncpu, llc_scratch)))return -EINVAL ; +calc_shard_layout (llc_scratch, shard_size, &cores_per_shard, &nr_large); + shard_cid = next_cid; + shard_idx = next_shard_idx++; + scx_shard_node[shard_idx] = nid; +while (!cpumask_empty (llc_scratch)) {s32 lcpu =cpumask_first (llc_scratch);const struct cpumask *sib =topology_sibling_cpumask (lcpu);s32 core_cid = next_cid;s32 core_idx = next_core_idx++;s32 ccpu; +u32 max_cores, cids_in_core;/* core_scratch = llc_scratch & this core */ cpumask_and (core_scratch, llc_scratch, sib);if (WARN_ON_ONCE (!cpumask_test_cpu (lcpu, core_scratch)))return -EINVAL ; +/* +* Advance to a new shard when either core or +* cid count reaches max. The latter bounds +* shard sizes under uneven SMT. Never start an +* empty shard. +*/ + cids_in_core =cpumask_weight (core_scratch); + max_cores = cores_per_shard + (shard_local < nr_large ?1 :0 ); +if (cores_in_shard && + (cores_in_shard >= max_cores || + cids_in_shard + cids_in_core > max_cids)) { + shard_local++; + cores_in_shard =0 ; + cids_in_shard =0 ; + shard_cid = next_cid; + shard_idx = next_shard_idx++; + scx_shard_node[shard_idx] = nid; + } + cores_in_shard++; + cids_in_shard += cids_in_core; +for_each_cpu (ccpu, core_scratch) {s32 cid = next_cid++; scx_cid_to_cpu_tbl[cid] = ccpu; scx_cpu_to_cid_tbl[ccpu] = cid; + scx_cid_to_shard[cid] = shard_idx; scx_cid_topo[cid] = (struct scx_cid_topo ){ .core_cid = core_cid, .core_idx = core_idx,
@@ -172,6 +258,8 @@ s32 scx_cid_init(struct scx_sched *sch) .llc_idx = llc_idx, .node_cid = node_cid, .node_idx = node_idx, + .shard_cid = shard_cid, + .shard_idx = shard_idx, };cpumask_clear_cpu (ccpu, llc_scratch);
@@ -184,12 +272,17 @@ s32 scx_cid_init(struct scx_sched *sch)/* * No-topo section: any possible cpu without a cid - normally just the -* not-online ones. Collect any currently-online cpus that land here in -* @online_no_topo so we can warn about them at the end. +* not-online ones. Pack into shards of up to min(@shard_size, +* SCX_CID_SHARD_MAX_CPUS) cids so that every cid has a valid shard +* assignment and the hard cap holds even with a large @shard_size. +* Collect any currently-online cpus that land here in @online_no_topo +* so we can warn about them at the end. */ -for_each_cpu (cpu, cpu_possible_mask) { -s32 cid; + notopo_in_shard =min_t (u32, shard_size,SCX_CID_SHARD_MAX_CPUS ); + notopo_shard_cid =-1 ; + notopo_shard_idx =-1 ; +for_each_cpu (cpu, cpu_possible_mask) {if (__scx_cpu_to_cid (cpu) !=-1 )continue ;if (cpu_online (cpu))
@@ -198,7 +291,18 @@ s32 scx_cid_init(struct scx_sched *sch) cid = next_cid++; scx_cid_to_cpu_tbl[cid] = cpu; scx_cpu_to_cid_tbl[cpu] = cid; + +if (notopo_in_shard >=min_t (u32, shard_size,SCX_CID_SHARD_MAX_CPUS )) { + notopo_shard_cid = cid; + notopo_shard_idx = next_shard_idx++; + notopo_in_shard =0 ; + } + notopo_in_shard++; + + scx_cid_to_shard[cid] = notopo_shard_idx; scx_cid_topo[cid] =SCX_CID_TOPO_NEG ; + scx_cid_topo[cid].shard_cid = notopo_shard_cid; + scx_cid_topo[cid].shard_idx = notopo_shard_idx; }if (!cpumask_empty (llc_fallback))
@@ -208,6 +312,20 @@ s32 scx_cid_init(struct scx_sched *sch)pr_warn ("scx_cid: online cpus with no usable topology: %*pbl\n" ,cpumask_pr_args (online_no_topo)); +/* +* Fill cid_shard_ranges[] from cid_to_shard[]. Shards are contiguous +* cid ranges by construction: base_cid is the first cid landing in a +* shard, nr_cids is the count. +*/ +for (cid =0 ; cid < next_cid; cid++) { +s32 sidx = scx_cid_to_shard[cid]; + +if (scx_cid_shard_ranges[sidx].nr_cids ==0 ) + scx_cid_shard_ranges[sidx].base_cid = cid; + scx_cid_shard_ranges[sidx].nr_cids++; + } + + scx_nr_cid_shards = next_shard_idx;return 0 ; }
kernel/sched/ext/cid.h +4 -0
@@ -48,8 +48,12 @@ struct scx_sched;* See the comment above the table definitions in cid.c for the * memory-ordering and visibility contract. */ +extern u32 scx_nr_cid_shards;extern s16 *scx_cid_to_cpu_tbl;extern s16 *scx_cpu_to_cid_tbl; +extern s32 *scx_cid_to_shard; +extern s32 *scx_shard_node; +extern struct scx_cid_shard *scx_cid_shard_ranges;extern struct scx_cid_topo *scx_cid_topo;extern struct btf_id_set8 scx_kfunc_ids_init_cids;
kernel/sched/ext/ext.c +4 -0
@@ -7135,6 +7135,9 @@ static int bpf_scx_init_member(const struct btf_type *t,case offsetof(struct sched_ext_ops , hotplug_seq): ops->hotplug_seq = *(u64 *)(udata + moff);return 1 ; +case offsetof(struct sched_ext_ops , cid_shard_size): + ops->cid_shard_size = *(u32 *)(udata + moff); +return 1 ;#ifdef CONFIG_EXT_SUB_SCHED case offsetof(struct sched_ext_ops , sub_cgroup_id): ops->sub_cgroup_id = *(u64 *)(udata + moff);
@@ -9832,6 +9835,7 @@ static int __init scx_init(void)CID_OFFSET_MATCH (timeout_ms, timeout_ms);CID_OFFSET_MATCH (exit_dump_len, exit_dump_len);CID_OFFSET_MATCH (hotplug_seq, hotplug_seq); +CID_OFFSET_MATCH (cid_shard_size, cid_shard_size);CID_OFFSET_MATCH (sub_cgroup_id, sub_cgroup_id);/* shared callbacks: the union view requires byte-for-byte offset match */ CID_OFFSET_MATCH (enqueue, enqueue);
kernel/sched/ext/internal.h +13 -0
@@ -845,6 +845,18 @@ struct sched_ext_ops {*/ u64 hotplug_seq; +/** +* @cid_shard_size: Target number of CIDs per shard +* +* Shards are contiguous CID ranges used as operation and locking +* domains for sub-scheduling. Each LLC is divided into ceil(nr_cpus / +* @cid_shard_size) shards, then cores are distributed across them +* evenly. If one core has more logical CPUs than @cid_shard_size, its +* shard will become larger than @cid_shard_size. Values above +* SCX_CID_SHARD_MAX_CPUS are capped. 0 means use the default (24). +*/ +u32 cid_shard_size; +/** * @cgroup_id: When >1, attach the scheduler as a sub-scheduler on the * specified cgroup.
@@ -977,6 +989,7 @@ struct sched_ext_ops_cid {u32 timeout_ms;u32 exit_dump_len;u64 hotplug_seq; +u32 cid_shard_size;u64 sub_cgroup_id;char name[SCX_OPS_NAME_LEN ];
kernel/sched/ext/types.h +30 -5
@@ -47,11 +47,16 @@ enum scx_consts { };/* -* Per-cid topology info. For each topology level (core, LLC, node), records -* the first cid in the unit and its global index. Global indices are -* consecutive integers assigned in cid-walk order, so e.g. core_idx ranges -* over [0, nr_cores_at_init) with no gaps. No-topo cids have all fields set -* to -1. +* Per-cid topology info. For each topology level (core, LLC, node) and shard, +* records the first cid in the unit and its global index. Global indices are +* consecutive integers assigned in cid-walk order, so e.g. core_idx ranges over +* [0, nr_cores_at_init) with no gaps. No-topo cids have core/LLC/node fields +* set to -1 but always have valid shard assignments. +* +* Shards are contiguous CID ranges used as scalable locking/work domains for +* sub-scheduler operations. By default each LLC becomes one shard, split into +* smaller shards if the LLC exceeds the target size. No-topo cids are packed +* into their own max-sized shards. * * @core_cid: first cid of this cid's core (smt-sibling group) * @core_idx: global index of that core, in [0, nr_cores_at_init)
@@ -59,6 +64,8 @@ enum scx_consts {* @llc_idx: global index of that LLC, in [0, nr_llcs_at_init) * @node_cid: first cid of this cid's NUMA node * @node_idx: global index of that node, in [0, nr_nodes_at_init) +* @shard_cid: first cid of this cid's shard +* @shard_idx: global index of that shard, in [0, scx_nr_cid_shards) */ struct scx_cid_topo {s32 core_cid;
@@ -67,6 +74,24 @@ struct scx_cid_topo {s32 llc_idx;s32 node_cid;s32 node_idx; +s32 shard_cid; +s32 shard_idx; +}; + +enum scx_cid_consts { +SCX_CID_SHARD_SIZE_DFL =24 , +SCX_CID_SHARD_MAX_CPUS =512 , +}; + +/* +* Per-shard metadata for O(1) shard->cid-range lookup. +* +* @base_cid: first cid of the shard +* @nr_cids: number of cids in the shard +*/ +struct scx_cid_shard { +s32 base_cid; +s32 nr_cids; };/*
Signature · 1 line
2.54.0
Tejun Heo
reviewerPATCH v4 13/40 sched_ext: Add shard boundaries to scx_bpf_cid_override()
In reply to Tejun HeoAn overridden cid mapping invalidates the auto-generated shard layout, so the override call has to provide both. Extend scx_bpf_cid_override() with a shard_start[] array that lists the first cid of each shard (starting at 0, strictly increasing, last shard implicitly extends to num_possible_cpus()).
A scheduler that wants only custom shards with the auto-generated cid mapping can read the current mapping and pass it back unchanged.
Overridden shards can span NUMA nodes, so scx_shard_node[] is rebuilt by majority count: each shard is assigned to the node that owns the most cpus in it.
v2: Snapshot the caller's cpu_to_cid/shard_start arrays before validating. (sashiko AI)
--- kernel/sched/ext/cid.c | 154 ++++++++++++++++++++--- tools/sched_ext/include/scx/compat.bpf.h | 11 +- tools/sched_ext/scx_qmap.bpf.c | 16 ++- tools/sched_ext/scx_qmap.c | 34 ++++- tools/sched_ext/scx_qmap.h | 1 + 5 files changed, 187 insertions(+), 29 deletions(-)
kernel/sched/ext/cid.c +137 -17
@@ -392,29 +392,66 @@ void scx_cpumask_to_cmask(const struct cpumask *src, struct scx_cmask *dst) } } +/* +* Return the index of the largest entry in @counts, or NUMA_NO_NODE if all +* entries are zero. Ties resolve to the lowest index. +*/ +static s32 pick_max_node (const u32 *counts,u32 n) +{ +s32 best =NUMA_NO_NODE ; +u32 best_count =0 , i; + +for (i =0 ; i < n; i++) { +if (counts[i] > best_count) { + best_count = counts[i]; + best = i; + } + } +return best; +} +__bpf_kfunc_start_defs ();/** -* scx_bpf_cid_override - Install an explicit cpu->cid mapping -* @cpu_to_cid: array of nr_cpu_ids s32 entries (cid for each cpu) -* @cpu_to_cid__sz: must be nr_cpu_ids * sizeof(s32) bytes +* scx_bpf_cid_override - Install an explicit cpu->cid mapping with shard info +* @cpu_to_cid_src: array of nr_cpu_ids s32 entries (cid for each cpu) +* @cpu_to_cid_src__sz: must be nr_cpu_ids * sizeof(s32) bytes +* @shard_start_src: array of first-cid-of-each-shard, strictly increasing from 0 +* @shard_start_src__sz: nr_shards * sizeof(s32) bytes * @aux: implicit BPF argument to access bpf_prog_aux hidden from BPF progs * * May only be called from ops.init_cids() of the root scheduler. Replace the -* topology-probed cid mapping with the caller-provided one. Each possible cpu -* must map to a unique cid in [0, num_possible_cpus()). Topo info is cleared. -* On invalid input, trigger scx_error() to abort the scheduler. +* topology-probed cid mapping and shard layout with caller-provided ones. Each +* possible cpu must map to a unique cid in [0, num_possible_cpus()). The shard +* starts must be strictly increasing with the first entry 0 and all values < +* num_possible_cpus(). The last shard extends to num_possible_cpus() and no +* shard may span more than SCX_CID_SHARD_MAX_CPUS cids. Topo info +* (core/LLC/node) is cleared and the shard layout is set from the input. On +* invalid input, abort the scheduler. */ -__bpf_kfunc void scx_bpf_cid_override(const s32 *cpu_to_cid, u32 cpu_to_cid__sz, -const struct bpf_prog_aux *aux) +__bpf_kfunc void scx_bpf_cid_override(const s32 *cpu_to_cid_src, u32 cpu_to_cid_src__sz, +const s32 *shard_start_src,u32 shard_start_src__sz, +const struct bpf_prog_aux *aux) { cpumask_var_t seen __free(free_cpumask_var) = CPUMASK_VAR_NULL; + u32 *node_counts __free(kfree) = NULL; + s32 *cpu_to_cid __free(kfree) = NULL; + s32 *shard_start __free(kfree) = NULL; +u32 npossible =num_possible_cpus ();struct scx_sched *sch; +u32 nr_shards;bool alloced; -s32 cpu, cid; +s32 cpu, cid, si; -/* GFP_KERNEL alloc must happen before the rcu read section */ +/* +* GFP_KERNEL allocs must happen before the rcu read section. Snapshot +* the BPF-supplied arrays so a concurrent map mutation can't change +* them between validation and use. +*/ alloced =zalloc_cpumask_var (&seen,GFP_KERNEL ); + node_counts =kcalloc (nr_node_ids,sizeof (*node_counts),GFP_KERNEL ); + cpu_to_cid =kmemdup (cpu_to_cid_src, cpu_to_cid_src__sz,GFP_KERNEL ); + shard_start =kmemdup (shard_start_src, shard_start_src__sz,GFP_KERNEL );guard (rcu)();
@@ -422,17 +459,57 @@ __bpf_kfunc void scx_bpf_cid_override(const s32 *cpu_to_cid, u32 cpu_to_cid__sz,if (unlikely (!sch))return ; -if (!alloced) { -scx_error (sch,"scx_bpf_cid_override: failed to allocate cpumask" ); +if (!alloced || !node_counts || !cpu_to_cid || !shard_start) { +scx_error (sch,"scx_bpf_cid_override: allocation failed" );return ; } -if (cpu_to_cid__sz != nr_cpu_ids *sizeof (s32)) { -scx_error (sch,"scx_bpf_cid_override: expected %zu bytes, got %u" , - nr_cpu_ids *sizeof (s32), cpu_to_cid__sz); +if (cpu_to_cid_src__sz != nr_cpu_ids *sizeof (s32)) { +scx_error (sch,"scx_bpf_cid_override: cpu_to_cid expected %zu bytes, got %u" , + nr_cpu_ids *sizeof (s32), cpu_to_cid_src__sz); +return ; + } + +if (!shard_start_src__sz || shard_start_src__sz %sizeof (s32)) { +scx_error (sch,"scx_bpf_cid_override: invalid shard_start size %u" , + shard_start_src__sz); +return ; + } + + nr_shards = shard_start_src__sz /sizeof (s32); + +/* validate shard_start[]: starts at 0, strictly increasing, in range */ +if (shard_start[0 ] !=0 ) { +scx_error (sch,"scx_bpf_cid_override: shard_start[0] must be 0, got %d" , + shard_start[0 ]); +return ; + } +for (si =1 ; si < nr_shards; si++) { +if (shard_start[si] <= shard_start[si -1 ]) { +scx_error (sch,"scx_bpf_cid_override: shard_start not increasing at [%d]" , + si); +return ; + } +if (shard_start[si] >= npossible) { +scx_error (sch,"scx_bpf_cid_override: shard_start[%d]=%d >= %u" , + si, shard_start[si], npossible); +return ; + } +if (shard_start[si] - shard_start[si -1 ] >SCX_CID_SHARD_MAX_CPUS ) { +scx_error (sch,"scx_bpf_cid_override: shard[%d] span %d exceeds max %d" , + si -1 , shard_start[si] - shard_start[si -1 ], +SCX_CID_SHARD_MAX_CPUS ); +return ; + } + } +if (npossible - shard_start[nr_shards -1 ] >SCX_CID_SHARD_MAX_CPUS ) { +scx_error (sch,"scx_bpf_cid_override: shard[%d] span %d exceeds max %d" , + nr_shards -1 , npossible - shard_start[nr_shards -1 ], +SCX_CID_SHARD_MAX_CPUS );return ; } +/* Validate first so that invalid input leaves globals untouched. */ for_each_possible_cpu (cpu) {s32 c = cpu_to_cid[cpu];
@@ -442,13 +519,56 @@ __bpf_kfunc void scx_bpf_cid_override(const s32 *cpu_to_cid, u32 cpu_to_cid__sz,scx_error (sch,"cid %d assigned to multiple cpus" , c);return ; } + } + +for_each_possible_cpu (cpu) { +s32 c = cpu_to_cid[cpu]; + scx_cpu_to_cid_tbl[cpu] = c; scx_cid_to_cpu_tbl[c] = cpu; } -/* Invalidate stale topo info - the override carries no topology. */ -for (cid =0 ; cid <num_possible_cpus (); cid++) +/* +* Derive scx_shard_node[] by majority count: an overridden shard may +* span NUMA nodes, so assign each to the node that owns the most cpus. +*/ +for (si =0 ; si < nr_shards; si++) { +u32 end = (si +1 < nr_shards) ? shard_start[si +1 ] : npossible; + +memset (node_counts,0 , nr_node_ids *sizeof (*node_counts)); +for (cid = shard_start[si]; cid < end; cid++) { +s32 node =cpu_to_node (scx_cid_to_cpu_tbl[cid]); + +if (numa_valid_node (node)) + node_counts[node]++; + } + scx_shard_node[si] =pick_max_node (node_counts, nr_node_ids); + } + +/* +* Invalidate stale topo info and install shard layout from +* @shard_start. Walk shards to derive shard_cid/shard_idx for each cid. +*/ + si =0 ; +for (cid =0 ; cid < npossible; cid++) { +if (si +1 < nr_shards && cid >= shard_start[si +1 ]) + si++; + scx_cid_to_shard[cid] = si; scx_cid_topo[cid] =SCX_CID_TOPO_NEG ; + scx_cid_topo[cid].shard_cid = shard_start[si]; + scx_cid_topo[cid].shard_idx = si; + } + +/* Rebuild scx_cid_shard_ranges[] for the new layout. */ +memset (scx_cid_shard_ranges,0 , npossible *sizeof (*scx_cid_shard_ranges)); +for (si =0 ; si < nr_shards; si++) { +u32 end = (si +1 < nr_shards) ? shard_start[si +1 ] : npossible; + + scx_cid_shard_ranges[si].base_cid = shard_start[si]; + scx_cid_shard_ranges[si].nr_cids = end - shard_start[si]; + } + + scx_nr_cid_shards = nr_shards; }/**
tools/sched_ext/include/scx/compat.bpf.h +7 -4
@@ -122,15 +122,18 @@ static inline bool scx_bpf_sub_dispatch(u64 cgroup_id) }/* -* v7.2: scx_bpf_cid_override() for explicit cpu->cid mapping. Ignore if +* v7.3: scx_bpf_cid_override() for explicit cid and shard mapping. Ignore if * missing. */ -void scx_bpf_cid_override___compat(const s32 *cpu_to_cid, u32 cpu_to_cid__sz) __ksym __weak; +void scx_bpf_cid_override___compat(const s32 *cpu_to_cid, u32 cpu_to_cid__sz, + const s32 *shard_start, u32 shard_start__sz) __ksym __weak; -static inline void scx_bpf_cid_override (const s32 *cpu_to_cid,u32 cpu_to_cid__sz) +static inline void scx_bpf_cid_override (const s32 *cpu_to_cid,u32 cpu_to_cid__sz, +const s32 *shard_start,u32 shard_start__sz) {if (bpf_ksym_exists (scx_bpf_cid_override___compat)) -return scx_bpf_cid_override___compat (cpu_to_cid, cpu_to_cid__sz); +scx_bpf_cid_override___compat (cpu_to_cid, cpu_to_cid__sz, + shard_start, shard_start__sz); }/**
tools/sched_ext/scx_qmap.bpf.c +10 -6
@@ -54,18 +54,20 @@ const volatile u32 max_tasks;/* * Optional cid-override test harness. When cid_override_mode is non-zero, -* qmap_init_cids() calls scx_bpf_cid_override() with the caller-supplied -* cpu_to_cid array to exercise the kfunc's acceptance and error paths. See enum +* qmap_init_cids() calls scx_bpf_cid_override() with the caller-supplied arrays +* to exercise the kfunc's acceptance and error paths. See enum * qmap_cid_override for the modes. */ const volatile u32 cid_override_mode; +const volatile u32 cid_override_nr_shards;/* -* Array lives in bss (writable) because scx_bpf_cid_override()'s BPF -* verifier signature treats its len-paired pointer as read/write - rodata +* Arrays live in bss (writable) because scx_bpf_cid_override()'s BPF +* verifier signature treats its len-paired pointers as read/write - rodata * fails verification with "write into map forbidden". Userspace populates -* it before SCX_OPS_LOAD, same as rodata, and nothing writes it after. +* them before SCX_OPS_LOAD, same as rodata, and nothing writes them after. */ s32 cid_override_cpu_to_cid[SCX_QMAP_MAX_CPUS ]; +s32 cid_override_shard_start[SCX_QMAP_MAX_CPUS ];UEI_DEFINE (uei);
@@ -1082,7 +1084,9 @@ s32 BPF_STRUCT_OPS_SLEEPABLE(qmap_init_cids) }scx_bpf_cid_override ((const s32 *)cid_override_cpu_to_cid, - nr_cpu_ids *sizeof (s32)); + nr_cpu_ids *sizeof (s32), + (const s32 *)cid_override_shard_start, + cid_override_nr_shards *sizeof (s32));return 0 ; }
tools/sched_ext/scx_qmap.c +32 -2
@@ -43,7 +43,7 @@ const char help_fmt[] =" -p Switch only tasks on SCHED_EXT policy instead of all\n" " -I Turn on SCX_OPS_ALWAYS_ENQ_IMMED\n" " -F COUNT IMMED stress: force every COUNT'th enqueue to a busy local DSQ (use with -I)\n" -" -C MODE cid-override test (shuffle|bad-dup|bad-range)\n" +" -C MODE cid-override test (shuffle|bad-dup|bad-range|bad-mono)\n" " -v Print libbpf debug messages\n" " -h Display this help and exit\n" ;
@@ -155,6 +155,7 @@ int main(int argc, char **argv)case 'C' : {u32 nr_cpus =libbpf_num_possible_cpus ();u32 mode, i; +s32 shard_sz =4 ;if (!strcmp (optarg,"shuffle" )) mode =QMAP_CID_OVR_SHUFFLE ;
@@ -162,13 +163,15 @@ int main(int argc, char **argv) mode =QMAP_CID_OVR_BAD_DUP ; else if (!strcmp(optarg, "bad-range")) mode =QMAP_CID_OVR_BAD_RANGE ; + else if (!strcmp(optarg, "bad-mono")) + mode =QMAP_CID_OVR_BAD_MONO ; else {fprintf (stderr,"unknown cid-override mode '%s'\n" , optarg);return 1 ; } skel->rodata->cid_override_mode = mode; -/* shuffle: reversed cpu_to_cid, bad-dup: dup cid 0, bad-range: identity */ +/* shuffle: reversed cpu_to_cid; others: identity */ for (i =0 ; i < nr_cpus; i++) {if (mode ==QMAP_CID_OVR_SHUFFLE ) skel->bss->cid_override_cpu_to_cid[i] = nr_cpus -1 - i;
@@ -179,6 +182,33 @@ int main(int argc, char **argv) skel->bss->cid_override_cpu_to_cid[1 ] =0 ;if (mode ==QMAP_CID_OVR_BAD_RANGE ) skel->bss->cid_override_cpu_to_cid[0 ] = (s32 )nr_cpus; + +/* +* bad-mono needs >= 3 shards to build a 0-based but +* non-monotonic shard_start. Shrink the shard size so +* the test runs on any machine with >= 3 cpus. +*/ +if (mode ==QMAP_CID_OVR_BAD_MONO ) { +if (nr_cpus <3 ) { +fprintf (stderr,"bad-mono needs >= 3 cpus (have %u)\n" , + nr_cpus); +return 1 ; + } + shard_sz = nr_cpus /3 ; + } + +/* shards of shard_sz each */ + skel->rodata->cid_override_nr_shards = (nr_cpus + shard_sz -1 ) / shard_sz; +for (i =0 ; i < skel->rodata->cid_override_nr_shards; i++) + skel->bss->cid_override_shard_start[i] = i * shard_sz; + +if (mode ==QMAP_CID_OVR_BAD_MONO ) { +/* swap [1] and [2] to break monotonicity */ +s32 tmp = skel->bss->cid_override_shard_start[1 ]; + skel->bss->cid_override_shard_start[1 ] = + skel->bss->cid_override_shard_start[2 ]; + skel->bss->cid_override_shard_start[2 ] = tmp; + }break ; }case 'v' :
tools/sched_ext/scx_qmap.h +1 -0
@@ -33,6 +33,7 @@ enum qmap_cid_override {QMAP_CID_OVR_SHUFFLE =1 ,/* valid reversed cpu->cid mapping */ QMAP_CID_OVR_BAD_DUP =2 ,/* invalid: duplicate cid assignment */ QMAP_CID_OVR_BAD_RANGE =3 ,/* invalid: out-of-range cid */ + QMAP_CID_OVR_BAD_MONO = 4, /* invalid: non-monotonic shard_start */ }; struct cpu_ctx {
Signature · 1 line
2.54.0
[email protected]
AI reviewRe: [PATCH v4 sched_ext/for-7.3 13/40] sched_ext: Add shard boundaries to scx_bpf_cid_override()
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] Unbounded allocation sizes for kmemdup before validation trigger kernel warnings and allocation…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] Unbounded allocation sizes for kmemdup before validation trigger kernel warnings and allocation failures. --
--- Patch [13]: [PATCH v4 sched_ext/for-7.3 13/40] sched_ext: Add shard boundaries to scx_bpf_cid_override() --- commit d662162dc44b1de0465d793b6ec2418996af7aef
Author: Tejun Heo <[email protected]>
sched_ext: Add shard boundaries to scx_bpf_cid_override()
This commit extends scx_bpf_cid_override() to accept explicit shard boundaries. It also snapshots the caller's arrays before validating them to prevent concurrent mutations from altering the data between validation and use.
Quoted 4 lines
kernel/sched/ext/cid.c +0
-0
[ ... ]
Quoted 1 line
kernel/sched/ext/cid.c +0
-0@@ -392,29 +392,66 @@ void scx_cpumask_to_cmask(const struct cpumask *src, struct scx_cmask *dst)
[ ... ]
kernel/sched/ext/cid.c void scx_cpumask_to_cmask(const struct cpumask *src, struct scx_cmask *dst) · line 454
Quoted 10 lines
kernel/sched/ext/cid.c +7
-1/* -/* GFP_KERNEL alloc must happen before the rcu read section */ +* GFP_KERNEL allocs must happen before the rcu read section. Snapshot +* the BPF-supplied arrays so a concurrent map mutation can't change +* them between validation and use. +*/ alloced =zalloc_cpumask_var (&seen,GFP_KERNEL ); + node_counts =kcalloc (nr_node_ids,sizeof (*node_counts),GFP_KERNEL ); + cpu_to_cid =kmemdup (cpu_to_cid_src, cpu_to_cid_src__sz,GFP_KERNEL ); + shard_start =kmemdup (shard_start_src, shard_start_src__sz,GFP_KERNEL );
[Severity: Medium]
Can a BPF program trigger a page allocator warning here by passing an unbounded size for cpu_to_cid_src__sz or shard_start_src__sz?
The sizes are passed directly to kmemdup() before the length validation checks occur below. Would it be safer to validate the expected array sizes before attempting to allocate memory for the snapshots?
Quoted 28 lines
kernel/sched/ext/cid.c +13
-5guard (rcu)();@@ -422,17 +459,57 @@ __bpf_kfunc void scx_bpf_cid_override(const s32 *cpu_to_cid, u32 cpu_to_cid__sz,if (unlikely (!sch))return ; -if (!alloced) { -scx_error (sch,"scx_bpf_cid_override: failed to allocate cpumask" ); +if (!alloced || !node_counts || !cpu_to_cid || !shard_start) { +scx_error (sch,"scx_bpf_cid_override: allocation failed" );return ; } -if (cpu_to_cid__sz != nr_cpu_ids *sizeof (s32)) { -scx_error (sch,"scx_bpf_cid_override: expected %zu bytes, got %u" , - nr_cpu_ids *sizeof (s32), cpu_to_cid__sz); +if (cpu_to_cid_src__sz != nr_cpu_ids *sizeof (s32)) { +scx_error (sch,"scx_bpf_cid_override: cpu_to_cid expected %zu bytes, got %u" , + nr_cpu_ids *sizeof (s32), cpu_to_cid_src__sz); +return ; + } + +if (!shard_start_src__sz || shard_start_src__sz %sizeof (s32)) { +scx_error (sch,"scx_bpf_cid_override: invalid shard_start size %u" , + shard_start_src__sz); +return ; + }
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=13
Tejun Heo
reviewerRe: [PATCH v4 sched_ext/for-7.3 13/40] sched_ext: Add shard boundaries to scx_bpf_cid_override()
Quoted 2 lines · On Wed, 08 Jul 2026 21:47:38 +0000, [email protected] wrote:
- [Medium] Unbounded allocation sizes for kmemdup before validation trigger kernel warnings and allocation failures.
The __sz arguments are verifier-bounded to the size of an actual BPF object, the kfunc is only reachable from a privileged scheduler's init_cids, and a failed allocation lands in the existing NULL bail which aborts the enable cleanly. Reordering wouldn't buy anything real. No change.
Thanks.
Signature · 1 line
tejun
Tejun Heo
reviewerPATCH v4 14/40 sched_ext: Defer scx_sched kobj sysfs add into the enable workfns
In reply to Tejun HeoSplit kobject_init_and_add() in scx_alloc_and_add_sched(): only kobject_init() runs there. A new scx_sched_sysfs_add() helper does kobject_add() (and creates sub_kset when the scheduler implements ops.sub_attach), called by both enable workfns once @sch is linked and its sysfs-visible state is initialized. Prep so a future caps attribute can rely on @sch being fully built by the time it's sysfs-visible. Add early enough that a stall later in enable still leaves sysfs inspectable.
--- kernel/sched/ext/ext.c | 73 +++++++++++++++++++++++-------------- kernel/sched/ext/internal.h | 1 + kernel/sched/ext/sub.c | 8 +++- 3 files changed, 53 insertions(+), 29 deletions(-)
kernel/sched/ext/ext.c +45 -28
@@ -5698,7 +5698,9 @@ static void scx_root_disable(struct scx_sched *sch)if (sch->sub_kset)kobject_del (&sch->sub_kset->kobj);#endif -kobject_del (&sch->kobj); +/* not added if enable failed before scx_sched_sysfs_add() */ +if (sch->kobj.state_in_sysfs) +kobject_del (&sch->kobj);free_kick_syncs ();
@@ -6415,36 +6417,15 @@ struct scx_sched *scx_alloc_and_add_sched(struct scx_enable_cmd *cmd,* disable. Released in scx_sched_free_rcu_work(). */ kobject_get(&parent->kobj); - ret =kobject_init_and_add (&sch->kobj, &scx_ktype, - &parent->sub_kset->kobj, - "sub-%llu", cgroup_id(cgrp)); - }else { - ret =kobject_init_and_add (&sch->kobj, &scx_ktype,NULL ,"root" ); - } - -if (ret <0 ) { -RCU_INIT_POINTER (ops->priv,NULL ); -kobject_put (&sch->kobj); -return ERR_PTR (ret); - } - -if (ops->sub_attach) { - sch->sub_kset =kset_create_and_add ("sub" ,NULL , &sch->kobj); -if (!sch->sub_kset) { -RCU_INIT_POINTER (ops->priv,NULL ); -kobject_put (&sch->kobj); -return ERR_PTR (-ENOMEM ); - } - } -#else /* CONFIG_EXT_SUB_SCHED */ - ret =kobject_init_and_add (&sch->kobj, &scx_ktype,NULL ,"root" ); -if (ret <0 ) { -RCU_INIT_POINTER (ops->priv,NULL ); -kobject_put (&sch->kobj); -return ERR_PTR (ret); }#endif /* CONFIG_EXT_SUB_SCHED */ +/* +* Init the kobj but don't add to sysfs yet. The enable path calls +* scx_sched_sysfs_add() once @sch's sysfs-visible state is initialized. +*/ +kobject_init (&sch->kobj, &scx_ktype); +/* * Consume the arena_map ref bpf_scx_reg_cid() took. Defer to here so * earlier failure paths leave cmd->arena_map set and bpf_scx_reg_cid
@@ -6502,6 +6483,36 @@ struct scx_sched *scx_alloc_and_add_sched(struct scx_enable_cmd *cmd,return ERR_PTR (ret); } +/* +* Add @sch's kobject to sysfs, and create its sub_kset if the scheduler +* implements ops.sub_attach. Called by the enable workfns once @sch's +* sysfs-visible state is initialized. +*/ +int scx_sched_sysfs_add (struct scx_sched *sch) +{ +#ifdef CONFIG_EXT_SUB_SCHED +struct scx_sched *parent =scx_parent (sch); +int ret; + +if (parent) + ret =kobject_add (&sch->kobj, &parent->sub_kset->kobj, +"sub-%llu" ,cgroup_id (sch_cgroup (sch))); +else + ret =kobject_add (&sch->kobj,NULL ,"root" ); +if (ret <0 ) +return ret; + +if (sch->ops.sub_attach) { + sch->sub_kset =kset_create_and_add ("sub" ,NULL , &sch->kobj); +if (!sch->sub_kset) +return -ENOMEM ; + } +return 0 ; +#else +return kobject_add (&sch->kobj,NULL ,"root" ); +#endif +} +static int check_hotplug_seq (struct scx_sched *sch,const struct sched_ext_ops *ops) {
@@ -6724,6 +6735,12 @@ static void scx_root_enable_workfn(struct kthread_work *work) sch->exit_info->flags |=SCX_EFLAG_INITIALIZED ; } + ret =scx_sched_sysfs_add (sch); +if (ret) { +cpus_read_unlock (); + goto err_disable; + } +for (i =SCX_OPI_CPU_HOTPLUG_BEGIN ; i <SCX_OPI_CPU_HOTPLUG_END ; i++)if (((void (**)(void ))ops)[i])set_bit (i, sch->has_op);
kernel/sched/ext/internal.h +1 -0
@@ -1690,6 +1690,7 @@ struct scx_sched *scx_alloc_and_add_sched(struct scx_enable_cmd *cmd,struct cgroup *cgrp, struct scx_sched *parent);int scx_validate_ops (struct scx_sched *sch,const struct sched_ext_ops *ops); +int scx_sched_sysfs_add (struct scx_sched *sch);extern raw_spinlock_t scx_sched_lock;extern struct mutex scx_enable_mutex;
kernel/sched/ext/sub.c +7 -1
@@ -270,7 +270,9 @@ void scx_sub_disable(struct scx_sched *sch)SCX_CALL_OP (sch, exit,NULL , sch->exit_info);if (sch->sub_kset)kobject_del (&sch->sub_kset->kobj); -kobject_del (&sch->kobj); +/* not added if enable failed before scx_sched_sysfs_add() */ +if (sch->kobj.state_in_sysfs) +kobject_del (&sch->kobj); }/* verify that a scheduler can be attached to @cgrp and return the parent */
@@ -363,6 +365,10 @@ void scx_sub_enable_workfn(struct kthread_work *work)if (ret) goto err_disable; + ret =scx_sched_sysfs_add (sch); +if (ret) + goto err_disable; +if (sch->level >=SCX_SUB_MAX_DEPTH ) {scx_error (sch,"max nesting depth %d violated" ,SCX_SUB_MAX_DEPTH );
Signature · 1 line
2.54.0
[email protected]
AI reviewRe: [PATCH v4 sched_ext/for-7.3 14/40] sched_ext: Defer scx_sched kobj sysfs add into the enable workfns
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] Meaningful code and commit message mismatch: In sub-schedulers, scx_sched_sysfs_add() is called…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] Meaningful code and commit message mismatch: In sub-schedulers, scx_sched_sysfs_add() is called before ops.init(), contradicting the root scheduler's behavior and the commit message's goal. --
--- Patch [14]: [PATCH v4 sched_ext/for-7.3 14/40] sched_ext: Defer scx_sched kobj sysfs add into the enable workfns --- commit 76fa4bf5abb621f0acd7524a77691d28f4fdf9b7
Author: Tejun Heo <[email protected]>
sched_ext: Defer scx_sched kobj sysfs add into the enable workfns
This commit splits kobject initialization and addition for schedulers, moving the sysfs addition to the enable work functions. This ensures the scheduler is fully built before becoming visible in sysfs.
Quoted 2 lines
Prep so a future caps attribute can rely on @sch being fully built by the time it's sysfs-visible.
Does the implementation in scx_sub_enable_workfn() match this description? The commit message states that sysfs addition happens once the scheduler is fully built and its sysfs-visible state is initialized.
Quoted 4 lines
kernel/sched/ext/sub.c +0
-0
[ ... ]
kernel/sched/ext/sub.c void scx_sub_enable_workfn(struct kthread_work *work) · line 374
Quoted 11 lines
kernel/sched/ext/sub.c +4
-0@@ -363,6 +365,10 @@ void scx_sub_enable_workfn(struct kthread_work *work)if (ret) goto err_disable; + ret =scx_sched_sysfs_add (sch); +if (ret) + goto err_disable; +if (sch->level >=SCX_SUB_MAX_DEPTH ) {scx_error (sch,"max nesting depth %d violated" ,SCX_SUB_MAX_DEPTH );
[Severity: Medium]
Is it intentional that scx_sched_sysfs_add() is called here in scx_sub_enable_workfn(), before ops.init() is executed?
In scx_root_enable_workfn(), the sysfs addition is deferred until after ops.init() runs and SCX_EFLAG_INITIALIZED is set. However, in this sub-scheduler path, it is added to sysfs earlier, before ops.init() is invoked.
Will this premature exposure cause issues for future caps attributes that expect sub-schedulers to be fully initialized when visible in sysfs?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=14
Tejun Heo
reviewerRe: [PATCH v4 sched_ext/for-7.3 14/40] sched_ext: Defer scx_sched kobj sysfs add into the enable workfns
Quoted 4 lines · On Wed, 08 Jul 2026 21:47:19 +0000, [email protected] wrote:
- [Medium] Meaningful code and commit message mismatch: In sub-schedulers,
scx_sched_sysfs_add()is called beforeops.init(), contradicting the root scheduler's behavior and the commit message's goal.
Intentional asymmetry: the kobj only exposes ops, events and caps, and a sub's caps come from parent grants at runtime, not from its own ops.init(). The pshard cmasks that caps reads are allocated and zeroed before the sysfs add, so a pre-init read shows the truthful "no caps yet". Root defers past ops.init() because it seeds its own caps there. No change.
Thanks.
Signature · 1 line
tejun
Tejun Heo
reviewerPATCH v4 15/40 sched_ext: Add per-shard scx_sched storage scaffolding
In reply to Tejun HeoAdd struct scx_pshard and sch->pshard[] indexed by shard_idx, each entry allocated on its shard's NUMA node from scx_shard_node[si]. The struct starts empty (one dummy field). Follow-up patches will grow it as shard-local state lands. Only cid-type schedulers with an arena pool get pshards.
Allocation happens after ops.init_cids() returns so any scx_bpf_cid_override() it issues has finalized scx_nr_cid_shards and scx_shard_node[]. sch->nr_pshards records the array size for the async RCU free path, which may run after a later scheduler's scx_cid_init() has rewritten the global.
v3: Build and publish pshard[] fully-formed here rather than a later patch.
v2: Free the partially-allocated pshard array on alloc failure. (sashiko AI)
--- kernel/sched/ext/ext.c | 8 ++++++ kernel/sched/ext/internal.h | 18 +++++++++++++ kernel/sched/ext/sub.c | 54 +++++++++++++++++++++++++++++++++++++ kernel/sched/ext/sub.h | 4 +++ 4 files changed, 84 insertions(+)
kernel/sched/ext/ext.c +8 -0
@@ -4644,6 +4644,8 @@ static void scx_sched_free_rcu_work(struct work_struct *work)free_pnode (sch->pnode[node]);kfree (sch->pnode); +scx_free_pshards (sch); +rhashtable_walk_enter (&sch->dsq_hash, &rht_iter); do { rhashtable_walk_start(&rht_iter);
@@ -6724,6 +6726,12 @@ static void scx_root_enable_workfn(struct kthread_work *work) goto err_disable; } + ret =scx_alloc_pshards (sch); +if (ret) { +cpus_read_unlock (); + goto err_disable; + } +if (sch->ops.init) { ret =SCX_CALL_OP_RET (sch, init,NULL );if (ret) {
kernel/sched/ext/internal.h +18 -0
@@ -1183,6 +1183,12 @@ struct scx_sched_pnode {struct scx_dispatch_q global_dsq; }; +#ifdef CONFIG_EXT_SUB_SCHED +struct scx_pshard { +int _dummy;/* until the first real field lands */ +}; +#endif +struct scx_sched {/* * cpu-form and cid-form ops share field offsets up to .priv (verified
@@ -1230,6 +1236,9 @@ struct scx_sched {*/ struct rhashtable dsq_hash;struct scx_sched_pnode **pnode; +#ifdef CONFIG_EXT_SUB_SCHED +struct scx_pshard **pshard;/* indexed by shard_idx */ +#endif struct scx_sched_pcpu __percpu *pcpu;u64 slice_dfl;
@@ -1245,6 +1254,15 @@ struct scx_sched {u32 dsp_max_batch;s32 level; +#ifdef CONFIG_EXT_SUB_SCHED +/* +* pshard[] size captured at enable for the async RCU free path - +* scx_nr_cid_shards may be rewritten by a later scx_cid_init() before +* free runs. While sch is active, use the global. +*/ +u32 nr_pshards; +#endif +/* * Updates to the following warned bitfields can race causing RMW issues * but it doesn't really matter.
kernel/sched/ext/sub.c +54 -0
60 lines not on this page · raw message
kernel/sched/ext/sub.h +4 -0
@@ -24,6 +24,8 @@ void drain_descendants(struct scx_sched *sch);void scx_sub_disable (struct scx_sched *sch);void scx_sub_enable_workfn (struct kthread_work *work);bool scx_bpf_sub_dispatch (u64 cgroup_id,const struct bpf_prog_aux *aux); +void scx_free_pshards (struct scx_sched *sch); +s32 scx_alloc_pshards (struct scx_sched *sch);#else /* CONFIG_EXT_SUB_SCHED */
@@ -33,6 +35,8 @@ static inline struct cgroup *sch_cgroup(struct scx_sched *sch) { return NULL; }static inline void set_cgroup_sched (struct cgroup *cgrp,struct scx_sched *sch) {}static inline void drain_descendants (struct scx_sched *sch) { }static inline void scx_sub_disable (struct scx_sched *sch) { } +static inline void scx_free_pshards (struct scx_sched *sch) {} +static inline s32 scx_alloc_pshards (struct scx_sched *sch) {return 0 ; }#endif /* CONFIG_EXT_SUB_SCHED */
Signature · 1 line
2.54.0
[email protected]
AI reviewRe: [PATCH v4 sched_ext/for-7.3 15/40] sched_ext: Add per-shard scx_sched storage scaffolding
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] Missing scx_alloc_pshards() call for sub-schedulers in scx_sub_enable_workfn(), breaking shard…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] Missing scx_alloc_pshards() call for sub-schedulers in scx_sub_enable_workfn(), breaking shard scaffolding. --
--- Patch [15]: [PATCH v4 sched_ext/for-7.3 15/40] sched_ext: Add per-shard scx_sched storage scaffolding --- commit 3777e1c0aa64eae381accea1de5bddc71fe6cf2d
Author: Tejun Heo <[email protected]>
sched_ext: Add per-shard scx_sched storage scaffolding
This commit adds the scx_pshard structure and sch->pshard[] array indexed by shard_idx, allocating each entry on its shard's NUMA node. The allocation is intended for cid-type schedulers with an arena pool, serving as scaffolding for upcoming shard-local state.
Quoted 4 lines
kernel/sched/ext/ext.c +0
-0
[ ... ]
kernel/sched/ext/ext.c static void scx_root_enable_workfn(struct kthread_work *work) · line 6734
Quoted 10 lines
kernel/sched/ext/ext.c +6
-0@@ -6724,6 +6726,12 @@ static void scx_root_enable_workfn(struct kthread_work *work) goto err_disable; } + ret =scx_alloc_pshards (sch); +if (ret) { +cpus_read_unlock (); + goto err_disable; + } +
[Severity: Medium]
Is it intentional that scx_alloc_pshards() is called here for the root scheduler, but omitted in scx_sub_enable_workfn() for sub-schedulers?
The commit message notes: "Only cid-type schedulers with an arena pool get pshards." Sub-schedulers are cid-form and initialize an arena pool in scx_sub_enable_workfn(), but appear to be missing the allocation call:
kernel/sched/ext/sub.c:scx_sub_enable_workfn() {
...
ret = scx_arena_pool_init(sch);
if (ret)
goto err_disable;
ret = scx_set_cmask_scratch_alloc(sch);
if (ret)
goto err_disable;
if (scx_validate_ops(sch, ops))
goto err_disable;
...
}Could this cause sub-schedulers to fail to allocate the pshard array, leaving sch->pshard as NULL and leading to crashes in follow-up patches that rely on this scaffolding?
Quoted 3 lines
kernel/sched/ext/ext.c +0
-0if (sch->ops.init) { ret =SCX_CALL_OP_RET (sch, init,NULL );if (ret) {
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=15
Tejun Heo
reviewerRe: [PATCH v4 sched_ext/for-7.3 15/40] sched_ext: Add per-shard scx_sched storage scaffolding
Quoted 2 lines · On Wed, 08 Jul 2026 21:44:18 +0000, [email protected] wrote:
- [Medium] Missing
scx_alloc_pshards()call for sub-schedulers inscx_sub_enable_workfn(), breaking shard scaffolding.
Same as the v3 round - scaffolding plus the root allocation only. Sub pshard allocation is wired up with the capability feature in a later patch and nothing in between touches a sub's pshard[]:
https://lore.kernel.org/r/[email protected]
Thanks.
Signature · 1 line
tejun
Tejun Heo
reviewerPATCH v4 16/40 sched_ext: Add scx_cmask_ref for validated arena cmask access
In reply to Tejun Heokfuncs taking struct scx_cmask * from BPF arena memory have two problems. The pointer can be any value the BPF prog hands in, and the header (@base, @nr_cids, @alloc_words) can be mutated by the prog concurrently with kernel access.
Add scx_cmask_ref, a validated handle. _init() normalizes the input pointer into the arena's kern_vm range via scx_arena_to_kaddr() and snapshots the header, rejecting a range outside the machine or a nr_cids whose words exceed the declared @alloc_words. Downstream sizing uses the snapshot, not the live header. _shard() reads slices while _or() and _copy() write back, all bounded by the snapshot. No callers yet.
struct scx_cmask's bits[] carried __counted_by(alloc_words), so UBSAN_BOUNDS and FORTIFY_SOURCE bound accesses to the array. That bound is read from @alloc_words at the access. For an arena cmask @alloc_words is BPF-writable. A prog that sets it larger than the real allocation makes the check pass on a genuine overrun, so the annotation catches nothing, and it only runs under those debug configs. Drop it - _init() validates @alloc_words explicitly, and kernel-owned cmasks set it themselves.
v2: Validate @alloc_words in _init(), drop __counted_by. (Andrea, sashiko AI)
--- kernel/sched/ext/cid.c | 133 +++++++++++++++++++++++++++++++++++++++ kernel/sched/ext/cid.h | 7 +++ kernel/sched/ext/types.h | 39 +++++++++++- 3 files changed, 178 insertions(+), 1 deletion(-)
kernel/sched/ext/cid.c +133 -0
@@ -641,6 +641,12 @@ enum cmask_op2 {/* predicates - short-circuit when the per-word result is true */ CMASK_OP2_SUBSET ,CMASK_OP2_INTERSECTS , +/* +* @a is a BPF-arena cmask. Words on @a use READ_ONCE/WRITE_ONCE since +* BPF may read/write concurrently. See scx_cmask_ref_or() / _copy(). +*/ +CMASK_OP2_REF_OR , + CMASK_OP2_REF_COPY, }; static __always_inline bool cmask_op2_is_pred(const enum cmask_op2 op)
@@ -669,6 +675,12 @@ static __always_inline bool cmask_word_op2(u64 *av, const u64 *bp, u64 mask,return (READ_ONCE (*bp) & ~READ_ONCE (*av)) & mask;case CMASK_OP2_INTERSECTS :return (READ_ONCE (*av) &READ_ONCE (*bp)) & mask; +case CMASK_OP2_REF_OR : +WRITE_ONCE (*av,READ_ONCE (*av) | (READ_ONCE (*bp) & mask)); +return false; +case CMASK_OP2_REF_COPY : +WRITE_ONCE (*av, (READ_ONCE (*av) & ~mask) | (READ_ONCE (*bp) & mask)); +return false; }unreachable (); }
@@ -899,6 +911,127 @@ static const struct btf_kfunc_id_set scx_kfunc_set_cid = { .set = &scx_kfunc_ids_cid, }; +/** +* scx_cmask_ref_init - Bind a scx_cmask_ref to a BPF-arena cmask +* @sch: scheduler whose arena hosts @src +* @src: BPF-supplied cmask pointer +* @ref: output ref +* +* Snapshot @src's @base, @nr_cids and @alloc_words. The snapshot is necessary +* because BPF may mutate the live header asynchronously. +* +* Return 0 on success, -EINVAL if the range is out of bounds or @alloc_words +* doesn't cover it. +*/ +int scx_cmask_ref_init (struct scx_sched *sch,const struct scx_cmask *src, +struct scx_cmask_ref *ref) +{ +struct scx_cmask *kern_src =scx_arena_to_kaddr (sch, src); +u32 base, nr_cids, alloc_words, npossible =num_possible_cpus (); + + base =READ_ONCE (kern_src->base); + nr_cids =READ_ONCE (kern_src->nr_cids); + alloc_words =READ_ONCE (kern_src->alloc_words); + +if (unlikely (base >= npossible || nr_cids > npossible - base || +SCX_CMASK_NR_WORDS (nr_cids) > alloc_words)) +return -EINVAL ; + + ref->sch = sch; + ref->src = kern_src; + ref->base = base; + ref->nr_cids = nr_cids; + + ref->shard_first = scx_cid_to_shard[base]; +if (likely (nr_cids)) + ref->shard_end = scx_cid_to_shard[base + nr_cids -1 ] +1 ; +else + ref->shard_end = ref->shard_first; + +return 0 ; +} + +/** +* scx_cmask_ref_shard - Read one shard from @ref into @out +* @ref: validated ref +* @shard_idx: target shard, in [@ref->shard_first, @ref->shard_end) +* @out: output cmask whose @out->alloc_words must hold the shard +* +* Set @out to the intersection of @ref's range with @shard_idx's cid range, +* with bits[] read from @ref->src via READ_ONCE. Empty intersection sets +* @out->nr_cids to 0. scx_error()s on @ref's sched if @out can't hold the +* shard. +*/ +void scx_cmask_ref_shard (const struct scx_cmask_ref *ref,s32 shard_idx, +struct scx_cmask *out) +{ +const struct scx_cid_shard *shard = &scx_cid_shard_ranges[shard_idx]; +u32 shard_base = shard->base_cid; +u32 shard_end = shard_base + shard->nr_cids; +u32 isect_base, isect_end, nr_words, src_off, wi; +u64 head_mask, tail_mask; + + isect_base =max (ref->base, shard_base); + isect_end =min (ref->base + ref->nr_cids, shard_end); + +if (isect_base >= isect_end) { + out->base = shard_base; + out->nr_cids =0 ; +return ; + } + + nr_words = ((isect_end -1 ) /64 ) - (isect_base /64 ) +1 ; +if (nr_words > out->alloc_words) { +scx_error (ref->sch,"scx_cmask_ref_shard: out alloc_words=%u < %u for shard %d" , + out->alloc_words, nr_words, shard_idx); + out->base = shard_base; + out->nr_cids =0 ; +return ; + } + + out->base = isect_base; + out->nr_cids = isect_end - isect_base; + src_off = (isect_base /64 ) - (ref->base /64 ); + +for (wi =0 ; wi < nr_words; wi++) + out->bits[wi] =READ_ONCE (ref->src->bits[src_off + wi]); + + head_mask =GENMASK_U64 (63 , isect_base &63 ); + out->bits[0 ] &= head_mask; + tail_mask =GENMASK_U64 ((isect_end -1 ) &63 ,0 ); + out->bits[nr_words -1 ] &= tail_mask; +} + +/** +* scx_cmask_ref_or - OR @src into the arena cmask referenced by @ref +* @ref: validated ref +* @src: stable kernel cmask +* +* Bits inside the intersection of @ref's snapshotted range with @src's range +* are OR'd into @ref->src and bits outside are left unchanged. Stores on +* @ref->src use WRITE_ONCE since BPF may read/write concurrently. +*/ +void scx_cmask_ref_or (const struct scx_cmask_ref *ref,const struct scx_cmask *src) +{ +cmask_walk_op2 (ref->src->bits, ref->base, ref->nr_cids, + src->bits, src->base, src->nr_cids,CMASK_OP2_REF_OR ); +} + +/** +* scx_cmask_ref_copy - Copy @src into the arena cmask referenced by @ref +* @ref: validated ref +* @src: stable kernel cmask +* +* Bits inside the intersection of @ref's snapshotted range with @src's range +* take @src's values and bits outside are left unchanged. Stores on @ref->src +* use WRITE_ONCE since BPF may read/write concurrently. +*/ +void scx_cmask_ref_copy (const struct scx_cmask_ref *ref,const struct scx_cmask *src) +{ +cmask_walk_op2 (ref->src->bits, ref->base, ref->nr_cids, + src->bits, src->base, src->nr_cids,CMASK_OP2_REF_COPY ); +} +int scx_cid_kfunc_init (void ) { return register_btf_kfunc_id_set(BPF_PROG_TYPE_STRUCT_OPS, &scx_kfunc_set_init_cids) ?:
kernel/sched/ext/cid.h +7 -0
@@ -293,4 +293,11 @@ static inline s32 scx_cpu_ret(struct scx_sched *sch, s32 cpu_or_cid) return scx_cid_to_cpu(sch, cpu_or_cid); } +int scx_cmask_ref_init (struct scx_sched *sch,const struct scx_cmask *src, +struct scx_cmask_ref *ref); +void scx_cmask_ref_shard (const struct scx_cmask_ref *ref,s32 shard_idx, +struct scx_cmask *out); +void scx_cmask_ref_or (const struct scx_cmask_ref *ref,const struct scx_cmask *src); +void scx_cmask_ref_copy (const struct scx_cmask_ref *ref,const struct scx_cmask *src); +#endif /* _KERNEL_SCHED_EXT_CID_H */
kernel/sched/ext/types.h +38 -1
@@ -116,7 +116,7 @@ struct scx_cmask {u32 base;u32 nr_cids;u32 alloc_words; - u64 bits[] __counted_by(alloc_words); +u64 bits[]; };/*
@@ -172,4 +172,41 @@ struct scx_cmask {#define SCX_CMASK_DEFINE_SHARD (NAME ,BASE ,NR_CIDS ) \ __SCX_CMASK_DEFINE(NAME, BASE, NR_CIDS, SCX_CID_SHARD_MAX_CPUS) +/* +* scx_cmask_ref: validated reference to a BPF-arena cmask. +* +* scx_cmask_ref_init() normalizes the pointer into the arena and snapshots +* @base/@nr_cids. The snapshot is what downstream code uses for sizing - the +* live header can be mutated concurrently by BPF. +* +* scx_cmask_ref_shard() reads one shard into a cmask. scx_cmask_ref_or() and +* scx_cmask_ref_copy() write back into the referenced arena cmask, bounded by +* the snapshot. +* +* Typical input use: +* +* struct scx_cmask_ref ref; +* SCX_CMASK_DEFINE(shard, 0, SCX_CID_SHARD_MAX_CPUS); +* s32 idx, ret; +* +* ret = scx_cmask_ref_init(sch, src, &ref); +* if (ret < 0) +* return ret; +* +* for (idx = ref.shard_first; idx < ref.shard_end; idx++) { +* scx_cmask_ref_shard(&ref, idx, shard); +* if (!shard->nr_cids) +* continue; +* ... use idx and shard ... +* } +*/ +struct scx_cmask_ref { +struct scx_sched *sch; +struct scx_cmask *src; +u32 base; +u32 nr_cids; +s32 shard_first; +s32 shard_end; +}; +#endif /* _KERNEL_SCHED_EXT_TYPES_H */
Signature · 1 line
2.54.0
Tejun Heo
reviewerPATCH v4 17/40 sched_ext: Build the set_cmask scratch from trusted geometry
In reply to Tejun Heoscx_call_op_set_cpumask() builds a per-cpu cmask in the set_cmask scratch, which lives in BPF-writable arena. A scheduler can corrupt the scratch's inline header (base, nr_cids, alloc_words) from another cpu, so sizing and indexing the write from it risks an out-of-bounds write.
Drive the build from kernel-known geometry instead. scx_cmask_ref_init_kern() imposes base and nr_cids rather than reading them, and scx_cmask_ref_from_cpumask() fills the scratch from the ref. Neither reads the header back.
--- kernel/sched/ext/cid.c | 70 ++++++++++++++++++++++++++++++++++++++++++ kernel/sched/ext/cid.h | 4 +++ kernel/sched/ext/ext.c | 11 ++++--- 3 files changed, 81 insertions(+), 4 deletions(-)
kernel/sched/ext/cid.c +70 -0
@@ -951,6 +951,38 @@ int scx_cmask_ref_init(struct scx_sched *sch, const struct scx_cmask *src,return 0 ; } +/** +* scx_cmask_ref_init_kern - Bind a scx_cmask_ref to a kernel-owned cmask +* @sch: scheduler the cmask belongs to +* @m: kernel address of the target cmask, storage sized for @nr_cids at @base +* @base: first cid of the active range +* @nr_cids: active range length +* @ref: output ref +* +* Like scx_cmask_ref_init() but the geometry is supplied by the caller, not +* read from @m's header, so a concurrent BPF write to the header can't steer +* later sizing or offsets. Rewrite the header from the trusted geometry and +* bind @ref to it. +*/ +void scx_cmask_ref_init_kern (struct scx_sched *sch,struct scx_cmask *m, +u32 base,u32 nr_cids,struct scx_cmask_ref *ref) +{ +WRITE_ONCE (m->base, base); +WRITE_ONCE (m->nr_cids, nr_cids); +WRITE_ONCE (m->alloc_words,SCX_CMASK_NR_WORDS (nr_cids)); + + ref->sch = sch; + ref->src = m; + ref->base = base; + ref->nr_cids = nr_cids; + + ref->shard_first = scx_cid_to_shard[base]; +if (likely (nr_cids)) + ref->shard_end = scx_cid_to_shard[base + nr_cids -1 ] +1 ; +else + ref->shard_end = ref->shard_first; +} +/** * scx_cmask_ref_shard - Read one shard from @ref into @out * @ref: validated ref
@@ -1032,6 +1064,44 @@ void scx_cmask_ref_copy(const struct scx_cmask_ref *ref, const struct scx_cmask src->bits, src->base, src->nr_cids, CMASK_OP2_REF_COPY); } +/** +* scx_cmask_ref_from_cpumask - Populate @ref's arena cmask from a cpumask +* @ref: kern-bound ref, see scx_cmask_ref_init_kern() +* @cpumask: cpus to translate into cids +* +* Write @ref's active range one word at a time, setting each cid's bit when +* its cpu is in @cpumask. Offsets and length come from @ref's trusted geometry +* and stores use WRITE_ONCE since BPF may read concurrently, so the arena +* header is never read. +*/ +void scx_cmask_ref_from_cpumask (const struct scx_cmask_ref *ref, +const struct cpumask *cpumask) +{ +struct scx_cmask *m = ref->src; +u32 base = ref->base, nr_cids = ref->nr_cids; +u32 wi, nr_words; + +if (!nr_cids) +return ; + + nr_words = (base + nr_cids -1 ) /64 - base /64 +1 ; +for (wi =0 ; wi < nr_words; wi++) { +u32 word_first_cid = (base /64 + wi) *64 ; +u64 word =0 ; +u32 bit; + +for (bit =0 ; bit <64 ; bit++) { +u32 cid = word_first_cid + bit; + +if (cid < base || cid >= base + nr_cids) +continue ; +if (cpumask_test_cpu (__scx_cid_to_cpu (cid), cpumask)) + word |=BIT_U64 (bit); + } +WRITE_ONCE (m->bits[wi], word); + } +} +int scx_cid_kfunc_init (void ) { return register_btf_kfunc_id_set(BPF_PROG_TYPE_STRUCT_OPS, &scx_kfunc_set_init_cids) ?:
kernel/sched/ext/cid.h +4 -0
@@ -295,8 +295,12 @@ static inline s32 scx_cpu_ret(struct scx_sched *sch, s32 cpu_or_cid)int scx_cmask_ref_init (struct scx_sched *sch,const struct scx_cmask *src,struct scx_cmask_ref *ref); +void scx_cmask_ref_init_kern (struct scx_sched *sch,struct scx_cmask *m, +u32 base,u32 nr_cids,struct scx_cmask_ref *ref);void scx_cmask_ref_shard (const struct scx_cmask_ref *ref,s32 shard_idx,struct scx_cmask *out); +void scx_cmask_ref_from_cpumask (const struct scx_cmask_ref *ref, +const struct cpumask *cpumask);void scx_cmask_ref_or (const struct scx_cmask_ref *ref,const struct scx_cmask *src);void scx_cmask_ref_copy (const struct scx_cmask_ref *ref,const struct scx_cmask *src);
kernel/sched/ext/ext.c +7 -4
@@ -412,12 +412,15 @@ static inline void scx_call_op_set_cpumask(struct scx_sched *sch, struct rq *rq, {if (scx_is_cid_type ()) {struct scx_cmask *kern_va = *this_cpu_ptr (sch->set_cmask_scratch); +struct scx_cmask_ref ref; +/* -* Build the per-CPU arena cmask and hand BPF its arena address. -* Caller holds the rq lock with IRQs disabled, which makes us -* the sole user of the scratch area. +* Build the per-cpu arena cmask from kernel geometry via @ref, +* never reading its BPF-writable header, and hand BPF the arena +* address. The rq lock makes this cpu the sole kernel writer. */ -scx_cpumask_to_cmask (cpumask, kern_va); +scx_cmask_ref_init_kern (sch, kern_va,0 ,num_possible_cpus (), &ref); +scx_cmask_ref_from_cpumask (&ref, cpumask);SCX_CALL_CID_OP_TASK (sch, set_cmask, rq, task,scx_kaddr_to_arena (sch, kern_va)); }else {
Signature · 1 line
2.54.0
Tejun Heo
reviewerPATCH v4 18/40 sched_ext: RCU-protect the sub-sched tree's children/sibling lists
In reply to Tejun HeoFuture kfuncs need to walk descendants without scx_sched_lock. Make the walker RCU-safe so that they can. A sub-sched's fields are initialized before it is linked, so a walk that observes a linked node also observes its setup. In-place changes after linking carry their own ordering.
Switch the children/sibling list ops to RCU and expand the descendant walker to accept rcu_read_lock as a valid read-side context. Walkers that mutate keep scx_sched_lock.
A sub-sched can be linked while an ancestor is bypassing, after the bypass walk that propagates the depth has passed its parent. Bypass state is a per-cpu flag plus a depth count and can't be established atomically at link time, so refuse to link under a bypassing ancestor. Take scx_bypass_lock across linking to check the parent's bypass state coherently.
v3: Reject linking under a bypassing ancestor instead of inheriting bypass_depth. (sashiko AI)
v2: Inherit bypass_depth before publishing @sch on the RCU sibling list.
--- kernel/sched/ext/ext.c | 18 +++++++++++++++--- kernel/sched/ext/sub.c | 11 +++++++---- kernel/sched/ext/sub.h | 4 ++-- 3 files changed, 24 insertions(+), 9 deletions(-)
kernel/sched/ext/ext.c +15 -3
@@ -5465,7 +5465,8 @@ s32 scx_link_sched(struct scx_sched *sch)const char *err_msg ="" ;s32 ret =0 ; - scoped_guard(raw_spinlock_irq, &scx_sched_lock) { + scoped_guard(raw_spinlock_irqsave, &scx_bypass_lock) /* for the parent bypass check */ + scoped_guard(raw_spinlock, &scx_sched_lock) {#ifdef CONFIG_EXT_SUB_SCHED struct scx_sched *parent =scx_parent (sch);
@@ -5482,6 +5483,17 @@ s32 scx_link_sched(struct scx_sched *sch)break ; } +/* +* Bypass state is spread across per-cpu flags and a +* depth count, so inheriting it is tricky and has no +* valid use case. Refuse it. +*/ +if (READ_ONCE (parent->bypass_depth)) { + err_msg ="parent bypassing" ; + ret = -EBUSY ; +break ; + } + ret =rhashtable_lookup_insert_fast (&scx_sched_hash, &sch->hash_node, scx_sched_hash_params);if (ret) {
@@ -5489,7 +5501,7 @@ s32 scx_link_sched(struct scx_sched *sch) break; } -list_add_tail (&sch->sibling, &parent->children); +list_add_tail_rcu (&sch->sibling, &parent->children); }#endif /* CONFIG_EXT_SUB_SCHED */
@@ -5516,7 +5528,7 @@ void scx_unlink_sched(struct scx_sched *sch)if (scx_parent (sch)) {rhashtable_remove_fast (&scx_sched_hash, &sch->hash_node, scx_sched_hash_params); -list_del_init (&sch->sibling); +list_del_rcu (&sch->sibling); }#endif /* CONFIG_EXT_SUB_SCHED */ list_del_rcu (&sch->all);
kernel/sched/ext/sub.c +7 -4
@@ -35,21 +35,24 @@ struct scx_sched *scx_next_descendant_pre(struct scx_sched *pos, struct scx_schestruct scx_sched *next;lockdep_assert (lockdep_is_held (&scx_enable_mutex) || -lockdep_is_held (&scx_sched_lock)); +lockdep_is_held (&scx_sched_lock) || +rcu_read_lock_any_held ());/* if first iteration, visit @root */ if (!pos)return root;/* visit the first child if exists */ - next = list_first_entry_or_null(&pos->children, struct scx_sched, sibling); + next = list_first_or_null_rcu(&pos->children, struct scx_sched, sibling);if (next)return next;/* no child, visit my or the closest ancestor's next sibling */ while (pos != root) { -if (!list_is_last (&pos->sibling, &scx_parent (pos)->children)) -return list_next_entry (pos, sibling); + next =list_next_or_null_rcu (&scx_parent (pos)->children, &pos->sibling, + struct scx_sched, sibling); +if (next) +return next; pos =scx_parent (pos); }
kernel/sched/ext/sub.h +2 -2
@@ -46,8 +46,8 @@ static inline s32 scx_alloc_pshards(struct scx_sched *sch) { return 0; }* @root: sched to walk the descendants of * * Walk @root's descendants. @root is included in the iteration and the first -* node to be visited. Must be called with either scx_enable_mutex or -* scx_sched_lock held. +* node to be visited. Must be called with scx_enable_mutex, scx_sched_lock, or +* RCU read lock. */ #define scx_for_each_descendant_pre (pos, root) \ for ((pos) = scx_next_descendant_pre(NULL, (root)); (pos); \
Signature · 1 line
2.54.0
Tejun Heo
reviewerPATCH v4 19/40 sched_ext: Add scx_skip_subtree_pre()
In reply to Tejun HeoFactor the sibling/ancestor portion of scx_next_descendant_pre() out as scx_skip_subtree_pre(), a pre-order walk primitive that skips @pos's subtree, and call it from scx_next_descendant_pre(). Same locking rules as the existing primitive.
Used in a follow-up to fast-skip subtrees that have nothing to do during a descendant walk.
--- kernel/sched/ext/sub.c | 37 ++++++++++++++++++++++++++++--------- kernel/sched/ext/sub.h | 2 ++ 2 files changed, 30 insertions(+), 9 deletions(-)
kernel/sched/ext/sub.c +28 -9
@@ -21,6 +21,33 @@#ifdef CONFIG_EXT_SUB_SCHED +/** +* scx_skip_subtree_pre - Skip @pos's subtree in a pre-order walk +* @pos: current position +* @root: walk root +* +* In a walk started by scx_next_descendant_pre(), continue past @pos's subtree: +* return @pos's next sibling, or the closest ancestor's next sibling, or NULL +* if @pos's subtree is the last under @root. Same locking rules. +*/ +struct scx_sched *scx_skip_subtree_pre (struct scx_sched *pos,struct scx_sched *root) +{ +struct scx_sched *next; + +lockdep_assert (lockdep_is_held (&scx_enable_mutex) || +lockdep_is_held (&scx_sched_lock) || +rcu_read_lock_any_held ()); + +while (pos != root) { + next =list_next_or_null_rcu (&scx_parent (pos)->children, &pos->sibling, + struct scx_sched, sibling); +if (next) +return next; + pos =scx_parent (pos); + } +return NULL ; +} +/** * scx_next_descendant_pre - find the next descendant for pre-order walk * @pos: the current position (%NULL to initiate traversal)
@@ -48,15 +75,7 @@ struct scx_sched *scx_next_descendant_pre(struct scx_sched *pos, struct scx_schereturn next;/* no child, visit my or the closest ancestor's next sibling */ -while (pos != root) { - next =list_next_or_null_rcu (&scx_parent (pos)->children, &pos->sibling, - struct scx_sched, sibling); -if (next) -return next; - pos =scx_parent (pos); - } - -return NULL ; +return scx_skip_subtree_pre (pos, root); }static struct scx_sched *scx_find_sub_sched (u64 cgroup_id)
kernel/sched/ext/sub.h +2 -0
@@ -15,6 +15,7 @@#ifdef CONFIG_EXT_SUB_SCHED +struct scx_sched *scx_skip_subtree_pre (struct scx_sched *pos,struct scx_sched *root);struct scx_sched *scx_next_descendant_pre (struct scx_sched *pos,struct scx_sched *root);void scx_set_task_sched (struct task_struct *p,struct scx_sched *sch);struct cgroup *sch_cgroup (struct scx_sched *sch);
@@ -30,6 +31,7 @@ s32 scx_alloc_pshards(struct scx_sched *sch);#else /* CONFIG_EXT_SUB_SCHED */ static inline struct scx_sched *scx_next_descendant_pre (struct scx_sched *pos,struct scx_sched *root) {return pos ?NULL : root; } +static inline struct scx_sched *scx_skip_subtree_pre (struct scx_sched *pos,struct scx_sched *root) {return NULL ; }static inline void scx_set_task_sched (struct task_struct *p,struct scx_sched *sch) {}static inline struct cgroup *sch_cgroup (struct scx_sched *sch) {return NULL ; }static inline void set_cgroup_sched (struct cgroup *cgrp,struct scx_sched *sch) {}
Signature · 1 line
2.54.0
Tejun Heo
reviewerPATCH v4 20/40 sched_ext: Stop resolving a disabled scheduler's programs
In reply to Tejun HeoA scheduler's BPF programs can outlive it. A timer it armed or a tracing program it loaded can fire after ops.exit() has run, before the programs are unloaded, and scx_prog_sched() still resolves the program to its scheduler through ops->priv. Harmless while kfuncs touch only lifetime-stable state, but a hazard once a kfunc reads global state a newly loaded scheduler can change underneath it.
Add scx_sched->dead, set right after ops.exit() and drained with synchronize_rcu(). It follows exit() rather than preceding it so exit()'s own kfunc calls still resolve to @sch. scx_prog_sched() returns NULL for a dead scheduler, so every kfunc's existing !sch bail rejects it at one choke point.
--- kernel/sched/ext/ext.c | 8 ++++++++ kernel/sched/ext/internal.h | 13 ++++++++++--- kernel/sched/ext/sub.c | 9 +++++++++ 3 files changed, 27 insertions(+), 3 deletions(-)
kernel/sched/ext/ext.c +8 -0
@@ -5696,6 +5696,14 @@ static void scx_root_disable(struct scx_sched *sch)if (sch->ops.exit)SCX_CALL_OP (sch, exit,NULL , sch->exit_info); +/* +* @sch's programs (an armed timer, a tracing prog) can fire after +* exit(). Now that exit()'s own kfuncs have resolved, stop +* scx_prog_sched() from resolving to @sch and drain in-flight resolvers. +*/ +WRITE_ONCE (sch->dead, true); +synchronize_rcu (); +scx_unlink_sched (sch);/*
kernel/sched/ext/internal.h +10 -3
@@ -1202,6 +1202,7 @@ struct scx_sched {struct sched_ext_ops_cid ops_cid; };bool is_cid_type;/* true if registered via bpf_sched_ext_ops_cid */ +bool dead;/* set after ops.exit(), gates scx_prog_sched() */ /* * Arena map auto-discovered from member progs at struct_ops attach.
@@ -1968,14 +1969,20 @@ static inline bool scx_task_on_sched(struct scx_sched *sch,static inline struct scx_sched *scx_prog_sched (const struct bpf_prog_aux *aux) {struct sched_ext_ops *ops; -struct scx_sched *root; +struct scx_sched *sch, *root; ops =bpf_prog_get_assoc_struct_ops (aux); -if (likely (ops)) -return rcu_dereference_all (ops->priv); +if (likely (ops)) { + sch =rcu_dereference_all (ops->priv); +if (sch &&unlikely (READ_ONCE (sch->dead))) +return NULL ; +return sch; + } root =rcu_dereference_all (scx_root);if (root) { +if (unlikely (READ_ONCE (root->dead))) +return NULL ;/* * COMPAT-v6.19: Schedulers built before sub-sched support was * introduced may have unassociated non-struct_ops programs.
kernel/sched/ext/sub.c +9 -0
@@ -344,6 +344,15 @@ void scx_sub_disable(struct scx_sched *sch)if (sch->ops.exit)SCX_CALL_OP (sch, exit,NULL , sch->exit_info); + +/* +* @sch's programs (an armed timer, a tracing prog) can fire after +* exit(). Now that exit()'s own kfuncs have resolved, stop +* scx_prog_sched() from resolving to @sch and drain in-flight resolvers. +*/ +WRITE_ONCE (sch->dead, true); +synchronize_rcu (); +if (sch->sub_kset)kobject_del (&sch->sub_kset->kobj);/* not added if enable failed before scx_sched_sysfs_add() */
Signature · 1 line
2.54.0
[email protected]
AI reviewRe: [PATCH v4 sched_ext/for-7.3 20/40] sched_ext: Stop resolving a disabled scheduler's programs
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
Pre-existing issues:
- [High] The sch->dead check was omitted from the !CONFIG_EXT_SUB_SCHED fallback version of scx_prog_sched(), leaving the Use-After-Exit hazard unfixed.
- [High] A parent scheduler can invoke a child scheduler's ops.dispatch() after the child's ops.exit() has already executed, bypassing the newly introduced dead flag protection. --
--- Patch [20]: [PATCH v4 sched_ext/for-7.3 20/40] sched_ext: Stop resolving a disabled scheduler's programs --- commit 4eb6882510c7d2ef68878ed595f1b7ecec9ef3cc
Author: Tejun Heo <[email protected]>
sched_ext: Stop resolving a disabled scheduler's programs
A scheduler's BPF programs can outlive it. A timer it armed or a tracing program it loaded can fire after ops.exit() has run, before the programs are unloaded, and scx_prog_sched() still resolves the program to its scheduler through ops->priv. Harmless while kfuncs touch only lifetime-stable state, but a hazard once a kfunc reads global state a newly loaded scheduler can change underneath it.
Add scx_sched->dead, set right after ops.exit() and drained with synchronize_rcu(). It follows exit() rather than preceding it so exit()'s own kfunc calls still resolve to @sch. scx_prog_sched() returns NULL for a dead scheduler, so every kfunc's existing !sch bail rejects it at one choke point.
Quoted 4 lines
kernel/sched/ext/internal.h +0
-0
[ ... ]
Quoted 14 lines
kernel/sched/ext/internal.h +5
-3@@ -1968,14 +1969,20 @@ static inline bool scx_task_on_sched(struct scx_sched *sch,static inline struct scx_sched *scx_prog_sched (const struct bpf_prog_aux *aux) {struct sched_ext_ops *ops; -struct scx_sched *root; +struct scx_sched *sch, *root; ops =bpf_prog_get_assoc_struct_ops (aux); -if (likely (ops)) -return rcu_dereference_all (ops->priv); +if (likely (ops)) { + sch =rcu_dereference_all (ops->priv); +if (sch &&unlikely (READ_ONCE (sch->dead))) +return NULL ;
kernel/sched/ext/internal.h static inline bool scx_task_on_sched(struct scx_sched *sch, · line 1988
Quoted 10 lines
kernel/sched/ext/internal.h +4
-0+return sch; + } root =rcu_dereference_all (scx_root);if (root) { +if (unlikely (READ_ONCE (root->dead))) +return NULL ;/* * COMPAT-v6.19: Schedulers built before sub-sched support was * introduced may have unassociated non-struct_ops programs.
[Severity: High]
This isn't a bug introduced by this patch, but does the fallback definition of scx_prog_sched() also need the sch->dead check?
When the kernel is built with CONFIG_EXT_SUB_SCHED=n, scx_prog_sched() in kernel/sched/ext/internal.h is defined as:
static inline struct scx_sched *scx_prog_sched(const struct bpf_prog_aux *aux)
{
return rcu_dereference_all(scx_root);
}If a BPF timer fires during the synchronize_rcu() window in scx_root_disable(), it seems this fallback will unconditionally return scx_root without checking its dead flag, allowing kfuncs to execute on torn-down scheduler state.
Quoted 7 lines
kernel/sched/ext/sub.c +0
-0@@ -344,6 +344,15 @@ void scx_sub_disable(struct scx_sched *sch)if (sch->ops.exit)
kernel/sched/ext/sub.c void scx_sub_disable(struct scx_sched *sch) · line 355
Quoted 10 lines
kernel/sched/ext/sub.c +9
-0SCX_CALL_OP (sch, exit,NULL , sch->exit_info); + +/* +* @sch's programs (an armed timer, a tracing prog) can fire after +* exit(). Now that exit()'s own kfuncs have resolved, stop +* scx_prog_sched() from resolving to @sch and drain in-flight resolvers. +*/ +WRITE_ONCE (sch->dead, true); +synchronize_rcu (); +
[Severity: High]
This is a pre-existing issue, but can a parent scheduler bypass the newly introduced dead flag and invoke a torn-down child scheduler's ops.dispatch()?
If a parent scheduler calls scx_dispatch_sched(child) from a BPF program holding the RCU read lock while scx_sub_disable() is waiting on synchronize_rcu():
Parent BPF program scx_sub_disable()
child->ops.exit()
WRITE_ONCE(child->dead, true);
synchronize_rcu(); (blocks on parent)scx_find_sub_sched()scx_dispatch_sched(child)
child->ops.dispatch()
It looks like scx_dispatch_sched() unconditionally invokes the dispatch operation without checking the dead flag. Does scx_dispatch_sched() or scx_bpf_sub_dispatch() need to check if the child is already dead to prevent access to torn-down state?
Quoted 3 lines
kernel/sched/ext/sub.c +0
-0if (sch->sub_kset)kobject_del (&sch->sub_kset->kobj);/* not added if enable failed before scx_sched_sysfs_add() */
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=20
Tejun Heo
reviewerRe: [PATCH v4 sched_ext/for-7.3 20/40] sched_ext: Stop resolving a disabled scheduler's programs
Quoted 3 lines · On Wed, 08 Jul 2026 21:55:20 +0000, [email protected] wrote:
- [High] The
sch->deadcheck was omitted from the!CONFIG_EXT_SUB_SCHEDfallback version ofscx_prog_sched(), leaving the Use-After-Exit hazard unfixed.
Good catch. The =n stub skips the dead check, making the post-exit drain this patch adds a no-op there. Fixed in v5 - the stub now returns NULL for a dead root like the =y version.
Quoted 3 lines
This is a pre-existing issue, but can a parent scheduler bypass the newly introduced dead flag and invoke a torn-down child scheduler's
ops.dispatch()?
Not reachable: scx_sub_disable() unlinks the child from the sub hash before setting dead, so scx_find_sub_sched() already returns NULL in the claimed window and scx_bpf_sub_dispatch() bails before ops.dispatch(). That path is gated by hash membership; a dead check there would be redundant.
Thanks.
Signature · 1 line
tejun
Tejun Heo
reviewerPATCH v4 21/40 sched_ext: Add per-shard cap delegation for sub-schedulers
In reply to Tejun HeoCaps are per-cid permissions parents delegate to direct children via scx_bpf_sub_grant() / scx_bpf_sub_revoke(). A child's cap set is always a subset of its parent's. Sub-scheds check their caps locally, and cross-sched communication is needed only when the delegation set itself changes.
Caps will be used to implement sub-sched scheduling on the enqueue path. Picking a cid for a task at a leaf depends on which cids the leaf is allowed to use, and resolving that programmatically on every enqueue would mean a cross-sched round-trip call chain, possibly retrying if the request can't be granted as-is. The dispatch path is different - it runs as top-down recursion via scx_bpf_sub_dispatch().
Locking is per shard. cid space is split into shards, and each sub-sched has its own pshard->lock for each shard. Operations are broken up on shard boundaries. Different shards never contend. Shards are expected to be topology-aligned and likely to serve as the locality unit when cids are allocated to schedulers, so per-shard lock granularity scales naturally with the allocation pattern.
This patch adds the framework with a single dummy cap. Real caps land in later patches.
The enable path is reordered for pshards. scx_arena_pool_init() moves ahead of scx_link_sched() so the pshards are allocated before the sched becomes reachable - scx_alloc_pshards() skips allocation when the arena pool isn't initialized.
- scx_bpf_sub_grant(): Per-cid all-or-nothing grant to direct child.
- scx_bpf_sub_revoke(): Clear caps on @cmask across @child and its subtree.
- scx_bpf_sub_caps(): Lockless snapshot of caps on a cid range.
/sys/kernel/sched_ext/SCHED/caps shows the caps each scheduler currently holds.
v4: Move the pshard[] full build/publish and the err_disable scx_error() recording to earlier patches. (sashiko AI)
v3: Build pshard[] fully before publishing it, read it with READ_ONCE. (sashiko AI)
v2: Validate ops before scx_link_sched() publishes the sub. (sashiko AI)
--- kernel/sched/ext/ext.c | 77 ++++- kernel/sched/ext/internal.h | 56 +++- kernel/sched/ext/sub.c | 342 ++++++++++++++++++++++- kernel/sched/ext/sub.h | 2 + tools/sched_ext/include/scx/common.bpf.h | 6 + 5 files changed, 468 insertions(+), 15 deletions(-)
kernel/sched/ext/ext.c +74 -3
@@ -4705,9 +4705,52 @@ static ssize_t scx_attr_events_show(struct kobject *kobj, }SCX_ATTR (events); +#ifdef CONFIG_EXT_SUB_SCHED +static const char *scx_cap_names[__SCX_NR_CAPS] = { + [__SCX_CAP_DUMMY] ="dummy" , +}; + +static ssize_t scx_attr_caps_show (struct kobject *kobj, +struct kobj_attribute *ka,char *buf) +{ + struct scx_sched *sch = container_of(kobj, struct scx_sched, kobj); +u32 npossible =num_possible_cpus (); + struct scx_cmask *agg __free(kfree) = +kzalloc (struct_size (agg, bits,SCX_CMASK_NR_WORDS (npossible)),GFP_KERNEL ); + unsigned long *agg_bm __free(bitmap) = bitmap_zalloc(npossible, GFP_KERNEL); +ssize_t count =0 ; +s32 cap, si; + +if (!agg || !agg_bm) +return -ENOMEM ; + +for (cap =0 ; cap < __SCX_NR_CAPS; cap++) { +SCX_CMASK_DEFINE (snap,0 ,SCX_CID_SHARD_MAX_CPUS ); + +scx_cmask_init (agg,0 , npossible); +for (si =0 ; si < sch->nr_pshards; si++) { +struct scx_cmask *cm = &sch->pshard[si]->caps[cap].cmask; + +scx_cmask_reframe (snap, cm->base, cm->nr_cids); +scx_cmask_copy (snap, cm); +scx_cmask_or (agg, snap); + } +/* %*pbl takes unsigned long bitmap layout, convert from u64 */ +bitmap_from_arr64 (agg_bm, agg->bits, npossible); + count +=sysfs_emit_at (buf, count,"%s: %*pbl\n" , + scx_cap_names[cap], npossible, agg_bm); + } +return count; +} +SCX_ATTR (caps); +#endif /* CONFIG_EXT_SUB_SCHED */ +static struct attribute *scx_sched_attrs[] = { &scx_attr_ops.attr, &scx_attr_events.attr, +#ifdef CONFIG_EXT_SUB_SCHED + &scx_attr_caps.attr, +#endifNULL , };ATTRIBUTE_GROUPS (scx_sched);
@@ -6724,8 +6767,8 @@ static void scx_root_enable_workfn(struct kthread_work *work)/* * A cid-form scheduler finalizes its cid layout in ops.init_cids(), -* which may call scx_bpf_cid_override(). Run it before ops.init() so -* the final layout is in effect. +* which may call scx_bpf_cid_override(). Run it before the caps and +* shard state are built so the final layout is in effect. */ if (sch->is_cid_type && sch->ops_cid.init_cids) { ret =SCX_CALL_OP_RET (sch, init_cids,NULL );
@@ -6755,6 +6798,9 @@ static void scx_root_enable_workfn(struct kthread_work *work) goto err_disable; } +scx_init_root_caps (sch); + +/* the cid caps and shards are live now, so ops.init() can query them */ if (sch->ops.init) { ret =SCX_CALL_OP_RET (sch, init,NULL );if (ret) {
@@ -7436,7 +7482,7 @@ static struct bpf_struct_ops bpf_sched_ext_ops = {/* * cid-form cfi stubs. Stubs whose signatures match the cpu-form (param types -* identical, only param names differ across structs) are reused; only +* identical, only param names differ across structs) are reused. Only * set_cmask needs a fresh stub since the second argument type differs. */ static void sched_ext_ops_cid__set_cmask(struct task_struct *p,
@@ -9637,6 +9683,28 @@ __bpf_kfunc struct cgroup *scx_bpf_task_cgroup(struct task_struct *p, }#endif /* CONFIG_CGROUP_SCHED */ +#ifndef CONFIG_EXT_SUB_SCHED +__bpf_kfunc s32 scx_bpf_sub_grant(u64 cgroup_id, u64 caps, +const struct scx_cmask *cmask__ign, +struct scx_cmask *denied_out__ign, +const struct bpf_prog_aux *aux) +{ +return -EOPNOTSUPP ; +} + +__bpf_kfunc void scx_bpf_sub_revoke(u64 cgroup_id, u64 caps, +const struct scx_cmask *cmask__ign, +const struct bpf_prog_aux *aux) +{ +} + +__bpf_kfunc s32 scx_bpf_sub_caps(u64 cgroup_id, u64 caps, struct scx_cmask *out__ign, +const struct bpf_prog_aux *aux) +{ +return -EOPNOTSUPP ; +} +#endif /* !CONFIG_EXT_SUB_SCHED */ +__bpf_kfunc_end_defs ();BTF_KFUNCS_START (scx_kfunc_ids_any)
@@ -9681,6 +9749,9 @@ BTF_ID_FLAGS(func, scx_bpf_events)#ifdef CONFIG_CGROUP_SCHED BTF_ID_FLAGS(func, scx_bpf_task_cgroup, KF_IMPLICIT_ARGS | KF_RCU | KF_ACQUIRE)#endif +BTF_ID_FLAGS(func, scx_bpf_sub_grant, KF_IMPLICIT_ARGS) +BTF_ID_FLAGS(func, scx_bpf_sub_revoke, KF_IMPLICIT_ARGS) +BTF_ID_FLAGS(func, scx_bpf_sub_caps, KF_IMPLICIT_ARGS)BTF_KFUNCS_END (scx_kfunc_ids_any) static const struct btf_kfunc_id_set scx_kfunc_set_any = {
kernel/sched/ext/internal.h +52 -4
70 lines not on this page · raw message
kernel/sched/ext/sub.c +334 -8
376 lines not on this page · raw message
kernel/sched/ext/sub.h +2 -0
@@ -27,6 +27,7 @@ void scx_sub_enable_workfn(struct kthread_work *work);bool scx_bpf_sub_dispatch (u64 cgroup_id,const struct bpf_prog_aux *aux);void scx_free_pshards (struct scx_sched *sch);s32 scx_alloc_pshards (struct scx_sched *sch); +void scx_init_root_caps (struct scx_sched *sch);#else /* CONFIG_EXT_SUB_SCHED */
@@ -39,6 +40,7 @@ static inline void drain_descendants(struct scx_sched *sch) { }static inline void scx_sub_disable (struct scx_sched *sch) { }static inline void scx_free_pshards (struct scx_sched *sch) {}static inline s32 scx_alloc_pshards (struct scx_sched *sch) {return 0 ; } +static inline void scx_init_root_caps (struct scx_sched *sch) {}#endif /* CONFIG_EXT_SUB_SCHED */
tools/sched_ext/include/scx/common.bpf.h +6 -0
@@ -114,6 +114,12 @@ u32 scx_bpf_cidperf_cap(s32 cid) __ksym __weak; u32 scx_bpf_cidperf_cur(s32 cid) __ksym __weak; void scx_bpf_cidperf_set(s32 cid, u32 perf) __ksym __weak; +/* sub-scheduler cap control, scx_bpf_sub_caps() cgroup_id 0 == self */ +s32 scx_bpf_sub_grant(u64 cgroup_id, u64 caps, const struct scx_cmask *cmask, + struct scx_cmask *denied) __ksym __weak; +void scx_bpf_sub_revoke(u64 cgroup_id, u64 caps, const struct scx_cmask *cmask) __ksym __weak; +s32 scx_bpf_sub_caps(u64 cgroup_id, u64 caps, struct scx_cmask *out) __ksym __weak; +/* * Use the following as @it__iter when calling scx_bpf_dsq_move[_vtime]() from * within bpf_for_each() loops.
Signature · 1 line
2.54.0
[email protected]
AI reviewRe: [PATCH v4 sched_ext/for-7.3 21/40] sched_ext: Add per-shard cap delegation for sub-schedulers
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] Missing validation for unallocated `pshard` arrays and missing arena maps in…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] Missing validation for unallocated pshard arrays and missing arena maps in scx_bpf_sub_grant(), scx_bpf_sub_revoke(), and scx_bpf_sub_caps() leads to NULL pointer dereferences and invalid memory access Oops. --
--- Patch [21]: [PATCH v4 sched_ext/for-7.3 21/40] sched_ext: Add per-shard cap delegation for sub-schedulers --- commit 809108a1ad8d6e1bc475090c3e5ff6fba86f6bad
Author: Tejun Heo <[email protected]>sched_ext: Add per-shard cap delegation for sub-schedulers
Caps are per-cid permissions parents delegate to direct children via scx_bpf_sub_grant() / scx_bpf_sub_revoke(). A child's cap set is always a subset of its parent's. Sub-scheds check their caps locally, and cross-sched communication is needed only when the delegation set itself changes.
Quoted 4 lines
kernel/sched/ext/sub.c +0
-0
[ ... ]
Quoted 8 lines
kernel/sched/ext/sub.c +4
-0@@ -760,6 +808,284 @@ __bpf_kfunc bool scx_bpf_sub_dispatch(u64 cgroup_id, const struct bpf_prog_aux * true); } +/* Validate common inputs. On success, *parent_out and *child_out are set. */ +static s32 sub_cap_preamble (u64 cgroup_id,u64 caps,const struct bpf_prog_aux *aux, +struct scx_sched **parent_out,struct scx_sched **child_out) +{
[ ... ]
Quoted 7 lines
kernel/sched/ext/sub.c +7
-0+ *parent_out = parent; + *child_out = child; +return 0 ; +} + +/** +* scx_bpf_sub_grant - Grant @caps on @cmask__ign's cids to a direct child
[ ... ]
Quoted 5 lines
kernel/sched/ext/sub.c +5
-0+__bpf_kfunc s32 scx_bpf_sub_grant(u64 cgroup_id, u64 caps, +const struct scx_cmask *cmask__ign, +struct scx_cmask *denied_out__ign, +const struct bpf_prog_aux *aux) +{
[ ... ]
kernel/sched/ext/sub.c __bpf_kfunc bool scx_bpf_sub_dispatch(u64 cgroup_id, const struct bpf_prog_aux * · line 898
Quoted 5 lines
kernel/sched/ext/sub.c +5
-0+/* apply the grant one shard at a time */ +for (si = ref.shard_first; si < ref.shard_end; si++) { +SCX_CMASK_DEFINE_SHARD (slice,0 ,SCX_CID_SHARD_MAX_CPUS ); +struct scx_pshard *pps = parent->pshard[si]; +struct scx_pshard *cps = child->pshard[si];
[Severity: High]
Does this code dereference a NULL pointer if the child scheduler was created without a BPF arena?
If the child scheduler has no arena pool, scx_alloc_pshards() skips allocating the pshard array and leaves it as NULL. When scx_bpf_sub_grant() accesses child->pshard[si], it would attempt to dereference that NULL pointer.
Quoted 5 lines
kernel/sched/ext/sub.c +5
-0+u32 cap_bit; + +scx_cmask_ref_shard (&ref, si, slice); +if (scx_cmask_empty (slice)) +continue ;
[ ... ]
Quoted 4 lines
kernel/sched/ext/sub.c +4
-0+__bpf_kfunc void scx_bpf_sub_revoke(u64 cgroup_id, u64 caps, +const struct scx_cmask *cmask__ign, +const struct bpf_prog_aux *aux) +{
[ ... ]
kernel/sched/ext/sub.c __bpf_kfunc bool scx_bpf_sub_dispatch(u64 cgroup_id, const struct bpf_prog_aux * · line 985
Quoted 4 lines
kernel/sched/ext/sub.c +4
-0+guard (raw_spinlock)(&parent->pshard[si]->lock); + pos =scx_next_descendant_pre (NULL , child); +while (pos) { +struct scx_pshard *ps = pos->pshard[si];
[Severity: High]
Can pos->pshard be NULL here if one of the descendant schedulers was created without an arena map?
If a descendant scheduler lacks an arena, pos->pshard is NULL, leading to a NULL pointer dereference when accessing pos->pshard[si].
Quoted 5 lines
kernel/sched/ext/sub.c +5
-0+u64 revoked_caps =0 ; +u32 cap_bit; + + scoped_guard (raw_spinlock_nested, &ps->lock) { +scx_for_each_cap_bit (cap_bit, caps) {
[ ... ]
Quoted 3 lines
kernel/sched/ext/sub.c +3
-0+__bpf_kfunc s32 scx_bpf_sub_caps(u64 cgroup_id, u64 caps, struct scx_cmask *out__ign, +const struct bpf_prog_aux *aux) +{
[ ... ]
kernel/sched/ext/sub.c __bpf_kfunc bool scx_bpf_sub_dispatch(u64 cgroup_id, const struct bpf_prog_aux * · line 1071
Quoted 7 lines
kernel/sched/ext/sub.c +7
-0+ pshard =READ_ONCE (target->pshard); +if (unlikely (!pshard)) { +scx_error (sch,"scx_bpf_sub_caps() called before caps storage is initialized" ); +return -ENODEV ; + } + + ret =scx_cmask_ref_init (sch, out__ign, &ref);
[Severity: High]
Can this lead to an invalid memory access oops if the calling scheduler (sch) has no arena?
If sch does not have an arena map, its arena_kern_base is 0. scx_cmask_ref_init() uses this base when resolving the out__ign arena pointer, resulting in a read from a user-space address and causing a kernel page fault.
Quoted 4 lines
kernel/sched/ext/sub.c +4
-0+if (ret) { +scx_error (sch,"invalid out (%d)" , ret); +return ret; + }
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=21
Tejun Heo
reviewerRe: [PATCH v4 sched_ext/for-7.3 21/40] sched_ext: Add per-shard cap delegation for sub-schedulers
Quoted 4 lines · On Wed, 08 Jul 2026 22:00:56 +0000, [email protected] wrote:
- [High] Missing validation for unallocated
pshardarrays and missing arena maps inscx_bpf_sub_grant(),scx_bpf_sub_revoke(), andscx_bpf_sub_caps()leads to NULL pointer dereferences and invalid memory access Oops.
Same as the v2/v3 rounds - sub-schedulers are always cid-form with an arena pool (enforced at attach) and their pshards are allocated before the sched is published, so these paths never see a NULL pshard or a zero arena base:
https://lore.kernel.org/r/[email protected]
Thanks.
Signature · 1 line
tejun
Tejun Heo
reviewerPATCH v4 22/40 sched_ext: Add coalescing sub_caps_updated() notifier for sub-schedulers
In reply to Tejun HeoWire up ops_cid.sub_caps_updated() to notify sub-scheds of cap changes.
Three constraints shape the design:
1. Static memory. Deliveries use a fixed-size buffer, both for runtime efficiency and so notifications can't be lost under memory pressure.
2. High-frequency updates. Grant/revoke can mutate caps in bursts, and the notifier path must absorb that without amplifying it.
3. Recursive grant/revoke from the callback. A child receiving a notification can call grant/revoke on its own children, which can cascade recursively down its subtree.
(1) and (2) lead to coalescing into a fixed payload. Each delivery carries a single (cmask, caps) pair covering every change since the previous one. Direction (set vs cleared) isn't encoded as it doesn't fit in the fixed-size summary. The callback queries scx_bpf_sub_caps() for current state. Only one delivery is in flight per shard. Further changes fold into the same buffer and ship as the next callback, so a shard's callbacks fire in order.
(3) leads to deferred delivery. Events accumulate during grant/revoke and are delivered after the shard lock is released.
v2:
- Request a private stack for ops.sub_caps_updated(). (sashiko AI)
- Build cmask_arena_out via scx_cmask_ref, not by re-reading its header.
--- kernel/sched/ext/ext.c | 13 ++- kernel/sched/ext/internal.h | 71 ++++++++++++++ kernel/sched/ext/sub.c | 178 ++++++++++++++++++++++++++++++++++-- kernel/sched/ext/sub.h | 1 + 4 files changed, 254 insertions(+), 9 deletions(-)
kernel/sched/ext/ext.c +11 -2
@@ -7281,6 +7281,11 @@ static int bpf_scx_check_member(const struct btf_type *t,case offsetof(struct sched_ext_ops , dispatch): prog->aux->priv_stack_requested = true; prog->aux->recursion_detected = scx_pstack_recursion_on_dispatch; +break ; +case offsetof(struct sched_ext_ops , sub_caps_updated): + prog->aux->priv_stack_requested = true; + prog->aux->recursion_detected = scx_pstack_recursion_on_caps_updated; + break; }#endif /* CONFIG_EXT_SUB_SCHED */
@@ -7482,11 +7487,13 @@ static struct bpf_struct_ops bpf_sched_ext_ops = {/* * cid-form cfi stubs. Stubs whose signatures match the cpu-form (param types -* identical, only param names differ across structs) are reused. Only -* set_cmask needs a fresh stub since the second argument type differs. +* identical, only param names differ across structs) are reused. Some need +* fresh stubs, set_cmask due to an argument type difference and the sub-sched +* notifiers because no cpu-form stub exists to reuse. */ static void sched_ext_ops_cid__set_cmask (struct task_struct *p,const struct scx_cmask *cmask) {} +static void sched_ext_ops__sub_caps_updated (const struct scx_cmask *cmask,u64 caps) {}static struct sched_ext_ops_cid __bpf_ops_sched_ext_ops_cid = { .select_cid = sched_ext_ops__select_cpu,
@@ -7519,6 +7526,7 @@ static struct sched_ext_ops_cid __bpf_ops_sched_ext_ops_cid = {#endif .sub_attach = sched_ext_ops__sub_attach, .sub_detach = sched_ext_ops__sub_detach, + .sub_caps_updated = sched_ext_ops__sub_caps_updated, .cid_online = sched_ext_ops__cpu_online, .cid_offline = sched_ext_ops__cpu_offline, .init_cids = sched_ext_ops__init_cids,
@@ -9977,6 +9985,7 @@ static int __init scx_init(void)CID_OFFSET_MATCH (dump_task, dump_task);CID_OFFSET_MATCH (sub_attach, sub_attach);CID_OFFSET_MATCH (sub_detach, sub_detach); +CID_OFFSET_MATCH (sub_caps_updated, sub_caps_updated);CID_OFFSET_MATCH (init_cids, init_cids);CID_OFFSET_MATCH (init, init);CID_OFFSET_MATCH (exit, exit);
kernel/sched/ext/internal.h +71 -0
98 lines not on this page · raw message
kernel/sched/ext/sub.c +171 -7
283 lines not on this page · raw message
kernel/sched/ext/sub.h +1 -0
@@ -21,6 +21,7 @@ void scx_set_task_sched(struct task_struct *p, struct scx_sched *sch);struct cgroup *sch_cgroup (struct scx_sched *sch);void set_cgroup_sched (struct cgroup *cgrp,struct scx_sched *sch);void scx_pstack_recursion_on_dispatch (struct bpf_prog *prog); +void scx_pstack_recursion_on_caps_updated (struct bpf_prog *prog);void drain_descendants (struct scx_sched *sch);void scx_sub_disable (struct scx_sched *sch);void scx_sub_enable_workfn (struct kthread_work *work);
Signature · 1 line
2.54.0
Tejun Heo
reviewerPATCH v4 23/40 sched_ext: Maintain per-cpu effective cap copies for single-read checks
In reply to Tejun HeoChecking a sched's caps on a cid would need to test several cap bits against caps[] to account for implied caps. Also, caps[] modifications aren't synchronized against scheduling operations on each cpu, which can lead to awkward race conditions.
Collect them per cpu instead. caps[] under pshard->lock stays the target configuration. scx_sched_pcpu->ecaps is added, the transposed effective
copy: the set of cap bits the sched holds on that cpu which can be accessed with a single read. It is stable under the rq lock. It can also be read locklessly with READ_ONCE().
Grant and revoke only mutate caps[]. They queue a sync request on the target cpu's rq->scx.ecaps_to_sync and kick it, and the cpu recomputes the queued scheds' ecaps from caps[] in balance_one() under its own rq lock. A dying sched runs the sync directly to retire its queued request before freeing. As held references can defer the freeing past the enclosing root scheduler's lifetime, root enable discards leftover sync requests before going live.
--- kernel/sched/ext/ext.c | 10 +++ kernel/sched/ext/internal.h | 21 +++++- kernel/sched/ext/sub.c | 133 +++++++++++++++++++++++++++++++++++- kernel/sched/ext/sub.h | 16 +++++ kernel/sched/sched.h | 3 + 5 files changed, 180 insertions(+), 3 deletions(-)
kernel/sched/ext/ext.c +10 -0
@@ -2595,6 +2595,8 @@ static int balance_one(struct rq *rq, struct task_struct *prev) rq->scx.flags |=SCX_RQ_IN_BALANCE ; rq->scx.flags &= ~SCX_RQ_BAL_KEEP ; +scx_process_sync_ecaps (rq); +if ((sch->ops.flags &SCX_OPS_HAS_CPU_PREEMPT ) &&unlikely (rq->scx.cpu_released)) {/*
@@ -4627,6 +4629,9 @@ static void scx_sched_free_rcu_work(struct work_struct *work)*/ WARN_ON_ONCE (!list_empty (&pcpu->deferred_reenq_local.node)); +/* flush the queued ecaps syncs */ +scx_discard_ecaps_to_sync (cpu, pcpu); +/* * Bypass blocks new kicks. Flush the kick irq_work so this * pcpu's to_kick_node is off the list before it is freed.
@@ -6389,6 +6394,9 @@ struct scx_sched *scx_alloc_and_add_sched(struct scx_enable_cmd *cmd, node =cpu_to_node (cpu); pcpu->sch = sch;INIT_LIST_HEAD (&pcpu->deferred_reenq_local.node); +#ifdef CONFIG_EXT_SUB_SCHED +init_llist_node (&pcpu->ecaps_to_sync_node); +#endif INIT_LIST_HEAD (&pcpu->to_kick_node); if (!zalloc_cpumask_var_node(&pcpu->cpus_to_kick, GFP_KERNEL, node) || !zalloc_cpumask_var_node(&pcpu->cpus_to_kick_if_idle, GFP_KERNEL, node) ||
@@ -6733,6 +6741,8 @@ static void scx_root_enable_workfn(struct kthread_work *work) rq->scx.cpuperf_target =SCX_CPUPERF_ONE ; } +scx_discard_stale_ecaps_syncs (); +/* * Keep CPUs stable during enable so that the BPF scheduler can track * online CPUs by watching ->on/offline_cpu() after ->init().
kernel/sched/ext/internal.h +20 -1
@@ -1182,6 +1182,24 @@ struct scx_sched_pcpu {cpumask_var_t cpus_to_wait;struct list_head to_kick_node; +#ifdef CONFIG_EXT_SUB_SCHED +/* +* pshard->caps[cap_bit] is the set of cids the sched holds that one +* cap on. ecaps is its transpose: the set of SCX_CAP_* bits the sched +* holds on this cpu, collected so that the hot-path check is a single +* read. +* +* While pshard->caps[] under pshard->lock is the target configuration, +* ecaps is the effective copy owned by the cpu. It is written under the +* rq lock while processing rq->ecaps_to_sync. Can also be read with +* READ_ONCE() outside rq lock. +* +* See queue_sync_ecaps() and scx_process_sync_ecaps(). +*/ +u64 ecaps; +struct llist_node ecaps_to_sync_node; +#endif +/* * The event counters are in a per-CPU variable to minimize the * accounting overhead. A system-wide view on the event counter is
@@ -1291,7 +1309,8 @@ struct scx_pshard {/* * Per-cap cmask, inline via TRAILING_OVERLAP so cmask.bits[] overlaps -* the trailing _bits[] storage. Access as &caps[i].cmask. +* the trailing _bits[] storage. Access as &caps[i].cmask. See +* scx_sched_pcpu->ecaps. */ TRAILING_OVERLAP(struct scx_cmask, cmask, bits, u64 _bits[SCX_CMASK_NR_WORDS(SCX_CID_SHARD_MAX_CPUS)];
kernel/sched/ext/sub.c +131 -2
155 lines not on this page · raw message
kernel/sched/ext/sub.h +16 -0
@@ -29,6 +29,9 @@ bool scx_bpf_sub_dispatch(u64 cgroup_id, const struct bpf_prog_aux *aux);void scx_free_pshards (struct scx_sched *sch);s32 scx_alloc_pshards (struct scx_sched *sch);void scx_init_root_caps (struct scx_sched *sch); +void scx_process_sync_ecaps (struct rq *rq); +void scx_discard_ecaps_to_sync (s32 cpu,struct scx_sched_pcpu *pcpu); +void scx_discard_stale_ecaps_syncs (void );#else /* CONFIG_EXT_SUB_SCHED */
@@ -42,6 +45,9 @@ static inline void scx_sub_disable(struct scx_sched *sch) { }static inline void scx_free_pshards (struct scx_sched *sch) {}static inline s32 scx_alloc_pshards (struct scx_sched *sch) {return 0 ; }static inline void scx_init_root_caps (struct scx_sched *sch) {} +static inline void scx_process_sync_ecaps (struct rq *rq) {} +static inline void scx_discard_ecaps_to_sync (s32 cpu,struct scx_sched_pcpu *pcpu) {} +static inline void scx_discard_stale_ecaps_syncs (void ) {}#endif /* CONFIG_EXT_SUB_SCHED */
@@ -58,6 +64,16 @@ static inline void scx_init_root_caps(struct scx_sched *sch) {}for ((pos) =scx_next_descendant_pre (NULL , (root)); (pos); \ (pos) = scx_next_descendant_pre((pos), (root))) +#ifdef CONFIG_EXT_SUB_SCHED + +/* caps implied by holding @cap */ +static inline u64 scx_caps_implied (u64 cap) +{ +return 0 ; +} + +#endif /* CONFIG_EXT_SUB_SCHED */ +/* * One user of this function is scx_bpf_dispatch() which can be called * recursively as sub-sched dispatches nest. Always inline to reduce stack usage
kernel/sched/sched.h +3 -0
@@ -806,6 +806,9 @@ struct scx_rq {u32 flags;u32 nr_immed;/* ENQ_IMMED tasks on local_dsq */ u64 clock;/* current per-rq clock -- see scx_bpf_now() */ +#ifdef CONFIG_EXT_SUB_SCHED +struct llist_head ecaps_to_sync;/* pending ecaps syncs */ +#endif cpumask_var_t cpus_to_sync;bool kick_sync_pending;unsigned long kick_sync;
Signature · 1 line
2.54.0
[email protected]
AI reviewRe: [PATCH v4 sched_ext/for-7.3 23/40] sched_ext: Maintain per-cpu effective cap copies for single-read checks
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [High] Unconditional evaluation of `__scx_cpu_to_cid(cpu_of(rq))` and `scx_cid_to_shard[cid]` in…
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [High] Unconditional evaluation of __scx_cpu_to_cid(cpu_of(rq)) and scx_cid_to_shard[cid] in scx_process_sync_ecaps() causes a kernel crash (NULL pointer dereference) during initialization failures.
- [High] Naked and unchecked use of the global scx_root pointer in queue_sync_ecaps() causes a NULL pointer dereference if the root scheduler is concurrently disabled. --
--- Patch [23]: [PATCH v4 sched_ext/for-7.3 23/40] sched_ext: Maintain per-cpu effective cap copies for single-read checks --- commit 0633bda49e9d687035b71703d07c991f9fd41b6c
Author: Tejun Heo <[email protected]>
sched_ext: Maintain per-cpu effective cap copies for single-read checks
This commit introduces per-cpu effective capability copies to optimize capability checking in the scheduler. It collects the capabilities per cpu so they can be accessed locklessly or under the runqueue lock. Grant and revoke operations now queue sync requests which are processed by the target cpu.
Quoted 4 lines
kernel/sched/ext/sub.c +0
-0
[ ... ]
Quoted 7 lines
kernel/sched/ext/sub.c +7
-0+static void queue_sync_ecaps (struct scx_sched *sch,s32 cid) +{ +s32 cpu =__scx_cid_to_cpu (cid); +struct scx_sched_pcpu *pcpu =per_cpu_ptr (sch->pcpu, cpu); + +/* +* Pairs with smp_mb() in scx_process_sync_ecaps(). Either the check
kernel/sched/ext/sub.c static void scx_sub_seed_caps(struct scx_sched *sch) · line 335
Quoted 10 lines
kernel/sched/ext/sub.c +10
-0+* below sees the node off the list and queues it, or the in-flight sync +* sees the caps[] update made before this call. +*/ +smp_mb (); + +/* @cid's pshard->lock excludes concurrent queueing attempts */ +if (llist_on_list (&pcpu->ecaps_to_sync_node)) +return ; +if (llist_add (&pcpu->ecaps_to_sync_node, &cpu_rq (cpu)->scx.ecaps_to_sync)) +scx_kick_cpu (scx_root, cpu,0 );
[Severity: High]
Is it safe to pass scx_root directly here without checking for NULL?
If the root scheduler is concurrently being disabled, scx_root might be reset to NULL. Because BPF kfuncs like scx_bpf_sub_grant() can still be executing in sleepable or tracing contexts during unregistration, could this lead to a NULL pointer dereference inside scx_kick_cpu()?
Would using the local sch parameter instead resolve this safely?
Quoted 1 line
+}
[ ... ]
kernel/sched/ext/sub.c static void scx_sub_seed_caps(struct scx_sched *sch) · line 360
Quoted 4 lines
kernel/sched/ext/sub.c +4
-0+void scx_process_sync_ecaps (struct rq *rq) +{ +s32 cid =__scx_cpu_to_cid (cpu_of (rq)); +s32 shard = scx_cid_to_shard[cid];
[Severity: High]
Can these array indexing operations cause a kernel crash during early setup failures?
If the BPF scheduler enable path fails before scx_cid_init() completes (e.g., due to memory allocation failure), the global pointers like scx_cpu_to_cid_tbl and scx_cid_to_shard remain NULL.
The error recovery path invokes scx_discard_ecaps_to_sync(), which calls scx_process_sync_ecaps(). Because the cid evaluation and scx_cid_to_shard dereference happen unconditionally before the llist_empty() check below, this looks like it will cause a NULL pointer dereference.
Would moving the cid and shard evaluations after the llist_empty() check avoid this?
Quoted 9 lines
kernel/sched/ext/sub.c +9
-0+struct llist_node *batch, *pos, *tmp; + +lockdep_assert_rq_held (rq); + +if (likely (llist_empty (&rq->scx.ecaps_to_sync))) +return ; + + batch =llist_del_all (&rq->scx.ecaps_to_sync); +llist_for_each_safe (pos, tmp, batch) {
[ ... ]
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=23
Tejun Heo
reviewerRe: [PATCH v4 sched_ext/for-7.3 23/40] sched_ext: Maintain per-cpu effective cap copies for single-read checks
Quoted 3 lines · On Wed, 08 Jul 2026 22:03:27 +0000, [email protected] wrote:
- [High] Unconditional evaluation of
__scx_cpu_to_cid(cpu_of(rq))andscx_cid_to_shard[cid]inscx_process_sync_ecaps()causes a kernel crash (NULL pointer dereference) during initialization failures.
The llist_empty() check is the first statement in the function - the cid and shard evaluations sit behind it, and the teardown/discard path doesn't call scx_process_sync_ecaps() at all. Same as the earlier rounds:
https://lore.kernel.org/r/[email protected]
Quoted 3 lines
- [High] Naked and unchecked use of the global
scx_rootpointer inqueue_sync_ecaps()causes a NULL pointer dereference if the root scheduler is concurrently disabled.
The sub-cap kfuncs run under a function-wide guard(irqsave), which is an RCU read section. Root disable runs drain_descendants() and then sets dead + synchronize_rcu() before clearing scx_root, so an in-flight kfunc can't observe the NULL and a later one bails at scx_prog_sched(). Passing @sch instead would be wrong the other way - the kick drives the root-owned balance_one() and wants root's bypass state.
Thanks.
Signature · 1 line
tejun
Tejun Heo
reviewerPATCH v4 24/40 sched_ext: Add sub_ecaps_updated() effective-cap change notifier
In reply to Tejun HeoA sub-scheduler that gains or loses effective caps on a cpu may want to act on it right away - e.g. place or preempt on a newly usable cpu. The existing ops.sub_caps_updated() doesn't fit as it is delivered asynchronously to scheduling operations and can arrive before the per-cpu effective caps go live.
Add ops.sub_ecaps_updated(cid, before, after), a cid-form callback fired from scx_process_sync_ecaps() when a sub-sched's effective caps on a cid change. It runs in dispatch context so the sched can insert, kick or preempt on the cid directly. @before is the caps as of the last delivery.
Cpu hotplug rides the same machinery. Going down zeroes each sched's ecaps on the cpu's cid, with queued syncs discarded at consumption while the cpu is inactive. Coming back up queues a sync for every sched. reported_ecaps is kept across the down/up cycle, so the resync fires the callback only if ownership actually changed while the cpu was down.
v2: Compute cid below the active-cpu guard; discard queued syncs on !cpu_active(). (sashiko AI)
--- kernel/sched/ext/ext.c | 13 +++- kernel/sched/ext/internal.h | 19 +++++- kernel/sched/ext/sub.c | 129 +++++++++++++++++++++++++++++++++--- kernel/sched/ext/sub.h | 8 ++- 4 files changed, 154 insertions(+), 15 deletions(-)
kernel/sched/ext/ext.c +11 -2
@@ -2595,7 +2595,7 @@ static int balance_one(struct rq *rq, struct task_struct *prev) rq->scx.flags |=SCX_RQ_IN_BALANCE ; rq->scx.flags &= ~SCX_RQ_BAL_KEEP ; -scx_process_sync_ecaps (rq); +scx_process_sync_ecaps (rq, prev);if ((sch->ops.flags &SCX_OPS_HAS_CPU_PREEMPT ) &&unlikely (rq->scx.cpu_released)) {
@@ -3120,6 +3120,11 @@ static void handle_hotplug(struct rq *rq, bool online)if (scx_enabled ())scx_idle_update_selcpu_topology (&sch->ops); +if (online) +scx_online_ecaps (rq); +else +scx_offline_ecaps (rq); +if (online &&SCX_HAS_OP (sch, cpu_online))SCX_CALL_OP (sch, cpu_online,NULL ,scx_cpu_arg (cpu));else if (!online &&SCX_HAS_OP (sch, cpu_offline))
@@ -4629,7 +4634,7 @@ static void scx_sched_free_rcu_work(struct work_struct *work)*/ WARN_ON_ONCE (!list_empty (&pcpu->deferred_reenq_local.node)); -/* flush the queued ecaps syncs */ +/* retire the queued ecaps syncs so the pcpu can be freed */ scx_discard_ecaps_to_sync (cpu, pcpu);/*
@@ -7504,6 +7509,7 @@ static struct bpf_struct_ops bpf_sched_ext_ops = {static void sched_ext_ops_cid__set_cmask (struct task_struct *p,const struct scx_cmask *cmask) {}static void sched_ext_ops__sub_caps_updated (const struct scx_cmask *cmask,u64 caps) {} +static void sched_ext_ops__sub_ecaps_updated (s32 cid,u64 before,u64 after) {}static struct sched_ext_ops_cid __bpf_ops_sched_ext_ops_cid = { .select_cid = sched_ext_ops__select_cpu,
@@ -7537,6 +7543,7 @@ static struct sched_ext_ops_cid __bpf_ops_sched_ext_ops_cid = { .sub_attach = sched_ext_ops__sub_attach, .sub_detach = sched_ext_ops__sub_detach, .sub_caps_updated = sched_ext_ops__sub_caps_updated, + .sub_ecaps_updated = sched_ext_ops__sub_ecaps_updated, .cid_online = sched_ext_ops__cpu_online, .cid_offline = sched_ext_ops__cpu_offline, .init_cids = sched_ext_ops__init_cids,
@@ -9857,6 +9864,7 @@ static const u32 scx_kf_allow_flags[] = { #endif /* CONFIG_EXT_GROUP_SCHED */ [SCX_OP_IDX(sub_attach)] = SCX_KF_ALLOW_UNLOCKED, [SCX_OP_IDX(sub_detach)] = SCX_KF_ALLOW_UNLOCKED, + [SCX_OP_IDX(sub_ecaps_updated)] = SCX_KF_ALLOW_ENQUEUE | SCX_KF_ALLOW_DISPATCH, [SCX_OP_IDX(cpu_online)] = SCX_KF_ALLOW_UNLOCKED, [SCX_OP_IDX(cpu_offline)] = SCX_KF_ALLOW_UNLOCKED, [SCX_OP_IDX(init_cids)] = SCX_KF_ALLOW_UNLOCKED | SCX_KF_ALLOW_INIT_CIDS,
@@ -9996,6 +10004,7 @@ static int __init scx_init(void)CID_OFFSET_MATCH (sub_attach, sub_attach);CID_OFFSET_MATCH (sub_detach, sub_detach);CID_OFFSET_MATCH (sub_caps_updated, sub_caps_updated); +CID_OFFSET_MATCH (sub_ecaps_updated, sub_ecaps_updated);CID_OFFSET_MATCH (init_cids, init_cids);CID_OFFSET_MATCH (init, init);CID_OFFSET_MATCH (exit, exit);
kernel/sched/ext/internal.h +18 -1
@@ -770,12 +770,26 @@ struct sched_ext_ops {* Delivered asynchronously after the change is recorded, and may run * before it takes effect on any given cpu. Use it to track which caps * the sub-sched holds and propagate to its own children, not to decide -* if a task can run on a cpu now. +* if a task can run on a cpu now. sub_ecaps_updated() reports that per +* cpu, once it is in effect. * * May call scx_bpf_sub_grant() / scx_bpf_sub_revoke() on children. */ void (*sub_caps_updated)(const struct scx_cmask *cmask,u64 caps); +/** +* @sub_ecaps_updated: This sub-sched's effective caps on a cid changed +* @cid: the cid whose effective caps changed +* @before: effective caps as of the last delivery +* @after: effective caps now +* +* Invoked when this sub-sched's effective caps on @cid change, once the +* change is in effect on the cpu. Runs in dispatch context with rq lock +* held, and can perform all operations allowed in ops.dispatch() +* including inserting/moving tasks. +*/ +void (*sub_ecaps_updated)(s32 cid,u64 before,u64 after); +/* * All online ops must come before ops.cpu_online(). */
@@ -997,6 +1011,7 @@ struct sched_ext_ops_cid {s32 (*sub_attach)(struct scx_sub_attach_args *args);void (*sub_detach)(struct scx_sub_detach_args *args);void (*sub_caps_updated)(const struct scx_cmask *cmask,u64 caps); +void (*sub_ecaps_updated)(s32 cid,u64 before,u64 after);void (*cid_online)(s32 cid);void (*cid_offline)(s32 cid);s32 (*init_cids)(void);
@@ -1198,6 +1213,8 @@ struct scx_sched_pcpu {*/ u64 ecaps;struct llist_node ecaps_to_sync_node; +/* effective caps as of the last sub_ecaps_updated() delivery */ +u64 reported_ecaps;#endif /*
kernel/sched/ext/sub.c +119 -10
173 lines not on this page · raw message
kernel/sched/ext/sub.h +6 -2
@@ -29,7 +29,9 @@ bool scx_bpf_sub_dispatch(u64 cgroup_id, const struct bpf_prog_aux *aux);void scx_free_pshards (struct scx_sched *sch);s32 scx_alloc_pshards (struct scx_sched *sch);void scx_init_root_caps (struct scx_sched *sch); -void scx_process_sync_ecaps (struct rq *rq); +void scx_process_sync_ecaps (struct rq *rq,struct task_struct *prev); +void scx_online_ecaps (struct rq *rq); +void scx_offline_ecaps (struct rq *rq);void scx_discard_ecaps_to_sync (s32 cpu,struct scx_sched_pcpu *pcpu);void scx_discard_stale_ecaps_syncs (void );
@@ -45,7 +47,9 @@ static inline void scx_sub_disable(struct scx_sched *sch) { }static inline void scx_free_pshards (struct scx_sched *sch) {}static inline s32 scx_alloc_pshards (struct scx_sched *sch) {return 0 ; }static inline void scx_init_root_caps (struct scx_sched *sch) {} -static inline void scx_process_sync_ecaps (struct rq *rq) {} +static inline void scx_process_sync_ecaps (struct rq *rq,struct task_struct *prev) {} +static inline void scx_online_ecaps (struct rq *rq) {} +static inline void scx_offline_ecaps (struct rq *rq) {}static inline void scx_discard_ecaps_to_sync (s32 cpu,struct scx_sched_pcpu *pcpu) {}static inline void scx_discard_stale_ecaps_syncs (void ) {}
Signature · 1 line
2.54.0
[email protected]
AI reviewRe: [PATCH v4 sched_ext/for-7.3 24/40] sched_ext: Add sub_ecaps_updated() effective-cap change notifier
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] `scx_discard_ecaps_to_sync()` loops infinitely if the target CPU is starved by RT/DL tasks…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] scx_discard_ecaps_to_sync() loops infinitely if the target CPU is starved by RT/DL tasks, stalling scheduler teardown. --
--- Patch [24]: [PATCH v4 sched_ext/for-7.3 24/40] sched_ext: Add sub_ecaps_updated() effective-cap change notifier --- commit acc58dd0c8619664f61b598c2dccb6ac82d1cc5a
Author: Tejun Heo <[email protected]>
Subject: sched_ext: Add sub_ecaps_updated() effective-cap change notifier
This patch adds a notifier for sub-scheduler effective capability changes on a CPU. It queues syncs and processes them in dispatch context, handling CPU hotplug by zeroing capabilities when offline and re-syncing when online.
Quoted 4 lines
kernel/sched/ext/sub.c +0
-0
[ ... ]
Quoted 22 lines
void
scx_discard_ecaps_to_sync(s32 cpu, struct scx_sched_pcpu *pcpu){kernel/sched/ext/sub.c +16
-3- scoped_guard (rq_lock_irqsave, cpu_rq(cpu)) -scx_process_sync_ecaps (cpu_rq (cpu)); +struct rq *rq =cpu_rq (cpu); -WARN_ON_ONCE (llist_on_list (&pcpu->ecaps_to_sync_node)); +while (true) { + scoped_guard (rq_lock_irqsave, rq) { +/* +* scx_process_sync_ecaps() takes the node off the list +* before it is done accessing @pcpu but does all of it +* under the rq lock. Off-list observed under the rq +* lock guarantees that the sync is complete. +*/ +if (!llist_on_list (&pcpu->ecaps_to_sync_node)) +return ; +/* +* Discard only when the cpu is truly down. cpu_active() +* is already set when scx_online_ecaps() queues an online +* resync while SCX_RQ_ONLINE is not - so test cpu_active(), +* or that resync would be dropped.
kernel/sched/ext/sub.c void scx_process_sync_ecaps(struct rq *rq) · line 504
Quoted 10 lines
kernel/sched/ext/sub.c +9
-0+*/ +if (!scx_enabled () || !cpu_active (cpu)) { +discard_queued_syncs (rq); +return ; + } + } +resched_cpu (cpu); +msleep (1 ); + } }
[Severity: High]
Could this regression lead to an indefinite hang in scx_discard_ecaps_to_sync() if the target CPU is starved by RT or DL tasks?
If a high-priority RT or DL task monopolizes the target CPU (for example, if RT throttling is disabled), pick_next_task() may never fall through to the ext_sched_class.
This prevents pick_task_scx() and balance_one() from being executed on the target CPU, so pcpu->ecaps_to_sync_node is never processed and removed from the list. The system_dfl_wq worker running scx_sched_free_rcu_work() would then block forever in this while (true) loop, leaking the scx_sched resources and stalling the sub-scheduler teardown.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=24
Tejun Heo
reviewerRe: [PATCH v4 sched_ext/for-7.3 24/40] sched_ext: Add sub_ecaps_updated() effective-cap change notifier
Quoted 2 lines · On Wed, 08 Jul 2026 22:02:19 +0000, [email protected] wrote:
- [High]
scx_discard_ecaps_to_sync()loops infinitely if the target CPU is starved by RT/DL tasks, stalling scheduler teardown.
The loop only spins while the target cpu never runs balance_one(), and the ext class has DL-server forward progress, so it gets cpu time and drains the node even under saturating RT/DL load. A class receiving zero runtime forever (RT throttling fully disabled plus a spinning FIFO task) is a pre-existing system-level failure that breaks far more than this teardown path. No change.
Thanks.
Signature · 1 line
tejun
Tejun Heo
reviewerPATCH v4 25/40 sched_ext: Generalize local-DSQ handling to rq-owned DSQs
In reply to Tejun HeoThe local DSQ is synchronized by the containing rq lock rather than its own dsq->lock. A later patch adds a second such DSQ. In preparation, factor the "rq owns the lock" test into dsq_is_rq_owned() and rename local_dsq_post_enq() to rq_owned_post_enq(), taking @rq explicitly.
No behavior change.
--- kernel/sched/ext/ext.c | 34 ++++++++++++++++++++++------------ 1 file changed, 22 insertions(+), 12 deletions(-)
kernel/sched/ext/ext.c +22 -12
@@ -96,6 +96,17 @@ static atomic_long_t scx_hotplug_seq = ATOMIC_LONG_INIT(0);/* Global cursor for the per-CPU tid allocator. Starts at 1; tid 0 is reserved. */ static atomic64_t scx_tid_cursor =ATOMIC64_INIT (1 ); +/* is @dsq synchronized by the containing rq lock instead of dsq->lock? */ +static bool dsq_is_rq_owned (struct scx_dispatch_q *dsq) +{ +switch (dsq->id) { +case SCX_DSQ_LOCAL : +return true; +default : +return false; + } +} +#ifdef CONFIG_EXT_SUB_SCHED /* * The sub sched being enabled. Used by scx_disable_and_exit_task() to exit
@@ -1258,11 +1269,10 @@ static void call_task_dequeue(struct scx_sched *sch, struct rq *rq, p->scx.flags &= ~SCX_TASK_IN_CUSTODY ; } -static void local_dsq_post_enq (struct scx_sched *sch,struct scx_dispatch_q *dsq, -struct task_struct *p,u64 enq_flags) +static void rq_owned_post_enq (struct scx_sched *sch,struct rq *rq, +struct scx_dispatch_q *dsq,struct task_struct *p, +u64 enq_flags) { - struct rq *rq = container_of(dsq, struct rq, scx.local_dsq); -call_task_dequeue (sch, rq, p,0 );/*
@@ -1322,13 +1332,13 @@ static void scx_dispatch_enqueue(struct scx_sched *sch, struct rq *rq, struct scx_dispatch_q *dsq, struct task_struct *p, u64 enq_flags) { - bool is_local = dsq->id == SCX_DSQ_LOCAL; + bool is_rq_owned = dsq_is_rq_owned(dsq);WARN_ON_ONCE (p->scx.dsq || !list_empty (&p->scx.dsq_list.node));WARN_ON_ONCE ((p->scx.dsq_flags &SCX_TASK_DSQ_ON_PRIQ ) || !RB_EMPTY_NODE (&p->scx.dsq_priq)); -if (!is_local) { +if (!is_rq_owned) {raw_spin_lock_nested (&dsq->lock, (enq_flags &SCX_ENQ_NESTED ) ?SINGLE_DEPTH_NESTING :0 );
@@ -1423,8 +1433,8 @@ static void scx_dispatch_enqueue(struct scx_sched *sch, struct rq *rq,* ops_state first, both sides would modify p->scx.flags * concurrently in a non-atomic way. */ -if (is_local) { -local_dsq_post_enq (sch, dsq, p, enq_flags); +if (is_rq_owned) { +rq_owned_post_enq (sch, rq, dsq, p, enq_flags); }else {/* * Task on global/bypass DSQ: leave custody, task on
@@ -1471,7 +1481,7 @@ static void task_unlink_from_dsq(struct task_struct *p,static void scx_dispatch_dequeue (struct rq *rq,struct task_struct *p) {struct scx_dispatch_q *dsq = p->scx.dsq; -bool is_local = dsq == &rq->scx.local_dsq; +bool is_rq_owned = dsq &&dsq_is_rq_owned (dsq);lockdep_assert_rq_held (rq);
@@ -1495,7 +1505,7 @@ static void scx_dispatch_dequeue(struct rq *rq, struct task_struct *p)return ; } -if (!is_local) +if (!is_rq_owned)raw_spin_lock (&dsq->lock);/*
@@ -1517,7 +1527,7 @@ static void scx_dispatch_dequeue(struct rq *rq, struct task_struct *p) } p->scx.dsq =NULL ; -if (!is_local) +if (!is_rq_owned)raw_spin_unlock (&dsq->lock); }
@@ -2069,7 +2079,7 @@ static void move_local_task_to_local_dsq(struct scx_sched *sch,dsq_inc_nr (dst_dsq, p, enq_flags); p->scx.dsq = dst_dsq; -local_dsq_post_enq (sch, dst_dsq, p, enq_flags); +rq_owned_post_enq (sch, dst_rq, dst_dsq, p, enq_flags); }/**
Signature · 1 line
2.54.0
Tejun Heo
reviewerPATCH v4 26/40 sched_ext: Add reject DSQ for cap-rejected dispatches
In reply to Tejun HeoWhen a sub-scheduler dispatches a task to a CPU it lacks the required capability on, the task must be rejected rather than allowed to run.
Add the machinery for that. Each rq gets a reject DSQ, a kernel-internal holding queue that is never run and that the BPF scheduler cannot reach. An insert that must be refused is diverted there instead of the local DSQ, and a deferred requeue then hands the parked tasks back to the BPF scheduler to re-decide. A cap revoke extends this to already-queued tasks. When the revoke reaches the cpu's effective caps, the cpu scans its local DSQ and reenqueues the tasks that no longer qualify.
A migration-disabled task must run on its cpu, so a capless one is admitted anyway and counted in the new SCX_EV_SUB_FORCED_ADMIT event.
This is preparation for the actual sub-sched cap enforcement. The divert is wired but inert here.
v2: Admit offline-rq and migration_pending inserts to local, not reject. (sashiko AI)
--- include/linux/sched/ext.h | 15 ++++- kernel/sched/ext/ext.c | 42 ++++++++++-- kernel/sched/ext/internal.h | 19 +++++- kernel/sched/ext/sub.c | 127 +++++++++++++++++++++++++++++++++++- kernel/sched/ext/sub.h | 49 ++++++++++++++ kernel/sched/sched.h | 3 + 6 files changed, 244 insertions(+), 11 deletions(-)
include/linux/sched/ext.h +13 -2
@@ -58,6 +58,7 @@ enum scx_dsq_id_flags {SCX_DSQ_GLOBAL =SCX_DSQ_FLAG_BUILTIN |1 ,SCX_DSQ_LOCAL =SCX_DSQ_FLAG_BUILTIN |2 ,SCX_DSQ_BYPASS =SCX_DSQ_FLAG_BUILTIN |3 , +SCX_DSQ_REJECT =SCX_DSQ_FLAG_BUILTIN |4 ,/* internal - see find_dsq_for_dispatch() */ SCX_DSQ_LOCAL_ON =SCX_DSQ_FLAG_BUILTIN |SCX_DSQ_FLAG_LOCAL_ON , SCX_DSQ_LOCAL_CPU_MASK = 0xffffffffLLU, };
@@ -124,7 +125,7 @@ enum scx_ent_flags { SCX_TASK_DEAD = 5 << SCX_TASK_STATE_SHIFT,/* -* Bits 12 and 13 are used to carry reenqueue reason. In addition to +* Bits 12 to 14 are used to carry reenqueue reason. In addition to * %SCX_ENQ_REENQ flag, ops.enqueue() can also test for * %SCX_TASK_REENQ_REASON_NONE to distinguish reenqueues. *
@@ -132,15 +133,17 @@ enum scx_ent_flags {* KFUNC reenqueued by scx_bpf_dsq_reenq() and friends * IMMED reenqueued due to failed ENQ_IMMED * PREEMPTED preempted while running +* CAP sub-sched cap miss, see p->scx.reenq_reason_* */ SCX_TASK_REENQ_REASON_SHIFT =12 , -SCX_TASK_REENQ_REASON_BITS =2 , +SCX_TASK_REENQ_REASON_BITS =3 ,SCX_TASK_REENQ_REASON_MASK = ((1 <<SCX_TASK_REENQ_REASON_BITS ) -1 ) <<SCX_TASK_REENQ_REASON_SHIFT ,SCX_TASK_REENQ_NONE =0 <<SCX_TASK_REENQ_REASON_SHIFT ,SCX_TASK_REENQ_KFUNC =1 <<SCX_TASK_REENQ_REASON_SHIFT ,SCX_TASK_REENQ_IMMED =2 <<SCX_TASK_REENQ_REASON_SHIFT ,SCX_TASK_REENQ_PREEMPTED =3 <<SCX_TASK_REENQ_REASON_SHIFT , +SCX_TASK_REENQ_CAP =4 <<SCX_TASK_REENQ_REASON_SHIFT ,/* iteration cursor, not a task */ SCX_TASK_CURSOR = 1 << 31,
@@ -239,6 +242,14 @@ struct sched_ext_entity {*/ u64 dsq_vtime; +/* +* Sub-sched cap rejected reenq context, valid only while +* %SCX_TASK_REENQ_CAP is set. @reenq_reason_caps is the SCX_CAP_* bits +* that were needed but missing. @reenq_reason_cid is the target cid. +*/ +u64 reenq_reason_caps; +s32 reenq_reason_cid; +/* * If set, reject future sched_setscheduler(2) calls updating the policy * to %SCHED_EXT with -%EACCES.
kernel/sched/ext/ext.c +35 -7
114 lines not on this page · raw message
kernel/sched/ext/internal.h +18 -1
@@ -1135,6 +1135,13 @@ struct scx_event_stats {* from sub_bypass_dsq's. */ s64 SCX_EV_SUB_BYPASS_DISPATCH ; + +/* +* The number of times a migration-disabled task lacking the cap for its +* cid was allowed onto the local DSQ. It must run on its pinned CPU, so +* it can't be rejected. The violation is counted here. +*/ +s64 SCX_EV_SUB_FORCED_ADMIT ; };#define SCX_EVENTS_LIST (SCX_EVENT ) \
@@ -1150,7 +1157,8 @@ struct scx_event_stats {SCX_EVENT (SCX_EV_BYPASS_DISPATCH ); \SCX_EVENT (SCX_EV_BYPASS_ACTIVATE ); \SCX_EVENT (SCX_EV_INSERT_NOT_OWNED ); \ -SCX_EVENT (SCX_EV_SUB_BYPASS_DISPATCH) +SCX_EVENT (SCX_EV_SUB_BYPASS_DISPATCH ); \ +SCX_EVENT (SCX_EV_SUB_FORCED_ADMIT) struct scx_sched;
@@ -1270,6 +1278,9 @@ enum scx_cap_flags { __SCX_CAP_ALL =BIT_U64 (__SCX_NR_CAPS) -1 ,SCX_CAP_DUMMY =BIT_U64 (__SCX_CAP_DUMMY), + +/* caps whose loss strands queued tasks, see scx_process_sync_ecaps() */ + SCX_CAPS_REENQ_ON_LOSS = 0, };#ifdef CONFIG_EXT_SUB_SCHED
@@ -1583,6 +1594,9 @@ enum scx_reenq_flags {/* low 16bits determine which tasks should be reenqueued */ SCX_REENQ_ANY =1LLU <<0 ,/* all tasks */ +/* internal: kernel-issued on cap revoke, not accepted from BPF */ +SCX_REENQ_CAP_REVOKE =1LLU <<1 , + __SCX_REENQ_FILTER_MASK =0xffffLLU , __SCX_REENQ_USER_MASK = SCX_REENQ_ANY,
@@ -1835,6 +1849,9 @@ void scx_task_iter_start(struct scx_task_iter *iter, struct cgroup *cgrp);void scx_task_iter_unlock (struct scx_task_iter *iter);void scx_task_iter_stop (struct scx_task_iter *iter);struct task_struct *scx_task_iter_next_locked (struct scx_task_iter *iter); +void scx_dispatch_dequeue (struct rq *rq,struct task_struct *p); +void scx_do_enqueue_task (struct rq *rq,struct task_struct *p,u64 enq_flags, +int sticky_cpu);bool scx_consume_dispatch_q (struct scx_sched *sch,struct rq *rq,struct scx_dispatch_q *dsq,u64 enq_flags);bool scx_consume_global_dsq (struct scx_sched *sch,struct rq *rq);
kernel/sched/ext/sub.c +126 -1
160 lines not on this page · raw message
kernel/sched/ext/sub.h +49 -0
73 lines not on this page · raw message
kernel/sched/sched.h +3 -0
@@ -794,6 +794,9 @@ enum scx_rq_flags {struct scx_rq {struct scx_dispatch_q local_dsq; +#ifdef CONFIG_EXT_SUB_SCHED +struct scx_dispatch_q reject_dsq;/* staging for cap-rejected tasks */ +#endif struct list_head runnable_list;/* runnable tasks on this rq */ struct list_head ddsp_deferred_locals;/* deferred ddsps from enq */ unsigned long ops_qseq;
Signature · 1 line
2.54.0
Tejun Heo
reviewerPATCH v4 27/40 sched_ext: Add the SCX_CAP_ENQ_IMMED cap
In reply to Tejun HeoReplace the __SCX_CAP_DUMMY placeholder with SCX_CAP_ENQ_IMMED, which gates inserting IMMED tasks onto a cid's local DSQ. An IMMED enqueue is guaranteed to either get its task running on the cpu at once or hand it back to the scheduler, so IMMED work can never pile up on the cpu's queue and a cpu can be shared across sub-scheds through IMMED access without any of them swamping it.
That makes ENQ_IMMED the natural baseline, the minimal cap to make any use of a cpu. SCX_CAP_BASE aliases it so gates on basic cpu access can state the intention instead of naming ENQ_IMMED.
Enforcement covers inserts and queued tasks. An insert without the cap is diverted to the reject DSQ, and queued tasks are reenqueued when the cap is lost. scx_bpf_sub_dispatch() skips a child that lacks the cap on the cpu, as its inserts would only be rejected. Vacating the running task on cap loss lands in a later patch.
--- kernel/sched/ext/ext.c | 2 +- kernel/sched/ext/internal.h | 11 ++++++++--- kernel/sched/ext/sub.c | 8 ++++++++ kernel/sched/ext/sub.h | 4 ++-- 4 files changed, 19 insertions(+), 6 deletions(-)
kernel/sched/ext/ext.c +1 -1
@@ -4752,7 +4752,7 @@ SCX_ATTR(events);#ifdef CONFIG_EXT_SUB_SCHED static const char *scx_cap_names[__SCX_NR_CAPS] = { - [__SCX_CAP_DUMMY] ="dummy" , + [__SCX_CAP_ENQ_IMMED] ="enq_immed" , }; static ssize_t scx_attr_caps_show(struct kobject *kobj,
kernel/sched/ext/internal.h +8 -3
@@ -1270,17 +1270,22 @@ struct scx_sched_pnode {* topology-aligned and likely to serve as the locality unit when cids are * allocated to schedulers, so per-shard lock granularity scales naturally with * the allocation pattern. +* +* ENQ_IMMED insert an IMMED task onto the cid's local DSQ */ enum scx_cap_flags { - __SCX_CAP_DUMMY =0 , + __SCX_CAP_ENQ_IMMED =0 , __SCX_NR_CAPS, __SCX_CAP_ALL =BIT_U64 (__SCX_NR_CAPS) -1 , -SCX_CAP_DUMMY =BIT_U64 (__SCX_CAP_DUMMY), +SCX_CAP_ENQ_IMMED =BIT_U64 (__SCX_CAP_ENQ_IMMED), + +/* alias for minimal cap to make any use of a cpu */ +SCX_CAP_BASE =SCX_CAP_ENQ_IMMED ,/* caps whose loss strands queued tasks, see scx_process_sync_ecaps() */ -SCX_CAPS_REENQ_ON_LOSS =0 , +SCX_CAPS_REENQ_ON_LOSS =SCX_CAP_ENQ_IMMED , };#ifdef CONFIG_EXT_SUB_SCHED
kernel/sched/ext/sub.c +8 -0
@@ -1279,6 +1279,14 @@ __bpf_kfunc bool scx_bpf_sub_dispatch(u64 cgroup_id, const struct bpf_prog_aux *return false; } +/* +* Skip a child that does not effectively hold the base cap on this cpu: +* its inserts would only be rejected. ecaps are synced at the top of +* balance_one() before dispatch, so this reflects the in-effect state. +*/ +if (scx_missing_caps (child,cpu_of (this_rq),SCX_CAP_BASE )) +return false; +return scx_dispatch_sched (child, this_rq, this_rq->scx.sub_dispatch_prev, true); }
kernel/sched/ext/sub.h +2 -2
@@ -106,13 +106,13 @@ static inline u64 scx_missing_caps(struct scx_sched *sch, s32 cpu, u64 needed)/* map @enq_flags to the SCX_CAP_* bit required for the local-DSQ insert */ static inline u64 scx_caps_for_enq (u64 enq_flags) { -return 0 ; +return SCX_CAP_ENQ_IMMED ; }/* map queued @p to the SCX_CAP_* bit required to stay on its local DSQ */ static inline u64 scx_caps_for_task (struct task_struct *p) { -return 0 ; +return SCX_CAP_ENQ_IMMED ; }/* caps implied by holding @cap */
Signature · 1 line
2.54.0
[email protected]
AI reviewRe: [PATCH v4 sched_ext/for-7.3 27/40] sched_ext: Add the SCX_CAP_ENQ_IMMED cap
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [High] scx_caps_for_enq() unconditionally maps all local DSQ enqueues to SCX_CAP_ENQ_IMMED, allowing…
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [High] scx_caps_for_enq() unconditionally maps all local DSQ enqueues to SCX_CAP_ENQ_IMMED, allowing sub-schedulers to bypass isolation guarantees and swamp the CPU with non-IMMED tasks.
- [High] Tasks in a bypassing descendant's runqueue can be permanently stranded if their nearest non-bypassing ancestor lacks SCX_CAP_BASE on the CPU.
--
--- Patch [27]: [PATCH v4 sched_ext/for-7.3 27/40] sched_ext: Add the SCX_CAP_ENQ_IMMED cap --- commit 5bf5e66c851f3131250d2bde5c5d59c7f1533314
Author: Tejun Heo <[email protected]>
sched_ext: Add the SCX_CAP_ENQ_IMMED cap
This commit introduces the SCX_CAP_ENQ_IMMED capability to replace the dummy placeholder, acting as a gate for inserting IMMED tasks into a local DSQ. It establishes ENQ_IMMED as the baseline capability required to use a CPU, enforcing that inserts without this cap are rejected or tasks reenqueued.
Quoted 9 lines
kernel/sched/ext/sub.c +1
-0@@ -1279,6 +1279,14 @@ __bpf_kfunc bool scx_bpf_sub_dispatch(u64 cgroup_id, const struct bpf_prog_aux *return false; } +/*
kernel/sched/ext/sub.c __bpf_kfunc bool scx_bpf_sub_dispatch(u64 cgroup_id, const struct bpf_prog_aux * · line 1292
Quoted 10 lines
kernel/sched/ext/sub.c +9
-0+* Skip a child that does not effectively hold the base cap on this cpu: +* its inserts would only be rejected. ecaps are synced at the top of +* balance_one() before dispatch, so this reflects the in-effect state. +*/ +if (scx_missing_caps (child,cpu_of (this_rq),SCX_CAP_BASE )) +return false; + +return scx_dispatch_sched (child, this_rq, this_rq->scx.sub_dispatch_prev, + true); }
[Severity: High]
Could this early return cause a regression that permanently strands tasks in a bypassing descendant's runqueue?
If a grandchild scheduler enters bypass mode, its enqueued tasks are routed to the per-CPU bypass DSQ of its nearest non-bypassing ancestor (the child scheduler). If that child scheduler then loses the SCX_CAP_BASE capability on this specific CPU, this check returns false and skips scx_dispatch_sched().
Since bypass DSQs are strictly per-CPU and only consumed via scx_dispatch_sched(), will the child's bypass DSQ remain unpumped?
Capability revocation via scx_process_sync_ecaps() only schedules a re-enqueue scan for the local DSQ, so it seems tasks trapped in the bypass DSQ would never be rescued.
[ ... ]
Quoted 8 lines
kernel/sched/ext/sub.h +0
-0@@ -106,13 +106,13 @@ static inline u64 scx_missing_caps(struct scx_sched *sch, s32 cpu, u64 needed)/* map @enq_flags to the SCX_CAP_* bit required for the local-DSQ insert */ static inline u64 scx_caps_for_enq (u64 enq_flags) {
kernel/sched/ext/sub.h static inline u64 scx_missing_caps(struct scx_sched *sch, s32 cpu, u64 needed) · line 116
Quoted 10 lines
kernel/sched/ext/sub.h +2
-2-return 0 ; +return SCX_CAP_ENQ_IMMED ; }/* map queued @p to the SCX_CAP_* bit required to stay on its local DSQ */ static inline u64 scx_caps_for_task (struct task_struct *p) { -return 0 ; +return SCX_CAP_ENQ_IMMED ; }
[Severity: High]
Does unconditionally returning SCX_CAP_ENQ_IMMED introduce a regression allowing sub-schedulers to bypass isolation guarantees?
Because scx_caps_for_enq() does not inspect enq_flags for SCX_ENQ_IMMED, a sub-scheduler holding only the baseline IMMED capability could call scx_bpf_dispatch() to enqueue a non-IMMED task to a local DSQ.
The task would then be placed in the local DSQ without the SCX_TASK_IMMED flag. This would mean it sits in the queue indefinitely rather than forcing a CPU eviction, which seems to contradict the commit message's goal of ensuring IMMED work can never pile up.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=27
Tejun Heo
reviewerRe: [PATCH v4 sched_ext/for-7.3 27/40] sched_ext: Add the SCX_CAP_ENQ_IMMED cap
Quoted 3 lines · On Wed, 08 Jul 2026 21:56:57 +0000, [email protected] wrote:
- [High] Tasks in a bypassing descendant's runqueue can be permanently stranded if their nearest non-bypassing ancestor lacks
SCX_CAP_BASEon the CPU.
Real limitation and already on the known-limitations list in the cover letter - there's no fallback yet for tasks a sub-scheduler can't run anywhere. The plan is a root-reserved fallback slice in a follow-up series. Not new to this patch.
Quoted 3 lines
- [High]
scx_caps_for_enq()unconditionally maps all local DSQ enqueues toSCX_CAP_ENQ_IMMED, allowing sub-schedulers to bypass isolation guarantees and swamp the CPU with non-IMMED tasks.
Same as the v3 round - at this patch ENQ_IMMED is the sole baseline cap; the ENQ vs ENQ_IMMED split that gates non-IMMED work lands later in the series:
https://lore.kernel.org/r/[email protected]
Thanks.
Signature · 1 line
tejun
Tejun Heo
reviewerPATCH v4 28/40 sched_ext: Assign a unique id to each scheduler instance
In reply to Tejun HeoNeither a scx_sched pointer nor its cgroup id uniquely identifies a scheduler instance. A freed sched's memory can be reallocated, and a cgroup can detach one sched and attach another. Add a monotonic, never-reused u64 id. A later patch compares it to drop a slice request that outlived a change of a task's owning scheduler.
--- kernel/sched/ext/ext.c | 4 ++++ kernel/sched/ext/internal.h | 3 +++ 2 files changed, 7 insertions(+)
kernel/sched/ext/ext.c +4 -0
@@ -108,6 +108,9 @@ static bool dsq_is_rq_owned(struct scx_dispatch_q *dsq) } } +/* Cursor for unique scx_sched instance ids. id 0 is reserved. */ +static atomic64_t scx_sched_id_cursor =ATOMIC64_INIT (0 ); +#ifdef CONFIG_EXT_SUB_SCHED /* * The sub sched being enabled. Used by scx_disable_and_exit_task() to exit
@@ -6460,6 +6463,7 @@ struct scx_sched *scx_alloc_and_add_sched(struct scx_enable_cmd *cmd, level * sizeof(parent->ancestors[0])); sch->ancestors[level] = sch; sch->level = level; + sch->id =atomic64_inc_return (&scx_sched_id_cursor);if (ops->timeout_ms) sch->watchdog_timeout =msecs_to_jiffies (ops->timeout_ms);
kernel/sched/ext/internal.h +3 -0
@@ -1445,6 +1445,9 @@ struct scx_sched {struct list_head all; +/* unique instance id, monotonic and never reused */ +u64 id; +#ifdef CONFIG_EXT_SUB_SCHED struct rhash_head hash_node;
Signature · 1 line
2.54.0
Tejun Heo
reviewerPATCH v4 29/40 sched_ext: Route task slice writes through set_task_slice()
In reply to Tejun HeoA later change makes set_task_slice() also drop a pending out-of-band slice request, so the BPF-triggered writes to p->scx.slice need to funnel through one place. Introduce set_task_slice() and route those writes through it. update_curr_scx() decrements curr->scx.slice directly for accounting and is left alone. No functional change - the helper only assigns p->scx.slice.
v2: Reword the set_task_slice comment to "BPF-triggered writes". (Andrea)
--- kernel/sched/ext/ext.c | 32 +++++++++++++++++++------------- 1 file changed, 19 insertions(+), 13 deletions(-)
kernel/sched/ext/ext.c +19 -13
@@ -1153,6 +1153,12 @@ static void touch_core_sched_dispatch(struct rq *rq, struct task_struct *p)#endif } +/* set @p's slice, BPF-triggered writes to p->scx.slice go through here */ +static void set_task_slice (struct task_struct *p,u64 slice) +{ + p->scx.slice = slice; +} +static void update_curr_scx (struct rq *rq) {struct task_struct *curr = rq->curr;
@@ -1238,7 +1244,7 @@ static void dsq_dec_nr(struct scx_dispatch_q *dsq, struct task_struct *p)static void refill_task_slice_dfl (struct scx_sched *sch,struct task_struct *p) { - p->scx.slice =READ_ONCE (sch->slice_dfl); +set_task_slice (p,READ_ONCE (sch->slice_dfl));__scx_add_event (sch,SCX_EV_REFILL_SLICE_DFL ,1 ); }
@@ -1333,7 +1339,7 @@ static void rq_owned_post_enq(struct scx_sched *sch, struct rq *rq,if ((enq_flags &SCX_ENQ_PREEMPT ) && p != rq->curr && rq->curr->sched_class == &ext_sched_class) { - rq->curr->scx.slice =0 ; +set_task_slice (rq->curr,0 );resched_curr (rq); } }
@@ -2039,7 +2045,7 @@ static void yield_task_scx(struct rq *rq)if (SCX_HAS_OP (sch, yield))SCX_CALL_OP_2TASKS_RET (sch, yield, rq, p,NULL );else - p->scx.slice =0 ; +set_task_slice (p,0 ); }static bool yield_to_task_scx (struct rq *rq,struct task_struct *to)
@@ -3273,7 +3279,7 @@ static void task_tick_scx(struct rq *rq, struct task_struct *curr, int queued)* we can't trust the slice management or ops.core_sched_before(). */ if (scx_bypassing (sch,cpu_of (rq))) { - curr->scx.slice =0 ; +set_task_slice (curr,0 );touch_core_sched (rq, curr); }else if (SCX_HAS_OP (sch, tick)) {SCX_CALL_OP_TASK (sch, tick, rq, curr);
@@ -3412,7 +3418,7 @@ static void scx_disable_task(struct scx_sched *sch, struct task_struct *p)* control, after ops.disable() has observed their final values. */ p->scx.dsq_vtime =0 ; - p->scx.slice =0 ; +set_task_slice (p,0 );/* * Verify the task is not in BPF scheduler's custody. If flag
@@ -7051,7 +7057,7 @@ static void scx_root_enable_workfn(struct kthread_work *work) queue_flags |=DEQUEUE_CLASS ; scoped_guard (sched_change, p, queue_flags) { - p->scx.slice =READ_ONCE (sch->slice_dfl); +set_task_slice (p,READ_ONCE (sch->slice_dfl)); p->sched_class = new_class; } }
@@ -7694,7 +7700,7 @@ static bool kick_one_cpu(s32 cpu, struct scx_sched_pcpu *pcpu, struct rq *this_r !sched_class_above(cur_class, &ext_sched_class)) {if (cpumask_test_cpu (cpu, pcpu->cpus_to_preempt)) {if (cur_class == &ext_sched_class) - rq->curr->scx.slice =0 ; +set_task_slice (rq->curr,0 );cpumask_clear_cpu (cpu, pcpu->cpus_to_preempt); }
@@ -8044,9 +8050,9 @@ __bpf_kfunc bool scx_bpf_dsq_insert___v2(struct task_struct *p, u64 dsq_id,return false;if (slice) - p->scx.slice = slice; +set_task_slice (p, slice);else - p->scx.slice = p->scx.slice ?:1 ; +set_task_slice (p, p->scx.slice ?:1 );scx_dsq_insert_commit (sch, p, dsq_id, enq_flags);
@@ -8070,9 +8076,9 @@ static bool scx_dsq_insert_vtime(struct scx_sched *sch, struct task_struct *p,return false;if (slice) - p->scx.slice = slice; +set_task_slice (p, slice);else - p->scx.slice = p->scx.slice ?:1 ; +set_task_slice (p, p->scx.slice ?:1 ); p->scx.dsq_vtime = vtime;
@@ -8256,7 +8262,7 @@ static bool scx_dsq_move(struct bpf_iter_scx_dsq_kern *kit,if (kit->cursor.flags & __SCX_DSQ_ITER_HAS_VTIME) p->scx.dsq_vtime = kit->vtime;if (kit->cursor.flags & __SCX_DSQ_ITER_HAS_SLICE) - p->scx.slice = kit->slice; +set_task_slice (p, kit->slice);/* execute move */ locked_rq =move_task_between_dsqs (sch, p, enq_flags, src_dsq, dst_dsq);
@@ -8651,7 +8657,7 @@ __bpf_kfunc bool scx_bpf_task_set_slice(struct task_struct *p, u64 slice,if (unlikely (!sch || !scx_task_on_sched (sch, p)))return false; - p->scx.slice = slice; +set_task_slice (p, slice);return true; }
Signature · 1 line
2.54.0
[email protected]
AI reviewRe: [PATCH v4 sched_ext/for-7.3 29/40] sched_ext: Route task slice writes through set_task_slice()
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Pre-existing issues:
- [High] Data race on a running task's slice (p->scx.slice) between the kernel runqueue update and concurrent BPF kfunc execution. --
--- Patch [29]: [PATCH v4 sched_ext/for-7.3 29/40] sched_ext: Route task slice writes through set_task_slice() --- commit 89b48440e365b5adf1131b995d2e06c22b4e11aa
Author: Tejun Heo <[email protected]>
sched_ext: Route task slice writes through set_task_slice()
This commit introduces set_task_slice() and routes BPF-triggered writes to p->scx.slice through it. This centralizes the writes in preparation for a later change to handle pending out-of-band slice requests.
Quoted 3 lines
kernel/sched/ext/ext.c +0
-0
[ ... ]
kernel/sched/ext/ext.c __bpf_kfunc bool scx_bpf_task_set_slice(struct task_struct *p, u64 slice, · line 8660
Quoted 6 lines
kernel/sched/ext/ext.c +1
-1@@ -8651,7 +8657,7 @@ __bpf_kfunc bool scx_bpf_task_set_slice(struct task_struct *p, u64 slice,if (unlikely (!sch || !scx_task_on_sched (sch, p)))return false; - p->scx.slice = slice; +set_task_slice (p, slice);
[Severity: High]
This is a pre-existing issue, but does this lockless write to the task's slice create a dangerous data race with update_curr_scx()?
When a BPF program locklessly calls scx_bpf_task_set_slice() on a running task, it races with the CPU executing update_curr_scx() under the runqueue lock. In update_curr_scx(), the slice is updated using min_t():
curr->scx.slice -= min_t(u64, curr->scx.slice, delta_exec);
Because the min_t() macro evaluates its arguments multiple times, curr->scx.slice can be read once for the comparison, then concurrently updated to a very small value by BPF, and read again for the subtraction. This could result in an integer underflow, giving the task a near-infinite slice and potentially monopolizing the CPU.
Since the commit message mentions this is in preparation for a later change that handles out-of-band slice requests, does that subsequent change mitigate this data race?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=29
Tejun Heo
reviewerRe: [PATCH v4 sched_ext/for-7.3 29/40] sched_ext: Route task slice writes through set_task_slice()
Quoted 2 lines · On Wed, 08 Jul 2026 21:53:32 +0000, [email protected] wrote:
This is a pre-existing issue, but does this lockless write to the task's slice create a dangerous data race with
update_curr_scx()?
The lockless write this race needs no longer exists after this and the following two patches: scx_bpf_task_set_slice() writes p->scx.slice directly only when the caller holds the task's current rq lock, and every other caller is routed through the deferred slice_oob path, drained under the rq lock at the top of update_curr_scx() before the min_t().
Thanks.
Signature · 1 line
tejun
Tejun Heo
reviewerPATCH v4 30/40 sched_ext: Track the cpu a task is runnable on
In reply to Tejun HeoAdd p->scx.runnable_cpu, the cpu @p is runnable on, or -1 when it is not. It is stamped as @p joins the runnable_list (set_task_runnable()) and cleared as it leaves (clr_task_runnable()), both under the rq lock.
task_cpu() can't answer "is @p on this rq" reliably: a remote wakeup changes it under @p's pi_lock alone, without the source rq lock, so it can read as the locked rq while @p is really elsewhere. runnable_cpu changes only under the rq lock, so a caller holding an rq lock can compare against it to know whether that is @p's current rq.
--- include/linux/sched/ext.h | 1 + init/init_task.c | 1 + kernel/sched/ext/ext.c | 9 +++++++++ 3 files changed, 11 insertions(+)
include/linux/sched/ext.h +1 -0
@@ -201,6 +201,7 @@ struct sched_ext_entity {s32 sticky_cpu;s32 holding_cpu;s32 selected_cpu; +s32 runnable_cpu;/* cpu @p is runnable on, -1 if not */ struct task_struct *kf_tasks[2 ];/* see SCX_CALL_OP_TASK() */ struct list_head runnable_node;/* rq->scx.runnable_list */
init/init_task.c +1 -0
@@ -145,6 +145,7 @@ struct task_struct init_task __aligned(L1_CACHE_BYTES) = { .dsq_list.node = LIST_HEAD_INIT(init_task.scx.dsq_list.node), .sticky_cpu = -1, .holding_cpu = -1, + .runnable_cpu = -1, .runnable_node = LIST_HEAD_INIT(init_task.scx.runnable_node), .runnable_at = INITIAL_JIFFIES, .ddsp_dsq_id = SCX_DSQ_INVALID,
kernel/sched/ext/ext.c +9 -0
@@ -1849,11 +1849,19 @@ static void set_task_runnable(struct rq *rq, struct task_struct *p)* appended to the runnable_list. */ list_add_tail (&p->scx.runnable_node, &rq->scx.runnable_list); + +/* +* Record the rq @p is runnable on, maintained under the rq lock so it +* stays valid unlike task_cpu(), which a remote wakeup can move under +* pi_lock alone. +*/ +WRITE_ONCE (p->scx.runnable_cpu,cpu_of (rq)); }static void clr_task_runnable (struct task_struct *p,bool reset_runnable_at) {list_del_init (&p->scx.runnable_node); +WRITE_ONCE (p->scx.runnable_cpu,-1 );if (reset_runnable_at) p->scx.flags |=SCX_TASK_RESET_RUNNABLE_AT ; }
@@ -3501,6 +3509,7 @@ void init_scx_entity(struct sched_ext_entity *scx)RB_CLEAR_NODE (&scx->dsq_priq); scx->sticky_cpu =-1 ; scx->holding_cpu =-1 ; + scx->runnable_cpu =-1 ;INIT_LIST_HEAD (&scx->runnable_node); scx->runnable_at = jiffies; scx->ddsp_dsq_id =SCX_DSQ_INVALID ;
Signature · 1 line
2.54.0
Tejun Heo
reviewerPATCH v4 31/40 sched_ext: Tie cpu occupancy to SCX_CAP_BASE through the task slice
In reply to Tejun HeoA task's slice grants it cpu occupancy - how long it holds its cpu. In a sub-scheduler hierarchy cpu access is delegated through revocable capabilities, so a task's occupancy must follow them. Only its own scheduler sets its slice, and extending the slice is allowed only while that scheduler holds baseline cpu access (SCX_CAP_BASE) on the cpu. Otherwise a scheduler could keep occupying a cpu it has been denied simply by handing out long slices.
The cap check reads effective caps, which are coherent only under the task's rq lock, and the kernel decrements the slice under that lock as the task runs, so a running task's slice can be changed only there while a queued task's can be set directly. Make scx_bpf_task_set_slice() apply the slice under the rq lock. Synchronously when the caller already holds it, otherwise by stashing it in the new p->scx.slice_oob, tagged with the scheduler's id so a request that outlived a reassignment is dropped. Whether the caller holds @p's current rq lock is tested with p->scx.runnable_cpu.
Revocation is enforced through the same grant. When a cpu's effective caps lose SCX_CAP_BASE, the cap-revoke reenq scan also checks the running task and zeroes its slice to evict it. The scan runs as a balance callback after the pick, so this catches both the task that was running when the revoke landed and a capless task the pick just promoted off the local DSQ. The paths that keep a task on its cpu - holding on to the last runnable task in balance, the ENQ_LAST reinsertion and the slice refill on pick - skip tasks lacking baseline access. A migration-disabled task is exempt, mirroring its capless admission on insert.
v4: Test rq ownership with p->scx.runnable_cpu, closing a remote-wakeup TOCTOU. (sashiko AI)
v3: Keep a pending out-of-band slice request across refill and preserve. (sashiko AI)
v2: Only write slice directly when @p is queued on the held rq. (sashiko AI)
--- include/linux/sched/ext.h | 17 ++- kernel/sched/ext/ext.c | 200 ++++++++++++++++++++++++++++++++++-- kernel/sched/ext/internal.h | 19 +++- kernel/sched/ext/sub.h | 11 ++ 4 files changed, 231 insertions(+), 16 deletions(-)
include/linux/sched/ext.h +13 -4
@@ -223,10 +223,11 @@ struct sched_ext_entity {/* BPF scheduler modifiable fields */ /* -* Runtime budget in nsecs. This is usually set through -* scx_bpf_dsq_insert() but can also be modified directly by the BPF -* scheduler. Automatically decreased by SCX as the task executes. On -* depletion, a scheduling event is triggered. +* Runtime budget in nsecs - how long the task may hold its cpu. Owned +* by the task's scheduler. Set it when enqueuing via +* scx_bpf_dsq_insert(), or otherwise via scx_bpf_task_set_slice(). +* Automatically decreased as the task executes. On depletion a +* scheduling event is triggered. * * This value is cleared to zero if the task is preempted by * %SCX_KICK_PREEMPT and shouldn't be used to determine how long the
@@ -243,6 +244,14 @@ struct sched_ext_entity {*/ u64 dsq_vtime; +/* +* Out-of-band slice request from scx_bpf_task_set_slice() when the +* caller does not hold the rq lock, applied under the rq lock at the +* next slice consideration. One atomic64 packs the pending flag, the +* issuing sch's id, and the requested slice. See scx_slice_oob_consts. +*/ +atomic64_t slice_oob; +/* * Sub-sched cap rejected reenq context, valid only while * %SCX_TASK_REENQ_CAP is set. @reenq_reason_caps is the SCX_CAP_* bits
kernel/sched/ext/ext.c +190 -10
284 lines not on this page · raw message
kernel/sched/ext/internal.h +17 -2
@@ -400,8 +400,9 @@ struct sched_ext_ops {* @p: task running currently * * This operation is called every 1/HZ seconds on CPUs which are -* executing an SCX task. Setting @p->scx.slice to 0 will trigger an -* immediate dispatch cycle on the CPU. +* executing an SCX task. Setting a slice of 0 for @p with +* scx_bpf_task_set_slice() will trigger an immediate dispatch cycle on +* the CPU. */ void (*tick)(struct task_struct *p);
@@ -1103,6 +1104,18 @@ struct scx_event_stats {*/ s64 SCX_EV_REFILL_SLICE_DFL ; +/* +* The number of times an out-of-band slice request exceeded the maximum +* representable value and was clamped. +*/ +s64 SCX_EV_SLICE_CLAMPED ; + +/* +* The number of times a slice extension was denied because the +* scheduler lacked baseline cpu access on the task's cpu. +*/ +s64 SCX_EV_SLICE_DENIED ; +/* * The total duration of bypass modes in nanoseconds. */
@@ -1153,6 +1166,8 @@ struct scx_event_stats {SCX_EVENT (SCX_EV_REENQ_IMMED ); \SCX_EVENT (SCX_EV_REENQ_LOCAL_REPEAT ); \SCX_EVENT (SCX_EV_REFILL_SLICE_DFL ); \ +SCX_EVENT (SCX_EV_SLICE_CLAMPED ); \ +SCX_EVENT (SCX_EV_SLICE_DENIED ); \SCX_EVENT (SCX_EV_BYPASS_DURATION ); \SCX_EVENT (SCX_EV_BYPASS_DISPATCH ); \SCX_EVENT (SCX_EV_BYPASS_ACTIVATE ); \
kernel/sched/ext/sub.h +11 -0
@@ -121,9 +121,20 @@ static inline u64 scx_caps_implied(u64 cap) return 0; } +/* may @p keep running on @rq's cpu? requires baseline cpu access */ +static inline bool scx_task_can_stay_on_cpu (struct rq *rq,struct task_struct *p) +{ +/* a migration-disabled task is let in without caps, keep it likewise */ +if (unlikely (is_migration_disabled (p))) +return true; + +return likely (!scx_missing_caps (scx_task_sched (p),cpu_of (rq),SCX_CAP_BASE )); +} +#else /* CONFIG_EXT_SUB_SCHED */ static inline u64 scx_missing_caps (struct scx_sched *sch,s32 cpu,u64 needed) {return 0 ; } +static inline bool scx_task_can_stay_on_cpu (struct rq *rq,struct task_struct *p) {return true; }#endif /* CONFIG_EXT_SUB_SCHED */
Signature · 1 line
2.54.0
Tejun Heo
reviewerPATCH v4 32/40 sched_ext: Add the SCX_CAP_ENQ cap
In reply to Tejun HeoAdd SCX_CAP_ENQ, which gates inserting tasks onto a cid's local DSQ. Unlike IMMED enqueue, plain enqueues can pile up, so ENQ is the stronger cap and implies ENQ_IMMED. Losing ENQ also triggers the reenq scan. The scan tests each queued task and the running task against the cap each needs via scx_caps_for_task(), so an ENQ-only loss reenqueues plain tasks, evicting a running one, while IMMED tasks, which need only ENQ_IMMED, stay put.
--- kernel/sched/ext/ext.c | 1 + kernel/sched/ext/internal.h | 14 +++++++++++--- kernel/sched/ext/sub.h | 12 ++++++++++-- 3 files changed, 22 insertions(+), 5 deletions(-)
kernel/sched/ext/ext.c +1 -0
@@ -4923,6 +4923,7 @@ SCX_ATTR(events);#ifdef CONFIG_EXT_SUB_SCHED static const char *scx_cap_names[__SCX_NR_CAPS] = { [__SCX_CAP_ENQ_IMMED] ="enq_immed" , + [__SCX_CAP_ENQ] ="enq" , }; static ssize_t scx_attr_caps_show(struct kobject *kobj,
kernel/sched/ext/internal.h +11 -3
@@ -1224,8 +1224,8 @@ struct scx_sched_pcpu {/* * pshard->caps[cap_bit] is the set of cids the sched holds that one * cap on. ecaps is its transpose: the set of SCX_CAP_* bits the sched -* holds on this cpu, collected so that the hot-path check is a single -* read. +* effectively holds on this cpu, with implied caps folded in, so that +* the hot-path check is a single read. * * While pshard->caps[] under pshard->lock is the target configuration, * ecaps is the effective copy owned by the cpu. It is written under the
@@ -1287,20 +1287,28 @@ struct scx_sched_pnode {* the allocation pattern. * * ENQ_IMMED insert an IMMED task onto the cid's local DSQ +* +* ENQ insert any task onto the cid's local DSQ (implies ENQ_IMMED) +* +* Implied caps apply to the holder's own use of a cid, not to delegation. +* scx_bpf_sub_grant() delegates literally-held caps, so a cap held only through +* implication is usable but cannot be re-delegated to a child. */ enum scx_cap_flags { __SCX_CAP_ENQ_IMMED =0 , + __SCX_CAP_ENQ =1 , __SCX_NR_CAPS, __SCX_CAP_ALL =BIT_U64 (__SCX_NR_CAPS) -1 ,SCX_CAP_ENQ_IMMED =BIT_U64 (__SCX_CAP_ENQ_IMMED), +SCX_CAP_ENQ =BIT_U64 (__SCX_CAP_ENQ),/* alias for minimal cap to make any use of a cpu */ SCX_CAP_BASE =SCX_CAP_ENQ_IMMED ,/* caps whose loss strands queued tasks, see scx_process_sync_ecaps() */ -SCX_CAPS_REENQ_ON_LOSS =SCX_CAP_ENQ_IMMED , +SCX_CAPS_REENQ_ON_LOSS =SCX_CAP_ENQ_IMMED |SCX_CAP_ENQ , };#ifdef CONFIG_EXT_SUB_SCHED
kernel/sched/ext/sub.h +10 -2
@@ -106,18 +106,26 @@ static inline u64 scx_missing_caps(struct scx_sched *sch, s32 cpu, u64 needed)/* map @enq_flags to the SCX_CAP_* bit required for the local-DSQ insert */ static inline u64 scx_caps_for_enq (u64 enq_flags) { -return SCX_CAP_ENQ_IMMED ; +if (enq_flags &SCX_ENQ_IMMED ) +return SCX_CAP_ENQ_IMMED ; +return SCX_CAP_ENQ ; }/* map queued @p to the SCX_CAP_* bit required to stay on its local DSQ */ static inline u64 scx_caps_for_task (struct task_struct *p) { -return SCX_CAP_ENQ_IMMED ; +if (p->scx.flags &SCX_TASK_IMMED ) +return SCX_CAP_ENQ_IMMED ; +return SCX_CAP_ENQ ; }/* caps implied by holding @cap */ static inline u64 scx_caps_implied (u64 cap) { +switch (cap) { +case SCX_CAP_ENQ : +return SCX_CAP_ENQ_IMMED ; + }return 0 ; }
Signature · 1 line
2.54.0
[email protected]
AI reviewRe: [PATCH v4 sched_ext/for-7.3 32/40] sched_ext: Add the SCX_CAP_ENQ cap
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] Tasks with carried-in IMMED state are spuriously rejected during internal movements (like RESTORE)…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] Tasks with carried-in IMMED state are spuriously rejected during internal movements (like RESTORE) if the scheduler lacks SCX_CAP_ENQ. --
--- Patch [32]: [PATCH v4 sched_ext/for-7.3 32/40] sched_ext: Add the SCX_CAP_ENQ cap --- commit ff6a1550bc7578cdbaff320215c7593bfdd63815
Author: Tejun Heo <[email protected]>
sched_ext: Add the SCX_CAP_ENQ cap
This commit adds SCX_CAP_ENQ, which gates inserting tasks onto a cid's local DSQ. Unlike IMMED enqueue, plain enqueues can pile up, so ENQ is the stronger cap and implies ENQ_IMMED.
Quoted 4 lines
kernel/sched/ext/sub.h +0
-0
[ ... ]
kernel/sched/ext/sub.h static inline u64 scx_missing_caps(struct scx_sched *sch, s32 cpu, u64 needed) · line 112
Quoted 9 lines
kernel/sched/ext/sub.h +3
-1@@ -106,18 +106,26 @@ static inline u64 scx_missing_caps(struct scx_sched *sch, s32 cpu, u64 needed)/* map @enq_flags to the SCX_CAP_* bit required for the local-DSQ insert */ static inline u64 scx_caps_for_enq (u64 enq_flags) { -return SCX_CAP_ENQ_IMMED ; +if (enq_flags &SCX_ENQ_IMMED ) +return SCX_CAP_ENQ_IMMED ; +return SCX_CAP_ENQ ; }
[Severity: High]
Does this code incorrectly reject tasks with a carried-in IMMED state during internal movements?
When an internal scheduling movement like ENQUEUE_RESTORE occurs for a running IMMED task, scx_do_enqueue_task() preserves the task's SCX_TASK_IMMED flag by jumping to the local_norefill path.
However, the enq_flags passed to scx_dispatch_enqueue() and subsequently scx_local_or_reject_dsq() do not have SCX_ENQ_IMMED set.
Since this function only checks enq_flags, it evaluates the requirement as SCX_CAP_ENQ when called from scx_local_or_reject_dsq():
kernel/sched/ext/sub.c:scx_local_or_reject_dsq() {
...
u64 missing = scx_missing_caps(sch, cpu_of(rq), scx_caps_for_enq(*enq_flags));
...
}If the sub-scheduler holds SCX_CAP_ENQ_IMMED but not SCX_CAP_ENQ, the capability check fails. This causes the task to be evicted to the reject DSQ and stripped of its IMMED protection.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=32
Tejun Heo
reviewerRe: [PATCH v4 sched_ext/for-7.3 32/40] sched_ext: Add the SCX_CAP_ENQ cap
Quoted 3 lines · On Wed, 08 Jul 2026 22:00:09 +0000, [email protected] wrote:
- [High] Tasks with carried-in IMMED state are spuriously rejected during internal movements (like RESTORE) if the scheduler lacks
SCX_CAP_ENQ.
Correct - an in-place RESTORE requeue of a running task re-runs cid admission requiring SCX_CAP_ENQ while the enqueue carries no SCX_ENQ_IMMED, so a sub holding only ENQ_IMMED bounces a task it's already running. The restore isn't a real scheduling event (set_next_task_scx() follows immediately) and shouldn't re-run admission at all. Fixed in v5: the restore path marks the enqueue with an internal IGNORE_CAPS flag which scx_caps_for_enq() maps to no required caps, split out as a prep patch.
Thanks.
Signature · 1 line
tejun
Tejun Heo
reviewerPATCH v4 33/40 sched_ext: Gate kicks on SCX_CAP_BASE and preemption on SCX_CAP_PREEMPT
In reply to Tejun HeoA kick forces a scheduling event on the target cpu, and a preemption also evicts the running task. Gate both on caps. Any kick requires baseline access on the cid, and preempting a task the sub-sched does not own - whether by a SCX_ENQ_PREEMPT insert or a SCX_KICK_PREEMPT kick - requires the new SCX_CAP_PREEMPT. Gating either alone would leave a hole - the weakest cap authorizing preempting kicks, or plain kicks disturbing cpus the kicker has no access to.
Preempting the sched's own subtree is always allowed, and the cap extends the right to any task on the cid. PREEMPT implies ENQ, and so ENQ_IMMED.
A preempting insert tests the running task under the target rq lock and is rejected and reenqueued unless the victim is in the inserter's subtree or it holds PREEMPT. A migration-disabled task is admitted regardless, but with SCX_ENQ_PREEMPT stripped.
Kicks are enforced on the delivery path, where the effective caps can be read coherently under the target rq's lock. A kick from a sub-sched lacking SCX_CAP_BASE on the cid is dropped, and a SCX_KICK_PREEMPT kick without PREEMPT for a task outside the kicker's subtree degrades to a plain reschedule.
Unlike the enqueue caps, PREEMPT is checked only at the instant of the insert or kick, never as a standing property of a queued task.
v2: Clear SCX_ENQ_PREEMPT on the offline and migration_pending force-admits.
--- kernel/sched/ext/ext.c | 34 +++++++++++++++++++++++++--------- kernel/sched/ext/internal.h | 24 ++++++++++++++++++++++-- kernel/sched/ext/sub.c | 19 +++++++++++++++---- kernel/sched/ext/sub.h | 15 +++++++++++++++ 4 files changed, 77 insertions(+), 15 deletions(-)
kernel/sched/ext/ext.c +25 -9
@@ -294,7 +294,7 @@ static bool u32_before(u32 a, u32 b)* * Test whether @sch is a descendant of @ancestor. */ -static bool scx_is_descendant (struct scx_sched *sch,struct scx_sched *ancestor) +bool scx_is_descendant (struct scx_sched *sch,struct scx_sched *ancestor) {if (sch->level < ancestor->level)return false;
@@ -4924,6 +4924,7 @@ SCX_ATTR(events);static const char *scx_cap_names[__SCX_NR_CAPS] = { [__SCX_CAP_ENQ_IMMED] ="enq_immed" , [__SCX_CAP_ENQ] ="enq" , + [__SCX_CAP_PREEMPT] ="preempt" , }; static ssize_t scx_attr_caps_show(struct kobject *kobj,
@@ -7856,13 +7857,22 @@ static bool kick_one_cpu(s32 cpu, struct scx_sched_pcpu *pcpu, struct rq *this_r* During CPU hotplug, a CPU may depend on kicking itself to make * forward progress. Allow kicking self regardless of online state. If * @cpu is running a higher class task, we have no control over @cpu. -* Skip kicking. +* Skip kicking. A sub-sched lacking baseline access on @cid has no +* business forcing a reschedule there - skip. This is the authoritative +* cap check: ecaps is read here under @rq's lock. */ if ((cpu_online (cpu) || cpu ==cpu_of (this_rq)) && - !sched_class_above (cur_class, &ext_sched_class)) { + !sched_class_above (cur_class, &ext_sched_class) && + !scx_missing_caps (pcpu->sch, cpu,SCX_CAP_BASE )) {if (cpumask_test_cpu (cpu, pcpu->cpus_to_preempt)) { -if (cur_class == &ext_sched_class) -set_task_slice (rq->curr,0 ); +if (cur_class == &ext_sched_class) { +if (likely (!scx_missing_caps (pcpu->sch, cpu, +scx_caps_for_preempt (pcpu->sch, rq)))) +set_task_slice (rq->curr,0 ); +else +__scx_add_event (pcpu->sch, +SCX_EV_SUB_PREEMPT_DENIED ,1 ); + }cpumask_clear_cpu (cpu, pcpu->cpus_to_preempt); }
@@ -7886,15 +7896,18 @@ static bool kick_one_cpu(s32 cpu, struct scx_sched_pcpu *pcpu, struct rq *this_rreturn should_wait; } -static void kick_one_cpu_if_idle (s32 cpu,struct rq *this_rq) +static void kick_one_cpu_if_idle (s32 cpu,struct scx_sched_pcpu *pcpu, +struct rq *this_rq) {struct rq *rq =cpu_rq (cpu);unsigned long flags;raw_spin_rq_lock_irqsave (rq, flags); +/* idle kicks need baseline access too, see kick_one_cpu() */ if (!can_skip_idle_kick (rq) && - (cpu_online (cpu) || cpu ==cpu_of (this_rq))) + (cpu_online (cpu) || cpu ==cpu_of (this_rq)) && + !scx_missing_caps (pcpu->sch, cpu,SCX_CAP_BASE ))resched_curr (rq);raw_spin_rq_unlock_irqrestore (rq, flags);
@@ -7931,7 +7944,7 @@ static void kick_cpus_irq_workfn(struct irq_work *irq_work) }for_each_cpu (cpu, pcpu->cpus_to_kick_if_idle) { -kick_one_cpu_if_idle (cpu, this_rq); +kick_one_cpu_if_idle (cpu, pcpu, this_rq);cpumask_clear_cpu (cpu, pcpu->cpus_to_kick_if_idle); } }
@@ -8972,7 +8985,10 @@ __bpf_kfunc void scx_bpf_kick_cpu(s32 cpu, u64 flags, const struct bpf_prog_aux* @aux: implicit BPF argument to access bpf_prog_aux hidden from BPF progs * * cid-addressed equivalent of scx_bpf_kick_cpu(). An invalid @cid aborts the -* scheduler via scx_cid_to_cpu(). +* scheduler via scx_cid_to_cpu(). Caps are enforced on the delivery path: a +* kick is dropped if the caller lacks baseline access on @cid, and a +* %SCX_KICK_PREEMPT degrades to a plain reschedule if the caller lacks +* %SCX_CAP_PREEMPT for a task outside its subtree. */ __bpf_kfunc void scx_bpf_kick_cid(s32 cid, u64 flags, const struct bpf_prog_aux *aux) {
kernel/sched/ext/internal.h +22 -2
@@ -1155,6 +1155,13 @@ struct scx_event_stats {* it can't be rejected. The violation is counted here. */ s64 SCX_EV_SUB_FORCED_ADMIT ; + +/* +* The number of times a preempting kick was refused because the +* sub-sched lacked SCX_CAP_PREEMPT for a task outside its subtree. The +* kick degrades to a plain reschedule. +*/ +s64 SCX_EV_SUB_PREEMPT_DENIED ; };#define SCX_EVENTS_LIST (SCX_EVENT ) \
@@ -1173,7 +1180,8 @@ struct scx_event_stats {SCX_EVENT (SCX_EV_BYPASS_ACTIVATE ); \SCX_EVENT (SCX_EV_INSERT_NOT_OWNED ); \SCX_EVENT (SCX_EV_SUB_BYPASS_DISPATCH ); \ -SCX_EVENT (SCX_EV_SUB_FORCED_ADMIT) +SCX_EVENT (SCX_EV_SUB_FORCED_ADMIT ); \ +SCX_EVENT (SCX_EV_SUB_PREEMPT_DENIED) struct scx_sched;
@@ -1287,22 +1295,33 @@ struct scx_sched_pnode {* the allocation pattern. * * ENQ_IMMED insert an IMMED task onto the cid's local DSQ +* - kick the cid's cpu (except SCX_KICK_PREEMPT) * * ENQ insert any task onto the cid's local DSQ (implies ENQ_IMMED) * +* PREEMPT preempt any task running on the cid regardless of the owning +* sched (implies ENQ). Preempting a task in the sched's own subtree +* doesn't require any cap. +* - SCX_ENQ_PREEMPT inserts +* - SCX_KICK_PREEMPT kicks +* * Implied caps apply to the holder's own use of a cid, not to delegation. * scx_bpf_sub_grant() delegates literally-held caps, so a cap held only through -* implication is usable but cannot be re-delegated to a child. +* implication is usable but cannot be re-delegated to a child. When granting a +* cap, it usually makes sense to delegate its implied caps explicitly alongside +* it. */ enum scx_cap_flags { __SCX_CAP_ENQ_IMMED =0 , __SCX_CAP_ENQ =1 , + __SCX_CAP_PREEMPT =2 , __SCX_NR_CAPS, __SCX_CAP_ALL =BIT_U64 (__SCX_NR_CAPS) -1 ,SCX_CAP_ENQ_IMMED =BIT_U64 (__SCX_CAP_ENQ_IMMED),SCX_CAP_ENQ =BIT_U64 (__SCX_CAP_ENQ), +SCX_CAP_PREEMPT =BIT_U64 (__SCX_CAP_PREEMPT),/* alias for minimal cap to make any use of a cpu */ SCX_CAP_BASE =SCX_CAP_ENQ_IMMED ,
@@ -1913,6 +1932,7 @@ struct scx_sched *scx_alloc_and_add_sched(struct scx_enable_cmd *cmd, struct scx_sched *parent);int scx_validate_ops (struct scx_sched *sch,const struct sched_ext_ops *ops);int scx_sched_sysfs_add (struct scx_sched *sch); +bool scx_is_descendant (struct scx_sched *sch,struct scx_sched *ancestor);extern raw_spinlock_t scx_sched_lock;extern struct mutex scx_enable_mutex;
kernel/sched/ext/sub.c +15 -4
@@ -221,11 +221,14 @@ void scx_init_root_caps(struct scx_sched *sch)* @sch: enqueuing sub-sched * @rq: rq whose local DSQ @p targets * @p: task being inserted -* @enq_flags: in/out; %SCX_ENQ_IMMED is cleared when diverting to reject +* @enq_flags: in/out, unhonored flags are cleared * * Return @rq's local DSQ if @sch holds the required caps on @rq's cid, * otherwise @rq's reject DSQ after recording the reenq reason on @p. * +* %SCX_ENQ_IMMED and %SCX_ENQ_PREEMPT are cleared when diverting to reject. +* %SCX_ENQ_PREEMPT is also cleared on a fallback migration-disabled admission. +* * Bypass doesn't need special-casing as a bypassing sched's tasks are enqueued * to and run by its nearest non-bypassing ancestor. If root is bypassing, it * always holds all caps.
@@ -234,7 +237,12 @@ struct scx_dispatch_q *scx_local_or_reject_dsq(struct scx_sched *sch, struct rq struct task_struct *p, u64 *enq_flags) { s32 cid = __scx_cpu_to_cid(cpu_of(rq)); -u64 missing =scx_missing_caps (sch,cpu_of (rq),scx_caps_for_enq (*enq_flags)); +u64 needed =scx_caps_for_enq (*enq_flags); +u64 missing; + +if (*enq_flags &SCX_ENQ_PREEMPT ) + needed |=scx_caps_for_preempt (sch, rq); + missing =scx_missing_caps (sch,cpu_of (rq), needed);/* requirements met */ if (likely (!missing))
@@ -244,10 +252,12 @@ struct scx_dispatch_q *scx_local_or_reject_dsq(struct scx_sched *sch, struct rq* The task must run on this CPU regardless of caps: the rq is draining * offline (BPF scheduler bypassed), the task is migration-disabled, or a * migration is pending. Admit despite the missing caps and count it. +* Refuse preemptions. */ if (unlikely (!scx_rq_online (rq) ||is_migration_disabled (p) || p->migration_pending)) {__scx_add_event (sch,SCX_EV_SUB_FORCED_ADMIT ,1 ); + *enq_flags &= ~SCX_ENQ_PREEMPT ;return &rq->scx.local_dsq; }
@@ -257,9 +267,10 @@ struct scx_dispatch_q *scx_local_or_reject_dsq(struct scx_sched *sch, struct rq/* * Only local DSQ can honor IMMED and dsq_inc_nr() WARNs on IMMED into * others. Strip both the enq flag and the sticky task flag - the -* latter can carry in from an earlier admitted IMMED insert. +* latter can carry in from an earlier admitted IMMED insert. Strip +* PREEMPT too. */ - *enq_flags &= ~SCX_ENQ_IMMED ; + *enq_flags &= ~(SCX_ENQ_IMMED |SCX_ENQ_PREEMPT ); p->scx.flags &= ~SCX_TASK_IMMED ;return &rq->scx.reject_dsq;
kernel/sched/ext/sub.h +15 -0
@@ -119,10 +119,24 @@ static inline u64 scx_caps_for_task(struct task_struct *p)return SCX_CAP_ENQ ; } +/* the cap @sch needs to preempt @rq's current task, 0 if none */ +static inline u64 scx_caps_for_preempt (struct scx_sched *sch,struct rq *rq) +{ +struct task_struct *curr = rq->curr; + +/* a non-ext task can't be preempted by ext, own-subtree needs no cap */ +if (curr->sched_class != &ext_sched_class || +scx_is_descendant (scx_task_sched (curr), sch)) +return 0 ; +return SCX_CAP_PREEMPT ; +} +/* caps implied by holding @cap */ static inline u64 scx_caps_implied (u64 cap) {switch (cap) { +case SCX_CAP_PREEMPT : +return SCX_CAP_ENQ |SCX_CAP_ENQ_IMMED ;case SCX_CAP_ENQ :return SCX_CAP_ENQ_IMMED ; }
@@ -142,6 +156,7 @@ static inline bool scx_task_can_stay_on_cpu(struct rq *rq, struct task_struct *p#else /* CONFIG_EXT_SUB_SCHED */ static inline u64 scx_missing_caps (struct scx_sched *sch,s32 cpu,u64 needed) {return 0 ; } +static inline u64 scx_caps_for_preempt (struct scx_sched *sch,struct rq *rq) {return 0 ; }static inline bool scx_task_can_stay_on_cpu (struct rq *rq,struct task_struct *p) {return true; }#endif /* CONFIG_EXT_SUB_SCHED */
Signature · 1 line
2.54.0
Tejun Heo
reviewerPATCH v4 34/40 sched_ext: Authorize remote-move inserts against the placing scheduler
In reply to Tejun Heoscx_local_or_reject_dsq() authorizes a local-DSQ insert against the caps of the scheduler doing the insert. On the consume/dispatch paths that is the scheduler running balance_one(), passed down through scx_consume_dispatch_q() and move_local_task_to_local_dsq(), so the check is correct.
The remote-move path loses it. move_remote_task_to_local_dsq() re-activates @p on the destination rq through enqueue_task_scx(), which reconstructs the scheduler from the task, i.e. @p's owner. When an ancestor places a descendant's task - e.g. draining a bypassed sub-scheduler - the owner is a sub-scheduler of the placer, so authorizing against the owner checks a narrower cap set and can spuriously reject a task the placer is entitled to run.
Carry the placing scheduler across the activate_task() boundary the same way enq_flags already are, via a per-rq field set only for the duration of the re-activation, and have scx_local_or_reject_dsq() authorize against it. The placer's caps are a superset of the owner's, so this admits what the placer may run and keeps rejecting what it may not.
--- kernel/sched/ext/ext.c | 27 +++++++++++++++------------ kernel/sched/ext/sub.c | 9 +++++++-- kernel/sched/sched.h | 3 ++- 3 files changed, 24 insertions(+), 15 deletions(-)
kernel/sched/ext/ext.c +15 -12
@@ -2256,7 +2256,8 @@ static void move_local_task_to_local_dsq(struct scx_sched *sch,* * Move @p which is currently on @src_rq to @dst_rq's local DSQ. */ -static void move_remote_task_to_local_dsq (struct task_struct *p,u64 enq_flags, +static void move_remote_task_to_local_dsq (struct scx_sched *sch, +struct task_struct *p,u64 enq_flags,struct rq *src_rq,struct rq *dst_rq) {lockdep_assert_rq_held (src_rq);
@@ -2273,15 +2274,19 @@ static void move_remote_task_to_local_dsq(struct task_struct *p, u64 enq_flags,raw_spin_rq_lock (dst_rq);/* -* We want to pass scx-specific enq_flags but activate_task() will -* truncate the upper 32 bit. As we own @rq, we can pass them through -* @rq->scx.remote_activate_enq_flags instead. +* activate_task() below truncates enq_flags to 32 bits and re-derives +* @p's owner, dropping our scx flags and the placing @sch. We own @rq, +* so stash both across the call. The enqueue reads them back, keeping +* the scx flags and checking caps against the placer, not the owner. */ WARN_ON_ONCE (!cpumask_test_cpu (cpu_of (dst_rq), p->cpus_ptr)); -WARN_ON_ONCE (dst_rq->scx.remote_activate_enq_flags); +WARN_ON_ONCE (dst_rq->scx.remote_activate_enq_flags || + dst_rq->scx.remote_activate_sch); dst_rq->scx.remote_activate_enq_flags = enq_flags; + dst_rq->scx.remote_activate_sch = sch;activate_task (dst_rq, p,0 ); dst_rq->scx.remote_activate_enq_flags =0 ; + dst_rq->scx.remote_activate_sch =NULL ; }/*
@@ -2410,14 +2415,14 @@ static bool unlink_dsq_and_lock_src_rq(struct task_struct *p, !WARN_ON_ONCE (src_rq !=task_rq (p)); } -static bool consume_remote_task (struct rq *this_rq, +static bool consume_remote_task (struct scx_sched *sch,struct rq *this_rq,struct task_struct *p,u64 enq_flags,struct scx_dispatch_q *dsq,struct rq *src_rq) {raw_spin_rq_unlock (this_rq);if (unlink_dsq_and_lock_src_rq (p, dsq, src_rq)) { -move_remote_task_to_local_dsq (p, enq_flags, src_rq, this_rq); +move_remote_task_to_local_dsq (sch, p, enq_flags, src_rq, this_rq);return true; }else {raw_spin_rq_unlock (src_rq);
@@ -2479,8 +2484,7 @@ static struct rq *move_task_between_dsqs(struct scx_sched *sch,raw_spin_unlock (&src_dsq->lock); }else {raw_spin_unlock (&src_dsq->lock); -move_remote_task_to_local_dsq (p, enq_flags, - src_rq, dst_rq); +move_remote_task_to_local_dsq (sch, p, enq_flags, src_rq, dst_rq); } }else {/*
@@ -2533,7 +2537,7 @@ bool scx_consume_dispatch_q(struct scx_sched *sch, struct rq *rq, }if (task_can_run_on_remote_rq (sch, p, rq, false)) { -if (likely (consume_remote_task (rq, p, enq_flags, dsq, task_rq))) +if (likely (consume_remote_task (sch, rq, p, enq_flags, dsq, task_rq)))return true; goto retry; }
@@ -2627,8 +2631,7 @@ static void dispatch_to_local_dsq(struct scx_sched *sch, struct rq *rq,scx_dispatch_enqueue (sch, src_rq,find_global_dsq (sch,task_cpu (p)), p, enq_flags |SCX_ENQ_GDSQ_FALLBACK ); }else { -move_remote_task_to_local_dsq (p, enq_flags, - src_rq, dst_rq); +move_remote_task_to_local_dsq (sch, p, enq_flags, src_rq, dst_rq);/* task has been moved to dst_rq, which is now locked */ locked_rq = dst_rq; }
kernel/sched/ext/sub.c +7 -2
@@ -237,12 +237,17 @@ struct scx_dispatch_q *scx_local_or_reject_dsq(struct scx_sched *sch, struct rq struct task_struct *p, u64 *enq_flags) { s32 cid = __scx_cpu_to_cid(cpu_of(rq)); +struct scx_sched *asch = rq->scx.remote_activate_sch ?: sch;u64 needed =scx_caps_for_enq (*enq_flags);u64 missing; +/* +* On a remote activation the scheduling sched (@asch) differs from +* @p's owner (@sch). Check caps against the scheduling sched. +*/ if (*enq_flags &SCX_ENQ_PREEMPT ) - needed |=scx_caps_for_preempt (sch, rq); - missing =scx_missing_caps (sch,cpu_of (rq), needed); + needed |=scx_caps_for_preempt (asch, rq); + missing =scx_missing_caps (asch,cpu_of (rq), needed);/* requirements met */ if (likely (!missing))
kernel/sched/sched.h +2 -1
@@ -800,8 +800,9 @@ struct scx_rq {struct list_head runnable_list;/* runnable tasks on this rq */ struct list_head ddsp_deferred_locals;/* deferred ddsps from enq */ unsigned long ops_qseq; -/* see move_remote_task_to_local_dsq() */ +/* both stashed across the activate_task() in move_remote_task_to_local_dsq() */ u64 remote_activate_enq_flags; +struct scx_sched *remote_activate_sch;u32 nr_running;u32 cpuperf_target;/* [0, SCHED_CAPACITY_SCALE] */ bool in_select_cpu;
Signature · 1 line
2.54.0
[email protected]
AI reviewRe: [PATCH v4 sched_ext/for-7.3 34/40] sched_ext: Authorize remote-move inserts against the placing scheduler
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] scx_local_or_reject_dsq aborts the task owner when the placing scheduler lacks capabilities, and…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] scx_local_or_reject_dsq aborts the task owner when the placing scheduler lacks capabilities, and scx_dispatch_enqueue conflates the owner and placer causing misattributed errors and erroneous ops.dequeue invocations. --
--- Patch [34]: [PATCH v4 sched_ext/for-7.3 34/40] sched_ext: Authorize remote-move inserts against the placing scheduler --- commit aba6c3faa0482889b73037930ca34ba762b24887
Author: Tejun Heo <[email protected]>
sched_ext: Authorize remote-move inserts against the placing scheduler
This commit addresses an issue in the remote-move path where tasks could be spuriously rejected because capabilities were checked against the task's owner rather than the placing scheduler. It stashes the placing scheduler across the activate_task() boundary to correctly authorize against it.
Quoted 4 lines
kernel/sched/ext/ext.c +0
-0
[ ... ]
kernel/sched/ext/ext.c bool scx_consume_dispatch_q(struct scx_sched *sch, struct rq *rq, · line 2543
Quoted 9 lines
kernel/sched/ext/ext.c +1
-1@@ -2533,7 +2537,7 @@ bool scx_consume_dispatch_q(struct scx_sched *sch, struct rq *rq, }if (task_can_run_on_remote_rq (sch, p, rq, false)) { -if (likely (consume_remote_task (rq, p, enq_flags, dsq, task_rq))) +if (likely (consume_remote_task (sch, rq, p, enq_flags, dsq, task_rq)))return true; goto retry; }
[Severity: High]
This is a pre-existing issue, but does passing the placing scheduler during local moves cause issues with ops.dequeue?
For local moves going through scx_consume_dispatch_q() into move_local_task_to_local_dsq(), the sch variable is the placing scheduler.
If call_task_dequeue() uses this, ops.dequeue is invoked on the placer instead of the task's owner, leaving the owner unaware that the task left custody.
[ ... ]
kernel/sched/ext/ext.c static void dispatch_to_local_dsq(struct scx_sched *sch, struct rq *rq, · line 2632
Quoted 3 lines
kernel/sched/ext/ext.c +0
-0@@ -2627,8 +2631,7 @@ static void dispatch_to_local_dsq(struct scx_sched *sch, struct rq *rq,scx_dispatch_enqueue (sch, src_rq,find_global_dsq (sch,task_cpu (p)), p, enq_flags |SCX_ENQ_GDSQ_FALLBACK );
[Severity: High]
This isn't a bug introduced by this patch, but it seems there is a similar conflation in scx_dispatch_enqueue() regarding error attribution.
If the placer passes invalid enq_flags or an invalid DSQ ID to a built-in DSQ, scx_dispatch_enqueue() triggers scx_error(sch, "cannot use vtime...").
For remote moves, sch here is the owner. Does this mean the owner scheduler is unfairly aborted for the placer's error?
Quoted 7 lines
kernel/sched/ext/ext.c +1
-2}else { -move_remote_task_to_local_dsq (p, enq_flags, - src_rq, dst_rq); +move_remote_task_to_local_dsq (sch, p, enq_flags, src_rq, dst_rq);/* task has been moved to dst_rq, which is now locked */ locked_rq = dst_rq; }
[ ... ]
Quoted 11 lines
kernel/sched/ext/sub.c +1
-0@@ -237,12 +237,17 @@ struct scx_dispatch_q *scx_local_or_reject_dsq(struct scx_sched *sch, struct rq struct task_struct *p, u64 *enq_flags) { s32 cid = __scx_cpu_to_cid(cpu_of(rq)); +struct scx_sched *asch = rq->scx.remote_activate_sch ?: sch;u64 needed =scx_caps_for_enq (*enq_flags);u64 missing;
kernel/sched/ext/sub.c struct scx_dispatch_q *scx_local_or_reject_dsq(struct scx_sched *sch, struct rq · line 250
Quoted 10 lines
kernel/sched/ext/sub.c +6
-2+/* +* On a remote activation the scheduling sched (@asch) differs from +* @p's owner (@sch). Check caps against the scheduling sched. +*/ if (*enq_flags &SCX_ENQ_PREEMPT ) - needed |=scx_caps_for_preempt (sch, rq); - missing =scx_missing_caps (sch,cpu_of (rq), needed); + needed |=scx_caps_for_preempt (asch, rq); + missing =scx_missing_caps (asch,cpu_of (rq), needed);
[Severity: High]
Does this incorrectly penalize the owner task if capabilities are missing?
If missing & SCX_CAP_BASE is true after this check, the subsequent error handling in scx_local_or_reject_dsq() calls scx_error(sch, ...).
Since capabilities are now correctly checked against the scheduling scheduler (asch), should the error attribution (and events like SCX_EV_SUB_FORCED_ADMIT) use asch instead of sch so the descendant scheduler isn't unfairly aborted when an ancestor lacks capabilities?
Quoted 3 lines
kernel/sched/ext/sub.c +0
-0/* requirements met */ if (likely (!missing))
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=34
Tejun Heo
reviewerRe: [PATCH v4 sched_ext/for-7.3 34/40] sched_ext: Authorize remote-move inserts against the placing scheduler
Quoted 2 lines · On Wed, 08 Jul 2026 22:04:44 +0000, [email protected] wrote:
This is a pre-existing issue, but does passing the placing scheduler during local moves cause issues with ops.dequeue?
The combination can't materialize: call_task_dequeue() no-ops unless SCX_TASK_IN_CUSTODY is set, custody is only entered for tasks placed on a user DSQ after an scx_task_on_sched() ownership check, and the one path where placer != owner (remote bypass drain) runs with custody already clear. That said, v5 adds a WARN_ON_ONCE to the per-task op macros asserting @sch is the task's owner, with the two intentional non-owner sites dispatching explicitly, so any future conflation surfaces instead of hiding.
Quoted 3 lines
If the placer passes invalid
enq_flagsor an invalid DSQ ID to a built-in DSQ,scx_dispatch_enqueue()triggersscx_error(sch, "cannot use vtime..."). For remote moves, sch here is the owner.
A remote re-activation always inserts into the local (or reject) DSQ, so the builtin-DSQ error branches aren't reachable from a placer move - they fire on a scheduler's own dispatches.
Quoted 3 lines
should the error attribution (and events like
SCX_EV_SUB_FORCED_ADMIT) use asch instead of sch so the descendant scheduler isn't unfairly aborted when an ancestor lacks capabilities?
scx_local_or_reject_dsq() records no error and aborts nothing - the missing-caps branch either force-admits or diverts to the reject DSQ. The only owner-vs-placer @sch use is the FORCED_ADMIT event counter, which is cosmetic. May sync it to @asch later.
Thanks.
Signature · 1 line
tejun
Andrea Righi
reviewerRe: [PATCH v4 sched_ext/for-7.3 34/40] sched_ext: Authorize remote-move inserts against the placing scheduler
In reply to Tejun HeoHi Tejun,
Quoted 32 lines · Tejun Heo
scx_local_or_reject_dsq()authorizes a local-DSQ insert against the caps of the scheduler doing the insert. On the consume/dispatch paths that is the scheduler runningbalance_one(), passed down throughscx_consume_dispatch_q()andmove_local_task_to_local_dsq(), so the check is correct.The remote-move path loses it.
move_remote_task_to_local_dsq()re-activates @p on the destination rq throughenqueue_task_scx(), which reconstructs the scheduler from the task, i.e. @p's owner. When an ancestor places a descendant's task - e.g. draining a bypassed sub-scheduler - the owner is a sub-scheduler of the placer, so authorizing against the owner checks a narrower cap set and can spuriously reject a task the placer is entitled to run.Carry the placing scheduler across the
activate_task()boundary the same wayenq_flagsalready are, via a per-rq field set only for the duration of the re-activation, and havescx_local_or_reject_dsq()authorize against it. The placer's caps are a superset of the owner's, so this admits what the placer may run and keeps rejecting what it may not.Signed-off-by Tejun Heo <[email protected]>--- kernel/sched/ext/ext.c | 27 +++++++++++++++------------ kernel/sched/ext/sub.c | 9 +++++++-- kernel/sched/sched.h | 3 ++- 3 files changed, 24 insertions(+), 15 deletions(-)kernel/sched/ext/ext.c +0
-0
kernel/sched/ext/ext.c static void move_local_task_to_local_dsq(struct scx_sched *sch, · line 2261
Quoted 7 lines
kernel/sched/ext/ext.c +2
-1@@ -2256,7 +2256,8 @@ static void move_local_task_to_local_dsq(struct scx_sched *sch,* * Move @p which is currently on @src_rq to @dst_rq's local DSQ. */ -static void move_remote_task_to_local_dsq (struct task_struct *p,u64 enq_flags, +static void move_remote_task_to_local_dsq (struct scx_sched *sch, +struct task_struct *p,u64 enq_flags,struct rq *src_rq,struct rq *dst_rq)
nit: since the function gained the new @sch parameter, maybe add "@sch: scx_sched being operated on" to the kernel-doc block.
Thanks,
-Andrea
Quoted 114 lines
kernel/sched/ext/ext.c +13
-11{lockdep_assert_rq_held (src_rq);@@ -2273,15 +2274,19 @@ static void move_remote_task_to_local_dsq(struct task_struct *p, u64 enq_flags,raw_spin_rq_lock (dst_rq);/* -* We want to pass scx-specific enq_flags but activate_task() will -* truncate the upper 32 bit. As we own @rq, we can pass them through -* @rq->scx.remote_activate_enq_flags instead. +* activate_task() below truncates enq_flags to 32 bits and re-derives +* @p's owner, dropping our scx flags and the placing @sch. We own @rq, +* so stash both across the call. The enqueue reads them back, keeping +* the scx flags and checking caps against the placer, not the owner. */ WARN_ON_ONCE (!cpumask_test_cpu (cpu_of (dst_rq), p->cpus_ptr)); -WARN_ON_ONCE (dst_rq->scx.remote_activate_enq_flags); +WARN_ON_ONCE (dst_rq->scx.remote_activate_enq_flags || + dst_rq->scx.remote_activate_sch); dst_rq->scx.remote_activate_enq_flags = enq_flags; + dst_rq->scx.remote_activate_sch = sch;activate_task (dst_rq, p,0 ); dst_rq->scx.remote_activate_enq_flags =0 ; + dst_rq->scx.remote_activate_sch =NULL ; }/* @@ -2410,14 +2415,14 @@ static bool unlink_dsq_and_lock_src_rq(struct task_struct *p, !WARN_ON_ONCE (src_rq !=task_rq (p)); } -static bool consume_remote_task (struct rq *this_rq, +static bool consume_remote_task (struct scx_sched *sch,struct rq *this_rq,struct task_struct *p,u64 enq_flags,struct scx_dispatch_q *dsq,struct rq *src_rq) {raw_spin_rq_unlock (this_rq);if (unlink_dsq_and_lock_src_rq (p, dsq, src_rq)) { -move_remote_task_to_local_dsq (p, enq_flags, src_rq, this_rq); +move_remote_task_to_local_dsq (sch, p, enq_flags, src_rq, this_rq);return true; }else {raw_spin_rq_unlock (src_rq);@@ -2479,8 +2484,7 @@ static struct rq *move_task_between_dsqs(struct scx_sched *sch,raw_spin_unlock (&src_dsq->lock); }else {raw_spin_unlock (&src_dsq->lock); -move_remote_task_to_local_dsq (p, enq_flags, - src_rq, dst_rq); +move_remote_task_to_local_dsq (sch, p, enq_flags, src_rq, dst_rq); } }else {/* @@ -2533,7 +2537,7 @@ bool scx_consume_dispatch_q(struct scx_sched *sch, struct rq *rq, }if (task_can_run_on_remote_rq (sch, p, rq, false)) { -if (likely (consume_remote_task (rq, p, enq_flags, dsq, task_rq))) +if (likely (consume_remote_task (sch, rq, p, enq_flags, dsq, task_rq)))return true; goto retry; }@@ -2627,8 +2631,7 @@ static void dispatch_to_local_dsq(struct scx_sched *sch, struct rq *rq,scx_dispatch_enqueue (sch, src_rq,find_global_dsq (sch,task_cpu (p)), p, enq_flags |SCX_ENQ_GDSQ_FALLBACK ); }else { -move_remote_task_to_local_dsq (p, enq_flags, - src_rq, dst_rq); +move_remote_task_to_local_dsq (sch, p, enq_flags, src_rq, dst_rq);/* task has been moved to dst_rq, which is now locked */ locked_rq = dst_rq; }kernel/sched/ext/sub.c +7
-2@@ -237,12 +237,17 @@ struct scx_dispatch_q *scx_local_or_reject_dsq(struct scx_sched *sch, struct rq struct task_struct *p, u64 *enq_flags) { s32 cid = __scx_cpu_to_cid(cpu_of(rq)); +struct scx_sched *asch = rq->scx.remote_activate_sch ?: sch;u64 needed =scx_caps_for_enq (*enq_flags);u64 missing; +/* +* On a remote activation the scheduling sched (@asch) differs from +* @p's owner (@sch). Check caps against the scheduling sched. +*/ if (*enq_flags &SCX_ENQ_PREEMPT ) - needed |=scx_caps_for_preempt (sch, rq); - missing =scx_missing_caps (sch,cpu_of (rq), needed); + needed |=scx_caps_for_preempt (asch, rq); + missing =scx_missing_caps (asch,cpu_of (rq), needed);/* requirements met */ if (likely (!missing))kernel/sched/sched.h +2
-1@@ -800,8 +800,9 @@ struct scx_rq {struct list_head runnable_list;/* runnable tasks on this rq */ struct list_head ddsp_deferred_locals;/* deferred ddsps from enq */ unsigned long ops_qseq; -/* see move_remote_task_to_local_dsq() */ +/* both stashed across the activate_task() in move_remote_task_to_local_dsq() */ u64 remote_activate_enq_flags; +struct scx_sched *remote_activate_sch;u32 nr_running;u32 cpuperf_target;/* [0, SCHED_CAPACITY_SCALE] */ bool in_select_cpu;Signature · 1 line
2.54.0
Tejun Heo
reviewerPATCH v4 35/40 sched_ext: Route ops.update_idle() to sub-schedulers and re-notify owed scheds
In reply to Tejun Heo__scx_update_idle() notified only the root scheduler. A sub-scheduler that holds a cid needs that cid's idle state to place and kick on it.
Deliver ops.update_idle() to every scheduler that holds SCX_CAP_BASE on the transitioning cid. The root holds every cap, so a real transition always reaches it.
Real transitions are not enough on their own. A cid that is already idle when a sub-sched gains baseline access produces no transition, so the new holder would never learn it is idle. The ecaps sync arms a re-notify on the gain, and the next idle pick delivers ops.update_idle() to just that sched, leaving holders that already track the cpu untouched. A matching loss of baseline access drops any pending re-notify.
Bypass suppresses ops.update_idle() too, so a cpu that goes idle during a bypass window and stays idle yields no transition to re-deliver on un-bypass. Arm the same re-notify for every sched leaving bypass. The acute case is a child granted cids during its own ops.sub_attach(). The grant lands while the child is bypassed and the notify walk skips it, so on un-bypass it holds cids it never saw go idle. The root is owed the same and is armed through a separate per-rq flag, which keeps this working when sub-schedulers are compiled out.
v2: Gate the idle catch-up in pick_task_idle() to avoid a double ops.update_idle(). (sashiko AI)
--- kernel/sched/ext/ext.c | 40 ++++++++++++++++- kernel/sched/ext/idle.c | 86 ++++++++++++++++++++++++++----------- kernel/sched/ext/internal.h | 2 + kernel/sched/ext/sub.c | 24 ++++++++++- kernel/sched/idle.c | 8 +++- kernel/sched/sched.h | 2 + 6 files changed, 133 insertions(+), 29 deletions(-)
kernel/sched/ext/ext.c +38 -2
@@ -5500,6 +5500,38 @@ void scx_disable_bypass_dsp(struct scx_sched *sch) } } +/** +* unbypass_renotify_idle - Arm an idle re-notify for a sched leaving bypass +* @rq: rq of the cpu leaving bypass +* @pos: scheduler that just left bypass on @rq's cpu +* @pcpu: @pos's per-cpu state for @rq's cpu +* +* A sched leaving bypass is owed the ops.update_idle() calls suppressed while +* bypassing. A cpu that goes idle during the bypass window and stays idle won't +* produce a notification. Arm a re-notify that scx_bypass()'s resched flushes +* on the next idle pick. +* +* An acute case is ops.sub_attach(). If the parent grants the child cids while +* attaching, when attach is complete and bypass is lifted, the child may hold +* idle cids it never saw go idle. +* +* The root is no exception as bypass suppresses its notifications the same way. +* However, the root uses a separate per-rq flag so its re-notify keeps working +* even when !CONFIG_EXT_SUB_SCHED. +*/ +static void unbypass_renotify_idle (struct rq *rq,struct scx_sched *pos, +struct scx_sched_pcpu *pcpu) +{ +if (pos == scx_root) { + rq->scx.flags |=SCX_RQ_ROOT_IDLE_RENOTIFY ; +return ; + } +#ifdef CONFIG_EXT_SUB_SCHED + pcpu->idle_renotify = true; + rq->scx.flags |=SCX_RQ_SUB_IDLE_RENOTIFY ; +#endif +} +/** * scx_bypass - [Un]bypass scx_ops and guarantee forward progress * @sch: sched to bypass
@@ -5583,11 +5615,15 @@ void scx_bypass(struct scx_sched *sch, bool bypass)scx_for_each_descendant_pre (pos, sch) {struct scx_sched_pcpu *pcpu =per_cpu_ptr (pos->pcpu, cpu); +bool was_bypassing = pcpu->flags &SCX_SCHED_PCPU_BYPASSING ; -if (pos->bypass_depth) +if (pos->bypass_depth) { pcpu->flags |=SCX_SCHED_PCPU_BYPASSING ; -else + }else { pcpu->flags &= ~SCX_SCHED_PCPU_BYPASSING ; +if (was_bypassing) +unbypass_renotify_idle (rq, pos, pcpu); + } }raw_spin_unlock (&scx_sched_lock);
kernel/sched/ext/idle.c +61 -25
117 lines not on this page · raw message
kernel/sched/ext/internal.h +2 -0
@@ -1244,6 +1244,8 @@ struct scx_sched_pcpu {*/ u64 ecaps;struct llist_node ecaps_to_sync_node; +/* owed a forced update_idle() re-notify on this cpu */ +bool idle_renotify;/* effective caps as of the last sub_ecaps_updated() delivery */ u64 reported_ecaps;#endif
kernel/sched/ext/sub.c +23 -1
@@ -483,6 +483,10 @@ static void discard_queued_syncs(struct rq *rq)* pshard->caps[] is the target configuration. pcpu->ecaps is the effective * transposed copy owned by the cid's cpu and written only here under @rq's * lock. +* +* A sched that newly gains baseline access here is owed an update_idle() so it +* learns the cid's idle state. Such a gain arms the per-rq +* %SCX_RQ_SUB_IDLE_RENOTIFY gate so the next idle pick delivers it. */ void scx_process_sync_ecaps (struct rq *rq,struct task_struct *prev) {
@@ -518,7 +522,7 @@ void scx_process_sync_ecaps(struct rq *rq, struct task_struct *prev)struct scx_sched_pcpu *pcpu = container_of(pos, struct scx_sched_pcpu, ecaps_to_sync_node);struct scx_pshard *ps = pcpu->sch->pshard[shard]; -u64 old, ecaps, lost; +u64 old, ecaps, lost, gained;init_llist_node (pos);
@@ -530,6 +534,7 @@ void scx_process_sync_ecaps(struct rq *rq, struct task_struct *prev)WRITE_ONCE (pcpu->ecaps, ecaps); lost = old & ~ecaps; + gained = ecaps & ~old; lost_all |= lost;/* tell the sched its effective caps on this cid changed */
@@ -547,6 +552,18 @@ void scx_process_sync_ecaps(struct rq *rq, struct task_struct *prev)scx_flush_dispatch_buf (pcpu->sch, rq); pcpu->reported_ecaps = ecaps; } + +/* +* Gaining baseline access owes an update_idle() so the sched +* learns the cpu's idle state. Arm the per-rq gate so the next +* idle pick flushes it. Losing access drops any pending notify. +*/ +if (gained &SCX_CAP_BASE ) { + pcpu->idle_renotify = true; + rq->scx.flags |=SCX_RQ_SUB_IDLE_RENOTIFY ; + }else if (lost &SCX_CAP_BASE ) { + pcpu->idle_renotify = false; + } }/*
@@ -1441,6 +1458,11 @@ __bpf_kfunc s32 scx_bpf_sub_grant(u64 cgroup_id, u64 caps,caps_updated_record (cps, changed_cids, granted_caps, &to_deliver); +/* +* The sync arms an update_idle() re-notify if +* the cid gains baseline access, so the holder +* learns of an already-idle cid. +*/ scx_cmask_for_each_cid (cid, changed_cids)queue_sync_ecaps (child, cid); }
kernel/sched/idle.c +7 -1
@@ -503,7 +503,13 @@ static void set_next_task_idle(struct rq *rq, struct task_struct *next, bool firstruct task_struct *pick_task_idle (struct rq *rq,struct rq_flags *rf) { -scx_update_idle (rq, true, false); +/* +* Notify scx only on an idle-to-idle re-pick (the cpu was already idle). +* A real task->idle transition is delivered by set_next_task_idle(), so +* calling here too would duplicate it. +*/ +if (scx_enabled () &&is_idle_task (rq->curr)) +scx_update_idle (rq, true, false);return rq->idle; }
kernel/sched/sched.h +2 -0
@@ -787,6 +787,8 @@ enum scx_rq_flags {SCX_RQ_BAL_KEEP =1 <<3 ,/* balance decided to keep current */ SCX_RQ_CLK_VALID =1 <<5 ,/* RQ clock is fresh and valid */ SCX_RQ_BAL_CB_PENDING =1 <<6 ,/* must queue a cb after dispatching */ +SCX_RQ_SUB_IDLE_RENOTIFY =1 <<7 ,/* sub-scheds are owed update_idle() */ +SCX_RQ_ROOT_IDLE_RENOTIFY =1 <<8 ,/* the root is owed update_idle() */ SCX_RQ_IN_WAKEUP =1 <<16 , SCX_RQ_IN_BALANCE = 1 << 17,
Signature · 1 line
2.54.0
Tejun Heo
reviewerPATCH v4 36/40 sched_ext: Replay ecaps notifications suppressed by bypass
In reply to Tejun Heoscx_process_sync_ecaps() consumes ecaps syncs while the sched is bypassing without delivering ops.sub_ecaps_updated(), leaving reported_ecaps stale. Nothing re-queued a sync when bypass lifted, so a cid whose caps never change again would never be notified. Attach-time initial grants hit this every time: they are consumed during the enable bypass window, so a sched never learned its initial effective caps through the callback.
Re-queue a sync for every (sched, cpu) with an undelivered delta at the per-cpu bypass exit in scx_bypass(), next to the idle renotify catch-up. The next balance on the cpu then delivers the pending delta with proper dispatch context.
--- kernel/sched/ext/ext.c | 4 +++- kernel/sched/ext/sub.c | 35 +++++++++++++++++++++++++++++++++++ kernel/sched/ext/sub.h | 2 ++ 3 files changed, 40 insertions(+), 1 deletion(-)
kernel/sched/ext/ext.c +3 -1
@@ -5621,8 +5621,10 @@ void scx_bypass(struct scx_sched *sch, bool bypass) pcpu->flags |=SCX_SCHED_PCPU_BYPASSING ; }else { pcpu->flags &= ~SCX_SCHED_PCPU_BYPASSING ; -if (was_bypassing) +if (was_bypassing) {unbypass_renotify_idle (rq, pos, pcpu); +scx_unbypass_replay_ecaps (rq, pos); + } } }
kernel/sched/ext/sub.c +35 -0
@@ -575,6 +575,41 @@ void scx_process_sync_ecaps(struct rq *rq, struct task_struct *prev)scx_schedule_reenq_local (rq,SCX_REENQ_CAP_REVOKE ); } +/** +* scx_unbypass_replay_ecaps - Replay a bypass-suppressed ecaps notification +* @rq: rq of the cpu leaving bypass +* @sch: scheduler that just left bypass on @rq's cpu +* +* scx_process_sync_ecaps() consumes syncs while bypassing without delivering +* ops.sub_ecaps_updated(), leaving reported_ecaps stale. Nothing re-queues a +* sync when bypass lifts, so without a replay a cid that never changes again +* would never be notified. The attach-time initial grants are the acute case +* as they are consumed during the enable bypass window. Re-queue a sync for +* any undelivered delta so the next balance delivers it. +*/ +void scx_unbypass_replay_ecaps (struct rq *rq,struct scx_sched *sch) +{ +s32 cpu =cpu_of (rq); +struct scx_sched_pcpu *pcpu =per_cpu_ptr (sch->pcpu, cpu); +struct scx_pshard *ps; +s32 cid; + +lockdep_assert_rq_held (rq); + +/* root holds every cap and never uses ecaps */ +if (!sch->level) +return ; + +if (READ_ONCE (pcpu->ecaps) == pcpu->reported_ecaps) +return ; + + cid =__scx_cpu_to_cid (cpu); + ps = sch->pshard[scx_cid_to_shard[cid]]; + +guard (raw_spinlock)(&ps->lock); +queue_sync_ecaps (sch, cid); +} +/* * A cpu came back. Re-seed each sub-sched's ecaps on the cpu's cid. The sync * recomputes effective caps from the pshard and fires ops.sub_ecaps_updated()
kernel/sched/ext/sub.h +2 -0
@@ -30,6 +30,7 @@ void scx_free_pshards(struct scx_sched *sch);s32 scx_alloc_pshards (struct scx_sched *sch);void scx_init_root_caps (struct scx_sched *sch);void scx_process_sync_ecaps (struct rq *rq,struct task_struct *prev); +void scx_unbypass_replay_ecaps (struct rq *rq,struct scx_sched *sch);void scx_online_ecaps (struct rq *rq);void scx_offline_ecaps (struct rq *rq);void scx_discard_ecaps_to_sync (s32 cpu,struct scx_sched_pcpu *pcpu);
@@ -52,6 +53,7 @@ static inline void scx_free_pshards(struct scx_sched *sch) {}static inline s32 scx_alloc_pshards (struct scx_sched *sch) {return 0 ; }static inline void scx_init_root_caps (struct scx_sched *sch) {}static inline void scx_process_sync_ecaps (struct rq *rq,struct task_struct *prev) {} +static inline void scx_unbypass_replay_ecaps (struct rq *rq,struct scx_sched *sch) {}static inline void scx_online_ecaps (struct rq *rq) {}static inline void scx_offline_ecaps (struct rq *rq) {}static inline void scx_discard_ecaps_to_sync (s32 cpu,struct scx_sched_pcpu *pcpu) {}
Signature · 1 line
2.54.0
Tejun Heo
reviewerPATCH v4 37/40 sched_ext: Add scx_bpf_sub_kill() to evict a child sub-scheduler
In reply to Tejun HeoA cid-form scheduler can grant caps to and revoke them from its child sub-schedulers but has no way to tear one down. Add scx_bpf_sub_kill() to evict a direct child with a printf-style reason that reaches the child's scx_exit_info. No exit code is taken because the child is a separate scheduler whose exit-code semantics the parent cannot know. The child and its subtree are disabled through the usual async path under a new exit kind, SCX_EXIT_PARENT_KILL.
The bstr formatting infrastructure in ext.c is exposed through internal.h with scx_ prefixes so the kfunc, which lives in sub.c, can format the reason.
--- kernel/sched/ext/ext.c | 28 +++++++----- kernel/sched/ext/internal.h | 11 +++++ kernel/sched/ext/sub.c | 57 ++++++++++++++++++++++++ tools/sched_ext/include/scx/common.bpf.h | 18 ++++++++ 4 files changed, 102 insertions(+), 12 deletions(-)
kernel/sched/ext/ext.c +16 -12
@@ -187,14 +187,8 @@ static const struct rhashtable_params dsq_hash_params = {static LLIST_HEAD (dsqs_to_free); -/* string formatting from BPF */ -struct scx_bstr_buf { -u64 data[MAX_BPRINTF_VARARGS ]; -char line[SCX_EXIT_MSG_LEN ]; -}; - -static DEFINE_RAW_SPINLOCK (scx_exit_bstr_buf_lock); -static struct scx_bstr_buf scx_exit_bstr_buf; +DEFINE_RAW_SPINLOCK (scx_exit_bstr_buf_lock); +struct scx_bstr_buf scx_exit_bstr_buf;/* ops debug dump */ static DEFINE_RAW_SPINLOCK (scx_dump_lock);
@@ -5714,6 +5708,8 @@ static const char *scx_exit_reason(enum scx_exit_kind kind)return "disabled by sysrq-S" ;case SCX_EXIT_PARENT :return "parent exiting" ; +case SCX_EXIT_PARENT_KILL : +return "killed by parent scheduler" ;case SCX_EXIT_ERROR :return "runtime error" ;case SCX_EXIT_ERROR_BPF :
@@ -9331,8 +9327,8 @@ static s32 __bstr_format(struct scx_sched *sch, u64 *data_buf, char *line_buf, } __printf(3, 0) -static s32 bstr_format (struct scx_sched *sch,struct scx_bstr_buf *buf, -char *fmt,unsigned long long *data,u32 data__sz) +s32 scx_bstr_format (struct scx_sched *sch,struct scx_bstr_buf *buf, +char *fmt,unsigned long long *data,u32 data__sz) {return __bstr_format (sch, buf->data, buf->line,sizeof (buf->line), fmt, data, data__sz);
@@ -9362,7 +9358,7 @@ __bpf_kfunc void scx_bpf_exit_bstr(s64 exit_code, char *fmt,raw_spin_lock_irqsave (&scx_exit_bstr_buf_lock, flags); sch =scx_prog_sched (aux);if (likely (sch) && -bstr_format (sch, &scx_exit_bstr_buf, fmt, data, data__sz) >=0 ) +scx_bstr_format (sch, &scx_exit_bstr_buf, fmt, data, data__sz) >=0 )scx_exit (sch,SCX_EXIT_UNREG_BPF , exit_code,"%s" , scx_exit_bstr_buf.line);raw_spin_unlock_irqrestore (&scx_exit_bstr_buf_lock, flags); }
@@ -9387,7 +9383,7 @@ __bpf_kfunc void scx_bpf_error_bstr(char *fmt, unsigned long long *data,raw_spin_lock_irqsave (&scx_exit_bstr_buf_lock, flags); sch =scx_prog_sched (aux);if (likely (sch) && -bstr_format (sch, &scx_exit_bstr_buf, fmt, data, data__sz) >=0 ) +scx_bstr_format (sch, &scx_exit_bstr_buf, fmt, data, data__sz) >=0 )scx_exit (sch,SCX_EXIT_ERROR_BPF ,0 ,"%s" , scx_exit_bstr_buf.line);raw_spin_unlock_irqrestore (&scx_exit_bstr_buf_lock, flags); }
@@ -10023,6 +10019,13 @@ __bpf_kfunc s32 scx_bpf_sub_caps(u64 cgroup_id, u64 caps, struct scx_cmask *out_ {return -EOPNOTSUPP ; } + +__bpf_kfunc s32 scx_bpf_sub_kill_bstr(u64 cgroup_id, char *fmt, +unsigned long long *data,u32 data__sz, +const struct bpf_prog_aux *aux) +{ +return -EOPNOTSUPP ; +}#endif /* !CONFIG_EXT_SUB_SCHED */ __bpf_kfunc_end_defs ();
@@ -10072,6 +10075,7 @@ BTF_ID_FLAGS(func, scx_bpf_task_cgroup, KF_IMPLICIT_ARGS | KF_RCU | KF_ACQUIRE) BTF_ID_FLAGS(func, scx_bpf_sub_grant, KF_IMPLICIT_ARGS) BTF_ID_FLAGS(func, scx_bpf_sub_revoke, KF_IMPLICIT_ARGS) BTF_ID_FLAGS(func, scx_bpf_sub_caps, KF_IMPLICIT_ARGS) +BTF_ID_FLAGS(func, scx_bpf_sub_kill_bstr, KF_IMPLICIT_ARGS)BTF_KFUNCS_END (scx_kfunc_ids_any) static const struct btf_kfunc_id_set scx_kfunc_set_any = {
kernel/sched/ext/internal.h +11 -0
@@ -51,6 +51,7 @@ enum scx_exit_kind {SCX_EXIT_UNREG_KERN ,/* kernel-initiated unregistration */ SCX_EXIT_SYSRQ ,/* requested by 'S' sysrq */ SCX_EXIT_PARENT ,/* parent exiting */ +SCX_EXIT_PARENT_KILL ,/* killed by parent scheduler */ SCX_EXIT_ERROR =1024 ,/* runtime error, error msg contains details */ SCX_EXIT_ERROR_BPF, /* ERROR but triggered through scx_bpf_error() */
@@ -1875,6 +1876,12 @@ struct scx_enable_cmd {int ret; }; +/* string formatting from BPF */ +struct scx_bstr_buf { +u64 data[MAX_BPRINTF_VARARGS ]; +char line[SCX_EXIT_MSG_LEN ]; +}; +extern struct scx_sched __rcu *scx_root; DECLARE_PER_CPU(struct rq *, scx_locked_rq_state);
@@ -1935,10 +1942,14 @@ struct scx_sched *scx_alloc_and_add_sched(struct scx_enable_cmd *cmd,int scx_validate_ops (struct scx_sched *sch,const struct sched_ext_ops *ops);int scx_sched_sysfs_add (struct scx_sched *sch);bool scx_is_descendant (struct scx_sched *sch,struct scx_sched *ancestor); +__printf(3, 0) s32 scx_bstr_format(struct scx_sched *sch, struct scx_bstr_buf *buf, +char *fmt,unsigned long long *data,u32 data__sz);extern raw_spinlock_t scx_sched_lock;extern struct mutex scx_enable_mutex;extern struct percpu_rw_semaphore scx_fork_rwsem; +extern raw_spinlock_t scx_exit_bstr_buf_lock; +extern struct scx_bstr_buf scx_exit_bstr_buf;#ifdef CONFIG_EXT_SUB_SCHED extern const struct rhashtable_params scx_sched_hash_params;extern struct rhashtable scx_sched_hash;
kernel/sched/ext/sub.c +57 -0
63 lines not on this page · raw message
tools/sched_ext/include/scx/common.bpf.h +18 -0
@@ -119,6 +119,8 @@ s32 scx_bpf_sub_grant(u64 cgroup_id, u64 caps, const struct scx_cmask *cmask, struct scx_cmask *denied) __ksym __weak; void scx_bpf_sub_revoke(u64 cgroup_id, u64 caps, const struct scx_cmask *cmask) __ksym __weak; s32 scx_bpf_sub_caps(u64 cgroup_id, u64 caps, struct scx_cmask *out) __ksym __weak; +s32 scx_bpf_sub_kill_bstr(u64 cgroup_id, char *fmt, + unsigned long long *data, u32 data__sz) __ksym __weak;/* * Use the following as @it__iter when calling scx_bpf_dsq_move[_vtime]() from
@@ -165,6 +167,22 @@ void ___scx_bpf_bstr_format_checker(const char *fmt, ...) {} ___scx_bpf_bstr_format_checker(fmt, ##args); \ }) +/* +* scx_bpf_sub_kill() wraps the scx_bpf_sub_kill_bstr() kfunc with variadic +* arguments instead of an array of u64. It kills the direct child sub-scheduler +* @cgid, passing the formatted reason to its user space, and evaluates to the +* kfunc's return value. On a kernel without sub-scheduler support the kfunc is +* absent and it returns -EOPNOTSUPP. +*/ +#define scx_bpf_sub_kill(cgid, fmt, args...) \ +({ \ + scx_bpf_bstr_preamble(fmt, args) \ + ___scx_bpf_bstr_format_checker(fmt, ##args); \ + bpf_ksym_exists(scx_bpf_sub_kill_bstr) ? \ + scx_bpf_sub_kill_bstr((cgid), ___fmt, ___param, \ + sizeof(___param)) : -EOPNOTSUPP; \ +}) +/* * scx_bpf_error() wraps the scx_bpf_error_bstr() kfunc with variadic arguments * instead of an array of u64. Invoking this macro will cause the scheduler to
Signature · 1 line
2.54.0
[email protected]
AI reviewRe: [PATCH v4 sched_ext/for-7.3 37/40] sched_ext: Add scx_bpf_sub_kill() to evict a child sub-scheduler
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] Missing `__str` suffix on the `fmt` parameter in `scx_bpf_sub_kill_bstr()` allows out-of-bounds…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] Missing __str suffix on the fmt parameter in scx_bpf_sub_kill_bstr() allows out-of-bounds kernel memory read. --
--- Patch [37]: [PATCH v4 sched_ext/for-7.3 37/40] sched_ext: Add scx_bpf_sub_kill() to evict a child sub-scheduler --- commit dffeca7158a063ab8ec3cb4f59c9c43d77050c68
Author: Tejun Heo <[email protected]>sched_ext: Add scx_bpf_sub_kill() to evict a child sub-scheduler
This commit adds scx_bpf_sub_kill() to allow a cid-form scheduler to evict a direct child sub-scheduler. The child is torn down asynchronously via the exit kind SCX_EXIT_PARENT_KILL. It exposes bstr formatting infrastructure to support this.
Quoted 4 lines
kernel/sched/ext/ext.c +0
-0
[ ... ]
kernel/sched/ext/ext.c __bpf_kfunc void scx_bpf_exit_bstr(s64 exit_code, char *fmt, · line 9362
Quoted 7 lines
kernel/sched/ext/ext.c +1
-1@@ -9362,7 +9358,7 @@ __bpf_kfunc void scx_bpf_exit_bstr(s64 exit_code, char *fmt,raw_spin_lock_irqsave (&scx_exit_bstr_buf_lock, flags); sch =scx_prog_sched (aux);if (likely (sch) && -bstr_format (sch, &scx_exit_bstr_buf, fmt, data, data__sz) >=0 ) +scx_bstr_format (sch, &scx_exit_bstr_buf, fmt, data, data__sz) >=0 )scx_exit (sch,SCX_EXIT_UNREG_BPF , exit_code,"%s" , scx_exit_bstr_buf.line);
[Severity: High]
This is a pre-existing issue, but does scx_bpf_exit_bstr() allow an out-of-bounds kernel memory read by omitting the __str suffix from the fmt parameter?
Since fmt does not have the __str suffix (i.e. fmt__str), the BPF verifier treats it as KF_ARG_PTR_TO_MEM instead of KF_ARG_PTR_TO_CONST_STR and does not check for a NUL terminator at load time.
If a BPF program passes a non-NUL-terminated buffer, the fmt pointer is ultimately passed to bpf_bprintf_prepare() in __bstr_format() with fmt_size = UINT_MAX, which calls strnchr(fmt, UINT_MAX, 0). This could loop over unmapped memory boundaries and cause a page fault.
kernel/sched/ext/ext.c __bpf_kfunc void scx_bpf_error_bstr(char *fmt, unsigned long long *data, · line 9387
Quoted 7 lines
kernel/sched/ext/ext.c +1
-1@@ -9387,7 +9383,7 @@ __bpf_kfunc void scx_bpf_error_bstr(char *fmt, unsigned long long *data,raw_spin_lock_irqsave (&scx_exit_bstr_buf_lock, flags); sch =scx_prog_sched (aux);if (likely (sch) && -bstr_format (sch, &scx_exit_bstr_buf, fmt, data, data__sz) >=0 ) +scx_bstr_format (sch, &scx_exit_bstr_buf, fmt, data, data__sz) >=0 )scx_exit (sch,SCX_EXIT_ERROR_BPF ,0 ,"%s" , scx_exit_bstr_buf.line);
[Severity: High]
This is a pre-existing issue, but does scx_bpf_error_bstr() also suffer from the same missing __str suffix on the fmt parameter?
[ ... ]
Quoted 4 lines
kernel/sched/ext/sub.c +0
-0
[ ... ]
kernel/sched/ext/sub.c __bpf_kfunc s32 scx_bpf_sub_caps(u64 cgroup_id, u64 caps, struct scx_cmask *out_ · line 1721
Quoted 5 lines
kernel/sched/ext/sub.c +5
-0+__printf(2, 0) +__bpf_kfunc s32 scx_bpf_sub_kill_bstr(u64 cgroup_id, char *fmt, +unsigned long long *data,u32 data__sz, +const struct bpf_prog_aux *aux) +{
[Severity: High]
Does scx_bpf_sub_kill_bstr() introduce the same vulnerability by missing the __str suffix on the fmt parameter?
Could we change this parameter to fmt__str to ensure the BPF verifier enforces a NUL-terminated constant string?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=37
Tejun Heo
reviewerRe: [PATCH v4 sched_ext/for-7.3 37/40] sched_ext: Add scx_bpf_sub_kill() to evict a child sub-scheduler
Quoted 2 lines · On Wed, 08 Jul 2026 22:11:10 +0000, [email protected] wrote:
- [High] Missing
__strsuffix on thefmtparameter inscx_bpf_sub_kill_bstr()allows out-of-bounds kernel memory read.
Same as the v2/v3 rounds - the fmt/data/data__sz triple follows the established scx _bstr kfunc convention, fmt is always a rodata literal from a privileged struct_ops program, and switching to __str would be a tree-wide change for the whole _bstr family rather than something for this patch:
https://lore.kernel.org/r/[email protected]
Thanks.
Signature · 1 line
tejun
Tejun Heo
reviewerPATCH v4 38/40 tools/sched_ext: Add three-mask cmask intersection iterator
In reply to Tejun HeoAdd cmask_next_and2_set() and its round-robin wrap, extending cmask_next_and_set() to a three-mask intersection: the next cid set in all three masks at or after @start. A caller iterating the intersection of three cmasks can then scan it in one pass, folding the third mask into the word-level AND rather than skipping non-members one candidate at a time.
--- tools/sched_ext/include/scx/cid.bpf.h | 77 +++++++++++++++++++++++++++ 1 file changed, 77 insertions(+)
tools/sched_ext/include/scx/cid.bpf.h +77 -0
@@ -668,6 +668,83 @@ static __always_inline u32 cmask_next_and_set_wrap(const struct scx_cmask __arenreturn found < start ? found : a_end; } +/* +* Like cmask_next_and_set() but over the intersection of THREE masks. Return +* a->base + a->nr_cids if no cid is set in all three at or after @start. +*/ +static __always_inline u32 cmask_next_and2_set(const struct scx_cmask __arena *a, + const struct scx_cmask __arena *b, + const struct scx_cmask __arena *c, +u32 start) +{ +u32 a_end = a->base + a->nr_cids; +u32 b_end = b->base + b->nr_cids; +u32 c_end = c->base + c->nr_cids; +u32 a_wbase = a->base /64 ; +u32 b_wbase = b->base /64 ; +u32 c_wbase = c->base /64 ; +u32 lo = a->base > b->base ? a->base : b->base; +u32 hi = a_end < b_end ? a_end : b_end; +u32 last_wi, start_wi, start_bit, i; + + lo = lo > c->base ? lo : c->base; + hi = hi < c_end ? hi : c_end; + +if (lo >= hi) +return a_end; +if (start < lo) + start = lo; +if (start >= hi) +return a_end; + + last_wi = (hi -1 ) /64 ; + start_wi = start /64 ; + start_bit = start &63 ; + + bpf_for(i, 0, CMASK_MAX_WORDS) { +u32 abs_wi = start_wi + i; +u64 word; +u32 found; + +if (abs_wi > last_wi) +break ; + + word = a->bits[abs_wi - a_wbase] & b->bits[abs_wi - b_wbase] & + c->bits[abs_wi - c_wbase]; +if (i ==0 ) + word &=GENMASK_U64 (63 , start_bit); +if (!word) +continue ; + + found = abs_wi *64 +ctzll (word); +if (found >= hi) +return a_end; +return found; + } +return a_end; +} + +/* +* Round-robin variant of cmask_next_and2_set(): wrap to @a->base if the +* three-way intersection has no cid in the forward half. Return a->base + +* a->nr_cids if empty. +*/ +static __always_inline u32 cmask_next_and2_set_wrap(const struct scx_cmask __arena *a, + const struct scx_cmask __arena *b, + const struct scx_cmask __arena *c, +u32 start) +{ +u32 a_end = a->base + a->nr_cids; +u32 found; + + found =cmask_next_and2_set (a, b, c, start); +if (found < a_end || start <= a->base) +return found; + + found =cmask_next_and2_set (a, b, c, a->base); +return found < start ? found : a_end; +} +/** * cmask_from_cpumask - translate a kernel cpumask to a cid-space cmask * @m: cmask to fill. Zeroed first; only bits within [@m->base, @m->base +
Signature · 1 line
2.54.0
Tejun Heo
reviewerPATCH v4 39/40 tools/sched_ext: scx_qmap - Expand hierarchical sub-scheduling
In reply to Tejun Heosched_ext sub-scheduling began as dispatch delegation only: a parent could call into a child cgroup sub-scheduler's ops.dispatch() from its own dispatch path, but could not delegate cpus to the child for enqueue and the other paths. sched_ext has since gained cap-based cid delegation, where a parent grants and revokes a child's per-cid caps. Expand scx_qmap to demonstrate it.
scx_qmap can now delegate the cids it holds exclusively, split among itself and its children by cpu.weight. Each gets the floor of its share as dedicated cids. The leftover from rounding forms a shared pool, round-robined among them as an ENQ_IMMED time-share.
This shape is deliberate. Exclusive cids exercise the basic grant and revoke of ownership, and the shared pool exercises time-sharing one cid across several schedulers. The implemented policy is impractical, but it covers most of what a practical sub-scheduler would need without overcomplicating qmap.
Delegation nests. A cid a node receives from its parent only as a round-robin share stays self-local and is never re-delegated. A node left with no exclusive cid, e.g. after its cpus went offline, evicts its children.
v4: Track all idle cids and mask with self_cids at the dispatch pick, dropping the reseed. (sashiko AI)
v3: Dispatch IMMED flags, repartition accounting order, partition-input snapshot. (sashiko AI)
v2: Use __sync_fetch_and_add() for the shared nr_dsps counter. (sashiko AI)
--- tools/sched_ext/scx_qmap.bpf.c | 746 ++++++++++++++++++++++++++++++--- tools/sched_ext/scx_qmap.c | 296 ++++++++++++- tools/sched_ext/scx_qmap.h | 98 ++++- 3 files changed, 1076 insertions(+), 64 deletions(-)
tools/sched_ext/scx_qmap.bpf.c +691 -55
@@ -1,21 +1,39 @@/* SPDX-License-Identifier: GPL-2.0 */ /* -* A simple five-level FIFO queue scheduler. +* scx_qmap: a demonstration and testing scheduler for sched_ext features. * -* There are five FIFOs implemented as arena-backed doubly-linked lists -* threaded through per-task context. A task gets assigned to one depending on -* its compound weight. Each CPU round robins through the FIFOs and dispatches -* more from FIFOs with higher indices - 1 from queue0, 2 from queue1, 4 from -* queue2 and so on. -* -* This scheduler demonstrates: +* A simple scheduler that exercises a broad set of sched_ext features. Unlikely +* to be useful for real workloads. It demonstrates: * * - BPF-side queueing using TIDs. * - BPF arena for scheduler state. * - Core-sched support. +* - Hierarchical sub-scheduling: delegating cpus to child cgroup schedulers. +* +* Base design: Five FIFOs (arena-backed doubly-linked lists through per-task +* context). A task is assigned to a FIFO by its compound weight. Each cpu +* round-robins the FIFOs, dispatching more from higher ones. +* +* Sub-scheduling: Any qmap sched can delegate cpus to its own child cgroup +* schedulers and keep the rest for its tasks. Terminology: +* +* excl - A cpu the delegatee owns wholly (ENQ_IMMED|ENQ|PREEMPT). +* shared - A cpu delegated as ENQ_IMMED only. Time-shared. +* held_excl / held_shared - What this node was handed by its parent. +* held-excl cpus are re-delegatable. A held-shared cpu is a +* time-share that stays self-local. +* self - The excl cpus the node kept for itself, plus all of held_shared. +* owner - Who holds a cid - a child slot, CID_SELF, or CID_NONE. * -* This scheduler is primarily for demonstration and testing of sched_ext -* features and unlikely to be useful for actual workloads. +* The scheduler splits its held-excl cpus among self and the children in +* proportion to each node's cpu.weight, handing each the floor of its share as +* excl cpus. The leftover from rounding forms a shared pool the round-robin +* timer hands around. With no excl cpu to delegate, the node evicts its +* children. +* +* This policy is a demonstration only, not a practical one. The split +* considers only direct children and is not work-conserving. It only exists to +* drive sub-sched primitives with as simple logic as possible. * * Copyright (c) 2022 Meta Platforms, Inc. and affiliates. * Copyright (c) 2022 Tejun Heo <[email protected]>
@@ -52,6 +70,9 @@ const volatile bool always_enq_immed;const volatile u32 immed_stress_nth;const volatile u32 max_tasks; +/* sub-sched: period for handing the round-robin cid pool to the next child */ +const volatile u64 round_robin_ns; +/* * Optional cid-override test harness. When cid_override_mode is non-zero, * qmap_init_cids() calls scx_bpf_cid_override() with the caller-supplied arrays
@@ -89,12 +110,12 @@ struct { struct qmap_arena __arena_global qa; -/* -* Global idle-cid tracking, maintained via update_idle / cpu_offline and -* scanned by the direct-dispatch path. Allocated in qmap_init() from one -* arena page, sized to the full cid space. -*/ -struct scx_cmask __arena *qa_idle_cids; +/* ensure that BPF and userspace are seeing the same size for qmap_cmask */ +_Static_assert (QMAP_CMASK_WORDS ==CMASK_NR_WORDS (SCX_QMAP_MAX_CPUS ), +"QMAP_CMASK_WORDS must equal CMASK_NR_WORDS(SCX_QMAP_MAX_CPUS)" ); +_Static_assert (sizeof (struct qmap_cmask ) == + struct_size_t(struct scx_cmask, bits, QMAP_CMASK_WORDS), +"qmap_cmask must be exactly sized to back a full scx_cmask" );/* Per-queue locks. Each in its own .data section as bpf_res_spin_lock requires. */ __hidden struct bpf_res_spin_lock qa_q_lock0 SEC(".data.qa_q_lock0");
@@ -196,7 +217,7 @@ static int qmap_spin_lock(struct bpf_res_spin_lock *lock) }/* -* Try prev_cid, then scan taskc->cpus_allowed AND qa_idle_cids round-robin +* Try prev_cid, then scan cpus_allowed AND idle_cids AND self_cids round-robin * from prev_cid + 1. Atomic claim retries on race; bounded by * IDLE_PICK_RETRIES to keep the verifier's insn budget in check. */
@@ -212,17 +233,19 @@ static s32 pick_direct_dispatch_cid(struct task_struct *p, s32 prev_cid,if (!always_enq_immed && p->nr_cpus_allowed ==1 )return prev_cid; -if (cmask_test_and_clear (prev_cid, qa_idle_cids)) +if (cmask_test (prev_cid, &qa.self_cids.mask) && +cmask_test_and_clear (prev_cid, &qa.idle_cids.mask))return prev_cid; cid = prev_cid; bpf_for(i, 0, IDLE_PICK_RETRIES) { - cid =cmask_next_and_set_wrap (&taskc->cpus_allowed, - qa_idle_cids, cid +1 ); + cid =cmask_next_and2_set_wrap (&taskc->cpus_allowed, + &qa.idle_cids.mask, + &qa.self_cids.mask, cid +1 );barrier_var (cid);if (cid >= nr_cids)return -1 ; -if (cmask_test_and_clear (cid, qa_idle_cids)) +if (cmask_test_and_clear (cid, &qa.idle_cids.mask))return cid; }return -1 ;
@@ -346,6 +369,15 @@ s32 BPF_STRUCT_OPS(qmap_select_cid, struct task_struct *p, } } +/* +* A received time-shared cid is held ENQ_IMMED-only, so inserts must set +* SCX_ENQ_IMMED. +*/ +static u64 needs_immed (s32 cid) +{ +return qa.cid_shared[cid] ?SCX_ENQ_IMMED :0 ; +} +static int weight_to_idx (u32 weight) {/* Coarsely map the compound weight to a FIFO. */
@@ -369,9 +401,16 @@ void BPF_STRUCT_OPS(qmap_enqueue, struct task_struct *p, u64 enq_flags)s32 cid;if (enq_flags &SCX_ENQ_REENQ ) { +u64 reason = p->scx.flags &SCX_TASK_REENQ_REASON_MASK ; +__sync_fetch_and_add (&qa.nr_reenqueued,1 );if (scx_bpf_task_cid (p) ==0 )__sync_fetch_and_add (&qa.nr_reenqueued_cid0,1 ); +/* cap-loss and IMMED-handback bounces, relocated below */ +if (reason ==SCX_TASK_REENQ_CAP ) +__sync_fetch_and_add (&qa.nr_reenq_cap,1 ); +else if (reason ==SCX_TASK_REENQ_IMMED ) +__sync_fetch_and_add (&qa.nr_reenq_immed,1 ); }if (p->flags &PF_KTHREAD ) {
@@ -394,6 +433,27 @@ void BPF_STRUCT_OPS(qmap_enqueue, struct task_struct *p, u64 enq_flags)*/ taskc->core_sched_seq = qa.core_sched_tail_seqs[idx]++; +/* +* A node with children delegates most cids. A task of ours that can run +* on none of our self cids (e.g. a per-NUMA kthread pinned to delegated +* cids) would starve in SHARED/FIFO since we never pull those on a +* delegated cid. Force it onto its first allowed cid's local DSQ with +* needs_immed(): if we hold access there it runs, else the kernel +* rejects and bounces it back via REENQ_CAP. Best-effort +* anti-starvation nudge. +*/ +if (qa.nr_sub_scheds && !(enq_flags &SCX_ENQ_REENQ ) && + !cmask_intersects (&taskc->cpus_allowed, &qa.self_cids.mask)) { +s32 c =cmask_next_set_wrap (&taskc->cpus_allowed,0 ); + +if (c >=0 && c <scx_bpf_nr_cids ()) { + taskc->force_local = false; +scx_bpf_dsq_insert (p,SCX_DSQ_LOCAL_ON | c, slice_ns, + enq_flags |needs_immed (c)); +return ; + } + } +/* * IMMED stress testing: Every immed_stress_nth'th enqueue, dispatch * directly to prev_cpu's local DSQ even when busy to force dsq->nr > 1
@@ -416,7 +476,8 @@ void BPF_STRUCT_OPS(qmap_enqueue, struct task_struct *p, u64 enq_flags)*/ if (taskc->force_local) { taskc->force_local = false; -scx_bpf_dsq_insert (p,SCX_DSQ_LOCAL , slice_ns, enq_flags); +scx_bpf_dsq_insert (p,SCX_DSQ_LOCAL , slice_ns, + enq_flags |needs_immed (scx_bpf_task_cid (p)));return ; }
@@ -431,7 +492,8 @@ void BPF_STRUCT_OPS(qmap_enqueue, struct task_struct *p, u64 enq_flags)if (!__COMPAT_is_enq_cpu_selected (enq_flags) && (cid =pick_direct_dispatch_cid (p,scx_bpf_task_cid (p), taskc)) >=0 ) {__sync_fetch_and_add (&qa.nr_ddsp_from_enq,1 ); -scx_bpf_dsq_insert (p,SCX_DSQ_LOCAL_ON | cid, slice_ns, enq_flags); +scx_bpf_dsq_insert (p,SCX_DSQ_LOCAL_ON | cid, slice_ns, + enq_flags |needs_immed (cid));return ; }
@@ -445,8 +507,9 @@ void BPF_STRUCT_OPS(qmap_enqueue, struct task_struct *p, u64 enq_flags)s32 cid;scx_bpf_dsq_insert (p,SHARED_DSQ ,0 , enq_flags); - cid =cmask_next_and_set_wrap (&taskc->cpus_allowed, - qa_idle_cids,0 ); + cid =cmask_next_and2_set_wrap (&taskc->cpus_allowed, + &qa.idle_cids.mask, + &qa.self_cids.mask,0 );if (cid <scx_bpf_nr_cids ())scx_bpf_kick_cid (cid,SCX_KICK_IDLE );return ;
@@ -541,7 +604,7 @@ static bool dispatch_highpri(bool from_timer)continue ;if (scx_bpf_dsq_move (BPF_FOR_EACH_ITER , p,SCX_DSQ_LOCAL_ON | cid, -SCX_ENQ_PREEMPT )) { +SCX_ENQ_PREEMPT |needs_immed (cid))) {if (cid == this_cid) { dispatched = true;__sync_fetch_and_add (&qa.nr_expedited_local,1 );
@@ -567,12 +630,35 @@ void BPF_STRUCT_OPS(qmap_dispatch, s32 cid, struct task_struct *prev) struct cpu_ctx __arena *cpuc;task_ctx_t *taskc;u32 batch = dsp_batch ?:1 ; +s32 owner;s32 i;if (dispatch_highpri (false))return ; -if (!qa.nr_highpri_queued &&scx_bpf_dsq_move_to_local (SHARED_DSQ ,0 )) +/* +* Sub-sched routing: a child-owned cid goes to its owner. Never run +* this node's own tasks on a delegated cid. Read without the guard. +*/ + owner = qa.part.cid_owner[cid]; +if (owner ==CID_SHARED ) { +/* route to the live rr holder (0 = self, runs below) */ +s32 pos = qa.part.rr_pos; +u64 holder_cgid = (pos >=0 && pos <MAX_PARTS ) ? + qa.part.rr_slots[pos] :0 ; + +if (holder_cgid) { +scx_bpf_sub_dispatch (holder_cgid); +return ; + } + }else if (owner >=0 && owner <MAX_SUB_SCHEDS && + qa.sub_sched_ctxs[owner].cgroup_id) { +if (scx_bpf_sub_dispatch (qa.sub_sched_ctxs[owner].cgroup_id)) +__sync_fetch_and_add (&qa.sub_sched_ctxs[owner].nr_dsps,1 ); +return ; + } + +if (!qa.nr_highpri_queued &&scx_bpf_dsq_move_to_local (SHARED_DSQ ,needs_immed (cid)))return ;if (dsp_inf_loop_after && qa.nr_dispatched > dsp_inf_loop_after) {
@@ -662,7 +748,7 @@ void BPF_STRUCT_OPS(qmap_dispatch, s32 cid, struct task_struct *prev)if (!batch || !scx_bpf_dispatch_nr_slots ()) {if (dispatch_highpri (false))return ; -scx_bpf_dsq_move_to_local (SHARED_DSQ ,0 ); +scx_bpf_dsq_move_to_local (SHARED_DSQ ,needs_immed (cid));return ; }if (!cpuc->dsp_cnt)
@@ -672,12 +758,6 @@ void BPF_STRUCT_OPS(qmap_dispatch, s32 cid, struct task_struct *prev) cpuc->dsp_cnt =0 ; } -for (i =0 ; i <MAX_SUB_SCHEDS ; i++) { -if (qa.sub_sched_cgroup_ids[i] && -scx_bpf_sub_dispatch (qa.sub_sched_cgroup_ids[i])) -return ; - } -/* * No other tasks. @prev will keep running. Update its core_sched_seq as * if the task were enqueued and dispatched immediately.
@@ -910,10 +990,18 @@ void BPF_STRUCT_OPS(qmap_cgroup_set_bandwidth, struct cgroup *cgrp,void BPF_STRUCT_OPS (qmap_update_idle ,s32 cid,bool idle) {QMAP_TOUCH_ARENA (); + +/* +* The kernel delivers update_idle() for every cid this node holds +* SCX_CAP_BASE on. Track every cid's idle state regardless of +* delegation: the direct-dispatch pick masks idle_cids with self_cids +* at selection, so a cid already idle when it returns to self needs no +* reseed here. +*/ if (idle) -cmask_set (cid, qa_idle_cids); +cmask_set (cid, &qa.idle_cids.mask);else -cmask_clear (cid, qa_idle_cids); +cmask_clear (cid, &qa.idle_cids.mask); } void BPF_STRUCT_OPS(qmap_set_cmask, struct task_struct *p,
@@ -1065,6 +1153,486 @@ static int lowpri_timerfn(void *map, int *key, struct bpf_timer *timer)return 0 ; } +struct round_robin_timer { +struct bpf_timer timer; +}; + +struct { + __uint(type, BPF_MAP_TYPE_ARRAY); + __uint(max_entries, 1); + __type(key, u32); + __type(value, struct round_robin_timer); +} round_robin_timer SEC(".maps"); + +/* +* Partition update synchronization. qa.part can be written from concurrent +* contexts. This single-runner guard admits one writer at a time without +* holding a lock across the grant/revoke kfuncs. part_pending coalesces +* repartition requests that arrive while it is held. +* +* They live in .bss, not the arena: rr_advance() runs from a bpf_timer +* callback, where the verifier rejects atomic ops on arena memory. +*/ +static u64 part_busy; +static u64 part_pending; + +static bool part_try_start (void ) +{ +/* set busy, report whether it was previously clear (we acquired it) */ +return !__sync_fetch_and_or (&part_busy,1 ); +} + +static void part_end (void ) +{ +__sync_fetch_and_and (&part_busy,0 ); +} + +/* +* compute_partition() scratch. +* +* The excl-held cids are handed out in cid order: position 0..nr_excl-1 over +* the held cids is split into contiguous ranges, one per participant that gets +* at least one excl cid. Range k is owned by cp_range_owner[k] and ends at the +* cumulative position cp_range_end[k]. +*/ +static s32 cp_range_owner[MAX_PARTS ];/* exclusive range k: its owner id ... */ +static s32 cp_range_end[MAX_PARTS ];/* ... and the cumulative position it ends at */ + +/* a participant in the partition: self or an attached child */ +struct participant { +s32 slot;/* child slot, or CID_SELF */ +u32 weight;/* cpu.weight */ +}; + +/** +* place_one - assign one excl-held cid to its owner +* @cid: the excl-held cid to place +* @n: its position among the excl-held cids, in [0, nr_excl) +* @total_excl: how many positions are owned exclusively (the rest are shared) +* +* Position @n below @total_excl is owned exclusively. It falls in the range +* whose cumulative end it is under, owned by cp_range_owner[]. A position at or +* above @total_excl is the rounding leftover which joins the shared pool. +* +* A separate __noinline function to help verification. +*/ +__noinline int place_one(s32 cid, s32 n, s32 total_excl) +{ +s32 owner =CID_SELF , i, s; + +if (cid <0 || cid >=SCX_QMAP_MAX_CPUS || n <0 || n >=SCX_QMAP_MAX_CPUS || + total_excl <0 ) { +scx_bpf_error ("-ERANGE" ); +return 0 ; + } + +if (n < total_excl) { +for (i =0 ; i <MAX_PARTS ; i++) { +if (n < cp_range_end[i]) { + owner = cp_range_owner[i]; +break ; + } + } + qa.part.cid_owner[cid] = owner; + }else { + s = n - total_excl; +if (s <0 || s >=MAX_PARTS ) { +scx_bpf_error ("-ERANGE" ); +return 0 ; + } + qa.part.shared_cids[s] = cid; +/* time-shared: dispatch resolves the live holder via rr_pos */ + qa.part.cid_owner[cid] =CID_SHARED ; + } +return 0 ; +} + +/** +* compute_partition - build the cid partition from this node's held caps +* +* Decide each cid's owner, the shared pool and the rr rotation. __noinline to +* help verification. See the comment at the top of the file. +*/ +__noinline void compute_partition(void) +{ +s32 nr_cids = qa.nr_cids; +s32 nr_excl, total_excl =0 , nr_rr =0 ; +s32 sum_w, i, cid, n =0 , share, self_w; +u64 cgid_snap[MAX_SUB_SCHEDS ]; +s32 w_snap[MAX_SUB_SCHEDS ]; + +if (nr_cids >SCX_QMAP_MAX_CPUS ) { +scx_bpf_error ("-ERANGE" ); +return ; + } + +/* find out the cids we hold */ +scx_bpf_sub_caps (0 ,SCX_CAP_ENQ , (void *)(long )&qa.held_excl.mask); +scx_bpf_sub_caps (0 ,SCX_CAP_ENQ_IMMED , (void *)(long )&qa.held_shared.mask); +cmask_andnot (&qa.held_shared.mask, &qa.held_excl.mask);/* held only as ENQ_IMMED */ + + qa.part.nr_shared =0 ; + qa.part.nr_rr =0 ; + qa.part.rr_pos =0 ; + + nr_excl =cmask_weight (&qa.held_excl.mask); + qa.part.nr_excl = nr_excl; + +/* no excl cid: held_shared stays self-local, the rest unheld */ +if (!nr_excl) { + bpf_for(cid, 0, nr_cids) { +if (cmask_test (cid, &qa.held_shared.mask)) + qa.part.cid_owner[cid] =CID_SELF ; +else + qa.part.cid_owner[cid] =CID_NONE ; + } +return ; + } + +/* +* Snapshot membership and weights so the sum_w and share loops agree. +* A mid-compute change would otherwise wrap nr_shared negative. +*/ + self_w = qa.self_weight ?:100 ; + bpf_for(i, 0, MAX_SUB_SCHEDS) { + cgid_snap[i] = qa.sub_sched_ctxs[i].cgroup_id; + w_snap[i] = cgid_snap[i] ? (qa.sub_sched_ctxs[i].weight ?:100 ) :0 ; + } + +/* +* Participants are self plus each child. Give each a fixed range/rr +* slot: self at slot 0, child i at slot i+1. +* +* sum_w totals every participant's weight. +*/ + sum_w = self_w; + bpf_for(i, 0, MAX_SUB_SCHEDS) { +barrier_var (sum_w); + sum_w += w_snap[i]; + } + +/* +* Split [0, nr_excl) into one contiguous range per participant, each +* the floor of its weight share. cp_range_owner[]/cp_range_end[] record +* each range's owner and cumulative end, total_excl counts the +* exclusive slots, and the rest (nr_excl - total_excl) are shared. +* rr_slots[] lists every participant for the round-robin. +*/ + share = (u64 )nr_excl * self_w / sum_w; + total_excl += share; + cp_range_owner[0 ] =CID_SELF ; + cp_range_end[0 ] = total_excl; + qa.part.rr_slots[nr_rr++] =0 ;/* self holds slot 0 (cgid 0 = no grant) */ + + bpf_for(i, 0, MAX_SUB_SCHEDS) { +u64 cgid = cgid_snap[i]; +s32 w = w_snap[i]; + +barrier_var (total_excl); + share = (u64 )nr_excl * w / sum_w; + total_excl += share; + cp_range_owner[i +1 ] = cgid ? i :CID_NONE ; + cp_range_end[i +1 ] = total_excl; + +if (cgid) { +barrier_var (nr_rr); +if (nr_rr <0 || nr_rr >=MAX_PARTS ) { +scx_bpf_error ("-ERANGE" ); +return ; + } + qa.part.rr_slots[nr_rr++] = cgid; + } + } + +/* assign each cid: held-excl by position, the rest self/none */ + bpf_for(cid, 0, nr_cids) { +if (cmask_test (cid, &qa.held_excl.mask)) { +place_one (cid, n, total_excl); + n++; +barrier_var (n); + }else if (cmask_test (cid, &qa.held_shared.mask)) { + qa.part.cid_owner[cid] =CID_SELF ;/* time-share, self-local */ + }else { + qa.part.cid_owner[cid] =CID_NONE ;/* not held */ + } + } + + qa.part.nr_shared = nr_excl - total_excl; + qa.part.nr_rr = nr_rr; +} + +/* +* Charge elapsed wall time to each cid's current owner. Runs under the +* partition guard before every ownership change and from the stats flush, so +* alloc_ns[] reflects the layout that was in effect. Shared-pool time is +* charged to the live round-robin holder. +*/ +static __noinline void account_alloc(void) +{ +u64 now =bpf_ktime_get_ns (); +s32 rr_owner =CID_SELF ; +s32 nr_cids = qa.nr_cids; +u64 delta; +s32 cid, i; + +if (nr_cids <0 || nr_cids >SCX_QMAP_MAX_CPUS ) { +scx_bpf_error ("-ERANGE" ); +return ; + } + +/* first call starts the clock */ +if (!qa.alloc_ts) { + qa.alloc_ts = now; +return ; + } + delta = now - qa.alloc_ts; + qa.alloc_ts = now; + qa.alloc_window_ns += delta; + +/* resolve the live shared-pool holder to an owner id */ +if (qa.part.nr_shared && qa.part.nr_rr) { +u32 pos = qa.part.rr_pos; +u64 cgid = pos <MAX_PARTS ? qa.part.rr_slots[pos] :0 ; + +if (cgid) { + rr_owner =CID_NONE ; + bpf_for(i, 0, MAX_SUB_SCHEDS) +if (qa.sub_sched_ctxs[i].cgroup_id == cgid) + rr_owner = i; + } + } + + bpf_for(cid, 0, nr_cids) { +s32 owner = qa.part.cid_owner[cid]; + +if (owner ==CID_SHARED ) + owner = rr_owner; +if (owner >=0 && owner <MAX_SUB_SCHEDS ) + qa.alloc_ns[owner] += delta; +else if (owner ==CID_SELF ) + qa.self_alloc_ns += delta; + } +} + +/* +* apply_partition - execute the plan compute_partition() built +* +* Turn the owner map into the per-child, shared and self cmasks and issue the +* grant/revoke kfuncs as a delta against each child's previous grant. If no +* excl cid, evict every child. +*/ +__noinline void apply_partition(void) +{ +s32 nr_cids = qa.nr_cids; +s32 nr_shared = qa.part.nr_shared; +s32 i, cid; + +if (nr_cids <0 || nr_cids >SCX_QMAP_MAX_CPUS || + nr_shared <0 || nr_shared >MAX_PARTS ) { +scx_bpf_error ("-ERANGE" ); +return ; + } + +/* no excl cpu: run own tasks on the held shares, evict children */ +if (!qa.part.nr_excl) { +cmask_copy (&qa.self_cids.mask, &qa.held_shared.mask); + bpf_for(i, 0, MAX_SUB_SCHEDS) +if (qa.sub_sched_ctxs[i].cgroup_id) +scx_bpf_sub_kill (qa.sub_sched_ctxs[i].cgroup_id, +"parent holds no excl cpu to distribute" ); +return ; + } + +/* +* Snapshot the old pool. The per-child revoke below clears ENQ_IMMED on +* the previously-granted pool, so a cid that left the pool (now a +* sibling's excl) doesn't keep a stale ENQ_IMMED on its last holder. +*/ +cmask_copy (&qa.prev_rr_cids.mask, &qa.rr_cids.mask); + +/* turn the owner map into the rr pool, per-child excl, and self sets */ +cmask_init (&qa.rr_cids.mask,0 , nr_cids); +cmask_init (&qa.self_cids.mask,0 , nr_cids); + +/* snapshot each child's grant, then rebuild the new sets below */ + bpf_for(i, 0, MAX_SUB_SCHEDS) { +cmask_copy (&qa.sub_sched_ctxs[i].prev_granted.mask, + &qa.sub_sched_ctxs[i].granted_cids.mask); +cmask_init (&qa.sub_sched_ctxs[i].granted_cids.mask,0 , nr_cids); + } + + bpf_for(i, 0, nr_shared) +cmask_set (qa.part.shared_cids[i], &qa.rr_cids.mask); + bpf_for(cid, 0, nr_cids) { +s32 o = qa.part.cid_owner[cid]; + +if (cmask_test (cid, &qa.rr_cids.mask)) +continue ; +if (o >=0 && o <MAX_SUB_SCHEDS ) +cmask_set (cid, &qa.sub_sched_ctxs[o].granted_cids.mask); +else if (o ==CID_SELF ) +cmask_set (cid, &qa.self_cids.mask); + } + +/* +* Apply each child's exclusive cids as a delta against its previous +* grant. Separately clear the previous shared grant (ENQ_IMMED on the +* old pool), covering cids still pooled and cids that left for a +* sibling's excl. The current holder is granted the new pool below. +*/ + bpf_for(i, 0, MAX_SUB_SCHEDS) { + struct sub_sched_ctx __arena *ssc = &qa.sub_sched_ctxs[i]; +u64 cgid = ssc->cgroup_id; + +if (!cgid) +continue ; + +cmask_copy (&qa.to_revoke_cids.mask, &ssc->prev_granted.mask); +cmask_andnot (&qa.to_revoke_cids.mask, &ssc->granted_cids.mask); +cmask_copy (&qa.to_grant_cids.mask, &ssc->granted_cids.mask); +cmask_andnot (&qa.to_grant_cids.mask, &ssc->prev_granted.mask); + +scx_bpf_sub_revoke (cgid,SCX_CAP_ENQ_IMMED , + (void *)(long )&qa.prev_rr_cids.mask); +scx_bpf_sub_revoke (cgid,SCX_CAP_ENQ |SCX_CAP_PREEMPT |SCX_CAP_ENQ_IMMED , + (void *)(long )&qa.to_revoke_cids.mask); +scx_bpf_sub_grant (cgid,SCX_CAP_ENQ |SCX_CAP_PREEMPT |SCX_CAP_ENQ_IMMED , + (void *)(long )&qa.to_grant_cids.mask,NULL ); + } + +/* the current holder of the shared pool gets ENQ_IMMED on all of it */ +if (nr_shared) { +s32 pos = qa.part.rr_pos; +u64 holder_cgid; + +if (pos <0 || pos >=MAX_PARTS ) { +scx_bpf_error ("-ERANGE" ); +return ; + } + + holder_cgid = qa.part.rr_slots[pos];/* 0 = self, nothing to grant */ +if (holder_cgid) +scx_bpf_sub_grant (holder_cgid,SCX_CAP_ENQ_IMMED , + (void *)(long )&qa.rr_cids.mask,NULL ); + } +} + +/* +* Recompute the split off the node's held caps and apply it. The contexts this +* runs from (the sub-sched callbacks, the userspace poke, the rr timer) are not +* serialized by the kernel, so a single runner does the work. A caller that +* finds the guard held leaves part_pending set; the holder drains it before +* releasing, with the rr timer as a backstop. +*/ +static void redistribute (void ) +{ +s32 i; + +__sync_fetch_and_or (&part_pending,1 ); + +if (!part_try_start ()) +return ; + + bpf_for(i, 0, 1024) { +__sync_fetch_and_and (&part_pending,0 ); +/* charge elapsed time to the current partition before rebuilding it */ +account_alloc (); +compute_partition (); +apply_partition (); +if (!__sync_fetch_and_or (&part_pending,0 )) +break ; + } + +part_end (); +} + +/* userspace pokes this (PROG_RUN) to resplit after a cpu.weight change */ +SEC("syscall") +int repartition (void *ctx) +{ +redistribute (); +return 0 ; +} + +/* +* Userspace pokes this (PROG_RUN) to bring alloc_ns[] current before reading +* it for the stats display. Skipping when the partition guard is held is +* fine - alloc_ts is untouched, so the elapsed time is charged next time. +*/ +SEC("syscall") +int flush_alloc (void *ctx) +{ +if (part_try_start ()) { +account_alloc (); +part_end (); + } +return 0 ; +} + +/* +* Hand the shared pool to the next participant in the rotation. Self's turn +* just revokes the pool back to this sched. A child's turn grants it ENQ_IMMED +* on the entire pool. As only excl-held cids are time-shared, a wall-clock +* rotation works. Driven by the round-robin timer. +*/ +static void rr_advance (void ) +{ +s32 nr_shared, old_pos, new_pos; +u64 old_cgid, new_cgid; +u32 nr_rr;/* unsigned for % */ + +/* a redistribute holds the partition and rebuilds the pool, so skip */ +if (!part_try_start ()) +return ; + + nr_rr = qa.part.nr_rr; + nr_shared = qa.part.nr_shared; + +if (nr_shared <0 || nr_shared >MAX_PARTS ) { +scx_bpf_error ("-ERANGE" ); +return ; + } + +if (nr_shared && nr_rr >=2 ) { +/* close out the outgoing holder's pool time */ +account_alloc (); + + old_pos = qa.part.rr_pos; + new_pos = (old_pos +1 ) % nr_rr; + old_cgid = qa.part.rr_slots[old_pos]; + new_cgid = qa.part.rr_slots[new_pos]; + qa.part.rr_pos = new_pos; + +/* +* Move the ENQ_IMMED cap to the next participant. The shared +* cids stay marked CID_SHARED. qmap_dispatch() resolves the +* live holder via rr_pos without the guard, so a dispatch +* racing this handoff may reenqueue a task once. Harmless for a +* time-share. +*/ +if (old_cgid) +scx_bpf_sub_revoke (old_cgid,SCX_CAP_ENQ_IMMED , + (void *)(long )&qa.rr_cids.mask); +if (new_cgid) +scx_bpf_sub_grant (new_cgid,SCX_CAP_ENQ_IMMED , + (void *)(long )&qa.rr_cids.mask,NULL ); + } + +part_end (); + +/* a resplit queued while we held the guard supersedes this rotation */ +if (__sync_fetch_and_or (&part_pending,0 )) +redistribute (); +} + +/* advance the time-shared cid pool every round_robin_ns */ +static int round_robin_timerfn (void *map,int *key,struct bpf_timer *timer) +{ +rr_advance (); +bpf_timer_start (timer, round_robin_ns,0 ); +return 0 ; +} +/* * Custom cid layout for the cid-override test. On invalid input the kfunc * scx_error()s and aborts the scheduler.
@@ -1142,16 +1710,43 @@ s32 BPF_STRUCT_OPS_SLEEPABLE(qmap_init) } qa.task_free_head = (task_ctx_t *)slab; +/* cache the cid count, trusted to be <= SCX_QMAP_MAX_CPUS hereafter */ + qa.nr_cids = nr_cids; + +/* cmasks are embedded in qa, so they only need initializing */ +cmask_init (&qa.idle_cids.mask,0 , nr_cids); +cmask_init (&qa.rr_cids.mask,0 , nr_cids); +cmask_init (&qa.prev_rr_cids.mask,0 , nr_cids); +cmask_init (&qa.self_cids.mask,0 , nr_cids); +cmask_init (&qa.to_revoke_cids.mask,0 , nr_cids); +cmask_init (&qa.to_grant_cids.mask,0 , nr_cids); +cmask_init (&qa.held_excl.mask,0 , nr_cids); +cmask_init (&qa.held_shared.mask,0 , nr_cids); + +scx_bpf_sub_caps (0 ,SCX_CAP_ENQ , (void *)(long )&qa.held_excl.mask); +scx_bpf_sub_caps (0 ,SCX_CAP_ENQ_IMMED , (void *)(long )&qa.held_shared.mask); +cmask_andnot (&qa.held_shared.mask, &qa.held_excl.mask); + + bpf_for(i, 0, MAX_SUB_SCHEDS) { +cmask_init (&qa.sub_sched_ctxs[i].granted_cids.mask,0 , nr_cids); +cmask_init (&qa.sub_sched_ctxs[i].prev_granted.mask,0 , nr_cids); + } +/* -* Allocate and initialize the idle cmask. Starts empty - update_idle -* fills it as cpus enter idle. +* The root starts holding every cid. qmap_sub_ecaps_updated() maintains +* per-cid shared state as effective caps settle, and redistribute() +* rebuilds owner and self from held caps. A non-root node starts with +* nothing. */ - qa_idle_cids =bpf_arena_alloc_pages (&arena,NULL ,1 ,NUMA_NO_NODE ,0 ); -if (!qa_idle_cids) { -scx_bpf_error ("failed to allocate idle cmask" ); -return -ENOMEM ; + bpf_for(i, 0, nr_cids) { +if (!sub_cgroup_id) { +cmask_set (i, &qa.self_cids.mask); + qa.part.cid_owner[i] =CID_SELF ; + }else { + qa.part.cid_owner[i] =CID_NONE ; + } } -cmask_init (qa_idle_cids,0 , nr_cids); + qa.part.nr_shared =0 ; ret =scx_bpf_create_dsq (SHARED_DSQ ,-1 );if (ret) {
@@ -1190,6 +1785,16 @@ s32 BPF_STRUCT_OPS_SLEEPABLE(qmap_init)return ret; } +/* sub-sched: drive the boundary-cid round-robin from a bpf timer */ + timer =bpf_map_lookup_elem (&round_robin_timer, &key); +if (!timer) +return -ESRCH ; +bpf_timer_init (timer, &round_robin_timer,CLOCK_MONOTONIC ); +bpf_timer_set_callback (timer, round_robin_timerfn); + ret =bpf_timer_start (timer, round_robin_ns,0 ); +if (ret) +return ret; +return 0 ; }
@@ -1202,13 +1807,20 @@ s32 BPF_STRUCT_OPS(qmap_sub_attach, struct scx_sub_attach_args *args) {s32 i; +/* as long as there is at least one excl cpu, children can attach */ +if (!cmask_weight (&qa.held_excl.mask)) +return -ENOSPC ; +for (i =0 ; i <MAX_SUB_SCHEDS ; i++) { -if (!qa.sub_sched_cgroup_ids[i]) { - qa.sub_sched_cgroup_ids[i] = args->ops->sub_cgroup_id; -bpf_printk ("attaching sub-sched[%d] on %s" , - i, args->cgroup_path); -return 0 ; - } +if (qa.sub_sched_ctxs[i].cgroup_id) +continue ; + + qa.sub_sched_ctxs[i].cgroup_id = args->ops->sub_cgroup_id; + qa.sub_sched_ctxs[i].weight =100 ;/* until userspace feeds it */ + qa.nr_sub_scheds++; +bpf_printk ("attaching sub-sched[%d] on %s" , i, args->cgroup_path); +redistribute (); +return 0 ; }return -ENOSPC ;
@@ -1219,15 +1831,37 @@ void BPF_STRUCT_OPS(qmap_sub_detach, struct scx_sub_detach_args *args)s32 i;for (i =0 ; i <MAX_SUB_SCHEDS ; i++) { -if (qa.sub_sched_cgroup_ids[i] == args->ops->sub_cgroup_id) { - qa.sub_sched_cgroup_ids[i] =0 ; -bpf_printk ("detaching sub-sched[%d] on %s" , - i, args->cgroup_path); -break ; - } +if (qa.sub_sched_ctxs[i].cgroup_id != args->ops->sub_cgroup_id) +continue ; + + qa.sub_sched_ctxs[i].cgroup_id =0 ; + qa.sub_sched_ctxs[i].weight =100 ; +cmask_init (&qa.sub_sched_ctxs[i].granted_cids.mask,0 , qa.nr_cids); + qa.nr_sub_scheds--; +bpf_printk ("detaching sub-sched[%d] on %s" , i, args->cgroup_path); +redistribute (); +break ; } } +void BPF_STRUCT_OPS (qmap_sub_caps_updated ,const struct scx_cmask *cmask,u64 caps) +{ +/* our held caps changed, redistribute */ +redistribute (); +} + +void BPF_STRUCT_OPS (qmap_sub_ecaps_updated ,s32 cid,u64 before,u64 after) +{ +/* +* Effective caps updated. Track which cids hold shared caps so a self +* task placed there enqueues IMMED. +*/ +if (after &SCX_CAP_ENQ_IMMED ) + qa.cid_shared[cid] = (after &SCX_CAP_ENQ ) ?0 :1 ; +else + qa.cid_shared[cid] =0 ; +} + SCX_OPS_CID_DEFINE(qmap_ops, .flags = SCX_OPS_ENQ_EXITING | SCX_OPS_TID_TO_TASK, .select_cid = (void *)qmap_select_cid,
@@ -1248,6 +1882,8 @@ SCX_OPS_CID_DEFINE(qmap_ops, .cgroup_set_bandwidth = (void *)qmap_cgroup_set_bandwidth, .sub_attach = (void *)qmap_sub_attach, .sub_detach = (void *)qmap_sub_detach, + .sub_caps_updated = (void *)qmap_sub_caps_updated, + .sub_ecaps_updated = (void *)qmap_sub_ecaps_updated, .init_cids = (void *)qmap_init_cids, .init = (void *)qmap_init, .exit = (void *)qmap_exit,
tools/sched_ext/scx_qmap.c +291 -5
378 lines not on this page · raw message
tools/sched_ext/scx_qmap.h +94 -4
134 lines not on this page · raw message
Signature · 1 line
2.54.0
[email protected]
AI reviewRe: [PATCH v4 sched_ext/for-7.3 39/40] tools/sched_ext: scx_qmap - Expand hierarchical sub-scheduling
Thank you for your contribution! Sashiko AI review found 5 potential issue(s) to consider: - [Medium] `dispatch_highpri()` ignores hierarchical CID delegation, allowing the parent scheduler to…
Thank you for your contribution! Sashiko AI review found 5 potential issue(s) to consider:
- [Medium] dispatch_highpri() ignores hierarchical CID delegation, allowing the parent scheduler to dispatch its high-priority tasks to CPUs exclusively delegated to child schedulers.
- [Medium] Unsynchronized modification of qa.sub_sched_ctxs array during qmap_sub_attach and qmap_sub_detach causes data races with partition rebuilding.
- [Medium] Double read (TOCTOU race) of qa.sub_sched_ctxs[owner].cgroup_id in qmap_dispatch can pass 0 to scx_bpf_sub_dispatch.
- [Medium] Userspace feed_weights() overwrites child weight with stale data due to concurrent sub-scheduler detach/attach (TOCTOU).
- [Medium] Negative error codes from scx_bpf_task_cid() are used directly as array and bitmask indices without bounds checking.
--
--- Patch [39]: [PATCH v4 sched_ext/for-7.3 39/40] tools/sched_ext: scx_qmap - Expand hierarchical sub-scheduling --- commit d8ef8ae12fe2035b97cf3007ec38b1e29eef5a35
Author: Tejun Heo <[email protected]>
tools/sched_ext: scx_qmap - Expand hierarchical sub-scheduling
This commit expands the scx_qmap test scheduler to demonstrate hierarchical sub-scheduling. It allows a parent to delegate exclusive and shared CPUs (cids) to child cgroups based on cpu.weight.
Quoted 4 lines
tools/sched_ext/scx_qmap.bpf.c +0
-0
[ ... ]
tools/sched_ext/scx_qmap.bpf.c void BPF_STRUCT_OPS(qmap_enqueue, struct task_struct *p, u64 enq_flags) · line 482
Quoted 9 lines
tools/sched_ext/scx_qmap.bpf.c +2
-1@@ -416,7 +476,8 @@ void BPF_STRUCT_OPS(qmap_enqueue, struct task_struct *p, u64 enq_flags)*/ if (taskc->force_local) { taskc->force_local = false; -scx_bpf_dsq_insert (p,SCX_DSQ_LOCAL , slice_ns, enq_flags); +scx_bpf_dsq_insert (p,SCX_DSQ_LOCAL , slice_ns, + enq_flags |needs_immed (scx_bpf_task_cid (p)));return ; }
[Severity: Medium]
The core kernel kfunc scx_bpf_task_cid() can return an s32 negative error code (such as -EINVAL) if the CID table is uninitialized or the CPU mapping is missing.
When this return value is passed directly to needs_immed(), the negative value is used to index into the qa.cid_shared array without bounds checking. Can this lead to an out-of-bounds memory read within the BPF arena?
Similarly, in pick_direct_dispatch_cid(), prev_cid is passed to bitmask operations like cmask_test_and_clear() and cmask_test(). Should there be a check to ensure the returned CID is >= 0 before using it as an index?
[ ... ]
Quoted 1 line
tools/sched_ext/scx_qmap.bpf.c +0
-0@@ -541,7 +604,7 @@ static bool dispatch_highpri(bool from_timer)
[ ... ]
tools/sched_ext/scx_qmap.bpf.c static bool dispatch_highpri(bool from_timer) · line 609
Quoted 5 lines
tools/sched_ext/scx_qmap.bpf.c +1
-1if (scx_bpf_dsq_move (BPF_FOR_EACH_ITER , p,SCX_DSQ_LOCAL_ON | cid, -SCX_ENQ_PREEMPT )) { +SCX_ENQ_PREEMPT |needs_immed (cid))) {if (cid == this_cid) { dispatched = true;
[Severity: Medium]
In qmap_dispatch(), dispatch_highpri(false) is called unconditionally before checking qa.part.cid_owner.
Inside dispatch_highpri(), tasks are moved to the local DSQ without verifying if the target cid is restricted by qa.self_cids.mask.
Does this sequence allow the parent scheduler's high-priority tasks to be dispatched to CPUs that are exclusively delegated to child sub-schedulers, thereby violating the delegation policy?
[ ... ]
Quoted 1 line
tools/sched_ext/scx_qmap.bpf.c +0
-0@@ -567,12 +630,35 @@ void BPF_STRUCT_OPS(qmap_dispatch, s32 cid, struct task_struct *prev)
[ ... ]
tools/sched_ext/scx_qmap.bpf.c void BPF_STRUCT_OPS(qmap_dispatch, s32 cid, struct task_struct *prev) · line 659
Quoted 6 lines
tools/sched_ext/scx_qmap.bpf.c +6
-0+ }else if (owner >=0 && owner <MAX_SUB_SCHEDS && + qa.sub_sched_ctxs[owner].cgroup_id) { +if (scx_bpf_sub_dispatch (qa.sub_sched_ctxs[owner].cgroup_id)) +__sync_fetch_and_add (&qa.sub_sched_ctxs[owner].nr_dsps,1 ); +return ; + }
[Severity: Medium]
The cgroup_id is read twice without protection here. The first read evaluates qa.sub_sched_ctxs[owner].cgroup_id as a condition for the if statement.
Immediately after, the same memory location is re-read to be passed to scx_bpf_sub_dispatch().
Because qa.sub_sched_ctxs can be concurrently modified by qmap_sub_detach() without locks, could the value become 0 between the condition check and the function call, causing scx_bpf_sub_dispatch() to be invoked with an invalid cgroup ID?
[ ... ]
Quoted 12 lines
tools/sched_ext/scx_qmap.bpf.c +2
-6@@ -1219,15 +1831,37 @@ void BPF_STRUCT_OPS(qmap_sub_detach, struct scx_sub_detach_args *args)s32 i;for (i =0 ; i <MAX_SUB_SCHEDS ; i++) { -if (qa.sub_sched_cgroup_ids[i] == args->ops->sub_cgroup_id) { - qa.sub_sched_cgroup_ids[i] =0 ; -bpf_printk ("detaching sub-sched[%d] on %s" , - i, args->cgroup_path); -break ; - } +if (qa.sub_sched_ctxs[i].cgroup_id != args->ops->sub_cgroup_id) +continue ;
tools/sched_ext/scx_qmap.bpf.c void BPF_STRUCT_OPS(qmap_sub_detach, struct scx_sub_detach_args *args) · line 1845
Quoted 10 lines
tools/sched_ext/scx_qmap.bpf.c +8
-0+ + qa.sub_sched_ctxs[i].cgroup_id =0 ; + qa.sub_sched_ctxs[i].weight =100 ; +cmask_init (&qa.sub_sched_ctxs[i].granted_cids.mask,0 , qa.nr_cids); + qa.nr_sub_scheds--; +bpf_printk ("detaching sub-sched[%d] on %s" , i, args->cgroup_path); +redistribute (); +break ; } }
[Severity: Medium]
During qmap_sub_detach() and qmap_sub_attach(), the global array qa.sub_sched_ctxs is modified directly (clearing cgroup_id and zeroing granted_cids.mask via cmask_init).
These modifications occur without acquiring the part_try_start() / part_end() synchronization guard used in apply_partition() and rr_advance().
If apply_partition() runs concurrently from the timer callback or sys_bpf calls, can this cause a data race that corrupts the partition state tracking masks?
Quoted 4 lines
tools/sched_ext/scx_qmap.c +0
-0
[ ... ]
Quoted 1 line
tools/sched_ext/scx_qmap.c +0
-0@@ -62,6 +92,213 @@ static void sigint_handler(int dummy)
[ ... ]
Quoted 10 lines
tools/sched_ext/scx_qmap.c +10
-0+/* read each direct child's cpu.weight into the arena, true if any changed */ +static bool feed_weights (struct qmap_arena *qa) +{ +bool changed = false; +int i; + +for (i =0 ; i <MAX_SUB_SCHEDS ; i++) { +u64 cgid = qa->sub_sched_ctxs[i].cgroup_id; +u32 w; +
tools/sched_ext/scx_qmap.c static void sigint_handler(int dummy) · line 162
Quoted 10 lines
tools/sched_ext/scx_qmap.c +10
-0+if (!cgid) +continue ; + w =read_cgroup_weight (cgid); +if (w && w != qa->sub_sched_ctxs[i].weight) { + qa->sub_sched_ctxs[i].weight = w; + changed = true; + } + } +return changed; +}
[Severity: Medium]
In feed_weights(), userspace reads the cgroup_id from the BPF arena memory and then performs a blocking sysfs read via read_cgroup_weight().
During this file I/O operation, the original child cgroup could detach and a new one could attach to the same slot. Will the userspace daemon then blindly overwrite the newly attached child's qa->sub_sched_ctxs[i].weight with the stale weight read from the previous child's sysfs file?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=39
Tejun Heo
reviewerRe: [PATCH v4 sched_ext/for-7.3 39/40] tools/sched_ext: scx_qmap - Expand hierarchical sub-scheduling
On Wed, 08 Jul 2026 22:03:52 +0000, [email protected] wrote:
1. Not reachable in qmap contexts - the cid table is populated for every possible cpu before a scheduler can run, so scx_bpf_task_cid() never returns negative once the callbacks fire. The cmask helpers also take u32 and bounds-check, so even a negative would fail the range test rather than access memory. No change.
2. Real policy nit - fixed in v5 by masking the highpri target pick with self_cids. Kernel-side enforcement is copy-down by design (an ancestor keeps its caps on delegated cids), so exclusivity is the demo's own policy to enforce.
3. Benign - a stale or zeroed id just makes scx_bpf_sub_dispatch() miss the hash lookup and return false, a wasted no-op call at worst. v5 reads the id once into a local anyway.
4. Same as the v3 round - self-healing, the fields are re-derived by the following redistribute() and the clean lock can't wrap the cmask ops as they call kfuncs:
https://lore.kernel.org/r/[email protected]
5. Known tolerated nit - the weight is advisory and self-corrects on the next pass. v5 adds a comment noting it.
Thanks.
Signature · 1 line
tejun
Tejun Heo
reviewerPATCH v4 40/40 tools/sched_ext: scx_qmap - Add sub-sched cap fault injection
In reply to Tejun HeoAdd a fault-injection mode to the scx_qmap sub-scheduler that deliberately dispatches one of its own tasks to a cid it does not hold. The kernel cap check must reject it and re-enqueue with SCX_TASK_REENQ_CAP, so the nr_inject_attempts counter tracks nr_reenq_cap one to one, exercising the delivery-time cap enforcement.
--- tools/sched_ext/scx_qmap.bpf.c | 40 ++++++++++++++++++++++++++++++++++ tools/sched_ext/scx_qmap.c | 20 +++++++++++++---- tools/sched_ext/scx_qmap.h | 8 +++++++ 3 files changed, 64 insertions(+), 4 deletions(-)
tools/sched_ext/scx_qmap.bpf.c +40 -0
@@ -378,6 +378,24 @@ static u64 needs_immed(s32 cid)return qa.cid_shared[cid] ?SCX_ENQ_IMMED :0 ; } +/* first cid this node does NOT hold for fault injection, -1 if none */ +static s32 first_unavail_cid (void ) +{ +s32 nr_cids = qa.nr_cids, c; + +if (nr_cids >SCX_QMAP_MAX_CPUS ) { +scx_bpf_error ("-ERANGE" ); +return -1 ; + } + + bpf_for(c, 0, nr_cids) { +if (!cmask_test (c, &qa.held_excl.mask) && + !cmask_test (c, &qa.held_shared.mask)) +return c; + } +return -1 ; +} +static int weight_to_idx (u32 weight) {/* Coarsely map the compound weight to a FIFO. */
@@ -454,6 +472,28 @@ void BPF_STRUCT_OPS(qmap_enqueue, struct task_struct *p, u64 enq_flags) } } +/* +* Fault injection: deliberately dispatch one of our own tasks to a cid +* we don't hold. The kernel cap check must reject it and re-enqueue +* with SCX_TASK_REENQ_CAP, so nr_inject_attempts tracks nr_reenq_cap +* and proves delivery-time enforcement. Throttled. +*/ +if (qa.inject_mode ==QMAP_INJ_WRONG_CID && p->nr_cpus_allowed >1 && + !(enq_flags &SCX_ENQ_REENQ )) { +static u32 inj_cnt; + +if (!(++inj_cnt %64 )) { +s32 bad =first_unavail_cid (); + +if (bad >=0 &&cmask_test (bad, &taskc->cpus_allowed)) { +__sync_fetch_and_add (&qa.nr_inject_attempts,1 ); +scx_bpf_dsq_insert (p,SCX_DSQ_LOCAL_ON | bad, + slice_ns, enq_flags); +return ; + } + } + } +/* * IMMED stress testing: Every immed_stress_nth'th enqueue, dispatch * directly to prev_cpu's local DSQ even when busy to force dsq->nr > 1
tools/sched_ext/scx_qmap.c +16 -4
@@ -52,7 +52,7 @@ const char help_fmt[] ="\n" "Usage: %s [-s SLICE_US] [-e COUNT] [-t COUNT] [-T COUNT] [-l COUNT] [-b COUNT]\n" " [-N COUNT] [-P] [-M] [-H] [-c CG_PATH] [-d PID] [-D LEN] [-S] [-p] [-I]\n" -" [-F COUNT] [-i SEC] [-R MS] [-v]\n" +" [-F COUNT] [-i SEC] [-R MS] [-J MODE] [-v]\n" "\n" " -s SLICE_US Override slice duration\n" " -e COUNT Trigger scx_bpf_error() after COUNT enqueues\n"
@@ -74,6 +74,7 @@ const char help_fmt[] =" -C MODE cid-override test (shuffle|bad-dup|bad-range|bad-mono)\n" " -i SEC Stats and weight-refresh interval, seconds (default 5)\n" " -R MS Round-robin period for time-shared cpus, ms (default 200)\n" +" -J MODE Fault injection (wrong-cid: dispatch to a cid not held)\n" " -v Print libbpf debug messages\n" " -h Display this help and exit\n" ;
@@ -183,6 +184,7 @@ struct hier_prev {u64 nr_dsps[MAX_SUB_SCHEDS ];u64 nr_reenq_cap;u64 nr_reenq_immed; +u64 nr_inject_attempts; };/* current wall-clock time as "HH:MM:SS" for the startup and interval headers */
@@ -264,12 +266,14 @@ static void print_hier(struct qmap_arena *qa, struct hier_prev *prev, u64 own_cg }format_cid_ranges (qa,CID_SHARED , ranges,sizeof (ranges)); -printf ("hier : nsub=%llu excl=%u shared=%s rr=%s reenq cap/immed +%llu/+%llu\n" , +printf ("hier : nsub=%llu excl=%u shared=%s rr=%s reenq cap/immed +%llu/+%llu inj=+%llu\n" , (unsigned long long )qa->nr_sub_scheds, qa->part.nr_excl, ranges, rr, (unsigned long long )(qa->nr_reenq_cap - prev->nr_reenq_cap), - (unsigned long long )(qa->nr_reenq_immed - prev->nr_reenq_immed)); + (unsigned long long )(qa->nr_reenq_immed - prev->nr_reenq_immed), + (unsigned long long )(qa->nr_inject_attempts - prev->nr_inject_attempts)); prev->nr_reenq_cap = qa->nr_reenq_cap; prev->nr_reenq_immed = qa->nr_reenq_immed; + prev->nr_inject_attempts = qa->nr_inject_attempts;printf ("hier : %-4s %10s %4s %6s %8s %s\n" ,"" ,"cgroup" ,"w" ,"alloc" ,"disp/s" ,"cids" );
@@ -310,6 +314,7 @@ int main(int argc, char **argv)struct hier_prev hprev = {};const char *sub_cg_path =NULL ;char tbuf[32 ]; +u32 inject_mode =0 ;u64 own_cgid =0 ;libbpf_set_print (libbpf_print_fn);
@@ -330,7 +335,7 @@ int main(int argc, char **argv) skel->rodata->slice_ns =__COMPAT_ENUM_OR_ZERO ("scx_public_consts" ,"SCX_SLICE_DFL" ); skel->rodata->max_tasks =16384 ; -while ((opt =getopt (argc, argv,"s:e:t:T:l:b:N:PMHc:d:D:SpIF:C:i:R:vh" )) !=-1 ) { +while ((opt =getopt (argc, argv,"s:e:t:T:l:b:N:PMHc:d:D:SpIF:C:i:R:J:vh" )) !=-1 ) {switch (opt) {case 's' : skel->rodata->slice_ns =strtoull (optarg,NULL ,0 ) *1000 ;
@@ -464,6 +469,12 @@ int main(int argc, char **argv)if (round_robin_ms <10 ) round_robin_ms =10 ;break ; +case 'J' : +if (!strcmp (optarg,"wrong-cid" )) + inject_mode =QMAP_INJ_WRONG_CID ; +else + inject_mode =strtoul (optarg,NULL ,0 ); +break ;case 'v' : verbose = true;break ;
@@ -480,6 +491,7 @@ int main(int argc, char **argv) qa = &skel->arena->qa; qa->test_error_cnt = test_error_cnt; + qa->inject_mode = inject_mode;if (sub_cg_path) printf("%s scx_qmap started: sub-scheduler on %s, stats every %ds\n",
tools/sched_ext/scx_qmap.h +8 -0
@@ -60,6 +60,12 @@ struct qmap_fifo {s32 idx; }; +/* -J fault-injection modes. Selects inject_mode in struct qmap_arena. */ +enum qmap_inject { +QMAP_INJ_OFF =0 , +QMAP_INJ_WRONG_CID =1 ,/* dispatch to a cid we don't hold */ +}; +/* * scx_cmask's are embedded in struct qmap_arena with inline backing storage. * The bpf side uses &field.mask with the normal cmask_* helpers. Userspace
@@ -167,6 +173,8 @@ struct qmap_arena {/* bpf -> userspace: stats */ u64 nr_reenq_cap;/* SCX_TASK_REENQ_CAP bounces */ u64 nr_reenq_immed;/* SCX_TASK_REENQ_IMMED bounces */ +u64 nr_inject_attempts;/* fault-injection: dispatches to an unheld cid */ + u32 inject_mode; /* fault-injection mode (QMAP_INJ_*) */ };#endif /* __SCX_QMAP_H */
Signature · 1 line
2.54.0
Tejun Heo
reviewerApplied 1-3 to sched_ext/for-7.2-fixes with the contexts adjusted for the pre-reorganization code layout. They will drop out of the next posting of this series.
Thanks.
--
tejun
Tejun Heo
reviewerApplied 4-9 to sched_ext/for-7.3. These will also drop out of the next posting of this series.
Thanks.
--
tejun