sched_ext: Misc fixes
Subsystems (2)
Tejun Heo
reviewerPATCH 1/1 sched_ext: Misc fixes
Hello,
Two unrelated small fixes.
0001-sched_ext-Drop-unlocked-scx_rq_clock_invalidate-from.patch 0002-sched_ext-Make-scx_bpf_events-read-the-calling-sched.patch
0001 drops the rq clock invalidation in the root disable path which was performed without holding the rq lock and is redundant anyway. 0002 makes scx_bpf_events() read the event counters of the scheduler associated with the calling program instead of always reading the root scheduler's.
diffstat follows.
kernel/sched/ext/ext.c | 31 +++++++++++++++++-------------- 1 file changed, 17 insertions(+), 14 deletions(-)
The patches are also applied to sched_ext/for-7.3.
git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext.git for-7.3
Thanks.
Signature · 1 line
tejun
Tejun Heo
reviewerPATCH 1/2 sched_ext: Drop unlocked scx_rq_clock_invalidate() from scx_root_disable()
scx_root_disable() invalidates each rq's clock before taking the rq lock. scx_rq_clock_invalidate() is a plain read-modify-write of rq->scx.flags and every other writer of the word runs under the rq lock, so the unlocked update can race a concurrent flags update and lose one side's bits.
The invalidation is also redundant. The dl_server rebalancing right below cycles each rq's lock and rq_unpin_lock() clears SCX_RQ_CLK_VALID on every unlock, while __scx_enabled is turned off only later in the function, so the clocks end up invalidated either way. Drop the call and note the invalidation in the comment.
--- kernel/sched/ext/ext.c | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-)
kernel/sched/ext/ext.c +6 -7
@@ -6476,12 +6476,13 @@ static void scx_root_disable(struct scx_sched *sch)percpu_up_write (&scx_fork_rwsem);/* -* Invalidate all the rq clocks to prevent getting outdated -* rq clocks from a previous scx scheduler. +* Re-balance the dl_server bandwidth reservations: detach ext_server +* (no more sched_ext tasks) and reinstate fair_server if it was +* previously detached because we were running in full mode. * -* Also re-balance the dl_server bandwidth reservations: detach -* ext_server (no more sched_ext tasks) and reinstate fair_server if it -* was previously detached because we were running in full mode. +* The rq lock cycle also invalidates each rq's clock (rq_unpin_lock() +* clears SCX_RQ_CLK_VALID on unlock), preventing the next enable from +* seeing outdated rq clocks from this scheduler. * * Unlike the enable path, this runs on a recovery path that cannot * fail, so we use dl_server_swap_bw() to atomically free ext_server's
@@ -6494,8 +6495,6 @@ static void scx_root_disable(struct scx_sched *sch)for_each_possible_cpu (cpu) {struct rq *rq =cpu_rq (cpu); -scx_rq_clock_invalidate (rq); - scoped_guard(rq_lock_irqsave, rq) {update_rq_clock (rq);if (was_switched_all) {
Signature · 1 line
2.55.0
Tejun Heo
reviewerPATCH 2/2 sched_ext: Make scx_bpf_events() read the calling scheduler's counters
In reply to Tejun Heoscx_bpf_events() always reads the root scheduler's event counters, so a sub-scheduler program querying its own events silently gets the root's instead and has no BPF-visible way to read its own (the per-scheduler sysfs "events" file is the only interface). Resolve the scheduler from the calling program with scx_prog_sched(). Unassociated programs follow the usual scx_prog_sched() resolution: the root scheduler under a pre-sub-attach compat root and zeroed counters otherwise.
Also fix up the malformed comment into proper kerneldoc.
--- kernel/sched/ext/ext.c | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-)
kernel/sched/ext/ext.c +11 -7
@@ -10608,19 +10608,23 @@ static void scx_read_events(struct scx_sched *sch, struct scx_event_stats *event } } -/* -* scx_bpf_events - Get a system-wide event counter to +/** +* scx_bpf_events - Read the event counters of the calling scheduler * @events: output buffer from a BPF program -* @events__sz: @events len, must end in '__sz'' for the verifier +* @events__sz: @events len, must end in '__sz' for the verifier +* @aux: implicit BPF argument to access bpf_prog_aux hidden from BPF progs +* +* Read the event counters of the scheduler associated with the calling program. +* @events is zeroed when no scheduler can be resolved. */ -__bpf_kfunc void scx_bpf_events(struct scx_event_stats *events, -size_t events__sz) +__bpf_kfunc void scx_bpf_events(struct scx_event_stats *events, size_t events__sz, +const struct bpf_prog_aux *aux) {struct scx_sched *sch;struct scx_event_stats e_sys;rcu_read_lock (); - sch =rcu_dereference (scx_root); + sch =scx_prog_sched (aux);if (sch)scx_read_events (sch, &e_sys);else
@@ -10743,7 +10747,7 @@ BTF_ID_FLAGS(func, scx_bpf_cpu_curr, KF_IMPLICIT_ARGS | KF_RET_NULL | KF_RCU_PRO BTF_ID_FLAGS(func, scx_bpf_cid_curr, KF_IMPLICIT_ARGS | KF_RET_NULL | KF_RCU_PROTECTED) BTF_ID_FLAGS(func, scx_bpf_tid_to_task, KF_RET_NULL | KF_RCU_PROTECTED) BTF_ID_FLAGS(func, scx_bpf_now) -BTF_ID_FLAGS(func, scx_bpf_events) +BTF_ID_FLAGS(func, scx_bpf_events, KF_IMPLICIT_ARGS)#ifdef CONFIG_CGROUP_SCHED BTF_ID_FLAGS(func, scx_bpf_task_cgroup, KF_IMPLICIT_ARGS | KF_RCU | KF_ACQUIRE)#endif
Signature · 1 line
2.55.0
[email protected]
AI reviewRe: [PATCH 1/2] sched_ext: Drop unlocked scx_rq_clock_invalidate() from scx_root_disable()
In reply to Tejun HeoThank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Pre-existing issues:
- [High] This problem wasn't introduced by this patch, but SCX_RQ_CLK_VALID can be permanently stranded as 1 during scx_root_disable(), causing the next SCX scheduler to read a massively outdated clock via scx_bpf_now(). --
--- Patch [1]: [PATCH 1/2] sched_ext: Drop unlocked scx_rq_clock_invalidate() from scx_root_disable() --- commit c8d4a8f1095e17e1c662f243f67711302652b61f
Author: Tejun Heo <[email protected]>
sched_ext: Drop unlocked scx_rq_clock_invalidate() from scx_root_disable()
This commit drops the unlocked scx_rq_clock_invalidate() call during scx_root_disable(), relying instead on the rq lock cycle and unlock path to clear the clock valid flags.
Quoted 10 lines
kernel/sched/ext/ext.c +0
-2@@ -6162,12 +6162,13 @@ static void scx_root_disable(struct scx_sched *sch)percpu_up_write (&scx_fork_rwsem);/* -* Invalidate all the rq clocks to prevent getting outdated -* rq clocks from a previous scx scheduler.
kernel/sched/ext/ext.c static void scx_root_disable(struct scx_sched *sch) · line 6171
Quoted 10 lines
kernel/sched/ext/ext.c +6
-3+* Re-balance the dl_server bandwidth reservations: detach ext_server +* (no more sched_ext tasks) and reinstate fair_server if it was +* previously detached because we were running in full mode. * -* Also re-balance the dl_server bandwidth reservations: detach -* ext_server (no more sched_ext tasks) and reinstate fair_server if it -* was previously detached because we were running in full mode. +* The rq lock cycle also invalidates each rq's clock (rq_unpin_lock() +* clears SCX_RQ_CLK_VALID on unlock), preventing the next enable from +* seeing outdated rq clocks from this scheduler.
[Severity: High]
This problem wasn't introduced by this patch, but is it possible for the SCX_RQ_CLK_VALID flag to be permanently stranded as 1 during scx_root_disable()?
The commit message and comment note that rq_unpin_lock() clears the flag on unlock, and __scx_enabled is turned off later. However, if a concurrent CPU takes the rq lock after the cycle but before __scx_enabled is disabled, it seems a race can occur:
CPU N (Normal scheduling):
Takes rq lockupdate_rq_clock() sets SCX_RQ_CLK_VALID (since __scx_enabled is still true)
CPU 0 (scx_root_disable):static_branch_disable(&__scx_enabled);
CPU N (Normal scheduling):
Releases rq lock -> rq_unpin_lock() -> scx_rq_clock_invalidate() scx_rq_clock_invalidate() in kernel/sched/sched.h checks:
if (!scx_enabled())
return;Because scx_enabled() now evaluates to false, the function returns early and fails to clear SCX_RQ_CLK_VALID.
When a new BPF scheduler is loaded later, early hooks like ops.select_cpu that run without holding the rq lock might call scx_bpf_now() and see SCX_RQ_CLK_VALID == 1. Will this cause them to use a massively outdated clock from the previous scheduler's run?
Quoted 12 lines
kernel/sched/ext/ext.c +0
-2* * Unlike the enable path, this runs on a recovery path that cannot * fail, so we use dl_server_swap_bw() to atomically free ext_server's @@ -6180,8 +6181,6 @@ static void scx_root_disable(struct scx_sched *sch)for_each_possible_cpu (cpu) {struct rq *rq =cpu_rq (cpu); -scx_rq_clock_invalidate (rq); - scoped_guard(rq_lock_irqsave, rq) {update_rq_clock (rq);if (was_switched_all) {
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
[email protected]
AI reviewRe: [PATCH 2/2] sched_ext: Make scx_bpf_events() read the calling scheduler's counters
In reply to Tejun HeoThank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Pre-existing issues:
- [High] A tracing or syscall BPF program can cause an out-of-bounds memory read by associating with a non-sched_ext struct_ops map (like tcp_congestion_ops), which is then incorrectly cast to struct sched_ext_ops * by scx_prog_sched(). --
--- Patch [2]: [PATCH 2/2] sched_ext: Make scx_bpf_events() read the calling scheduler's counters --- commit c53feaf881922f5f2912216d45a7b765df97ad06
Author: Tejun Heo <[email protected]>
sched_ext: Make scx_bpf_events() read the calling scheduler's counters
scx_bpf_events() now resolves the scheduler from the calling program with scx_prog_sched(), instead of always reading the root scheduler's event counters.
Quoted 4 lines
kernel/sched/ext/ext.c +0
-0
[ ... ]
Quoted 17 lines
kernel/sched/ext/ext.c +7
-4@@ -10418,19 +10418,23 @@ static void scx_read_events(struct scx_sched *sch, struct scx_event_stats *event } } -/* -* scx_bpf_events - Get a system-wide event counter to +/** +* scx_bpf_events - Read the event counters of the calling scheduler * @events: output buffer from a BPF program -* @events__sz: @events len, must end in '__sz'' for the verifier +* @events__sz: @events len, must end in '__sz' for the verifier +* @aux: implicit BPF argument to access bpf_prog_aux hidden from BPF progs +* +* Read the event counters of the scheduler associated with the calling program. +* @events is zeroed when no scheduler can be resolved. */ -__bpf_kfunc void scx_bpf_events(struct scx_event_stats *events,
kernel/sched/ext/ext.c static void scx_read_events(struct scx_sched *sch, struct scx_event_stats *event · line 10437
Quoted 10 lines
kernel/sched/ext/ext.c +3
-2- size_t events__sz) +__bpf_kfunc void scx_bpf_events(struct scx_event_stats *events, size_t events__sz, +const struct bpf_prog_aux *aux) {struct scx_sched *sch;struct scx_event_stats e_sys;rcu_read_lock (); - sch =rcu_dereference (scx_root); + sch =scx_prog_sched (aux);
[Severity: High]
This is a pre-existing issue, but does this expose us to a type confusion and out-of-bounds read?
Looking at scx_prog_sched() in kernel/sched/ext/internal.h, it retrieves associated struct_ops:
kernel/sched/ext/internal.h:scx_prog_sched() {
...
ops = bpf_prog_get_assoc_struct_ops(aux);
if (likely(ops))
return rcu_dereference_all(ops->priv);
...
}If a BPF_PROG_TYPE_TRACING or BPF_PROG_TYPE_SYSCALL program is associated with a different struct_ops map (like tcp_congestion_ops), the pointer is blindly cast to struct sched_ext_ops *.
Since struct sched_ext_ops is much larger than other struct_ops structures, reading ops->priv could read memory well past the end of the map allocation and cause a kernel panic when the garbage value is dereferenced as a struct scx_sched *.
Does scx_prog_sched() need to verify the type of the associated struct_ops before accessing ops->priv?
Quoted 3 lines
kernel/sched/ext/ext.c +0
-0if (sch)scx_read_events (sch, &e_sys);else
[ ... ]
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
Tejun Heo
reviewerRe: [PATCH 2/2] sched_ext: Make scx_bpf_events() read the calling scheduler's counters
Hello,
Quoted 8 lines
If a
BPF_PROG_TYPE_TRACINGorBPF_PROG_TYPE_SYSCALLprogram is associated with a differentstruct_opsmap (liketcp_congestion_ops), the pointer is blindly cast to structsched_ext_ops*.Since struct
sched_ext_opsis much larger than otherstruct_opsstructures, readingops->privcould read memory well past the end of the map allocation and cause a kernel panic when the garbage value is dereferenced as a structscx_sched*.
The hole exists but reaching it requires the privileged userspace to deliberately associate its own tracing or syscall program with a wrong type of struct_ops map and then call SCX kfuncs from it. Association is an explicit per-program operation, so this can't happen accidentally. The right fix is rejecting the association of a program which uses SCX kfuncs with a non-sched_ext struct_ops map at association time, which the association path has enough information to do. Will worry about it later.
Thanks.
Signature · 1 line
tejun
Tejun Heo
reviewerPATCH v2 1/2 sched_ext: Drop unlocked scx_rq_clock_invalidate() from scx_root_disable()
In reply to Tejun Heoscx_root_disable() invalidates each rq's clock before taking the rq lock. scx_rq_clock_invalidate() is a plain read-modify-write of rq->scx.flags and every other writer of the word runs under the rq lock, so the unlocked update can race a concurrent flags update and lose one side's bits.
The invalidation doesn't matter in the first place. The cached clock is read only by scx_bpf_now() from a loaded scheduler's BPF programs, nothing can re-validate the clock while sched_ext is disabled as scx_rq_clock_update() is gated on scx_enabled() too, and the usual rq lock cycles under the next scheduler refresh or invalidate it before it's practically observable. Drop the invalidation instead of fixing the locking.
v2: Description and comment updated - the invalidation is unnecessary rather
than subsumed by the rq lock cycle below.
--- kernel/sched/ext/ext.c | 11 +++-------- 1 file changed, 3 insertions(+), 8 deletions(-)
kernel/sched/ext/ext.c +3 -8
@@ -6367,12 +6367,9 @@ static void scx_root_disable(struct scx_percpu_up_write (&scx_fork_rwsem);/* -* Invalidate all the rq clocks to prevent getting outdated -* rq clocks from a previous scx scheduler. -* -* Also re-balance the dl_server bandwidth reservations: detach -* ext_server (no more sched_ext tasks) and reinstate fair_server if it -* was previously detached because we were running in full mode. +* Re-balance the dl_server bandwidth reservations: detach ext_server +* (no more sched_ext tasks) and reinstate fair_server if it was +* previously detached because we were running in full mode. * * Unlike the enable path, this runs on a recovery path that cannot * fail, so we use dl_server_swap_bw() to atomically free ext_server's
@@ -6385,8 +6382,6 @@ static void scx_root_disable(struct scx_for_each_possible_cpu (cpu) {struct rq *rq =cpu_rq (cpu); -scx_rq_clock_invalidate (rq); - scoped_guard(rq_lock_irqsave, rq) {update_rq_clock (rq);if (was_switched_all) {
Applied to sched_ext/for-7.3 with the v2 description for the first patch.
Thanks.
Signature · 1 line
tejun