sched_ext: Allow ops.cgroup_set_weight/idle() to be sleepable
Subsystems (3)
PATCH 1/1 sched_ext: Allow ops.cgroup_set_weight/idle() to be sleepable
From: Tao Cui <[email protected]>
ops.cgroup_set_weight() and ops.cgroup_set_idle() are delivered from scx_group_set_weight() and scx_group_set_idle(), which run from the cpu.weight (and v1 cpu.shares) and cpu.idle cgroup interface write paths in process context. Both hold percpu_down_read(&scx_cgroup_ops_rwsem), whose read side may sleep. The call sites are therefore sleepable, like ops.cgroup_set_bandwidth(), which was recently added to the sleepable allow-list.
bpf_scx_check_member() rejects a sleepable program on any member not on its allow-list, so a BPF scheduler cannot allocate -- which is sleepable -- when a cgroup's weight or idle state changes at runtime. Add cgroup_set_weight() and cgroup_set_idle() to the allow-list so these callbacks can allocate on demand, and document that they may block.
Also add the matching compatibility markers, so userspace can detect this support via BTF, mirroring scx_compat_marker_cgroup_set_bandwidth_may_sleep().
To size the alternative, a scheduler that gives each cgroup a
dedicated idle DSQ must create it in ops.cgroup_init() for every
cgroup up front. In a VM with 2000 cgroups that is 2000+ standing
DSQs, each a struct scx_dispatch_q plus a per-CPU area. With the
allow-list entries the same scheduler can create the DSQ lazily on
the first cpu.idle=1 write of a cgroup: 4 allocations for the 4
cgroups marked idle at runtime, and repeated cpu.idle writes do not re-allocate. Verified with a probe scheduler on an unpatched kernel (load rejected with -EINVAL) and on a patched one.
--- kernel/sched/ext/ext.c | 18 ++++++++++++++++++ kernel/sched/ext/internal.h | 9 +++++---- 2 files changed, 23 insertions(+), 4 deletions(-)
kernel/sched/ext/ext.c +18 -0
@@ -8079,7 +8079,9 @@ static int bpf_scx_check_member(const struct btf_type *t,case offsetof(struct sched_ext_ops , cgroup_init):case offsetof(struct sched_ext_ops , cgroup_exit):case offsetof(struct sched_ext_ops , cgroup_prep_move): +case offsetof(struct sched_ext_ops , cgroup_set_weight):case offsetof(struct sched_ext_ops , cgroup_set_bandwidth): +case offsetof(struct sched_ext_ops , cgroup_set_idle):#endif case offsetof(struct sched_ext_ops , cpu_online):case offsetof(struct sched_ext_ops , cpu_offline):
@@ -11055,3 +11057,19 @@ __initcall(scx_init);#ifdef CONFIG_EXT_GROUP_SCHED DEFINE_SCX_COMPAT_MARKER (cgroup_set_bandwidth_may_sleep);#endif /* CONFIG_EXT_GROUP_SCHED */ + +/* +* scx_compat_marker_cgroup_set_weight_may_sleep: advertises that +* ops.cgroup_set_weight() may be implemented as a sleepable callback. +*/ +#ifdef CONFIG_EXT_GROUP_SCHED +DEFINE_SCX_COMPAT_MARKER (cgroup_set_weight_may_sleep); +#endif /* CONFIG_EXT_GROUP_SCHED */ + +/* +* scx_compat_marker_cgroup_set_idle_may_sleep: advertises that +* ops.cgroup_set_idle() may be implemented as a sleepable callback. +*/ +#ifdef CONFIG_EXT_GROUP_SCHED +DEFINE_SCX_COMPAT_MARKER (cgroup_set_idle_may_sleep); +#endif /* CONFIG_EXT_GROUP_SCHED */
kernel/sched/ext/internal.h +5 -4
@@ -736,7 +736,7 @@ struct sched_ext_ops {* @cgrp: cgroup whose weight is being updated * @weight: new weight [1..10000] * -* Update @cgrp's weight to @weight. +* Update @cgrp's weight to @weight. This operation may block. * * Knobs of a cgroup belong to the parent, so the set_* ops are * delivered to @cgrp's parent's sched. That sched may never have seen
@@ -773,9 +773,10 @@ struct sched_ext_ops {* @cgrp: cgroup whose idle state is being updated * @idle: whether the cgroup is entering or exiting idle state * -* Update @cgrp's idle state to @idle. This callback is invoked when -* a cgroup transitions between idle and non-idle states, allowing the -* BPF scheduler to adjust its behavior accordingly. +* Update @cgrp's idle state to @idle. This operation may block. This +* callback is invoked when a cgroup transitions between idle and +* non-idle states, allowing the BPF scheduler to adjust its behavior +* accordingly. * * Delivery follows the same rule as cgroup_set_weight(). */
Signature · 1 line
2.43.0
[email protected]
AI reviewThank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Pre-existing issues:
- [High] Concurrent cgroup writes to cpu.weight, cpu.shares, or cpu.idle can permanently desynchronize the core scheduler and BPF scheduler state, and cause data races on SCX fields. --
--- Patch [1]: [PATCH] sched_ext: Allow ops.cgroup_set_weight/idle() to be sleepable --- commit 1a01a0e2918fcf8ca79c6152b56ecfa3e223935c
Author: Tao Cui <[email protected]>
sched_ext: Allow ops.cgroup_set_weight/idle() to be sleepable
ops.cgroup_set_weight() and ops.cgroup_set_idle() are delivered from scx_group_set_weight() and scx_group_set_idle(), which run from the cpu.weight and cpu.idle cgroup interface write paths. This patch adds them to the allow-list so these callbacks can allocate on demand.
kernel/sched/ext/ext.c static int bpf_scx_check_member(const struct btf_type *t, · line 8084
Quoted 11 lines
kernel/sched/ext/ext.c +2
-0@@ -8079,7 +8079,9 @@ static int bpf_scx_check_member(const struct btf_type *t,case offsetof(struct sched_ext_ops , cgroup_init):case offsetof(struct sched_ext_ops , cgroup_exit):case offsetof(struct sched_ext_ops , cgroup_prep_move): +case offsetof(struct sched_ext_ops , cgroup_set_weight):case offsetof(struct sched_ext_ops , cgroup_set_bandwidth): +case offsetof(struct sched_ext_ops , cgroup_set_idle):
[Severity: High]
This isn't a bug introduced by this patch, but does making these operations sleepable widen a race condition where concurrent cgroup writes could permanently desynchronize the core scheduler and BPF scheduler state?
Looking at kernel/sched/core.c:cpu_weight_write_u64(), the core scheduler is updated inside sched_group_set_shares() (which takes and drops shares_mutex), leaving a gap before the BPF scheduler is updated:
kernel/sched/core.c:cpu_weight_write_u64() {
...
ret = sched_group_set_shares(css_tg(css), scale_load(weight));
if (!ret)
scx_group_set_weight(css_tg(css), cgrp_weight);
...
}Inside scx_group_set_weight(), the SCX scheduler state and tg->scx.weight are updated without serialization (holding only a read lock):
kernel/sched/ext/ext.c:scx_group_set_weight() {
...
SCX_CALL_OP(sch, cgroup_set_weight, NULL, tg_cgrp(tg), weight);
tg->scx.weight = weight;
...
}The same pattern exists in kernel/sched/core.c:cpu_idle_write_s64() calling scx_group_set_idle() and updating tg->scx.idle.
If two threads concurrently write to cpu.weight or cpu.idle, they serialize on shares_mutex but drop it before calling the SCX functions. Because this patch allows ops.cgroup_set_weight and ops.cgroup_set_idle to sleep, could the threads execute out of order?
If Thread B's core scheduler update runs after Thread A's, but Thread B's BPF callback finishes before Thread A's, the final state in the core scheduler would be B, while the SCX state ends up as A. Can this lead to incorrect functional behavior and data races on tg->scx.weight and tg->scx.idle?
Quoted 2 lines
kernel/sched/ext/ext.c +0
-0#endif case offsetof(struct sched_ext_ops , cpu_online):
[ ... ]
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Hello,
Quoted 65 lines · 在 2026/8/25 13:36, [email protected] 写道:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Pre-existing issues:
- [High] Concurrent cgroup writes tocpu.weight,cpu.shares, orcpu.idlecan permanently desynchronize the core scheduler and BPF scheduler state, and cause data races on SCX fields. ----- Patch [1]: [PATCH]
sched_ext: Allow ops.cgroup_set_weight/idle() to be sleepable --- commit1a01a0e2918fcf8ca79c6152b56ecfa3e223935c
Author: Tao Cui <[email protected]>
sched_ext: Allow ops.cgroup_set_weight/idle() to be sleepable
ops.cgroup_set_weight()andops.cgroup_set_idle()are delivered fromscx_group_set_weight()andscx_group_set_idle(), which run from the cpu.weight and cpu.idle cgroup interface write paths. This patch adds them to the allow-list so these callbacks can allocate on demand.kernel/sched/ext/ext.c static int bpf_scx_check_member(const struct btf_type *t, · line 8084
Quoted 11 lines
kernel/sched/ext/ext.c +2
-0@@ -8079,7 +8079,9 @@ static int bpf_scx_check_member(const struct btf_type *t,case offsetof(struct sched_ext_ops , cgroup_init):case offsetof(struct sched_ext_ops , cgroup_exit):case offsetof(struct sched_ext_ops , cgroup_prep_move): +case offsetof(struct sched_ext_ops , cgroup_set_weight):case offsetof(struct sched_ext_ops , cgroup_set_bandwidth): +case offsetof(struct sched_ext_ops , cgroup_set_idle):[Severity: High]
This isn't a bug introduced by this patch, but does making these operations sleepable widen a race condition where concurrent cgroup writes could permanently desynchronize the core scheduler and BPF scheduler state?Looking at kernel/sched/core.c:cpu_weight_write_u64(), the core scheduler is updated inside
sched_group_set_shares()(which takes and dropsshares_mutex), leaving a gap before the BPF scheduler is updated:kernel/sched/core.c:cpu_weight_write_u64() { ... ret = sched_group_set_shares(css_tg(css), scale_load(weight)); if (!ret) scx_group_set_weight(css_tg(css), cgrp_weight); ... }Inside
scx_group_set_weight(), the SCX scheduler state andtg->scx.weightare updated without serialization (holding only a read lock):kernel/sched/ext/ext.c:scx_group_set_weight() { ... SCX_CALL_OP(sch, cgroup_set_weight, NULL, tg_cgrp(tg), weight); tg->scx.weight = weight; ... }The same pattern exists in kernel/sched/core.c:cpu_idle_write_s64() calling
scx_group_set_idle()and updatingtg->scx.idle.If two threads concurrently write to cpu.weight or cpu.idle, they serialize on
shares_mutexbut drop it before calling the SCX functions. Because this patch allows ops.cgroup_set_weight and ops.cgroup_set_idle to sleep, could the threads execute out of order?
Quoted 5 lines
If Thread B's core scheduler update runs after Thread A's, but Thread B's BPF callback finishes before Thread A's, the final state in the core scheduler would be B, while the SCX state ends up as A. Can this lead to incorrect functional behavior and data races on
tg->scx.weightandtg->scx.idle?
That race is pre-existing: the window between sched_group_set_shares() dropping its lock and scx_group_set_weight() running is unserialized even without a sleepable callback. Making the callbacks sleepable only widens the window.
Michal's "Lift cgroup update locking to core to prevent CFS/SCX divergence" series covers both updates with core-side locking and fixes it, so this patch doesn't grow a fix for it.
Thanks.
Quoted 5 lines
Quoted 2 lines
kernel/sched/ext/ext.c +0
-0#endif case offsetof(struct sched_ext_ops , cpu_online):[ ... ]
Andrea Righi
reviewerHi Tao,
Quoted 9 lines · Tao Cui
From: Tao Cui <[email protected]>
ops.cgroup_set_weight()andops.cgroup_set_idle()are delivered fromscx_group_set_weight()andscx_group_set_idle(), which run from the cpu.weight (and v1 cpu.shares) and cpu.idle cgroup interface write paths in process context. Both holdpercpu_down_read(&scx_cgroup_ops_rwsem), whose read side may sleep. The call sites are therefore sleepable, likeops.cgroup_set_bandwidth(), which was recently added to the sleepable allow-list.
The change itself makes sense, but I think the serialization issue pointed out by sashiko is valid. Concurrent cgroup knob updates can complete their sched_ext notifications out of order, potentially leaving the core scheduler and BPF scheduler with inconsistent state.
We should probably address the race first. I may have a fix and will post it shortly (testing right now). Then we can apply this sleepable callback change on top.
Quoted 11 lines
bpf_scx_check_member()rejects a sleepable program on any member not on its allow-list, so a BPF scheduler cannot allocate -- which is sleepable -- when a cgroup's weight or idle state changes at runtime. Addcgroup_set_weight()andcgroup_set_idle()to the allow-list so these callbacks can allocate on demand, and document that they may block.Also add the matching compatibility markers, so userspace can detect this support via BTF, mirroring
scx_compat_marker_cgroup_set_bandwidth_may_sleep().
Quoted 8 lines
To size the alternative, a scheduler that gives each cgroup a
dedicated idle DSQ must create it inops.cgroup_init()for every
cgroup up front. In a VM with 2000 cgroups that is 2000+ standing
DSQs, each a structscx_dispatch_qplus a per-CPU area. With the
allow-list entries the same scheduler can create the DSQ lazily on
the first cpu.idle=1 write of a cgroup: 4 allocations for the 4
cgroups marked idle at runtime, and repeated cpu.idle writes do not re-allocate. Verified with a probe scheduler on an unpatched kernel (load rejected with-EINVAL) and on a patched one.
nit: this paragraph needs a rewrapping (line too long).
Besides rewrapping, could we simplify this part and rephrase it in terms of DSQs rather than allocations? For example (something along these lines):
Without this support, a scheduler that uses a dedicated idle DSQ for each cgroup must create it eagerly from ops.cgroup_init(). With a sleepable ops.cgroup_set_idle() callback, it can instead create the DSQ lazily when the cgroup is first marked idle, avoiding unnecessary DSQs for cgroups that never use cpu.idle.
Thanks,
-Andrea
Quoted 71 lines
Signed-off-by Tao Cui <[email protected]>--- kernel/sched/ext/ext.c | 18 ++++++++++++++++++ kernel/sched/ext/internal.h | 9 +++++---- 2 files changed, 23 insertions(+), 4 deletions(-)kernel/sched/ext/ext.c +18
-0@@ -8079,7 +8079,9 @@ static int bpf_scx_check_member(const struct btf_type *t,case offsetof(struct sched_ext_ops , cgroup_init):case offsetof(struct sched_ext_ops , cgroup_exit):case offsetof(struct sched_ext_ops , cgroup_prep_move): +case offsetof(struct sched_ext_ops , cgroup_set_weight):case offsetof(struct sched_ext_ops , cgroup_set_bandwidth): +case offsetof(struct sched_ext_ops , cgroup_set_idle):#endif case offsetof(struct sched_ext_ops , cpu_online):case offsetof(struct sched_ext_ops , cpu_offline):@@ -11055,3 +11057,19 @@ __initcall(scx_init);#ifdef CONFIG_EXT_GROUP_SCHED DEFINE_SCX_COMPAT_MARKER (cgroup_set_bandwidth_may_sleep);#endif /* CONFIG_EXT_GROUP_SCHED */ + +/* +* scx_compat_marker_cgroup_set_weight_may_sleep: advertises that +* ops.cgroup_set_weight() may be implemented as a sleepable callback. +*/ +#ifdef CONFIG_EXT_GROUP_SCHED +DEFINE_SCX_COMPAT_MARKER (cgroup_set_weight_may_sleep); +#endif /* CONFIG_EXT_GROUP_SCHED */ + +/* +* scx_compat_marker_cgroup_set_idle_may_sleep: advertises that +* ops.cgroup_set_idle() may be implemented as a sleepable callback. +*/ +#ifdef CONFIG_EXT_GROUP_SCHED +DEFINE_SCX_COMPAT_MARKER (cgroup_set_idle_may_sleep); +#endif /* CONFIG_EXT_GROUP_SCHED */ kernel/sched/ext/internal.h +5
-4@@ -736,7 +736,7 @@ struct sched_ext_ops {* @cgrp: cgroup whose weight is being updated * @weight: new weight [1..10000] * -* Update @cgrp's weight to @weight. +* Update @cgrp's weight to @weight. This operation may block. * * Knobs of a cgroup belong to the parent, so the set_* ops are * delivered to @cgrp's parent's sched. That sched may never have seen @@ -773,9 +773,10 @@ struct sched_ext_ops {* @cgrp: cgroup whose idle state is being updated * @idle: whether the cgroup is entering or exiting idle state * -* Update @cgrp's idle state to @idle. This callback is invoked when -* a cgroup transitions between idle and non-idle states, allowing the -* BPF scheduler to adjust its behavior accordingly. +* Update @cgrp's idle state to @idle. This operation may block. This +* callback is invoked when a cgroup transitions between idle and +* non-idle states, allowing the BPF scheduler to adjust its behavior +* accordingly. * * Delivery follows the same rule as cgroup_set_weight(). */ Signature · 1 line
2.43.0
Hi Andrea,
Quoted 13 lines · Andrea Righi
Hi Tao,
Quoted 9 lines · Tao Cui
From: Tao Cui <[email protected]>
ops.cgroup_set_weight()andops.cgroup_set_idle()are delivered fromscx_group_set_weight()andscx_group_set_idle(), which run from the cpu.weight (and v1 cpu.shares) and cpu.idle cgroup interface write paths in process context. Both holdpercpu_down_read(&scx_cgroup_ops_rwsem), whose read side may sleep. The call sites are therefore sleepable, likeops.cgroup_set_bandwidth(), which was recently added to the sleepable allow-list.
Quoted 9 lines
The change itself makes sense, but I think the serialization issue pointed out by sashiko is valid. Concurrent cgroup knob updates can complete their
sched_extnotifications out of order, potentially leaving the core scheduler and BPF scheduler with inconsistent state.We should probably address the race first. I may have a fix and will post it shortly (testing right now). Then we can apply this sleepable callback change on top.
Thanks for the review.
Agreed on the ordering: I'll wait for the serialization fix to land and rebase this on top before sending v2.
Quoted 22 lines
Quoted 11 lines
bpf_scx_check_member()rejects a sleepable program on any member not on its allow-list, so a BPF scheduler cannot allocate -- which is sleepable -- when a cgroup's weight or idle state changes at runtime. Addcgroup_set_weight()andcgroup_set_idle()to the allow-list so these callbacks can allocate on demand, and document that they may block.Also add the matching compatibility markers, so userspace can detect this support via BTF, mirroring
scx_compat_marker_cgroup_set_bandwidth_may_sleep().Quoted 8 lines
To size the alternative, a scheduler that gives each cgroup a
dedicated idle DSQ must create it inops.cgroup_init()for every
cgroup up front. In a VM with 2000 cgroups that is 2000+ standing
DSQs, each a structscx_dispatch_qplus a per-CPU area. With the
allow-list entries the same scheduler can create the DSQ lazily on
the first cpu.idle=1 write of a cgroup: 4 allocations for the 4
cgroups marked idle at runtime, and repeated cpu.idle writes do not re-allocate. Verified with a probe scheduler on an unpatched kernel (load rejected with-EINVAL) and on a patched one.nit: this paragraph needs a rewrapping (line too long).
Quoted 9 lines
Besides rewrapping, could we simplify this part and rephrase it in terms of DSQs rather than allocations? For example (something along these lines):
Without this support, a scheduler that uses a dedicated idle DSQ for each cgroup must create it eagerly from
ops.cgroup_init(). With a sleepableops.cgroup_set_idle()callback, it can instead create the DSQ lazily when the cgroup is first marked idle, avoiding unnecessary DSQs for cgroups that never use cpu.idle.
For the commit message, v2 will rewrap the last paragraph and rephrase it in terms of DSQs following your suggestion.
Thanks,
Tao
Quoted 74 lines
Thanks,
-AndreaQuoted 71 lines
Signed-off-by Tao Cui <[email protected]>--- kernel/sched/ext/ext.c | 18 ++++++++++++++++++ kernel/sched/ext/internal.h | 9 +++++---- 2 files changed, 23 insertions(+), 4 deletions(-)kernel/sched/ext/ext.c +18
-0@@ -8079,7 +8079,9 @@ static int bpf_scx_check_member(const struct btf_type *t,case offsetof(struct sched_ext_ops , cgroup_init):case offsetof(struct sched_ext_ops , cgroup_exit):case offsetof(struct sched_ext_ops , cgroup_prep_move): +case offsetof(struct sched_ext_ops , cgroup_set_weight):case offsetof(struct sched_ext_ops , cgroup_set_bandwidth): +case offsetof(struct sched_ext_ops , cgroup_set_idle):#endif case offsetof(struct sched_ext_ops , cpu_online):case offsetof(struct sched_ext_ops , cpu_offline):@@ -11055,3 +11057,19 @@ __initcall(scx_init);#ifdef CONFIG_EXT_GROUP_SCHED DEFINE_SCX_COMPAT_MARKER (cgroup_set_bandwidth_may_sleep);#endif /* CONFIG_EXT_GROUP_SCHED */ + +/* +* scx_compat_marker_cgroup_set_weight_may_sleep: advertises that +* ops.cgroup_set_weight() may be implemented as a sleepable callback. +*/ +#ifdef CONFIG_EXT_GROUP_SCHED +DEFINE_SCX_COMPAT_MARKER (cgroup_set_weight_may_sleep); +#endif /* CONFIG_EXT_GROUP_SCHED */ + +/* +* scx_compat_marker_cgroup_set_idle_may_sleep: advertises that +* ops.cgroup_set_idle() may be implemented as a sleepable callback. +*/ +#ifdef CONFIG_EXT_GROUP_SCHED +DEFINE_SCX_COMPAT_MARKER (cgroup_set_idle_may_sleep); +#endif /* CONFIG_EXT_GROUP_SCHED */ kernel/sched/ext/internal.h +5
-4@@ -736,7 +736,7 @@ struct sched_ext_ops {* @cgrp: cgroup whose weight is being updated * @weight: new weight [1..10000] * -* Update @cgrp's weight to @weight. +* Update @cgrp's weight to @weight. This operation may block. * * Knobs of a cgroup belong to the parent, so the set_* ops are * delivered to @cgrp's parent's sched. That sched may never have seen @@ -773,9 +773,10 @@ struct sched_ext_ops {* @cgrp: cgroup whose idle state is being updated * @idle: whether the cgroup is entering or exiting idle state * -* Update @cgrp's idle state to @idle. This callback is invoked when -* a cgroup transitions between idle and non-idle states, allowing the -* BPF scheduler to adjust its behavior accordingly. +* Update @cgrp's idle state to @idle. This operation may block. This +* callback is invoked when a cgroup transitions between idle and +* non-idle states, allowing the BPF scheduler to adjust its behavior +* accordingly. * * Delivery follows the same rule as cgroup_set_weight(). */ Signature · 1 line
2.43.0