sched_ext: warn when cpu.max is set but the BPF scheduler doesn't implement bandwidth control
Subsystems (3)
RFC PATCH 1/1 sched_ext: warn when cpu.max is set but the BPF scheduler doesn't implement bandwidth control
From: Tao Cui <[email protected]>
The kernel stores cpu.max bandwidth parameters in the task_group and passes them to the BPF scheduler via ops.cgroup_set_bandwidth() and scx_cgroup_init_args, but does not enforce the quota itself. If the loaded BPF scheduler doesn't implement the callback, cpu.max is silently ignored -- the cgroup gets unlimited CPU regardless of the configured quota.
Of the example schedulers, only scx_qmap implements the callback -- and only to bpf_printk() the parameters, so no in-tree scheduler actually enforces the quota. Measured with scx_simple: a cgroup with cpu.max = "50000 100000" (50% of one CPU) and one busy task used 9946ms of CPU in 10 seconds with nr_throttled remaining 0.
Print a one-time warning when a finite quota is configured on a cgroup while the active scheduler lacks the callback, so users and container orchestrators know the quota is not enforced.
--- kernel/sched/ext/ext.c | 6 ++++++ 1 file changed, 6 insertions(+)
kernel/sched/ext/ext.c +6 -0
@@ -4953,6 +4953,12 @@ void scx_group_set_bandwidth(struct task_group *tg, tg->scx.bw_burst_us != burst_us))SCX_CALL_OP (sch, cgroup_set_bandwidth,NULL ,tg_cgrp (tg), period_us, quota_us, burst_us); + else if (scx_cgroup_enabled && sch && + !SCX_HAS_OP(sch, cgroup_set_bandwidth) && + quota_us != RUNTIME_INF) + pr_warn_once("sched_ext: BPF scheduler \"%s\" does not implement " +"ops.cgroup_set_bandwidth(); cpu.max will not be enforced\n" , + sch->ops.name); tg->scx.bw_period_us = period_us; tg->scx.bw_quota_us = quota_us;
Signature · 1 line
2.43.0
Hi, tejun
Quoted 16 lines · Tao Cui
From: Tao Cui <[email protected]>
The kernel stores cpu.max bandwidth parameters in the
task_groupand passes them to the BPF scheduler viaops.cgroup_set_bandwidth()andscx_cgroup_init_args, but does not enforce the quota itself. If the loaded BPF scheduler doesn't implement the callback, cpu.max is silently ignored -- the cgroup gets unlimited CPU regardless of the configured quota.Of the example schedulers, only
scx_qmapimplements the callback -- and only tobpf_printk()the parameters, so no in-tree scheduler actually enforces the quota. Measured withscx_simple: a cgroup with cpu.max = "50000 100000" (50% of one CPU) and one busy task used 9946ms of CPU in 10 seconds withnr_throttledremaining 0.
Quoted 4 lines
Print a one-time warning when a finite quota is configured on a cgroup while the active scheduler lacks the callback, so users and container orchestrators know the quota is not enforced.
Some background on how I found this: I was testing how sched_ext interacts with cgroup CPU controls in a VM, and the cpu.max case stood out:
cgroup with cpu.max = "50000 100000" (50% of one CPU), one busy task, scx_simple loaded
sched_ext : 9946ms of CPU in 10s, nr_throttled = 0
CFS : ~5000ms in 10s, throttling as expected
The same happens with scx_flatcg and scx_central -- neither they nor scx_simple implement ops.cgroup_set_bandwidth(), so the quota is silently ignored. grep shows scx_qmap is the only in-tree user of the callback -- and its implementation just bpf_printk()s the parameters, so even there the quota is not enforced. Outside the
tree, lavd implements its own bandwidth accounting, but as far as I can tell rusty and bpfland don't, which suggests users on those schedulers are running containers with cpu.max that does nothing.
That's why I drafted the warning patch -- but I'm not sure a warning is the right approach. Some alternatives I can think of:
1. pr_warn_once() as in this patch (minimal, but the quota is still not enforced)
2. refuse to enable sched_ext (or the cgroup support) when a finite quota exists and the callback is missing
3. kernel-side fallback enforcement, e.g. throttle in
scx_next_task_picked()/dispatch path based on tg->scx.bw_quota_us
Is the missing enforcement considered the BPF scheduler's responsibility by design (and just under-documented), or would a kernel fallback be welcome? If it's the former, maybe sched-ext.rst should mention that cpu.max requires ops.cgroup_set_bandwidth() from the loaded scheduler.
Happy to work on whichever direction you prefer.
Thanks,
Tao
Quoted 22 lines
Signed-off-by Tao Cui <[email protected]>--- kernel/sched/ext/ext.c | 6 ++++++ 1 file changed, 6 insertions(+)kernel/sched/ext/ext.c +6
-0@@ -4953,6 +4953,12 @@ void scx_group_set_bandwidth(struct task_group *tg, tg->scx.bw_burst_us != burst_us))SCX_CALL_OP (sch, cgroup_set_bandwidth,NULL ,tg_cgrp (tg), period_us, quota_us, burst_us); + else if (scx_cgroup_enabled && sch && + !SCX_HAS_OP(sch, cgroup_set_bandwidth) && + quota_us != RUNTIME_INF) + pr_warn_once("sched_ext: BPF scheduler \"%s\" does not implement " +"ops.cgroup_set_bandwidth(); cpu.max will not be enforced\n" , + sch->ops.name); tg->scx.bw_period_us = period_us; tg->scx.bw_quota_us = quota_us;
[email protected]
AI reviewThank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] The new warning for unenforced `cpu.max` is missing during BPF scheduler initialization, meaning…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] The new warning for unenforced cpu.max is missing during BPF scheduler initialization, meaning preexisting cgroup quotas are silently ignored without notification. --
--- Patch [1]: [PATCH RFC] sched_ext: warn when cpu.max is set but the BPF scheduler doesn't implement bandwidth control --- commit 88bd45e8c02c048f00992a1d03614c8ef709c738
Author: Tao Cui <[email protected]>
sched_ext: warn when cpu.max is set but the BPF scheduler doesn't implement bandwidth control
This commit adds a one-time warning when a finite cpu.max quota is configured on a cgroup but the active BPF scheduler lacks the cgroup_set_bandwidth callback. The intent is to notify users and orchestrators that the quota is silently ignored and not enforced by the scheduler.
Quoted 5 lines
kernel/sched/ext/ext.c +0
-0
kernel/sched/ext/ext.c void scx_group_set_bandwidth(struct task_group *tg, · line 4573
Quoted 9 lines
kernel/sched/ext/ext.c +6
-0@@ -4565,6 +4565,12 @@ void scx_group_set_bandwidth(struct task_group *tg, tg->scx.bw_burst_us != burst_us))SCX_CALL_OP (sch, cgroup_set_bandwidth,NULL ,tg_cgrp (tg), period_us, quota_us, burst_us); + else if (scx_cgroup_enabled && sch && + !SCX_HAS_OP(sch, cgroup_set_bandwidth) && + quota_us != RUNTIME_INF) + pr_warn_once("sched_ext: BPF scheduler \"%s\" does not implement " +"ops.cgroup_set_bandwidth(); cpu.max will not be enforced\n" , + sch->ops.name);
[Severity: Medium]
Will this warning also catch preexisting cgroup bandwidth configurations?
If a user configures a finite cpu.max limit on a cgroup before the BPF scheduler is loaded, the initialization path in scx_cgroup_init() passes the existing quota to the scheduler:
kernel/sched/ext/ext.c:scx_cgroup_init() {
...
struct scx_cgroup_init_args args = {
.weight = tg->scx.weight,
.bw_period_us = tg->scx.bw_period_us,
.bw_quota_us = tg->scx.bw_quota_us,
.bw_burst_us = tg->scx.bw_burst_us,
};
...
ret = SCX_CALL_OP_RET(sch, cgroup_init, NULL,
css->cgroup, &args);
...
}Should scx_cgroup_init() also check for an unenforced quota and emit this warning, to ensure users are notified when preexisting configurations are silently ignored?
Quoted 3 lines
kernel/sched/ext/ext.c +0
-0tg->scx.bw_period_us = period_us; tg->scx.bw_quota_us = quota_us;
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Tejun Heo
reviewerHello,
Quoted 16 lines · Tao Cui
From: Tao Cui <[email protected]>
The kernel stores cpu.max bandwidth parameters in the
task_groupand passes them to the BPF scheduler viaops.cgroup_set_bandwidth()andscx_cgroup_init_args, but does not enforce the quota itself. If the loaded BPF scheduler doesn't implement the callback, cpu.max is silently ignored -- the cgroup gets unlimited CPU regardless of the configured quota.Of the example schedulers, only
scx_qmapimplements the callback -- and only tobpf_printk()the parameters, so no in-tree scheduler actually enforces the quota. Measured withscx_simple: a cgroup with cpu.max = "50000 100000" (50% of one CPU) and one busy task used 9946ms of CPU in 10 seconds withnr_throttledremaining 0.
Quoted 3 lines
Print a one-time warning when a finite quota is configured on a cgroup while the active scheduler lacks the callback, so users and container orchestrators know the quota is not enforced.
We had something similar with cpu.weight and it created more annoaynces than helping anything. cgroup bw control isn't the only thing the BPF scheduler may skip to implement. It can also choose to ignore e.g. nice levels completely too and there's no way to detect things like that. Documentation is probably the right way to handle this.
Thanks.
Signature · 1 line
tejun
Hello,
Quoted 23 lines · Tejun Heo
Hello,
Quoted 16 lines · Tao Cui
From: Tao Cui <[email protected]>
The kernel stores cpu.max bandwidth parameters in the
task_groupand passes them to the BPF scheduler viaops.cgroup_set_bandwidth()andscx_cgroup_init_args, but does not enforce the quota itself. If the loaded BPF scheduler doesn't implement the callback, cpu.max is silently ignored -- the cgroup gets unlimited CPU regardless of the configured quota.Of the example schedulers, only
scx_qmapimplements the callback -- and only tobpf_printk()the parameters, so no in-tree scheduler actually enforces the quota. Measured withscx_simple: a cgroup with cpu.max = "50000 100000" (50% of one CPU) and one busy task used 9946ms of CPU in 10 seconds withnr_throttledremaining 0.Quoted 3 lines
Print a one-time warning when a finite quota is configured on a cgroup while the active scheduler lacks the callback, so users and container orchestrators know the quota is not enforced.
Quoted 6 lines
We had something similar with cpu.weight and it created more annoaynces than helping anything. cgroup bw control isn't the only thing the BPF scheduler may skip to implement. It can also choose to ignore e.g. nice levels completely too and there's no way to detect things like that. Documentation is probably the right way to handle this.
Understood, the cpu.weight precedent makes sense: the scheduler may ignore a whole set of knobs, and warning on just one of them would be arbitrary.
I'll follow up with a patch to sched-ext.rst instead: a note that cgroup CPU knobs like cpu.max only take effect if the loaded scheduler implements the corresponding callbacks, and that schedulers may also ignore things like nice levels.
Thanks,
Tao
Quoted 2 lines
Thanks.