sched_ext/scx_flatcg: expire cached hweights on weight changes
Subsystems (2)
PATCH 1/1 sched_ext/scx_flatcg: expire cached hweights on weight changes
From: Tao Cui <[email protected]>
fcg_cgroup_set_weight() updates cgc->weight and the parent's child_weight_sum but doesn't bump hweight_gen, so the hweights cached by cgrp_refresh_hweight() stay stale until some task activation bumps the generation. For cgroups whose tasks never go through a 0->n runnable transition (e.g. persistently busy ones), a cpu.weight change never propagates to scheduling at all.
Bump hweight_gen on weight changes so the next refresh recomputes with the new weight.
Verified on a flatcg VM: a live cpu.weight 100->800 change on a busy cgroup leaves HWT update at 0 and the distribution unchanged; with it, hweight_gen increments and the refresh recomputes.
--- tools/sched_ext/scx_flatcg.bpf.c | 3 +++ 1 file changed, 3 insertions(+)
tools/sched_ext/scx_flatcg.bpf.c +3 -0
@@ -605,6 +605,9 @@ void BPF_STRUCT_OPS(fcg_cgroup_set_weight, struct cgroup *cgrp, u32 weight) pcgc->child_weight_sum += (s64 )weight - cgc->weight; cgc->weight = weight;bpf_spin_unlock (&cgv_tree_lock); + +/* expire cached hweights so the new weight propagates */ +__sync_fetch_and_add (&hweight_gen,1 ); }static bool try_pick_next_cgroup (u64 *cgidp)
Signature · 1 line
2.43.0
Quoted 9 lines · Tao Cui
From: Tao Cui <[email protected]>
fcg_cgroup_set_weight()updatescgc->weightand the parent'schild_weight_sumbut doesn't bumphweight_gen, so the hweights cached bycgrp_refresh_hweight()stay stale until some task activation bumps the generation. For cgroups whose tasks never go through a 0->n runnable transition (e.g. persistently busy ones), a cpu.weight change never propagates to scheduling at all.
Quoted 7 lines
Bump
hweight_genon weight changes so the next refresh recomputes with the new weight.Verified on a flatcg VM: a live cpu.weight 100->800 change on a busy cgroup leaves HWT update at 0 and the distribution unchanged; with it,
hweight_genincrements and the refresh recomputes.
Hello,
Some background on how I found this: I was running scx_flatcg in a VM with a simple cgroup hierarchy to check how it distributes CPU under different cpu.weight values --
A (100) -> A1 (100), A2 (100)
D (weight varies)
with 3 busy tasks in each leaf cgroup, 4 vCPUs, measuring cpu.stat usage_usec deltas over 15 seconds.
D=200 gave D roughly 47%; D=800 gave 46% -- the distribution barely responded to weight at all. That led me to the missing hweight_gen bump, which this patch fixes.
But even with the fix applied, the distribution still doesn't follow the documented compounding model. With D=800 (compounded share should be 800/900 = 89%), D only gets around 49%. Removing the /2 in cgrp_cap_budget()'s max_budget didn't help either (50%).
From what I can tell, the budget clamp is proportional to hweight, so light cgroups get a very small budget and are clamped right behind the cvtime leader, which keeps them competitive regardless of their weight. Under saturation with persistently busy tasks, the distribution compresses to near-equal no matter what the weights are.
Is this the intended behavior of the budget clamping, or should the steady-state distribution converge to the compounded shares over time? The header comment's model doesn't seem to hold in this scenario.
Quoted 19 lines
Signed-off-by Tao Cui <[email protected]>--- tools/sched_ext/scx_flatcg.bpf.c | 3 +++ 1 file changed, 3 insertions(+)tools/sched_ext/scx_flatcg.bpf.c +3
-0@@ -605,6 +605,9 @@ void BPF_STRUCT_OPS(fcg_cgroup_set_weight, struct cgroup *cgrp, u32 weight) pcgc->child_weight_sum += (s64 )weight - cgc->weight; cgc->weight = weight;bpf_spin_unlock (&cgv_tree_lock); + +/* expire cached hweights so the new weight propagates */ +__sync_fetch_and_add (&hweight_gen,1 ); }static bool try_pick_next_cgroup (u64 *cgidp)
Tejun Heo
reviewerHello,
Quoted 3 lines · Tao Cui
Is this the intended behavior of the budget clamping, or should the steady-state distribution converge to the compounded shares over time? The header comment's model doesn't seem to hold in this scenario.
The clamp isn't the culprit. I reproduced your setup (3 busy tasks per leaf, 4 CPUs) and dug in.
The dominant factor is that D is runnable-task limited. At weight 800 its compounded share is 3.55 CPUs but it only has 3 tasks, so 75% is the best it can do. Below that cap, the window granting loses more:
whenever all three of D's tasks are already running and another CPU picks D, the pop from D's DSQ comes up empty, the cgv_node gets stashed and that CPU grants a full cgrp_slice_ns window to another cgroup. The loss scales with the window size. Measured D shares at weight 800:
3 tasks/leaf 55-59% 3 tasks/leaf, 100ms slices 52% 5 tasks/leaf 84-86%
Disabling the clamp entirely is the 59% above, so it barely matters. With enough runnable tasks per cgroup, the distribution converges to the documented compounding. The model holds, but only when no cgroup is runnable-task limited, and the window granting degrades sooner than per-task fair queueing would as that limit is approached. That's an inherent simplification of this example scheduler.
Separately, while digging into this, I found that the true-up in fcg_dispatch() is broken:
__sync_fetch_and_add(&cgc->cvtime_delta, (cpuc->cur_at + cgrp_slice_ns - now) * FCG_HWEIGHT_ONE / (cgc->hweight ?: 1));
In the CNS_EXPIRE case, now is past cur_at + cgrp_slice_ns, so the u64 subexpression wraps. The multiplication preserves the two's complement encoding but the unsigned division by hweight destroys it, adding about 2^64 / hweight per expiry instead of a small correction. The sign is also inverted. The true-up should be actual minus charged, so the expiry overrun should be added and the CNS_EMPTY unused portion subtracted. Under saturation the budget clamp mostly masks the garbage, which is why the numbers above barely move with it fixed (the 86% in the 5 tasks/leaf row), but the accounting is broken all the same. The following fixes it and tests fine (BPF division is unsigned, keep the dividends positive):
s64 delta = now - cpuc->cur_at - cgrp_slice_ns; if (delta >= 0) __sync_fetch_and_add(&cgc->cvtime_delta, (u64)delta * FCG_HWEIGHT_ONE / (cgc->hweight ?: 1)); else __sync_fetch_and_sub(&cgc->cvtime_delta, (u64)-delta * FCG_HWEIGHT_ONE / (cgc->hweight ?: 1));
Care to send a patch?
Thanks.
--
tejun
Quoted 60 lines · Tejun Heo
Hello,
Quoted 3 lines · Tao Cui
Is this the intended behavior of the budget clamping, or should the steady-state distribution converge to the compounded shares over time? The header comment's model doesn't seem to hold in this scenario.
The clamp isn't the culprit. I reproduced your setup (3 busy tasks per leaf, 4 CPUs) and dug in.
The dominant factor is that D is runnable-task limited. At weight 800 its compounded share is 3.55 CPUs but it only has 3 tasks, so 75% is the best it can do. Below that cap, the window granting loses more:
whenever all three of D's tasks are already running and another CPU picks D, the pop from D's DSQ comes up empty, thecgv_nodegets stashed and that CPU grants a fullcgrp_slice_nswindow to another cgroup. The loss scales with the window size. Measured D shares at weight 800:3 tasks/leaf 55-59% 3 tasks/leaf, 100ms slices 52% 5 tasks/leaf 84-86%Disabling the clamp entirely is the 59% above, so it barely matters. With enough runnable tasks per cgroup, the distribution converges to the documented compounding. The model holds, but only when no cgroup is runnable-task limited, and the window granting degrades sooner than per-task fair queueing would as that limit is approached. That's an inherent simplification of this example scheduler.
Separately, while digging into this, I found that the true-up in
fcg_dispatch()is broken:__sync_fetch_and_add(&cgc->cvtime_delta, (cpuc->cur_at + cgrp_slice_ns - now) * FCG_HWEIGHT_ONE / (cgc->hweight ?: 1));In the
CNS_EXPIREcase, now is pastcur_at+cgrp_slice_ns, so the u64 subexpression wraps. The multiplication preserves the two's complement encoding but the unsigned division by hweight destroys it, adding about 2^64 / hweight per expiry instead of a small correction. The sign is also inverted. The true-up should be actual minus charged, so the expiry overrun should be added and theCNS_EMPTYunused portion subtracted. Under saturation the budget clamp mostly masks the garbage, which is why the numbers above barely move with it fixed (the 86% in the 5 tasks/leaf row), but the accounting is broken all the same. The following fixes it and tests fine (BPF division is unsigned, keep the dividends positive):s64 delta = now - cpuc->cur_at - cgrp_slice_ns; if (delta >= 0) __sync_fetch_and_add(&cgc->cvtime_delta, (u64)delta * FCG_HWEIGHT_ONE / (cgc->hweight ?: 1)); else __sync_fetch_and_sub(&cgc->cvtime_delta, (u64)-delta * FCG_HWEIGHT_ONE / (cgc->hweight ?: 1));
Quoted 2 lines
Care to send a patch?
Hi, tejun
Thanks for the detailed walkthrough. I was chasing the clamp, but the runnable-task limit and the window granting behavior explain it much better, learned a lot from your analysis.
I sent a fix based on your suggestion, and saw it's been applied to your tree:
sched_ext/scx_flatcg: Fix cvtime true-up on slice overrun
https://lore.kernel.org/all/[email protected]/
Kept your approach (s64 delta, keep dividends positive) and added A/B numbers from the same VM setup in the commit message.
Looking forward to more of this while digging through sched_ext.
Thanks,
Tao
Quoted 4 lines
Thanks.
--
tejun
[email protected]
AI reviewThis looks like a bug fix. Should this include a Fixes: tag pointing to the commit that introduced the scx_flatcg scheduler?
Quoted 9 lines
sched_ext/scx_flatcg: expire cached hweights on weight changes
fcg_cgroup_set_weight()updatescgc->weightand the parent'schild_weight_sumbut doesn't bumphweight_gen, so the hweights cached bycgrp_refresh_hweight()stay stale until some task activation bumps the generation. For cgroups whose tasks never go through a 0->n runnable transition (e.g. persistently busy ones), a cpu.weight change never propagates to scheduling at all.
Quoted 6 lines
Bump
hweight_genon weight changes so the next refresh recomputes with the new weight.Verified on a flatcg VM: a live cpu.weight 100->800 change on a busy cgroup leaves HWT update at 0 and the distribution unchanged; with it,
hweight_genincrements and the refresh recomputes.
This looks like a bug fix. Should this include a Fixes: tag pointing to the commit that introduced the scx_flatcg scheduler?
This isn't a bug, but should the verification sentence say "without this change" to make the "with it" contrast more explicit? The first clause lacks a matching counterpart, so on first read the "with it" appears to refer back to something unstated.
Quoted 8 lines
Signed-off-by Tao Cui <[email protected]>tools/sched_ext/scx_flatcg.bpf.c +0
-0
tools/sched_ext/scx_flatcg.bpf.c void BPF_STRUCT_OPS(fcg_cgroup_set_weight, struct cgroup *cgrp, u32 weight) · line 612
Quoted 9 lines
tools/sched_ext/scx_flatcg.bpf.c +3
-0@@ -604,6 +604,9 @@ void BPF_STRUCT_OPS(fcg_cgroup_set_weight, struct cgroup *cgrp, u32 weight) pcgc->child_weight_sum += (s64 )weight - cgc->weight; cgc->weight = weight;bpf_spin_unlock (&cgv_tree_lock); + +/* expire cached hweights so the new weight propagates */ +__sync_fetch_and_add (&hweight_gen,1 ); }static bool try_pick_next_cgroup (u64 *cgidp)
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/31820156939
Quoted 6 lines · Tao Cui
fcg_cgroup_set_weight()updatescgc->weightand the parent'schild_weight_sumbut doesn't bumphweight_gen, so the hweights cached bycgrp_refresh_hweight()stay stale until some task activation bumps the generation. For cgroups whose tasks never go through a 0->n runnable transition (e.g. persistently busy ones), a cpu.weight change never propagates to scheduling at all.
Applied to sched_ext/for-7.3.
Thanks.
--
tejun