sched/fair: cpu.max.runtime and idle refill for CPU credits

This series adds cpu.max.runtime, a cgroup v2 file to read and set the runtime the CFS bandwidth controller has for a cgroup, lets cpu.max.burst exceed the quota, and makes an idle group keep earning quota up to quota + burst. Patch 3 passes the writes on to BPF schedulers, and patch 4 lets them charge their tasks to the same runtime.

Use case
========

We run virtual machines whose CPU time is governed by CPU credits, as burstable instances are. Each instance has a baseline, which is the share of its vCPUs it may always use. Running below the baseline earns credits, running above it spends them, and the credits an instance holds are capped. When it has none left, the instance is held to its baseline. The starting credits, the baseline and the cap are part of the instance type's guarantee and must hold whatever the host does to the instance. An instance must neither lose credits nor gain credits it did not earn.

Each instance's vCPU threads run in their own cgroup, and the CFS bandwidth controller is that bucket of credits as it stands:

- quota per period is the rate credits are earned at, the baseline;
- quota + burst is the cap, the most credits an instance can hold;
- the runtime is the credit level. The controller refills it by the quota every period, up to quota + burst, and throttles the cgroup when it runs out.

The controller already does the per-CPU charging and the throttling in the scheduler, at a granularity that userspace accounting cannot match. Three things are missing. The cap cannot exceed one quota, because burst is limited to the quota (patch 1 drops that limit). The level cannot be read or set (patch 1). An idle group does not earn (patch 2), because the period timer stops after a period in which the group did not run, and the periods it sleeps through refill nothing. For example, an instance that idles for a minute with a 100ms period earns one quota instead of up to 600.

The level has to be read and set by the host's control software:

- at launch, an instance starts with the credits its type grants, not

    with one quota;

- across a host kernel live update (kexec) or a reboot, the cgroups are created anew, so their levels have to be read before and written back after, or every instance loses or gains credits;
- a live migration moves the level with the instance to the new host;
- the control plane raises or lowers the level, for example when credits move between instances.

Ben pointed out that a cpu.max or cpu.max.burst write already adds a quota to the runtime, and that cpuacct gives the usage. That falls short of the cases above:

- it can only add, and only one quota per write. Setting a level requires writing a temporary quota equal to the difference and then the real quota back. Lowering a level, which a restore or a transfer may need, requires shrinking the burst until the cap cuts the runtime and then widening it again. Each of these leaves the cgroup running with the wrong bandwidth in between, and each write goes through tg_set_cfs_bandwidth(), which visits every CPU's cfs_rq;
- usage is not the level. Deriving the level from the usage requires replaying every period's refill in userspace, including the periods the timer slept through, which duplicates the controller's accounting.

Keeping the credits in a BPF scheduler instead (patch 3 lets sched_ext follow the level) does not remove the need. The credit guarantee has to hold before a BPF scheduler loads, after it exits or errors out, and on hosts that run no BPF scheduler. The tasks are then on CFS, where the bandwidth controller is the only enforcement, and it has no BPF hook to refill or set the runtime.

Patch 2 changes behavior for existing cpu.max.burst users. A group with a burst that becomes runnable after idling has up to quota + burst, as if it had been running below its quota throughout, instead of the runtime it had when the timer stopped. Groups without a burst see no change.

Testing
=======

On tip/sched/core in qemu with lockdep, the cgroup test_cpu selftests pass, including eight new ones for patches 1 and 2, and so does the new sched_ext selftest cgroup_runtime, which covers patches 3 and 4. Every patch builds with W=1 and no new warnings, with and without CONFIG_CFS_BANDWIDTH and on 32-bit.

An AI coding assistant (Claude Code) helped write the selftests and changelogs and ran the build and test matrix; I reviewed all of it.

Changes in v2
=============

- New patch 2: refill the periods the period timer slept through.
- New patch 4: with the new SCX_OPS_CHARGE_CPU_MAX flag, a BPF scheduler's tasks are charged to the fair class's runtime, read with the new kfunc scx_bpf_task_cpu_max_runtime(); selftest cgroup_runtime.
- Patch 1: a write distributes the runtime to throttled cfs_rqs at once (Ben); a write moves runtime_snap so it is not counted as burst usage; a read is capped at quota + burst so it can be written back; a write needs a quota; burst is bounded so quota + burst cannot overflow (Ben); cpu.max.runtime needs CONFIG_CFS_BANDWIDTH (v1 did not build with CONFIG_EXT_GROUP_SCHED=y and CONFIG_CFS_BANDWIDTH=n); documented in cgroup-v2.rst and sched-bwc.rst; four more selftests.
- Patch 3: the callback is invoked on every write, since writing the same value again adds back what the cgroup ran; it is serialized with cpu.max writes and may be sleepable; the cached value is dropped; the cid-form variant, ops.cpuctl_set_runtime(), is added. Both writes hold a new mutex across the fair class's update and the callback, so cpu.max and cpu.max.runtime writes reach the scheduler in the order the fair class applied them.
- Use case described (Tejun). Rebased on tip/sched/core.

v1: https://lore.kernel.org/all/[email protected]/

Fernand Sieber (4):
sched/fair: expose cpu.max.runtime to set bandwidth runtime directly sched/fair: refill bandwidth for the periods the timer slept through
sched_ext: add cgroup_set_runtime ops callback
sched_ext: charge BPF scheduler tasks to cpu.max bandwidth

 Documentation/admin-guide/cgroup-v2.rst       |  27 +-
 Documentation/scheduler/sched-bwc.rst         |  14 +-
 kernel/sched/core.c                           |  61 +-
 kernel/sched/ext/ext.c                        |  53 ++
 kernel/sched/ext/ext.h                        |   2 +
 kernel/sched/ext/internal.h                   |  45 +-
 kernel/sched/fair.c                           | 260 ++++++-
 kernel/sched/sched.h                          |  15 +
 tools/sched_ext/include/scx/common.bpf.h      |   1 +
 tools/sched_ext/include/scx/compat.h          |   1 +
 tools/testing/selftests/cgroup/test_cpu.c     | 519 ++++++++++++++
 tools/testing/selftests/sched_ext/Makefile    |   1 +
 .../selftests/sched_ext/cgroup_runtime.bpf.c  | 195 ++++++
 .../selftests/sched_ext/cgroup_runtime.c      | 652 ++++++++++++++++++
 tools/testing/selftests/sched_ext/config      |   1 +
 15 files changed, 1836 insertions(+), 11 deletions(-)
 create mode 100644 tools/testing/selftests/sched_ext/cgroup_runtime.bpf.c
 create mode 100644 tools/testing/selftests/sched_ext/cgroup_runtime.c

base-commit: d7e670601af6c564ccf6d3b2ddf70ee4f41a6d1c

2.47.3




Amazon Development Centre (South Africa) (Proprietary) Limited
29 Gogosoa Street, Observatory, Cape Town, Western Cape, 7925, South Africa
Registration Number: 2004 / 034463 / 07

Add cpu.max.runtime, a cgroup v2 file that reads and sets the CFS bandwidth runtime, cfs_b->runtime, in microseconds. Userspace can then grant a cgroup runtime beyond its quota. Refills still add only the quota, capped at quota + burst, so the excess is used up over time and the group returns to its quota per period.

A write is rejected without a quota or above quota + burst, the most the group can accumulate. A read is capped at quota + burst so it can be written back; uncapped slack from __return_cfs_rq_runtime() can exceed that until the next refill. Runtime the CPUs already took, at most a slice each, is not included and a write leaves it in place.

A write distributes the new runtime to throttled cfs_rqs at once, as the slack timer does, rather than waiting for the unsynchronized period timer. It also shifts runtime_snap by the change, so cpu.stat burst accounting neither counts the write nor hides what the group ran since the last refill.

Also drop the burst <= quota limit and require only quota + burst <= max_bw_runtime_us (about 203 days), the quota's own limit, so a group can accumulate many periods of quota. This applies to cgroup v1 cpu.cfs_burst_us too, which shares tg_set_bandwidth().

The file exists with CONFIG_CFS_BANDWIDTH only.

---
 Documentation/admin-guide/cgroup-v2.rst   |  22 +-
 Documentation/scheduler/sched-bwc.rst     |  13 +-
 kernel/sched/core.c                       |  34 ++-
 kernel/sched/fair.c                       |  61 ++++
 kernel/sched/sched.h                      |   2 +
 tools/testing/selftests/cgroup/test_cpu.c | 336 ++++++++++++++++++++++
 6 files changed, 459 insertions(+), 9 deletions(-)

index 9e3c67b02ae2e..c5a02057f0242 100644
@@ -1231,12 +1231,32 @@ will be referred to. All time durations are in microseconds.
 	A read-write single value file which exists on non-root
 	cgroups.  The default is "0".
 
-	The burst in the range [0, $MAX].
+	The burst. The cgroup can accumulate the runtime it does not
+	use, up to $MAX + cpu.max.burst. The burst may exceed $MAX, as
+	long as $MAX + cpu.max.burst is at most 2^44 - 1 (about 203
+	days).
 
 	This file affects only processes under the fair-class scheduler and a BPF
 	scheduler with the ``cgroup_set_bandwidth`` callback depending on what
 	the callback actually does.
 
+  cpu.max.runtime
+	A read-write single value file which exists on non-root
+	cgroups if the kernel is built with CONFIG_CFS_BANDWIDTH.
+
+	The runtime the cgroup has accumulated and not used yet, in
+	microseconds. Each period refills it by $MAX, up to $MAX +
+	cpu.max.burst, and a read returns at most $MAX + cpu.max.burst.
+	A write sets it, up to $MAX + cpu.max.burst, and lets a cgroup
+	that ran out of runtime run again at once. A write fails with
+	EINVAL while $MAX is "max".
+
+	The runtime does not include what the cgroup's CPUs already took
+	from it, at most a slice each (sched_cfs_bandwidth_slice_us).
+	A write leaves that with the CPUs.
+
+	This file affects only processes under the fair-class scheduler.
+
   cpu.pressure
 	A read-write nested-keyed file.
 

index e881a945c1888..c22b3c8c14e29 100644
@@ -90,8 +90,8 @@ bandwidth restriction in place, such a group is described as an unconstrained
 bandwidth group. This represents the traditional work-conserving behavior for
 CFS.
 
-Writing any (valid) positive value(s) no smaller than cpu.cfs_burst_us will
-enact the specified bandwidth limit. The minimum quota allowed for the quota or
+Writing any (valid) positive value(s) will enact the specified bandwidth
+limit. The minimum quota allowed for the quota or
 period is 1ms. There is also an upper bound on the period length of 1s.
 Additional restrictions exist when bandwidth limits are used in a hierarchical
 fashion, these are explained in more detail below.

@@ -101,9 +101,10 @@ and return the group to an unconstrained state once more.
 
 A value of 0 for cpu.cfs_burst_us indicates that the group can not accumulate
 any unused bandwidth. It makes the traditional bandwidth control behavior for
-CFS unchanged. Writing any (valid) positive value(s) no larger than
-cpu.cfs_quota_us into cpu.cfs_burst_us will enact the cap on unused bandwidth
-accumulation.
+CFS unchanged. Writing any (valid) positive value(s) into cpu.cfs_burst_us
+will enact the cap on unused bandwidth accumulation: the group can accumulate
+up to cpu.cfs_quota_us + cpu.cfs_burst_us. The burst may be larger than the
+quota, as long as their sum is at most 2^44 - 1 microseconds (about 203 days).
 
 Any updates to a group's bandwidth specification will result in it becoming
 unthrottled if it is in a constrained state.

@@ -243,4 +244,4 @@ Examples
 	# echo 50000 > cpu.cfs_period_us /* period = 50ms */
 	# echo 10000 > cpu.cfs_burst_us /* burst = 10ms */
 
-   Larger buffer setting (no larger than quota) allows greater burst capacity.
+   Larger buffer setting allows greater burst capacity.

index 175b52a7ea793..76fcd70ff207a 100644
@@ -10131,8 +10131,7 @@ static int tg_set_bandwidth(struct task_group *tg,
 	if (quota_us != RUNTIME_INF && quota_us > max_bw_runtime_us)
 		return -EINVAL;
 
-	if (quota_us != RUNTIME_INF && (burst_us > quota_us ||
-					burst_us + quota_us > max_bw_runtime_us))
+	if (quota_us != RUNTIME_INF && burst_us > max_bw_runtime_us - quota_us)
 		return -EINVAL;
 
 #ifdef CONFIG_CFS_BANDWIDTH

@@ -10193,6 +10192,29 @@ static int cpu_burst_write_u64(struct cgroup_subsys_state *css,
 	tg_bandwidth(tg, &period_us, &quota_us, NULL);
 	return tg_set_bandwidth(tg, period_us, quota_us, burst_us);
 }
+
+#ifdef CONFIG_CFS_BANDWIDTH
+static int cpu_runtime_write_u64(struct cgroup_subsys_state *css,
+				 struct cftype *cftype, u64 runtime_us)
+{
+	struct task_group *tg = css_tg(css);
+
+	if (runtime_us > max_bw_runtime_us)
+		return -EINVAL;
+
+	return cfs_bandwidth_set_runtime(&tg->cfs_bandwidth,
+					 runtime_us * NSEC_PER_USEC);
+}
+
+static u64 cpu_runtime_read_u64(struct cgroup_subsys_state *css,
+				struct cftype *cftype)
+{
+	struct task_group *tg = css_tg(css);
+
+	return div_u64(cfs_bandwidth_get_runtime(&tg->cfs_bandwidth),
+		       NSEC_PER_USEC);
+}
+#endif /* CONFIG_CFS_BANDWIDTH */
 #endif /* CONFIG_GROUP_SCHED_BANDWIDTH */
 
 #ifdef CONFIG_RT_GROUP_SCHED

@@ -10544,7 +10566,15 @@ static struct cftype cpu_files[] = {
 		.read_u64 = cpu_burst_read_u64,
 		.write_u64 = cpu_burst_write_u64,
 	},
+#ifdef CONFIG_CFS_BANDWIDTH
+	{
+		.name = "max.runtime",
+		.flags = CFTYPE_NOT_ON_ROOT,
+		.read_u64 = cpu_runtime_read_u64,
+		.write_u64 = cpu_runtime_write_u64,
+	},
 #endif /* CONFIG_CFS_BANDWIDTH */
+#endif /* CONFIG_GROUP_SCHED_BANDWIDTH */
 #ifdef CONFIG_UCLAMP_TASK_GROUP
 	{
 		.name = "uclamp.min",

index a4df59c9c9b60..1f60c84e0dc38 100644
@@ -578,6 +578,8 @@ extern void init_cfs_bandwidth(struct cfs_bandwidth *cfs_b, struct cfs_bandwidth
 
 extern void __refill_cfs_bandwidth_runtime(struct cfs_bandwidth *cfs_b);
 extern void start_cfs_bandwidth(struct cfs_bandwidth *cfs_b);
+u64 cfs_bandwidth_get_runtime(struct cfs_bandwidth *cfs_b);
+int cfs_bandwidth_set_runtime(struct cfs_bandwidth *cfs_b, u64 runtime);
 extern void unthrottle_cfs_rq(struct cfs_rq *cfs_rq);
 extern bool cfs_task_bw_constrained(struct task_struct *p);
 

2.47.3




Amazon Development Centre (South Africa) (Proprietary) Limited
29 Gogosoa Street, Observatory, Cape Town, Western Cape, 7925, South Africa
Registration Number: 2004 / 034463 / 07

cpu.max.burst lets a group accumulate unused quota up to quota + burst. But the period timer stops after a period in which the group took no runtime, and start_cfs_bandwidth() restarts it with hrtimer_forward_now() and discards the periods that passed. A group that sits idle stops accumulating, and one that wakes after idling for a long time has only what it had when the timer stopped.

Count the missed periods instead. While the timer is stopped, its expiry is the first period boundary it missed, so the overrun at restart is the number of refills owed. Add quota for each, capped at quota + burst as __refill_cfs_bandwidth_runtime() caps it.

Without a burst the cap is quota. The timer stops right after a refill that nothing drew from, so the runtime is at quota then, and groups without a burst see no change. Groups with a burst do see a change. A group that has been idle now has up to quota + burst when it runs again. The documentation of cpu.max.burst, cpu.max.runtime and cpu.cfs_burst_us says so.

A change of the bandwidth (cpu.max, cpu.max.burst) applies from that moment, and the group keeps what it earned before it. The periods owed are first refilled at the bandwidth they were earned under, capped at its quota + burst, and then the new bandwidth applies, with its own cap. A group that had no quota earned nothing, so one given a quota after a long time unconstrained does not start full. A cpu.max.runtime write also refills the periods owed first, so that it replaces them, and only the periods after the write refill.

cpu.max.runtime reports the runtime the group has now, owed refills included, without starting the timer.

The idle periods are not added to cpu.stat's nr_periods, which counts the periods in which the timer ran. A timer that runs late by several periods still refills only once, unlike a stopped one; this patch leaves that as it is.

---
 Documentation/admin-guide/cgroup-v2.rst   |  15 +-
 Documentation/scheduler/sched-bwc.rst     |   5 +-
 kernel/sched/core.c                       |   6 +
 kernel/sched/fair.c                       |  79 +++++++++-
 kernel/sched/sched.h                      |   1 +
 tools/testing/selftests/cgroup/test_cpu.c | 183 ++++++++++++++++++++++
 6 files changed, 278 insertions(+), 11 deletions(-)

index c5a02057f0242..2c220e5998639 100644
@@ -1232,9 +1232,9 @@ will be referred to. All time durations are in microseconds.
 	cgroups.  The default is "0".
 
 	The burst. The cgroup can accumulate the runtime it does not
-	use, up to $MAX + cpu.max.burst. The burst may exceed $MAX, as
-	long as $MAX + cpu.max.burst is at most 2^44 - 1 (about 203
-	days).
+	use, up to $MAX + cpu.max.burst, also in the periods in which it
+	does not run. The burst may exceed $MAX, as long as $MAX +
+	cpu.max.burst is at most 2^44 - 1 (about 203 days).
 
 	This file affects only processes under the fair-class scheduler and a BPF
 	scheduler with the ``cgroup_set_bandwidth`` callback depending on what

@@ -1246,10 +1246,11 @@ will be referred to. All time durations are in microseconds.
 
 	The runtime the cgroup has accumulated and not used yet, in
 	microseconds. Each period refills it by $MAX, up to $MAX +
-	cpu.max.burst, and a read returns at most $MAX + cpu.max.burst.
-	A write sets it, up to $MAX + cpu.max.burst, and lets a cgroup
-	that ran out of runtime run again at once. A write fails with
-	EINVAL while $MAX is "max".
+	cpu.max.burst, whether the cgroup runs in it or not. A read
+	returns at most $MAX + cpu.max.burst. A write sets it, up to
+	$MAX + cpu.max.burst, and lets a cgroup that ran out of runtime
+	run again at once. A write fails with EINVAL while $MAX is
+	"max".
 
 	The runtime does not include what the cgroup's CPUs already took
 	from it, at most a slice each (sched_cfs_bandwidth_slice_us).

index c22b3c8c14e29..ccf7434ca73f0 100644
@@ -103,8 +103,9 @@ A value of 0 for cpu.cfs_burst_us indicates that the group can not accumulate
 any unused bandwidth. It makes the traditional bandwidth control behavior for
 CFS unchanged. Writing any (valid) positive value(s) into cpu.cfs_burst_us
 will enact the cap on unused bandwidth accumulation: the group can accumulate
-up to cpu.cfs_quota_us + cpu.cfs_burst_us. The burst may be larger than the
-quota, as long as their sum is at most 2^44 - 1 microseconds (about 203 days).
+up to cpu.cfs_quota_us + cpu.cfs_burst_us, also over periods in which it does
+not run. The burst may be larger than the quota, as long as their sum is at
+most 2^44 - 1 microseconds (about 203 days).
 
 Any updates to a group's bandwidth specification will result in it becoming
 unthrottled if it is in a constrained state.

index 76fcd70ff207a..75a5f5d1b8f54 100644
@@ -9856,6 +9856,12 @@ static int tg_set_cfs_bandwidth(struct task_group *tg,
 		cfs_bandwidth_usage_inc();
 
 	scoped_guard (raw_spinlock_irq, &cfs_b->lock) {
+		/*
+		 * The new bandwidth applies from now on. The group keeps what
+		 * it earned while idle under the old one.
+		 */
+		cfs_bandwidth_refill_idle_periods(cfs_b);
+
 		cfs_b->period = ns_to_ktime(period);
 		cfs_b->quota = quota;
 		cfs_b->burst = burst;

index 1f60c84e0dc38..676e6371d0265 100644
@@ -580,6 +580,7 @@ extern void __refill_cfs_bandwidth_runtime(struct cfs_bandwidth *cfs_b);
 extern void start_cfs_bandwidth(struct cfs_bandwidth *cfs_b);
 u64 cfs_bandwidth_get_runtime(struct cfs_bandwidth *cfs_b);
 int cfs_bandwidth_set_runtime(struct cfs_bandwidth *cfs_b, u64 runtime);
+void cfs_bandwidth_refill_idle_periods(struct cfs_bandwidth *cfs_b);
 extern void unthrottle_cfs_rq(struct cfs_rq *cfs_rq);
 extern bool cfs_task_bw_constrained(struct task_struct *p);
 

2.47.3




Amazon Development Centre (South Africa) (Proprietary) Limited
29 Gogosoa Street, Observatory, Cape Town, Western Cape, 7925, South Africa
Registration Number: 2004 / 034463 / 07

Add a sched_ext_ops callback, and its cid-form variant, that is invoked when userspace writes cpu.max.runtime. A BPF scheduler that implements ops.cgroup_set_bandwidth() to enforce a cgroup's CPU bandwidth also needs the written runtime, so that its budget for the cgroup matches the fair class's, for example after userspace restores a saved runtime or adds to it.

The callback is invoked on every write, after the fair class has applied it. Unlike cgroup_set_bandwidth(), it does not skip a write of the value last written. The runtime goes down as the cgroup runs, so writing the same value again is a top-up.

A new tg_bandwidth_mutex is held across the fair class's update and the callback, by the cpu.max.runtime write and by tg_set_bandwidth(), which the cpu.max and cgroup v1 cpu.cfs_* writes go through and which calls cgroup_set_bandwidth(). The runtime passed is then at most the quota + burst the scheduler was last given, and cpu.max and cpu.max.runtime writes reach it in the order the fair class applied them. The mutex is taken before, and the callbacks run after, the fair class's cpus_read_lock() and cfs_constraints_mutex sections, so cgroup_set_bandwidth() runs as it did before.

Like cgroup_set_bandwidth(), the callback runs in process context and may be sleepable.

---
 Documentation/admin-guide/cgroup-v2.rst |  4 +++-
 kernel/sched/core.c                     | 25 +++++++++++++++++++++++--
 kernel/sched/ext/ext.c                  | 19 +++++++++++++++++++
 kernel/sched/ext/ext.h                  |  2 ++
 kernel/sched/ext/internal.h             | 21 +++++++++++++++++++++
 5 files changed, 68 insertions(+), 3 deletions(-)

index 2c220e5998639..b77ca6f8ca548 100644
@@ -1256,7 +1256,9 @@ will be referred to. All time durations are in microseconds.
 	from it, at most a slice each (sched_cfs_bandwidth_slice_us).
 	A write leaves that with the CPUs.
 
-	This file affects only processes under the fair-class scheduler.
+	This file affects only processes under the fair-class scheduler and a BPF
+	scheduler with the ``cgroup_set_runtime`` callback depending on what
+	the callback actually does.
 
   cpu.pressure
 	A read-write nested-keyed file.

index 75a5f5d1b8f54..25a4acb72106c 100644
@@ -10099,6 +10099,14 @@ static u64 cpu_period_read_u64(struct cgroup_subsys_state *css,
 	return period_us;
 }
 
+/*
+ * Held across a bandwidth or runtime write and the sched_ext callback that
+ * follows it, so that the BPF scheduler gets the writes in the order the fair
+ * class applied them. The callback runs after the fair class's update has
+ * dropped cpus_read_lock() and cfs_constraints_mutex.
+ */
+static DEFINE_MUTEX(tg_bandwidth_mutex);
+
 static int tg_set_bandwidth(struct task_group *tg,
 			    u64 period_us, u64 quota_us, u64 burst_us)
 {

@@ -10140,6 +10148,8 @@ static int tg_set_bandwidth(struct task_group *tg,
 	if (quota_us != RUNTIME_INF && burst_us > max_bw_runtime_us - quota_us)
 		return -EINVAL;
 
+	guard(mutex)(&tg_bandwidth_mutex);
+
 #ifdef CONFIG_CFS_BANDWIDTH
 	ret = tg_set_cfs_bandwidth(tg, period_us, quota_us, burst_us);
 #endif /* CONFIG_CFS_BANDWIDTH */

@@ -10204,12 +10214,23 @@ static int cpu_runtime_write_u64(struct cgroup_subsys_state *css,
 				 struct cftype *cftype, u64 runtime_us)
 {
 	struct task_group *tg = css_tg(css);
+	int ret;
 
 	if (runtime_us > max_bw_runtime_us)
 		return -EINVAL;
 
-	return cfs_bandwidth_set_runtime(&tg->cfs_bandwidth,
-					 runtime_us * NSEC_PER_USEC);
+	/*
+	 * Serialize against tg_set_bandwidth(), so that the BPF scheduler gets
+	 * the runtime checked against the current quota + burst, and this and
+	 * cpu.max writes reach it in the order the fair class took them.
+	 */
+	guard(mutex)(&tg_bandwidth_mutex);
+
+	ret = cfs_bandwidth_set_runtime(&tg->cfs_bandwidth,
+					runtime_us * NSEC_PER_USEC);
+	if (!ret)
+		scx_group_set_runtime(tg, runtime_us);
+	return ret;
 }
 
 static u64 cpu_runtime_read_u64(struct cgroup_subsys_state *css,

index ebdf7b91f9d37..265a80dd93282 100644
@@ -5817,6 +5817,19 @@ void scx_group_set_bandwidth(struct task_group *tg,
 
 	percpu_up_read(&scx_cgroup_ops_rwsem);
 }
+
+void scx_group_set_runtime(struct task_group *tg, u64 runtime_us)
+{
+	struct scx_sched *sch;
+
+	percpu_down_read(&scx_cgroup_ops_rwsem);
+	sch = scx_tg_knob_sched(tg);
+
+	if (scx_cgroup_enabled && sch && SCX_HAS_OP(sch, cgroup_set_runtime))
+		SCX_CALL_OP(sch, cgroup_set_runtime, NULL, tg_cgrp(tg), runtime_us);
+
+	percpu_up_read(&scx_cgroup_ops_rwsem);
+}
 #endif	/* CONFIG_EXT_GROUP_SCHED */
 
 #if defined(CONFIG_EXT_GROUP_SCHED) || defined(CONFIG_EXT_SUB_SCHED)

@@ -9230,6 +9243,7 @@ static int bpf_scx_check_member(const struct btf_type *t,
 	case offsetof(struct sched_ext_ops, cgroup_exit):
 	case offsetof(struct sched_ext_ops, cgroup_prep_move):
 	case offsetof(struct sched_ext_ops, cgroup_set_bandwidth):
+	case offsetof(struct sched_ext_ops, cgroup_set_runtime):
 #endif
 	case offsetof(struct sched_ext_ops, cpu_online):
 	case offsetof(struct sched_ext_ops, cpu_offline):

@@ -9392,6 +9406,7 @@ static void sched_ext_ops__cgroup_cancel_move(struct task_struct *p, struct cgro
 static void sched_ext_ops__cgroup_set_weight(struct cgroup *cgrp, u32 weight) {}
 static void sched_ext_ops__cgroup_set_bandwidth(struct cgroup *cgrp, u64 period_us, u64 quota_us, u64 burst_us) {}
 static void sched_ext_ops__cgroup_set_idle(struct cgroup *cgrp, bool idle) {}
+static void sched_ext_ops__cgroup_set_runtime(struct cgroup *cgrp, u64 runtime_us) {}
 #endif	/* CONFIG_EXT_GROUP_SCHED */
 static s32 sched_ext_ops__sub_attach(struct scx_sub_attach_args *args) { return -EINVAL; }
 static void sched_ext_ops__sub_detach(struct scx_sub_detach_args *args) {}

@@ -9434,6 +9449,7 @@ static struct sched_ext_ops __bpf_ops_sched_ext_ops = {
 	.cgroup_set_weight	= sched_ext_ops__cgroup_set_weight,
 	.cgroup_set_bandwidth	= sched_ext_ops__cgroup_set_bandwidth,
 	.cgroup_set_idle	= sched_ext_ops__cgroup_set_idle,
+	.cgroup_set_runtime	= sched_ext_ops__cgroup_set_runtime,
 #endif
 	.sub_attach		= sched_ext_ops__sub_attach,
 	.sub_detach		= sched_ext_ops__sub_detach,

@@ -9502,6 +9518,7 @@ static struct sched_ext_ops_cid __bpf_ops_sched_ext_ops_cid = {
 	.cpuctl_set_weight	= sched_ext_ops__cgroup_set_weight,
 	.cpuctl_set_bandwidth	= sched_ext_ops__cgroup_set_bandwidth,
 	.cpuctl_set_idle	= sched_ext_ops__cgroup_set_idle,
+	.cpuctl_set_runtime	= sched_ext_ops__cgroup_set_runtime,
 #endif
 	.sub_attach		= sched_ext_ops__sub_attach,
 	.sub_detach		= sched_ext_ops__sub_detach,

@@ -12226,6 +12243,7 @@ static const u32 scx_kf_allow_flags[] = {
 	[SCX_OP_IDX(cgroup_set_weight)]	= SCX_KF_ALLOW_UNLOCKED,
 	[SCX_OP_IDX(cgroup_set_bandwidth)] = SCX_KF_ALLOW_UNLOCKED,
 	[SCX_OP_IDX(cgroup_set_idle)]	= SCX_KF_ALLOW_UNLOCKED,
+	[SCX_OP_IDX(cgroup_set_runtime)] = SCX_KF_ALLOW_UNLOCKED,
 #endif	/* CONFIG_EXT_GROUP_SCHED */
 	[SCX_OP_IDX(sub_attach)]	= SCX_KF_ALLOW_UNLOCKED,
 	[SCX_OP_IDX(sub_detach)]	= SCX_KF_ALLOW_UNLOCKED,

@@ -12395,6 +12413,7 @@ static int __init scx_init(void)
 	CID_OFFSET_MATCH(cgroup_set_weight, cpuctl_set_weight);
 	CID_OFFSET_MATCH(cgroup_set_bandwidth, cpuctl_set_bandwidth);
 	CID_OFFSET_MATCH(cgroup_set_idle, cpuctl_set_idle);
+	CID_OFFSET_MATCH(cgroup_set_runtime, cpuctl_set_runtime);
 #endif
 	/* @priv tail must align since both share the same data block */
 	CID_OFFSET_MATCH(priv, priv);

index d32bc01071b01..6d1ce582bf724 100644
@@ -94,6 +94,7 @@ void scx_cgroup_cancel_attach(struct cgroup_taskset *tset);
 void scx_group_set_weight(struct task_group *tg, unsigned long cgrp_weight);
 void scx_group_set_idle(struct task_group *tg, bool idle);
 void scx_group_set_bandwidth(struct task_group *tg, u64 period_us, u64 quota_us, u64 burst_us);
+void scx_group_set_runtime(struct task_group *tg, u64 runtime_us);
 #else	/* CONFIG_EXT_GROUP_SCHED */
 static inline void scx_tg_init(struct task_group *tg) {}
 static inline int scx_tg_online(struct task_group *tg) { return 0; }

@@ -104,5 +105,6 @@ static inline void scx_cgroup_cancel_attach(struct cgroup_taskset *tset) {}
 static inline void scx_group_set_weight(struct task_group *tg, unsigned long cgrp_weight) {}
 static inline void scx_group_set_idle(struct task_group *tg, bool idle) {}
 static inline void scx_group_set_bandwidth(struct task_group *tg, u64 period_us, u64 quota_us, u64 burst_us) {}
+static inline void scx_group_set_runtime(struct task_group *tg, u64 runtime_us) {}
 #endif	/* CONFIG_EXT_GROUP_SCHED */
 #endif	/* CONFIG_CGROUP_SCHED */

index ff0a3d508533f..e794507596d6f 100644
@@ -863,6 +863,26 @@ struct sched_ext_ops {
 	 */
 	void (*cgroup_set_idle)(struct cgroup *cgrp, bool idle);
 
+	/**
+	 * @cgroup_set_runtime: A cgroup's runtime is being set
+	 * @cgrp: cgroup whose runtime is being set
+	 * @runtime_us: runtime in microseconds
+	 *
+	 * Set the runtime @cgrp has accumulated and not used yet to
+	 * @runtime_us. This is from the cpu.max.runtime cgroup interface,
+	 * which exists with CONFIG_CFS_BANDWIDTH only. It is called on every
+	 * write, also of an unchanged value. The runtime goes down as @cgrp
+	 * runs, so a write of the same value is a top-up.
+	 *
+	 * @cgrp has a quota, and @runtime_us is at most the quota + burst the
+	 * scheduler was last given, by @cgroup_init or @cgroup_set_bandwidth:
+	 * cpu.max and cpu.max.runtime writes reach the scheduler in the order
+	 * the fair class applied them.
+	 *
+	 * Delivery follows the same rule as cgroup_set_weight().
+	 */
+	void (*cgroup_set_runtime)(struct cgroup *cgrp, u64 runtime_us);
+
 #endif	/* CONFIG_EXT_GROUP_SCHED */
 
 	/**

@@ -1239,6 +1259,7 @@ struct sched_ext_ops_cid {
 	void (*cpuctl_set_bandwidth)(struct cgroup *cgrp, u64 period_us, u64 quota_us,
 				     u64 burst_us);
 	void (*cpuctl_set_idle)(struct cgroup *cgrp, bool idle);
+	void (*cpuctl_set_runtime)(struct cgroup *cgrp, u64 runtime_us);
 #endif	/* CONFIG_EXT_GROUP_SCHED */
 	s32 (*sub_attach)(struct scx_sub_attach_args *args);
 	void (*sub_detach)(struct scx_sub_detach_args *args);

2.47.3




Amazon Development Centre (South Africa) (Proprietary) Limited
29 Gogosoa Street, Observatory, Cape Town, Western Cape, 7925, South Africa
Registration Number: 2004 / 034463 / 07

A BPF scheduler that enforces cpu.max keeps each cgroup's budget itself, and no runtime moves between it and the fair class. One loaded after a cpu.max.runtime write starts the cgroup on a fresh quota, losing the unspent grant.

Conversely, a read reports the fair class's runtime, which refills to quota + burst meanwhile. On disable, the fair class takes the tasks back with that full runtime, so a cgroup that used up its budget runs over its quota until the period accounting catches up.

Under SCX_OPS_SWITCH_PARTIAL, both budgets apply at once: a cgroup with tasks in both classes runs up to its quota in each, which no handover at load or unload can fix. Keep one budget, the fair class's, and let a BPF scheduler opt in to drawing from it with the new SCX_OPS_CHARGE_CPU_MAX flag.

With SCX_OPS_CHARGE_CPU_MAX, update_curr_scx() charges a task's runtime to its cgroups' cfs_rqs as a fair task's would, without throttling it. A task the scheduler stops late leaves its CPU in debt, which the next periods pay, also once the task has moved; debt past minus a slice moves to cfs_b, whose next refill pays up to a quota of it. The new kfunc scx_bpf_task_cpu_max_runtime() returns the runtime left less that debt, and the scheduler runs the task only while it is positive. Cgroups whose period timer is stopped are skipped, as only a charge restarts it.

Both classes then draw from one budget: under SCX_OPS_SWITCH_PARTIAL a cgroup gets its quota per period however its tasks are split, and nothing is handed over at load or unload. A task returning to the fair class leaves its CPU at most a slice in debt; schedulers without the flag behave as before.

cfs_b->lock is taken at most twice per millisecond of run time per CPU. The cgroup_runtime selftest covers these cases.

---
 Documentation/admin-guide/cgroup-v2.rst       |   4 +-
 kernel/sched/ext/ext.c                        |  34 +
 kernel/sched/ext/internal.h                   |  24 +-
 kernel/sched/fair.c                           | 122 ++++
 kernel/sched/sched.h                          |  12 +
 tools/sched_ext/include/scx/common.bpf.h      |   1 +
 tools/sched_ext/include/scx/compat.h          |   1 +
 tools/testing/selftests/sched_ext/Makefile    |   1 +
 .../selftests/sched_ext/cgroup_runtime.bpf.c  | 195 ++++++
 .../selftests/sched_ext/cgroup_runtime.c      | 652 ++++++++++++++++++
 tools/testing/selftests/sched_ext/config      |   1 +
 11 files changed, 1045 insertions(+), 2 deletions(-)
 create mode 100644 tools/testing/selftests/sched_ext/cgroup_runtime.bpf.c
 create mode 100644 tools/testing/selftests/sched_ext/cgroup_runtime.c

index b77ca6f8ca548..db9fa6e6dbd49 100644
@@ -1258,7 +1258,9 @@ will be referred to. All time durations are in microseconds.
 
 	This file affects only processes under the fair-class scheduler and a BPF
 	scheduler with the ``cgroup_set_runtime`` callback depending on what
-	the callback actually does.
+	the callback actually does. A BPF scheduler with the
+	``SCX_OPS_CHARGE_CPU_MAX`` flag draws what its processes run from
+	the same runtime, so the cgroup's processes in both share it.
 
   cpu.pressure
 	A read-write nested-keyed file.

index 265a80dd93282..6ffc9ca2c5c9f 100644
@@ -1625,6 +1625,11 @@ static void expire_task_slice(struct task_struct *p)
 		scx_set_task_slice(p, 0);
 }
 
+static bool scx_task_charges_cpu_max(const struct task_struct *p)
+{
+	return scx_task_sched(p)->ops.flags & SCX_OPS_CHARGE_CPU_MAX;
+}
+
 static void update_curr_scx(struct rq *rq)
 {
 	struct task_struct *donor;

@@ -1650,6 +1655,9 @@ static void update_curr_scx(struct rq *rq)
 	if (unlikely(donor == scx_rescuee(rq)))
 		scx_rescue_charge(rq, delta_exec);
 
+	if (scx_task_charges_cpu_max(donor))
+		cfs_bandwidth_charge_task(rq, donor, delta_exec);
+
 	dl_server_update(&rq->ext_server, delta_exec);
 }
 

@@ -2605,6 +2613,13 @@ static bool dequeue_task_scx(struct rq *rq, struct task_struct *p, int core_deq_
 
 	ops_dequeue(rq, p, deq_flags);
 
+	/* charge what @p ran, then return what @rq holds for its cgroups */
+	if (scx_task_charges_cpu_max(p)) {
+		if (task_current_donor(rq, p))
+			update_curr_scx(rq);
+		cfs_bandwidth_return_task(rq, p);
+	}
+
 	/*
 	 * A current scheduling context which is going off @rq first gets
 	 * dequeued and then stops running. As we want running <-> stopping

@@ -11793,6 +11808,24 @@ __bpf_kfunc s32 scx_bpf_task_cpu(const struct task_struct *p)
 	return task_cpu(p);
 }
 
+/**
+ * scx_bpf_task_cpu_max_runtime - Runtime a task has left under cpu.max
+ * @p: task of interest
+ *
+ * Return the runtime in nsecs that @p's cgroup and its ancestors have left
+ * under their cpu.max quotas for @p on its current CPU, the least over those
+ * with a quota: the cgroup's cpu.max.runtime plus what the CPU holds for
+ * the cgroup, less what it owes. U64_MAX if none limits it. Read without
+ * locks.
+ *
+ * With %SCX_OPS_CHARGE_CPU_MAX, the scheduler should run @p only while this
+ * returns more than 0. The runtime grows when a period refills it.
+ */
+__bpf_kfunc u64 scx_bpf_task_cpu_max_runtime(struct task_struct *p)
+{
+	return cfs_bandwidth_task_runtime(p);
+}
+
 /**
  * scx_bpf_task_cid - cid a task is currently associated with
  * @p: task of interest

@@ -12119,6 +12152,7 @@ BTF_ID_FLAGS(func, scx_bpf_get_online_cpumask, KF_ACQUIRE)
 BTF_ID_FLAGS(func, scx_bpf_put_cpumask, KF_RELEASE)
 BTF_ID_FLAGS(func, scx_bpf_task_running, KF_RCU)
 BTF_ID_FLAGS(func, scx_bpf_task_cpu, KF_RCU)
+BTF_ID_FLAGS(func, scx_bpf_task_cpu_max_runtime, KF_RCU)
 BTF_ID_FLAGS(func, scx_bpf_task_cid, KF_RCU)
 BTF_ID_FLAGS(func, scx_bpf_locked_rq, KF_IMPLICIT_ARGS | KF_RET_NULL)
 BTF_ID_FLAGS(func, scx_bpf_cpu_curr, KF_IMPLICIT_ARGS | KF_RET_NULL | KF_RCU_PROTECTED)

index e794507596d6f..e8c0b573d0a43 100644
@@ -243,6 +243,27 @@ enum scx_ops_flags {
 	 */
 	SCX_OPS_ENQ_BLOCKED		= 1LLU << 10,
 
+	/*
+	 * If set, what the scheduler's tasks run is charged to the cpu.max
+	 * bandwidth of their cgroups, the runtime the fair class keeps and
+	 * cpu.max.runtime reads, as what fair tasks run is. The tasks of a
+	 * cgroup in both classes then share one quota, also with
+	 * %SCX_OPS_SWITCH_PARTIAL, and nothing is handed over when tasks
+	 * switch class.
+	 *
+	 * The kernel never throttles the scheduler's tasks. The scheduler
+	 * enforces the quota by running a task only while
+	 * scx_bpf_task_cpu_max_runtime() returns more than 0. What a task runs
+	 * beyond that is charged: up to a slice (sched_cfs_bandwidth_slice_us)
+	 * per CPU as a debt the cgroup's fair tasks on that CPU pay back, and up
+	 * to a quota more from the cgroup's next period. The scheduler does not
+	 * need ops.cgroup_set_runtime(), as a write of cpu.max.runtime sets that
+	 * runtime.
+	 *
+	 * Without CONFIG_CFS_BANDWIDTH, this flag has no effect.
+	 */
+	SCX_OPS_CHARGE_CPU_MAX		= 1LLU << 11,
+
 	SCX_OPS_ALL_FLAGS		= SCX_OPS_KEEP_BUILTIN_IDLE |
 					  SCX_OPS_ENQ_LAST |
 					  SCX_OPS_ENQ_EXITING |

@@ -253,7 +274,8 @@ enum scx_ops_flags {
 					  SCX_OPS_ALWAYS_ENQ_IMMED |
 					  SCX_OPS_TID_TO_TASK |
 					  SCX_OPS_LAZY_RESCHED |
-					  SCX_OPS_ENQ_BLOCKED,
+					  SCX_OPS_ENQ_BLOCKED |
+					  SCX_OPS_CHARGE_CPU_MAX,
 
 	/* high 8 bits are internal, don't include in SCX_OPS_ALL_FLAGS */
 	__SCX_OPS_INTERNAL_MASK		= 0xffLLU << 56,

index 676e6371d0265..fd98000ffd744 100644
@@ -456,6 +456,7 @@ struct cfs_bandwidth {
 	u64			runtime;
 	u64			burst;
 	u64			runtime_snap;
+	u64			debt;
 	s64			hierarchical_quota;
 
 	u8			idle;

@@ -621,6 +622,17 @@ static inline bool cfs_task_bw_constrained(struct task_struct *p) { return false
 
 #endif /* !CONFIG_CGROUP_SCHED */
 
+#ifdef CONFIG_CFS_BANDWIDTH
+void cfs_bandwidth_charge_task(struct rq *rq, struct task_struct *p, u64 delta_exec);
+void cfs_bandwidth_return_task(struct rq *rq, struct task_struct *p);
+u64 cfs_bandwidth_task_runtime(struct task_struct *p);
+#else
+static inline void cfs_bandwidth_charge_task(struct rq *rq, struct task_struct *p,
+					     u64 delta_exec) { }
+static inline void cfs_bandwidth_return_task(struct rq *rq, struct task_struct *p) { }
+static inline u64 cfs_bandwidth_task_runtime(struct task_struct *p) { return RUNTIME_INF; }
+#endif
+
 /*
  * A weight of 0 or 1 can cause arithmetics problems.
  * A weight of a cfs_rq is the sum of weights of which entities

index 6194484d4d9a9..cfb7f1b70665f 100644
@@ -98,6 +98,7 @@ s32 scx_bpf_pick_any_cpu_node(const cpumask_t *cpus_allowed, int node, u64 flags
 s32 scx_bpf_pick_any_cpu(const cpumask_t *cpus_allowed, u64 flags) __ksym;
 bool scx_bpf_task_running(const struct task_struct *p) __ksym;
 s32 scx_bpf_task_cpu(const struct task_struct *p) __ksym;
+u64 scx_bpf_task_cpu_max_runtime(struct task_struct *p) __ksym __weak;
 struct rq *scx_bpf_cpu_rq(s32 cpu) __ksym __weak;
 struct rq *scx_bpf_locked_rq(void) __ksym;
 struct task_struct *scx_bpf_cpu_curr(s32 cpu) __ksym __weak;

index 07fb55e63c57c..60b9972f1a5dc 100644
@@ -215,6 +215,7 @@ static inline bool __COMPAT_struct_has_field(const char *type, const char *field
 #define SCX_OPS_BUILTIN_IDLE_PER_NODE SCX_OPS_FLAG(SCX_OPS_BUILTIN_IDLE_PER_NODE)
 #define SCX_OPS_ALWAYS_ENQ_IMMED SCX_OPS_FLAG(SCX_OPS_ALWAYS_ENQ_IMMED)
 #define SCX_OPS_ENQ_BLOCKED SCX_OPS_FLAG(SCX_OPS_ENQ_BLOCKED)
+#define SCX_OPS_CHARGE_CPU_MAX SCX_OPS_FLAG(SCX_OPS_CHARGE_CPU_MAX)
 
 #define SCX_PICK_IDLE_FLAG(name) __COMPAT_ENUM_OR_ZERO("scx_pick_idle_cpu_flags", #name)
 

index c5d3a2eaea7da..4a727b8dc7aab 100644
@@ -165,6 +165,7 @@ all_test_bpfprogs := $(foreach prog,$(wildcard *.bpf.c),$(INCLUDE_DIR)/$(patsubs
 
 auto-test-targets :=			\
 	cgroup_nr_cpus			\
+	cgroup_runtime			\
 	create_dsq			\
 	dequeue				\
 	dequeue_iter			\

index 8173f9170ffe8..495b140403a18 100644
@@ -9,3 +9,4 @@ CONFIG_DEBUG_INFO=y
 CONFIG_DEBUG_INFO_BTF=y
 CONFIG_EXPERT=y
 CONFIG_SCHED_PROXY_EXEC=y
+CONFIG_CFS_BANDWIDTH=y

2.47.3




Amazon Development Centre (South Africa) (Proprietary) Limited
29 Gogosoa Street, Observatory, Cape Town, Western Cape, 7925, South Africa
Registration Number: 2004 / 034463 / 07

Hello, Fernand.

I think this patchset is too maximalist. As described, the use case is specific to selling burstable CPU budget. Nobody else has asked for this. There may be other use cases which haven't been stated yet but the description doesn't surface any, so as it stands this is pretty niche.

On top of that, the series integrates into both fair and sched_ext and then makes the two work together so that the handoff is seamless. That's too much for something which doesn't have a strong use case or an established generic need. As with scx_bpf_cgroup_report_throttled(), patch 4 modifies fair's internal state from sched_ext, which I don't think is a good idea as the cgroup piping is split at the interface level:

https://lore.kernel.org/all/[email protected]/

Seamlessly switching tasks between the classes in partial mode has a lot of other integration problems beyond this one and at least I'm not aiming to achieve it. It'd cost a lot more than it'd be useful.

The sched_ext part can just be a feature of your own BPF scheduler, which you seem to be using already. There, the credit level is just state in your BPF maps which your userspace can read and set, so the maximalist version can wait until the need is proven. If you want a handover between your scheduler and fair, that can be done from userspace.

If a burstable budget is something useful to add to fair, can you do that first and separately? It should be able to stand on its own.

Thanks.

--
tejun

Hi Tejun,

Thank you for your prompt feedback.

That works for me as a starting point. I will resend just patches 1 and 2 as v3.

The minimal viable bar for us is to never lose track of CPU credits. With patches 1 and 2, the userspace handover you suggest covers the planned transitions: on roll out and roll back, userspace reads cpu.max.runtime before the switch and writes it back after. The one case it cannot cover is an unplanned exit, a BPF scheduler erroring out or killed by the watchdog and falling back to fair, where there is no window for userspace to hand the level back.

Thanks,
Fernand

Amazon Development Centre (South Africa) (Proprietary) Limited 29 Gogosoa Street, Observatory, Cape Town, Western Cape, 7925, South Africa Registration Number: 2004 / 034463 / 07

This series makes the CFS bandwidth controller's bucket a burstable CPU budget that userspace can save and restore. It lets cpu.max.burst exceed the quota, so the bucket can hold more than one quota, and adds cpu.max.runtime to read and set what it holds (patch 1). It makes a group that is not running keep earning quota up to quota + burst (patch 2).

v2 had two further patches that passed the level on to a BPF scheduler. Tejun asked for the fair side first and on its own, so they are dropped here and will be proposed separately if this is accepted.

Use case
========

We run virtual machines whose CPU time is governed by CPU credits, as burstable instances are. An instance earns credits while it runs below its baseline, spends them above it, and holds at most a capped amount; with none left it is held to the baseline. The starting credits, the baseline and the cap are part of the instance type's guarantee and must hold whatever the host does to the instance.

Each instance's vCPU threads run in their own cgroup, and the controller already is that bucket of credits: the quota per period is the rate credits are earned at, quota + burst is the cap, and the runtime is the level. It does the per-CPU charging and the throttling in the scheduler, at a granularity userspace accounting cannot match. What is missing is a cap above one quota, a way to read and write the level, and earning while idle. An instance that idles for a minute with a 100ms period earns one quota instead of up to 600.

The level has to be saved and restored by the host's control software:
an instance starts with the credits its type grants, not with one quota; a host kernel live update (kexec) or a reboot creates the cgroups anew, so every level has to be read before and written back after, or every instance loses or gains credits; a live migration moves the level with the instance; and the control plane raises or lowers it when credits move between instances. A container checkpoint and restore has the same problem.

Ben pointed out that a cpu.max or cpu.max.burst write already adds a quota to the runtime, and that cpuacct gives the usage. That falls short. A write can only add, and only one quota, so setting a level means writing a temporary quota equal to the difference and then the real quota back, and lowering one means shrinking the burst until the cap cuts the runtime and widening it again; each leaves the cgroup running with the wrong bandwidth in between, and each write visits every CPU's cfs_rq. And usage is not the level: deriving it means replaying every period's refill in userspace, including the periods the timer slept through, which duplicates the controller's accounting.

Patch 2 changes behavior for existing cpu.max.burst users. A group with a burst that becomes runnable after idling has up to quota + burst, as if it had been running below its quota throughout, instead of the runtime it had when the timer stopped. Groups without a burst see no change.

Testing
=======

On tip/sched/core in qemu with lockdep, the cgroup test_cpu selftests pass, 18 of 18, including the nine new ones for these two patches, with no lockdep report in dmesg. Both patches build with W=1 and no warning in kernel/sched, with and without CONFIG_CFS_BANDWIDTH and on 32-bit, and kernel-doc reports nothing on the files they touch.

An AI coding assistant (Claude Code) helped write the selftests and changelogs and ran the build and test matrix; I reviewed all of it.

Changes in v3
=============

- Patches 3 and 4, the sched_ext side, dropped (Tejun).
- Patch 1: cpu.max.runtime reads 0 while cpu.max is "max" instead of a level the group no longer has, and taking a group's quota away drops its runtime instead of keeping it for the next quota (kernel-patches CI); the lockdep_assert_irqs_disabled() in distribute_cfs_runtime() is gone, as both bandwidth timers run with hardirqs enabled on PREEMPT_RT, and a comment states the real precondition; the burst_kept selftest grants the runtime after a period has started, so a hog cannot burn it while the test waits; CONFIG_CFS_BANDWIDTH added to tools/testing/selftests/cgroup/config.
- Patch 2: a cpu.max write starts the next period from the write, so the first refill under a new period is one new period away and not up to one old period away; selftest for it.

v2: https://lore.kernel.org/all/[email protected]/
v1: https://lore.kernel.org/all/[email protected]/

Fernand Sieber (2):
sched/fair: expose cpu.max.runtime to set bandwidth runtime directly sched/fair: refill bandwidth for the periods the timer slept through

 Documentation/admin-guide/cgroup-v2.rst   |  24 +-
 Documentation/scheduler/sched-bwc.rst     |  14 +-
 kernel/sched/core.c                       |  56 ++-
 kernel/sched/fair.c                       | 163 +++++-
 kernel/sched/sched.h                      |   4 +
 tools/testing/selftests/cgroup/config     |   1 +
 tools/testing/selftests/cgroup/test_cpu.c | 587 ++++++++++++++++++++++
 7 files changed, 839 insertions(+), 10 deletions(-)

base-commit: d7e670601af6c564ccf6d3b2ddf70ee4f41a6d1c

2.47.3




Amazon Development Centre (South Africa) (Proprietary) Limited
29 Gogosoa Street, Observatory, Cape Town, Western Cape, 7925, South Africa
Registration Number: 2004 / 034463 / 07

Add cpu.max.runtime, a cgroup v2 file that reads and sets the CFS bandwidth runtime, cfs_b->runtime, in microseconds. Userspace can then grant a cgroup runtime beyond its quota. Refills still add only the quota, capped at quota + burst, so the excess is used up over time and the group returns to its quota per period.

A write is rejected without a quota or above quota + burst, the most the group can accumulate. A read is capped at quota + burst so it can be written back; uncapped slack from __return_cfs_rq_runtime() can exceed that until the next refill. Runtime the CPUs already took, at most a slice each, is not included and a write leaves it in place. Without a quota a read reports 0, and taking a group's quota away drops the runtime it had, rather than keeping it for the next time the group is given a quota.

A write distributes the new runtime to throttled cfs_rqs at once, as the slack timer does, rather than waiting for the unsynchronized period timer. It also shifts runtime_snap by the change, so cpu.stat burst accounting neither counts the write nor hides what the group ran since the last refill.

Also drop the burst <= quota limit and require only quota + burst <= max_bw_runtime_us (about 203 days), the quota's own limit, so a group can accumulate many periods of quota. This applies to cgroup v1 cpu.cfs_burst_us too, which shares tg_set_bandwidth().

The file exists with CONFIG_CFS_BANDWIDTH only.

---
 Documentation/admin-guide/cgroup-v2.rst   |  23 +-
 Documentation/scheduler/sched-bwc.rst     |  13 +-
 kernel/sched/core.c                       |  44 ++-
 kernel/sched/fair.c                       |  65 ++++
 kernel/sched/sched.h                      |   2 +
 tools/testing/selftests/cgroup/config     |   1 +
 tools/testing/selftests/cgroup/test_cpu.c | 356 ++++++++++++++++++++++
 7 files changed, 495 insertions(+), 9 deletions(-)

index 9e3c67b02ae2e..eb802f8ba877f 100644
@@ -1231,12 +1231,33 @@ will be referred to. All time durations are in microseconds.
 	A read-write single value file which exists on non-root
 	cgroups.  The default is "0".
 
-	The burst in the range [0, $MAX].
+	The burst. The cgroup can accumulate the runtime it does not
+	use, up to $MAX + cpu.max.burst. The burst may exceed $MAX, as
+	long as $MAX + cpu.max.burst is at most 2^44 - 1 (about 203
+	days).
 
 	This file affects only processes under the fair-class scheduler and a BPF
 	scheduler with the ``cgroup_set_bandwidth`` callback depending on what
 	the callback actually does.
 
+  cpu.max.runtime
+	A read-write single value file which exists on non-root
+	cgroups if the kernel is built with CONFIG_CFS_BANDWIDTH.
+
+	The runtime the cgroup has accumulated and not used yet, in
+	microseconds. Each period refills it by $MAX, up to $MAX +
+	cpu.max.burst, and a read returns at most $MAX + cpu.max.burst.
+	A write sets it, up to $MAX + cpu.max.burst, and lets a cgroup
+	that ran out of runtime run again at once. A write fails with
+	EINVAL while $MAX is "max", a read then returns 0, and setting
+	$MAX to "max" drops the runtime the cgroup had.
+
+	The runtime does not include what the cgroup's CPUs already took
+	from it, at most a slice each (sched_cfs_bandwidth_slice_us).
+	A write leaves that with the CPUs.
+
+	This file affects only processes under the fair-class scheduler.
+
   cpu.pressure
 	A read-write nested-keyed file.
 

index e881a945c1888..c22b3c8c14e29 100644
@@ -90,8 +90,8 @@ bandwidth restriction in place, such a group is described as an unconstrained
 bandwidth group. This represents the traditional work-conserving behavior for
 CFS.
 
-Writing any (valid) positive value(s) no smaller than cpu.cfs_burst_us will
-enact the specified bandwidth limit. The minimum quota allowed for the quota or
+Writing any (valid) positive value(s) will enact the specified bandwidth
+limit. The minimum quota allowed for the quota or
 period is 1ms. There is also an upper bound on the period length of 1s.
 Additional restrictions exist when bandwidth limits are used in a hierarchical
 fashion, these are explained in more detail below.

@@ -101,9 +101,10 @@ and return the group to an unconstrained state once more.
 
 A value of 0 for cpu.cfs_burst_us indicates that the group can not accumulate
 any unused bandwidth. It makes the traditional bandwidth control behavior for
-CFS unchanged. Writing any (valid) positive value(s) no larger than
-cpu.cfs_quota_us into cpu.cfs_burst_us will enact the cap on unused bandwidth
-accumulation.
+CFS unchanged. Writing any (valid) positive value(s) into cpu.cfs_burst_us
+will enact the cap on unused bandwidth accumulation: the group can accumulate
+up to cpu.cfs_quota_us + cpu.cfs_burst_us. The burst may be larger than the
+quota, as long as their sum is at most 2^44 - 1 microseconds (about 203 days).
 
 Any updates to a group's bandwidth specification will result in it becoming
 unthrottled if it is in a constrained state.

@@ -243,4 +244,4 @@ Examples
 	# echo 50000 > cpu.cfs_period_us /* period = 50ms */
 	# echo 10000 > cpu.cfs_burst_us /* burst = 10ms */
 
-   Larger buffer setting (no larger than quota) allows greater burst capacity.
+   Larger buffer setting allows greater burst capacity.

index a4df59c9c9b60..1f60c84e0dc38 100644
@@ -578,6 +578,8 @@ extern void init_cfs_bandwidth(struct cfs_bandwidth *cfs_b, struct cfs_bandwidth
 
 extern void __refill_cfs_bandwidth_runtime(struct cfs_bandwidth *cfs_b);
 extern void start_cfs_bandwidth(struct cfs_bandwidth *cfs_b);
+u64 cfs_bandwidth_get_runtime(struct cfs_bandwidth *cfs_b);
+int cfs_bandwidth_set_runtime(struct cfs_bandwidth *cfs_b, u64 runtime);
 extern void unthrottle_cfs_rq(struct cfs_rq *cfs_rq);
 extern bool cfs_task_bw_constrained(struct task_struct *p);
 

index 39f979690dd3b..aaf5872c71410 100644
@@ -1,3 +1,4 @@
+CONFIG_CFS_BANDWIDTH=y
 CONFIG_CGROUPS=y
 CONFIG_CGROUP_CPUACCT=y
 CONFIG_CGROUP_FREEZER=y

2.47.3




Amazon Development Centre (South Africa) (Proprietary) Limited
29 Gogosoa Street, Observatory, Cape Town, Western Cape, 7925, South Africa
Registration Number: 2004 / 034463 / 07

cpu.max.burst lets a group accumulate unused quota up to quota + burst. But the period timer stops after a period in which the group took no runtime, and start_cfs_bandwidth() restarts it with hrtimer_forward_now() and discards the periods that passed. A group that sits idle stops accumulating, and one that wakes after idling for a long time has only what it had when the timer stopped.

Count the missed periods instead. While the timer is stopped, its expiry is the first period boundary it missed, so the overrun at restart is the number of refills owed. Add quota for each, capped at quota + burst as __refill_cfs_bandwidth_runtime() caps it.

Without a burst the cap is quota. The timer stops right after a refill that nothing drew from, so the runtime is at quota then, and groups without a burst see no change. Groups with a burst do see a change. A group that has been idle now has up to quota + burst when it runs again. The documentation of cpu.max.burst, cpu.max.runtime and cpu.cfs_burst_us says so.

A change of the bandwidth (cpu.max, cpu.max.burst) applies from that moment, and the group keeps what it earned before it. The periods owed are first refilled at the bandwidth they were earned under, capped at its quota + burst, and then the new bandwidth applies, with its own cap. A group that had no quota earned nothing, so one given a quota after a long time unconstrained does not start full. The next period starts from the change too: a stopped timer's expiry is a boundary of the period the group had, so leaving it would put the first refill under the new bandwidth up to one old period away. A cpu.max.runtime write also refills the periods owed first, so that it replaces them, and only the periods after the write refill.

cpu.max.runtime reports the runtime the group has now, owed refills included, without starting the timer.

The idle periods are not added to cpu.stat's nr_periods, which counts the periods in which the timer ran. A timer that runs late by several periods still refills only once, unlike a stopped one; this patch leaves that as it is.

---
 Documentation/admin-guide/cgroup-v2.rst   |  17 +-
 Documentation/scheduler/sched-bwc.rst     |   5 +-
 kernel/sched/core.c                       |  12 ++
 kernel/sched/fair.c                       | 100 +++++++++-
 kernel/sched/sched.h                      |   2 +
 tools/testing/selftests/cgroup/test_cpu.c | 231 ++++++++++++++++++++++
 6 files changed, 355 insertions(+), 12 deletions(-)

index eb802f8ba877f..12d40b55f2dbb 100644
@@ -1232,9 +1232,9 @@ will be referred to. All time durations are in microseconds.
 	cgroups.  The default is "0".
 
 	The burst. The cgroup can accumulate the runtime it does not
-	use, up to $MAX + cpu.max.burst. The burst may exceed $MAX, as
-	long as $MAX + cpu.max.burst is at most 2^44 - 1 (about 203
-	days).
+	use, up to $MAX + cpu.max.burst, also in the periods in which it
+	does not run. The burst may exceed $MAX, as long as $MAX +
+	cpu.max.burst is at most 2^44 - 1 (about 203 days).
 
 	This file affects only processes under the fair-class scheduler and a BPF
 	scheduler with the ``cgroup_set_bandwidth`` callback depending on what

@@ -1246,11 +1246,12 @@ will be referred to. All time durations are in microseconds.
 
 	The runtime the cgroup has accumulated and not used yet, in
 	microseconds. Each period refills it by $MAX, up to $MAX +
-	cpu.max.burst, and a read returns at most $MAX + cpu.max.burst.
-	A write sets it, up to $MAX + cpu.max.burst, and lets a cgroup
-	that ran out of runtime run again at once. A write fails with
-	EINVAL while $MAX is "max", a read then returns 0, and setting
-	$MAX to "max" drops the runtime the cgroup had.
+	cpu.max.burst, whether the cgroup runs in it or not. A read
+	returns at most $MAX + cpu.max.burst. A write sets it, up to
+	$MAX + cpu.max.burst, and lets a cgroup that ran out of runtime
+	run again at once. A write fails with EINVAL while $MAX is
+	"max", a read then returns 0, and setting $MAX to "max" drops
+	the runtime the cgroup had.
 
 	The runtime does not include what the cgroup's CPUs already took
 	from it, at most a slice each (sched_cfs_bandwidth_slice_us).

index c22b3c8c14e29..ccf7434ca73f0 100644
@@ -103,8 +103,9 @@ A value of 0 for cpu.cfs_burst_us indicates that the group can not accumulate
 any unused bandwidth. It makes the traditional bandwidth control behavior for
 CFS unchanged. Writing any (valid) positive value(s) into cpu.cfs_burst_us
 will enact the cap on unused bandwidth accumulation: the group can accumulate
-up to cpu.cfs_quota_us + cpu.cfs_burst_us. The burst may be larger than the
-quota, as long as their sum is at most 2^44 - 1 microseconds (about 203 days).
+up to cpu.cfs_quota_us + cpu.cfs_burst_us, also over periods in which it does
+not run. The burst may be larger than the quota, as long as their sum is at
+most 2^44 - 1 microseconds (about 203 days).
 
 Any updates to a group's bandwidth specification will result in it becoming
 unthrottled if it is in a constrained state.

index e068d81e1fc5f..265088af6a561 100644
@@ -9856,10 +9856,22 @@ static int tg_set_cfs_bandwidth(struct task_group *tg,
 		cfs_bandwidth_usage_inc();
 
 	scoped_guard (raw_spinlock_irq, &cfs_b->lock) {
+		/*
+		 * The new bandwidth applies from now on. The group keeps what
+		 * it earned while idle under the old one.
+		 */
+		cfs_bandwidth_refill_idle_periods(cfs_b);
+
 		cfs_b->period = ns_to_ktime(period);
 		cfs_b->quota = quota;
 		cfs_b->burst = burst;
 
+		/*
+		 * The next period starts from here too, rather than at the
+		 * boundary left over from the old one.
+		 */
+		cfs_bandwidth_reanchor_period(cfs_b);
+
 		/*
 		 * A group without a quota accumulates no runtime, so drop what
 		 * it had rather than report it and add to it once it has a

index 1f60c84e0dc38..a03ac077a8f49 100644
@@ -580,6 +580,8 @@ extern void __refill_cfs_bandwidth_runtime(struct cfs_bandwidth *cfs_b);
 extern void start_cfs_bandwidth(struct cfs_bandwidth *cfs_b);
 u64 cfs_bandwidth_get_runtime(struct cfs_bandwidth *cfs_b);
 int cfs_bandwidth_set_runtime(struct cfs_bandwidth *cfs_b, u64 runtime);
+void cfs_bandwidth_refill_idle_periods(struct cfs_bandwidth *cfs_b);
+void cfs_bandwidth_reanchor_period(struct cfs_bandwidth *cfs_b);
 extern void unthrottle_cfs_rq(struct cfs_rq *cfs_rq);
 extern bool cfs_task_bw_constrained(struct task_struct *p);
 

2.47.3




Amazon Development Centre (South Africa) (Proprietary) Limited
29 Gogosoa Street, Observatory, Cape Town, Western Cape, 7925, South Africa
Registration Number: 2004 / 034463 / 07