docs/sched_ext: document that cgroup CPU knobs are scheduler-dependent
Subsystems (3)
PATCH v3 1/1 docs/sched_ext: document that cgroup CPU knobs are scheduler-dependent
From: Tao Cui <[email protected]>
The fair class enforces cpu controller knobs such as cpu.max, cpu.weight and cpu.idle in the kernel. sched_ext only passes them to the BPF scheduler through the ops.cgroup_set_*() callbacks. Whether and how a knob takes effect is up to the loaded scheduler: it may implement the corresponding callback partially or not at all. The same applies to other knobs like nice levels.
Document this in the basics section so users and container orchestrators know what to expect from a BPF scheduler.
---
v2 -> v3: Drop the scheduler list and the nr_throttled example, keep the section concise and generic, per review.
v2: https://lore.kernel.org/r/[email protected]
Documentation/scheduler/sched-ext.rst | 12 ++++++++++++ 1 file changed, 12 insertions(+)
Documentation/scheduler/sched-ext.rst +12 -0
@@ -242,6 +242,18 @@ optional. The following modified excerpt is from .name = "simple", }; +Scheduler-Dependent Knobs +------------------------- + +The fair class enforces cpu controller knobs such as ``cpu.max``, +``cpu.weight`` and ``cpu.idle`` in the kernel. sched_ext only passes +them to the BPF scheduler through ``ops.cgroup_set_weight()``, +``ops.cgroup_set_idle()``, ``ops.cgroup_set_bandwidth()`` and friends. +Whether and how a knob takes effect is up to the loaded scheduler: it +may implement the corresponding callback partially or not at all. The +same applies to other knobs like nice levels. When relying on these +knobs, check the documentation or source of the loaded scheduler. + Dispatch Queues ---------------
Signature · 1 line
2.43.0
Andrea Righi
reviewerHi Tao,
Quoted 27 lines · Tao Cui
From: Tao Cui <[email protected]>
The fair class enforces cpu controller knobs such as cpu.max, cpu.weight and cpu.idle in the kernel.
sched_extonly passes them to the BPF scheduler through the ops.cgroup_set_*() callbacks. Whether and how a knob takes effect is up to the loaded scheduler: it may implement the corresponding callback partially or not at all. The same applies to other knobs like nice levels.Document this in the basics section so users and container orchestrators know what to expect from a BPF scheduler.
Signed-off-by Tao Cui <[email protected]>---
v2 -> v3: Drop the scheduler list and thenr_throttledexample, keep the section concise and generic, per review.v2: https://lore.kernel.org/r/[email protected]
Documentation/scheduler/sched-ext.rst | 12 ++++++++++++ 1 file changed, 12 insertions(+)Documentation/scheduler/sched-ext.rst +0
-0
Documentation/scheduler/sched-ext.rst optional. The following modified excerpt is from · line 251
Quoted 10 lines
Documentation/scheduler/sched-ext.rst +7
-0@@ -242,6 +242,18 @@ optional. The following modified excerpt is from .name = "simple", }; +Scheduler-Dependent Knobs +------------------------- + +The fair class enforces cpu controller knobs such as ``cpu.max``, +``cpu.weight`` and ``cpu.idle`` in the kernel. sched_ext only passes +them to the BPF scheduler through ``ops.cgroup_set_weight()``, +``ops.cgroup_set_idle()``, ``ops.cgroup_set_bandwidth()`` and friends.
Existing cpu.weight and bandwidth values are delivered via ops.cgroup_init(), ops.cgroup_set_*() callbacks handle later changes.
Moreover, cpu.idle state is not included in scx_cgroup_init_args, so apparently BPF schedulers don't receive an initial value (only subsequent writes). This should be probably fixed separately.
Documentation/scheduler/sched-ext.rst optional. The following modified excerpt is from · line 256
Quoted 5 lines
Documentation/scheduler/sched-ext.rst +5
-0+Whether and how a knob takes effect is up to the loaded scheduler: it +may implement the corresponding callback partially or not at all. The +same applies to other knobs like nice levels. When relying on these +knobs, check the documentation or source of the loaded scheduler. +
"nice levels" can be a bit ambiguous, per-task nice changes are converted to weights and reported through ops.set_weight(), writes to the cgroup file cpu.weight.nice are reported through ops.cgroup_set_weight().
Maybe we can rephrase the whole paragraph as following, or something along these lines:
The fair-class scheduler enforces CPU controller settings such as cpu.max, cpu.weight, and cpu.idle. For sched_ext tasks, the scheduler core communicates these settings to the BPF scheduler through ops.cgroup_init() and reports subsequent changes through the corresponding ops.cgroup_set_*() callbacks. Similarly, per-task nice changes are converted to weights and reported through ops.set_weight().
Each BPF scheduler is responsible for implementing the scheduling semantics of these settings and may choose to ignore them. Consult the loaded scheduler's documentation before relying on these controls.
Thanks,
-Andrea
Hello, Andrea,
Quoted 68 lines · Andrea Righi
Hi Tao,
Quoted 27 lines · Tao Cui
From: Tao Cui <[email protected]>
The fair class enforces cpu controller knobs such as cpu.max, cpu.weight and cpu.idle in the kernel.
sched_extonly passes them to the BPF scheduler through the ops.cgroup_set_*() callbacks. Whether and how a knob takes effect is up to the loaded scheduler: it may implement the corresponding callback partially or not at all. The same applies to other knobs like nice levels.Document this in the basics section so users and container orchestrators know what to expect from a BPF scheduler.
Signed-off-by Tao Cui <[email protected]>---
v2 -> v3: Drop the scheduler list and thenr_throttledexample, keep the section concise and generic, per review.v2: https://lore.kernel.org/r/[email protected]
Documentation/scheduler/sched-ext.rst | 12 ++++++++++++ 1 file changed, 12 insertions(+)Documentation/scheduler/sched-ext.rst +0
-0Documentation/scheduler/sched-ext.rst optional. The following modified excerpt is from · line 251
Quoted 10 lines
Documentation/scheduler/sched-ext.rst +7
-0@@ -242,6 +242,18 @@ optional. The following modified excerpt is from .name = "simple", }; +Scheduler-Dependent Knobs +------------------------- + +The fair class enforces cpu controller knobs such as ``cpu.max``, +``cpu.weight`` and ``cpu.idle`` in the kernel. sched_ext only passes +them to the BPF scheduler through ``ops.cgroup_set_weight()``, +``ops.cgroup_set_idle()``, ``ops.cgroup_set_bandwidth()`` and friends.Existing cpu.weight and bandwidth values are delivered via
ops.cgroup_init(), ops.cgroup_set_*() callbacks handle later changes.Moreover, cpu.idle state is not included in
scx_cgroup_init_args, so apparently BPF schedulers don't receive an initial value (only subsequent writes). This should be probably fixed separately.Documentation/scheduler/sched-ext.rst optional. The following modified excerpt is from · line 256
Quoted 5 lines
Documentation/scheduler/sched-ext.rst +5
-0+Whether and how a knob takes effect is up to the loaded scheduler: it +may implement the corresponding callback partially or not at all. The +same applies to other knobs like nice levels. When relying on these +knobs, check the documentation or source of the loaded scheduler. +"nice levels" can be a bit ambiguous, per-task nice changes are converted to weights and reported through
ops.set_weight(), writes to the cgroup file cpu.weight.nice are reported throughops.cgroup_set_weight().Maybe we can rephrase the whole paragraph as following, or something along these lines:
The fair-class scheduler enforces CPU controller settings such as cpu.max, cpu.weight, and cpu.idle. For
sched_exttasks, the scheduler core communicates these settings to the BPF scheduler throughops.cgroup_init()and reports subsequent changes through the corresponding ops.cgroup_set_*() callbacks. Similarly, per-task nice changes are converted to weights and reported throughops.set_weight().
Quoted 4 lines
Each BPF scheduler is responsible for implementing the scheduling semantics of these settings and may choose to ignore them. Consult the loaded scheduler's documentation before relying on these controls.
Thanks for the review. v4 adopts your wording: ops.cgroup_init() for the initial values, ops.cgroup_set_*() only for subsequent changes, and per-task nice disambiguated from cpu.weight.nice via ops.set_weight().
On the missing cpu.idle initial value in scx_cgroup_init_args: I can look into a separate fix for that.
Thanks,
Tao
Quoted 2 lines
Thanks,
-Andrea