Add utilization clamping support (CGroups API)
Subsystems (4)
v13 cover Add utilization clamping support (CGroups API)
Hi all, this is a respin of:
https://lore.kernel.org/lkml/[email protected]/
which introduces these small changes:
- move common code from cpu_uclamp_{min,max}_write() into cpu_uclamp_write(clamp_id)
- comment rewording: update all RUNNABLE tasks on system default changes only when cgroups are in use.
The series is based on top of today's tip/sched/core:
commit a1dc0446d649 ("sched/core: Silence a warning in sched_init()")
Thanks Michal for your additional review!
Since Tejun seems to be happy regarding cgroup integration:
https://lore.kernel.org/lkml/[email protected]/
and the are no substantial changes in the code functionalities with respect to v12, this series adds Tejun's ACK tag.
Cheers,
Patrick
Series Organization
===================
The full tree is available here:
git://linux-arm.org/linux-pb.git lkml/utilclamp_v13
http://www.linux-arm.org/git?p=linux-pb.git;a=shortlog;h=refs/heads/lkml/utilclamp_v13
Newcomer's Short Abstract
=========================
The Linux scheduler tracks a "utilization" signal for each scheduling entity (SE), e.g. tasks, to know how much CPU time they use. This signal allows the scheduler to know how "big" a task is and, in principle, it can support advanced task placement strategies by selecting the best CPU to run a task. Some of these strategies are represented by the Energy Aware Scheduler [1].
When the schedutil cpufreq governor is in use, the utilization signal allows the Linux scheduler to also drive frequency selection. The CPU utilization signal, which represents the aggregated utilization of tasks scheduled on that CPU, is used to select the frequency which best fits the workload generated by the tasks.
The current translation of utilization values into a frequency selection is
simple: we go to max for RT tasks or to the minimum frequency which can accommodate the utilization of DL+FAIR tasks.
However, utilization values by themselves cannot convey the desired power/performance behaviors of each task as intended by user-space. As such they are not ideally suited for task placement decisions.
Task placement and frequency selection policies in the kernel can be improved by taking into consideration hints coming from authorized user-space elements, like for example the Android middleware or more generally any "System Management Software" (SMS) framework.
Utilization clamping is a mechanism which allows to "clamp" (i.e. filter) the utilization generated by RT and FAIR tasks within a range defined by user-space. The clamped utilization value can then be used, for example, to enforce a minimum and/or maximum frequency depending on which tasks are active on a CPU.
The main use-cases for utilization clamping are:
- boosting: better interactive response for small tasks which are affecting the user experience.
Consider for example the case of a small control thread for an external accelerator (e.g. GPU, DSP, other devices). Here, from the task utilization the scheduler does not have a complete view of what the task's requirements are and, if it's a small utilization task, it keeps selecting a more energy efficient CPU, with smaller capacity and lower frequency, thus negatively impacting the overall time required to complete task activations.
- capping: increase energy efficiency for background tasks not affecting the user experience.
Since running on a lower capacity CPU at a lower frequency is more energy efficient, when the completion time is not a main goal, then capping the utilization considered for certain (maybe big) tasks can have positive effects, both on energy consumption and thermal headroom. This feature allows also to make RT tasks more energy friendly on mobile systems where running them on high capacity CPUs and at the maximum frequency is not required.
From these two use-cases, it's worth noticing that frequency selection biasing, introduced by patches 9 and 10 of this series, is just one possible usage of utilization clamping. Another compelling extension of utilization clamping is in helping the scheduler in making tasks placement decisions.
Utilization is (also) a task specific property the scheduler uses to know how much CPU bandwidth a task requires, at least as long as there is idle time. Thus, the utilization clamp values, defined either per-task or per-task_group, can represent tasks to the scheduler as being bigger (or smaller) than what they actually are.
Utilization clamping thus enables interesting additional optimizations, for example on asymmetric capacity systems like Arm big.LITTLE and DynamIQ CPUs, where:
- boosting: try to run small/foreground tasks on higher-capacity CPUs to complete them faster despite being less energy efficient.
- capping: try to run big/background tasks on low-capacity CPUs to save power and thermal headroom for more important tasks
This series does not present this additional usage of utilization clamping but it's an integral part of the EAS feature set, where [2] is one of its main components.
Android kernels use SchedTune, a solution similar to utilization clamping, to bias both 'frequency selection' and 'task placement'. This series provides the foundation to add similar features to mainline while focusing, for the time being, just on schedutil integration.
References
==========
[1] Energy Aware Scheduling
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/scheduler/sched-energy.txt?h=v5.1
[2] Expressing per-task/per-cgroup performance hints Linux Plumbers Conference 2018
https://linuxplumbersconf.org/event/2/contributions/128/
Patrick Bellasi (6):
sched/core: uclamp: Extend CPU's cgroup controller sched/core: uclamp: Propagate parent clamps sched/core: uclamp: Propagate system defaults to root group sched/core: uclamp: Use TG's clamps to restrict TASK's clamps sched/core: uclamp: Update CPU's refcount on TG's clamp changes sched/core: uclamp: always use enum uclamp_id for clamp_id values
Documentation/admin-guide/cgroup-v2.rst | 34 +++ init/Kconfig | 22 ++ kernel/sched/core.c | 375 ++++++++++++++++++++++-- kernel/sched/sched.h | 12 +- 4 files changed, 421 insertions(+), 22 deletions(-)
Signature · 1 line
2.22.0
PATCH v13 1/6 sched/core: uclamp: Extend CPU's cgroup controller
The cgroup CPU bandwidth controller allows to assign a specified (maximum) bandwidth to the tasks of a group. However this bandwidth is defined and enforced only on a temporal base, without considering the actual frequency a CPU is running on. Thus, the amount of computation completed by a task within an allocated bandwidth can be very different depending on the actual frequency the CPU is running that task. The amount of computation can be affected also by the specific CPU a task is running on, especially when running on asymmetric capacity systems like Arm's big.LITTLE.
With the availability of schedutil, the scheduler is now able to drive frequency selections based on actual task utilization. Moreover, the utilization clamping support provides a mechanism to bias the frequency selection operated by schedutil depending on constraints assigned to the tasks currently RUNNABLE on a CPU.
Giving the mechanisms described above, it is now possible to extend the cpu controller to specify the minimum (or maximum) utilization which should be considered for tasks RUNNABLE on a cpu.
This makes it possible to better defined the actual computational power assigned to task groups, thus improving the cgroup CPU bandwidth controller which is currently based just on time constraints.
Extend the CPU controller with a couple of new attributes uclamp.{min,max} which allow to enforce utilization boosting and capping for all the tasks in a group.
Specifically:
- uclamp.min: defines the minimum utilization which should be considered i.e. the RUNNABLE tasks of this group will run at least at a minimum frequency which corresponds to the uclamp.min utilization
- uclamp.max: defines the maximum utilization which should be considered i.e. the RUNNABLE tasks of this group will run up to a maximum frequency which corresponds to the uclamp.max utilization
These attributes:
a) are available only for non-root nodes, both on default and legacy hierarchies, while system wide clamps are defined by a generic interface which does not depends on cgroups. This system wide interface enforces constraints on tasks in the root node.
b) enforce effective constraints at each level of the hierarchy which are a restriction of the group requests considering its parent's effective constraints. Root group effective constraints are defined by the system wide interface.
This mechanism allows each (non-root) level of the hierarchy to:
- request whatever clamp values it would like to get
- effectively get only up to the maximum amount allowed by its parent
c) have higher priority than task-specific clamps, defined via sched_setattr(), thus allowing to control and restrict task requests.
Add two new attributes to the cpu controller to collect "requested" clamp values. Allow that at each non-root level of the hierarchy. Validate local consistency by enforcing uclamp.min < uclamp.max. Keep it simple by not caring now about "effective" values computation and propagation along the hierarchy.
---
Changes in v13:
Message-ID: <[email protected]>
- move common code from cpu_uclamp_{min,max}_write() into cpu_uclamp_write(clamp_id)
--- Documentation/admin-guide/cgroup-v2.rst | 34 +++++ init/Kconfig | 22 ++++ kernel/sched/core.c | 167 +++++++++++++++++++++++- kernel/sched/sched.h | 8 ++ 4 files changed, 230 insertions(+), 1 deletion(-)
Documentation/admin-guide/cgroup-v2.rst +34 -0
@@ -951,6 +951,13 @@ controller implements weight and absolute bandwidth limit models for normal scheduling policy and absolute bandwidth allocation model for realtime scheduling policy. +In all the above models, cycles distribution is defined only on a temporal +base and it does not account for the frequency at which tasks are executed. +The (optional) utilization clamping support allows to hint the schedutil +cpufreq governor about the minimum desired frequency which should always be +provided by a CPU, as well as the maximum desired frequency, which should not +be exceeded by a CPU. + WARNING: cgroup2 doesn't yet support control of realtime processes and the cpu controller can only be enabled when all RT processes are in the root cgroup. Be aware that system management software may already
@@ -1016,6 +1023,33 @@ All time durations are in microseconds. Shows pressure stall information for CPU. See Documentation/accounting/psi.rst for details. + cpu.uclamp.min + A read-write single value file which exists on non-root cgroups. + The default is "0", i.e. no utilization boosting. + + The requested minimum utilization (protection) as a percentage + rational number, e.g. 12.34 for 12.34%. + + This interface allows reading and setting minimum utilization clamp + values similar to the sched_setattr(2). This minimum utilization + value is used to clamp the task specific minimum utilization clamp. + + The requested minimum utilization (protection) is always capped by + the current value for the maximum utilization (limit), i.e. + `cpu.uclamp.max`. + + cpu.uclamp.max + A read-write single value file which exists on non-root cgroups. + The default is "max". i.e. no utilization capping + + The requested maximum utilization (limit) as a percentage rational + number, e.g. 98.76 for 98.76%. + + This interface allows reading and setting maximum utilization clamp + values similar to the sched_setattr(2). This maximum utilization + value is used to clamp the task specific maximum utilization clamp. + + Memory ------
init/Kconfig +22 -0
@@ -928,6 +928,28 @@ config RT_GROUP_SCHED endif #CGROUP_SCHED +config UCLAMP_TASK_GROUP + bool "Utilization clamping per group of tasks" + depends on CGROUP_SCHED + depends on UCLAMP_TASK + default n + help + This feature enables the scheduler to track the clamped utilization + of each CPU based on RUNNABLE tasks currently scheduled on that CPU. + + When this option is enabled, the user can specify a min and max + CPU bandwidth which is allowed for each single task in a group. + The max bandwidth allows to clamp the maximum frequency a task + can use, while the min bandwidth allows to define a minimum + frequency a task will always use. + + When task group based utilization clamping is enabled, an eventually + specified task-specific clamp value is constrained by the cgroup + specified clamp value. Both minimum and maximum task clamping cannot + be bigger than the corresponding clamping defined at task group level. + + If in doubt, say N. + config CGROUP_PIDS bool "PIDs controller" help
kernel/sched/core.c +166 -1
204 lines not on this page · raw message
kernel/sched/sched.h +8 -0
@@ -393,6 +393,14 @@ struct task_group {#endif struct cfs_bandwidth cfs_bandwidth; + +#ifdef CONFIG_UCLAMP_TASK_GROUP +/* The two decimal precision [%] value requested from user-space */ +unsigned int uclamp_pct[UCLAMP_CNT ]; +/* Clamp values requested for a task group */ +struct uclamp_se uclamp_req[UCLAMP_CNT ]; +#endif + };#ifdef CONFIG_FAIR_GROUP_SCHED
Signature · 1 line
2.22.0
Michal Koutný
maintainerRe: [PATCH v13 1/6] sched/core: uclamp: Extend CPU's cgroup controller
kernel/sched/core.c static void cpu_cgroup_attach(struct cgroup_taskset *tset) · line 7098
Quoted 12 lines · Patrick Bellasi
kernel/sched/core.c +12
-0+static ssize_t cpu_uclamp_write (struct kernfs_open_file *of,char *buf, +size_t nbytes,loff_t off, +enum uclamp_id clamp_id) +{ +struct uclamp_request req; +struct task_group *tg; + + req =capacity_from_percent (buf); +if (req.ret) +return req.ret; + +rcu_read_lock ();
This should be the uclamp_mutex.
(The compound results of the series is correct as the lock is introduced in "sched/core: uclamp: Propagate parent clamps".
This is just for the happiness of cherry-pickers/bisectors.)
kernel/sched/core.c static void cpu_cgroup_attach(struct cgroup_taskset *tset) · line 7140
Quoted 8 lines
kernel/sched/core.c +7
-0+static inline void cpu_uclamp_print (struct seq_file *sf, +enum uclamp_id clamp_id) +{ +rcu_read_lock (); + tg =css_tg (seq_css (sf)); + util_clamp = tg->uclamp_req[clamp_id].value; +rcu_read_unlock ();
Why is the rcu_read_lock() needed here? (I'm considering the comment in of_css() that should apply here (and it seems that similar uses in other seq_file handlers also skip this).)
Review static struct cftype cpu_legacy_files[] = {
Quoted 5 lines
(quoted diff) +2
-0@@ -7369,6 +7506,20 @@ static struct cftype cpu_legacy_files[] = { + .name = "uclamp.min", + .name = "uclamp.max",
I don't see technical reasons why uclamp couldn't work on legacy hierarchy and Tejun acked the series, despite that I'll ask -- should the new attributes be exposed in v1 controller hierarchy (taking into account the legacy API is sort of frozen and potential maintenance needs spanning both hierarchies)?
Re: [PATCH v13 1/6] sched/core: uclamp: Extend CPU's cgroup controller
On Tue, Aug 06, 2019 at 17:11:34 +0100, Michal Koutný wrote...
Quoted 15 lines · Patrick Bellasi
kernel/sched/core.c static void cpu_cgroup_attach(struct cgroup_taskset *tset) · line 7098
Quoted 12 lines
kernel/sched/core.c +12
-0+static ssize_t cpu_uclamp_write (struct kernfs_open_file *of,char *buf, +size_t nbytes,loff_t off, +enum uclamp_id clamp_id) +{ +struct uclamp_request req; +struct task_group *tg; + + req =capacity_from_percent (buf); +if (req.ret) +return req.ret; + +rcu_read_lock ();This should be the
uclamp_mutex.
Quoted 3 lines
(The compound results of the series is correct as the lock is introduced in "sched/core: uclamp: Propagate parent clamps".
This is just for the happiness of cherry-pickers/bisectors.)
Right, will move the uclamp_mutex introduction in this patch instead of in the following one.
Quoted 11 lines
kernel/sched/core.c static void cpu_cgroup_attach(struct cgroup_taskset *tset) · line 7140
Quoted 8 lines
kernel/sched/core.c +7
-0+static inline void cpu_uclamp_print (struct seq_file *sf, +enum uclamp_id clamp_id) +{ +rcu_read_lock (); + tg =css_tg (seq_css (sf)); + util_clamp = tg->uclamp_req[clamp_id].value; +rcu_read_unlock ();Why is the
rcu_read_lock()needed here? (I'm considering the comment inof_css()that should apply here (and it seems that similar uses in otherseq_filehandlers also skip this).)
So, looks like that since we are in the context of a file operation, all the cgroup's attribute read/write functions are implicitly save.
IOW, we don't need an RCU lock since the TG data structures are granted to be always available till the end of the read/write operation.
That seems to make sense... I'm wondering if keeping the RCU look is still a precaution for possible future code/assumption changes or just an unnecessary overhead?
Quoted 10 lines
Review static struct cftype cpu_legacy_files[] = {
Quoted 5 lines
(quoted diff) +2
-0@@ -7369,6 +7506,20 @@ static struct cftype cpu_legacy_files[] = { + .name = "uclamp.min", + .name = "uclamp.max",I don't see technical reasons why uclamp couldn't work on legacy hierarchy and Tejun acked the series, despite that I'll ask -- should the new attributes be exposed in v1 controller hierarchy (taking into account the legacy API is sort of frozen and potential maintenance needs spanning both hierarchies)?
Not sure to get what you mean here: I'm currently exposing uclamp to both v1 and v2 hierarchies.
Best,
Patrick
--
#include <best/regards.h>
Patrick Bellasi
Michal Koutný
maintainerRe: [PATCH v13 1/6] sched/core: uclamp: Extend CPU's cgroup controller
Quoted 2 lines · Patrick Bellasi
Not sure to get what you mean here: I'm currently exposing uclamp to both v1 and v2 hierarchies.
cpu controller has different API for v1 and v2 hierarchies. My question reworded is -- are the new knobs exposed in the legacy API intentionally/for a reason?
Michal
PATCH v13 2/6 sched/core: uclamp: Propagate parent clamps
In reply to Patrick BellasiIn order to properly support hierarchical resources control, the cgroup delegation model requires that attribute writes from a child group never fail but still are locally consistent and constrained based on parent's assigned resources. This requires to properly propagate and aggregate parent attributes down to its descendants.
Implement this mechanism by adding a new "effective" clamp value for each task group. The effective clamp value is defined as the smaller value between the clamp value of a group and the effective clamp value of its parent. This is the actual clamp value enforced on tasks in a task group.
Since it's possible for a cpu.uclamp.min value to be bigger than the cpu.uclamp.max value, ensure local consistency by restricting each "protection" (i.e. min utilization) with the corresponding "limit" (i.e. max utilization).
Do that at effective clamps propagation to ensure all user-space write never fails while still always tracking the most restrictive values.
Update sysctl_sched_uclamp_handler() to use the newly introduced uclamp_mutex so that we serialize system default updates with cgroup relate updates.
--- kernel/sched/core.c | 65 ++++++++++++++++++++++++++++++++++++++++++-- kernel/sched/sched.h | 2 ++ 2 files changed, 64 insertions(+), 3 deletions(-)
kernel/sched/core.c +62 -3
@@ -773,6 +773,18 @@ static void set_load_weight(struct task_struct *p, bool update_load) }#ifdef CONFIG_UCLAMP_TASK +/* +* Serializes updates of utilization clamp values +* +* The (slow-path) user-space triggers utilization clamp value updates which +* can require updates on (fast-path) scheduler's data structures used to +* support enqueue/dequeue operations. +* While the per-CPU rq lock protects fast-path update operations, user-space +* requests are serialized using a mutex to reduce the risk of conflicting +* updates or API abuses. +*/ +static DEFINE_MUTEX (uclamp_mutex); +/* Max allowed minimum utilization */ unsigned int sysctl_sched_uclamp_util_min =SCHED_CAPACITY_SCALE ;
@@ -1010,10 +1022,9 @@ int sysctl_sched_uclamp_handler(struct ctl_table *table, int write, loff_t *ppos) {int old_min, old_max; -static DEFINE_MUTEX (mutex);int result; -mutex_lock (&mutex); +mutex_lock (&uclamp_mutex); old_min = sysctl_sched_uclamp_util_min; old_max = sysctl_sched_uclamp_util_max;
@@ -1048,7 +1059,7 @@ int sysctl_sched_uclamp_handler(struct ctl_table *table, int write, sysctl_sched_uclamp_util_min = old_min; sysctl_sched_uclamp_util_max = old_max; done: -mutex_unlock (&mutex); +mutex_unlock (&uclamp_mutex);return result; }
@@ -1137,6 +1148,8 @@ static void __init init_uclamp(void)unsigned int clamp_id;int cpu; +mutex_init (&uclamp_mutex); +for_each_possible_cpu (cpu) {memset (&cpu_rq (cpu)->uclamp,0 ,sizeof (struct uclamp_rq ));cpu_rq (cpu)->uclamp_flags =0 ;
@@ -1153,6 +1166,7 @@ static void __init init_uclamp(void) uclamp_default[clamp_id] = uc_max;#ifdef CONFIG_UCLAMP_TASK_GROUP root_task_group.uclamp_req[clamp_id] = uc_max; + root_task_group.uclamp[clamp_id] = uc_max;#endif } }
@@ -6799,6 +6813,7 @@ static inline void alloc_uclamp_sched_group(struct task_group *tg,for_each_clamp_id (clamp_id) {uclamp_se_set (&tg->uclamp_req[clamp_id],uclamp_none (clamp_id), false); + tg->uclamp[clamp_id] = parent->uclamp[clamp_id]; }#endif }
@@ -7045,6 +7060,45 @@ static void cpu_cgroup_attach(struct cgroup_taskset *tset) }#ifdef CONFIG_UCLAMP_TASK_GROUP +static void cpu_util_update_eff (struct cgroup_subsys_state *css) +{ +struct cgroup_subsys_state *top_css = css; +struct uclamp_se *uc_parent =NULL ; +struct uclamp_se *uc_se =NULL ; +unsigned int eff[UCLAMP_CNT ]; +unsigned int clamp_id; +unsigned int clamps; + +css_for_each_descendant_pre (css, top_css) { + uc_parent =css_tg (css)->parent + ?css_tg (css)->parent->uclamp :NULL ; + +for_each_clamp_id (clamp_id) { +/* Assume effective clamps matches requested clamps */ + eff[clamp_id] =css_tg (css)->uclamp_req[clamp_id].value; +/* Cap effective clamps with parent's effective clamps */ +if (uc_parent && + eff[clamp_id] > uc_parent[clamp_id].value) { + eff[clamp_id] = uc_parent[clamp_id].value; + } + } +/* Ensure protection is always capped by limit */ + eff[UCLAMP_MIN ] =min (eff[UCLAMP_MIN ], eff[UCLAMP_MAX ]); + +/* Propagate most restrictive effective clamps */ + clamps =0x0 ; + uc_se =css_tg (css)->uclamp; +for_each_clamp_id (clamp_id) { +if (eff[clamp_id] == uc_se[clamp_id].value) +continue ; + uc_se[clamp_id].value = eff[clamp_id]; + uc_se[clamp_id].bucket_id =uclamp_bucket_id (eff[clamp_id]); + clamps |= (0x1 << clamp_id); + } +if (!clamps) + css =css_rightmost_descendant (css); + } +}#define _POW10 (exp) ((unsigned int)1e##exp)#define POW10 (exp) _POW10(exp)
@@ -7095,6 +7149,7 @@ static ssize_t cpu_uclamp_write(struct kernfs_open_file *of, char *buf,if (req.ret)return req.ret; +mutex_lock (&uclamp_mutex);rcu_read_lock (); tg =css_tg (of_css (of));
@@ -7107,7 +7162,11 @@ static ssize_t cpu_uclamp_write(struct kernfs_open_file *of, char *buf,*/ tg->uclamp_pct[clamp_id] = req.percent; +/* Update effective clamps to track the most restrictive value */ +cpu_util_update_eff (of_css (of)); +rcu_read_unlock (); +mutex_unlock (&uclamp_mutex);return nbytes; }
kernel/sched/sched.h +2 -0
@@ -399,6 +399,8 @@ struct task_group {unsigned int uclamp_pct[UCLAMP_CNT ];/* Clamp values requested for a task group */ struct uclamp_se uclamp_req[UCLAMP_CNT ]; +/* Effective clamp values used for a task group */ +struct uclamp_se uclamp[UCLAMP_CNT ];#endif };
Signature · 1 line
2.22.0
Michal Koutný
maintainerRe: [PATCH v13 2/6] sched/core: uclamp: Propagate parent clamps
Quoted 9 lines · Patrick Bellasi
kernel/sched/core.c +1
-0@@ -7095,6 +7149,7 @@ static ssize_t cpu_uclamp_write(struct kernfs_open_file *of, char *buf,if (req.ret)return req.ret; +mutex_lock (&uclamp_mutex);rcu_read_lock (); tg =css_tg (of_css (of));
kernel/sched/core.c static ssize_t cpu_uclamp_write(struct kernfs_open_file *of, char *buf, · line 7169
Quoted 8 lines
kernel/sched/core.c +4
-0@@ -7107,7 +7162,11 @@ static ssize_t cpu_uclamp_write(struct kernfs_open_file *of, char *buf,*/ tg->uclamp_pct[clamp_id] = req.percent; +/* Update effective clamps to track the most restrictive value */ +cpu_util_update_eff (of_css (of)); +rcu_read_unlock (); +mutex_unlock (&uclamp_mutex);
Following my remarks to "[PATCH v13 1/6] sched/core: uclamp: Extend CPU's cgroup", I wonder if the rcu_read_lock() couldn't be moved right before cpu_util_update_eff(). And by extension rcu_read_(un)lock could be hidden into cpu_util_update_eff() closer to its actual need.
Re: [PATCH v13 2/6] sched/core: uclamp: Propagate parent clamps
On Tue, Aug 06, 2019 at 17:11:53 +0100, Michal Koutný wrote...
Quoted 13 lines · Patrick Bellasi
Quoted 12 lines
kernel/sched/core.c +1
-0@@ -7095,6 +7149,7 @@ static ssize_t cpu_uclamp_write(struct kernfs_open_file *of, char *buf,if (req.ret)return req.ret; +mutex_lock (&uclamp_mutex);rcu_read_lock (); tg =css_tg (of_css (of));@@ -7107,7 +7162,11 @@ static ssize_t cpu_uclamp_write(struct kernfs_open_file *of, char *buf,*/ tg->uclamp_pct[clamp_id] = req.percent;
Quoted 9 lines
kernel/sched/core.c static ssize_t cpu_uclamp_write(struct kernfs_open_file *of, char *buf, · line 7169
Quoted 5 lines
kernel/sched/core.c +4
-0+/* Update effective clamps to track the most restrictive value */ +cpu_util_update_eff (of_css (of)); +rcu_read_unlock (); +mutex_unlock (&uclamp_mutex);Following my remarks to "[PATCH v13 1/6] sched/core: uclamp: Extend CPU's cgroup", I wonder if the
rcu_read_lock()couldn't be moved right beforecpu_util_update_eff(). And by extension rcu_read_(un)lock could be hidden intocpu_util_update_eff()closer to its actual need.
Well, if I've got correctly your comment in the previous message, I would say that at this stage we don't need RCU looks at all.
Reason being that cpu_util_update_eff() gets called only from cpu_uclamp_write() which is from an ongoing write operation on a cgroup attribute and thus granted to be available.
We will eventually need to move the RCU look only down the stack when uclamp_update_active_tasks() gets called to update the RUNNABLE tasks on a RQ... or perhaps we don't need them since we already get the task_rq_lock() for each task we visit.
Is that correct?
Cheers,
Patrick
Signature · 3 lines
#include <best/regards.h> Patrick Bellasi
Michal Koutný
maintainerRe: [PATCH v13 2/6] sched/core: uclamp: Propagate parent clamps
Quoted 2 lines · Patrick Bellasi
Well, if I've got correctly your comment in the previous message, I would say that at this stage we don't need RCU looks at all.
Agreed.
Quoted 8 lines
Reason being that
cpu_util_update_eff()gets called only fromcpu_uclamp_write()which is from an ongoing write operation on a cgroup attribute and thus granted to be available.We will eventually need to move the RCU look only down the stack when
uclamp_update_active_tasks()gets called to update the RUNNABLE tasks on a RQ... or perhaps we don't need them since we already get thetask_rq_lock()for each task we visit.
Unless you remove css_for_each_descendant_pre() in cpu_util_update_eff(), the rcu_read_lock() cannot go below it. (You'd be RCU-accessing other csses that aren't pinned in the write.)
Michal
PATCH v13 3/6 sched/core: uclamp: Propagate system defaults to root group
In reply to Patrick BellasiThe clamp values are not tunable at the level of the root task group. That's for two main reasons:
- the root group represents "system resources" which are always entirely available from the cgroup standpoint.
- when tuning/restricting "system resources" makes sense, tuning must be done using a system wide API which should also be available when control groups are not.
When a system wide restriction is available, cgroups should be aware of its value in order to know exactly how much "system resources" are available for the subgroups.
Utilization clamping supports already the concepts of:
- system defaults: which define the maximum possible clamp values usable by tasks.
- effective clamps: which allows a parent cgroup to constraint (maybe temporarily) its descendants without losing the information related to the values "requested" from them.
Exploit these two concepts and bind them together in such a way that, whenever system default are tuned, the new values are propagated to (possibly) restrict or relax the "effective" value of nested cgroups.
When cgroups are in use, force an update of all the RUNNABLE tasks. Otherwise, keep things simple and do just a lazy update next time each task will be enqueued.
Do that since we assume a more strict resource control is required when cgroups are in use. This allows also to keep "effective" clamp values updated in case we need to expose them to user-space.
---
Changes in v13:
Message-ID: <[email protected]>
- comment rewording: update all RUNNABLE tasks on system default changes only when cgroups are in use.
--- kernel/sched/core.c | 31 +++++++++++++++++++++++++++++-- 1 file changed, 29 insertions(+), 2 deletions(-)
kernel/sched/core.c +29 -2
@@ -1017,10 +1017,30 @@ static inline void uclamp_rq_dec(struct rq *rq, struct task_struct *p)uclamp_rq_dec_id (rq, p, clamp_id); } +#ifdef CONFIG_UCLAMP_TASK_GROUP +static void cpu_util_update_eff (struct cgroup_subsys_state *css); +static void uclamp_update_root_tg (void ) +{ +struct task_group *tg = &root_task_group; + +uclamp_se_set (&tg->uclamp_req[UCLAMP_MIN ], + sysctl_sched_uclamp_util_min, false); +uclamp_se_set (&tg->uclamp_req[UCLAMP_MAX ], + sysctl_sched_uclamp_util_max, false); + +rcu_read_lock (); +cpu_util_update_eff (&root_task_group.css); +rcu_read_unlock (); +} +#else +static void uclamp_update_root_tg (void ) { } +#endif +int sysctl_sched_uclamp_handler (struct ctl_table *table,int write,void __user *buffer,size_t *lenp,loff_t *ppos) { +bool update_root_tg = false;int old_min, old_max;int result;
@@ -1043,16 +1063,23 @@ int sysctl_sched_uclamp_handler(struct ctl_table *table, int write,if (old_min != sysctl_sched_uclamp_util_min) {uclamp_se_set (&uclamp_default[UCLAMP_MIN ], sysctl_sched_uclamp_util_min, false); + update_root_tg = true; }if (old_max != sysctl_sched_uclamp_util_max) {uclamp_se_set (&uclamp_default[UCLAMP_MAX ], sysctl_sched_uclamp_util_max, false); + update_root_tg = true; } +if (update_root_tg) +uclamp_update_root_tg (); +/* -* Updating all the RUNNABLE task is expensive, keep it simple and do -* just a lazy update at each next enqueue time. +* We update all RUNNABLE tasks only when task groups are in use. +* Otherwise, keep it simple and do just a lazy update at each next +* task enqueue time. */ + goto done; undo:
Signature · 1 line
2.22.0
PATCH v13 4/6 sched/core: uclamp: Use TG's clamps to restrict TASK's clamps
In reply to Patrick BellasiWhen a task specific clamp value is configured via sched_setattr(2), this value is accounted in the corresponding clamp bucket every time the task is {en,de}qeued. However, when cgroups are also in use, the task specific clamp values could be restricted by the task_group (TG) clamp values.
Update uclamp_cpu_inc() to aggregate task and TG clamp values. Every time a task is enqueued, it's accounted in the clamp bucket tracking the smaller clamp between the task specific value and its TG effective value. This allows to:
1. ensure cgroup clamps are always used to restrict task specific requests, i.e. boosted not more than its TG effective protection and capped at least as its TG effective limit.
2. implement a "nice-like" policy, where tasks are still allowed to request less than what enforced by their TG effective limits and protections
Do this by exploiting the concept of "effective" clamp, which is already used by a TG to track parent enforced restrictions.
Apply task group clamp restrictions only to tasks belonging to a child group. While, for tasks in the root group or in an autogroup, system defaults are still enforced.
--- kernel/sched/core.c | 28 +++++++++++++++++++++++++++- 1 file changed, 27 insertions(+), 1 deletion(-)
kernel/sched/core.c +27 -1
@@ -873,16 +873,42 @@ unsigned int uclamp_rq_max_value(struct rq *rq, unsigned int clamp_id,return uclamp_idle_value (rq, clamp_id, clamp_value); } +static inline struct uclamp_se +uclamp_tg_restrict (struct task_struct *p,unsigned int clamp_id) +{ +struct uclamp_se uc_req = p->uclamp_req[clamp_id]; +#ifdef CONFIG_UCLAMP_TASK_GROUP +struct uclamp_se uc_max; + +/* +* Tasks in autogroups or root task group will be +* restricted by system defaults. +*/ +if (task_group_is_autogroup (task_group (p))) +return uc_req; +if (task_group (p) == &root_task_group) +return uc_req; + + uc_max =task_group (p)->uclamp[clamp_id]; +if (uc_req.value > uc_max.value || !uc_req.user_defined) +return uc_max; +#endif + +return uc_req; +} +/* * The effective clamp bucket index of a task depends on, by increasing * priority: * - the task specific clamp value, when explicitly requested from userspace +* - the task group effective clamp value, for tasks not either in the root +* group or in an autogroup * - the system default clamp value, defined by the sysadmin */ static inline struct uclamp_se uclamp_eff_get (struct task_struct *p,unsigned int clamp_id) { -struct uclamp_se uc_req = p->uclamp_req[clamp_id]; +struct uclamp_se uc_req =uclamp_tg_restrict (p, clamp_id);struct uclamp_se uc_max = uclamp_default[clamp_id];/* System default restrictions always apply */
Signature · 1 line
2.22.0
PATCH v13 5/6 sched/core: uclamp: Update CPU's refcount on TG's clamp changes
In reply to Patrick BellasiOn updates of task group (TG) clamp values, ensure that these new values are enforced on all RUNNABLE tasks of the task group, i.e. all RUNNABLE tasks are immediately boosted and/or capped as requested.
Do that each time we update effective clamps from cpu_util_update_eff(). Use the *cgroup_subsys_state (css) to walk the list of tasks in each affected TG and update their RUNNABLE tasks.
Update each task by using the same mechanism used for cpu affinity masks updates, i.e. by taking the rq lock.
--- kernel/sched/core.c | 58 ++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 57 insertions(+), 1 deletion(-)
kernel/sched/core.c +57 -1
@@ -1043,6 +1043,57 @@ static inline void uclamp_rq_dec(struct rq *rq, struct task_struct *p)uclamp_rq_dec_id (rq, p, clamp_id); } +static inline void +uclamp_update_active (struct task_struct *p,unsigned int clamp_id) +{ +struct rq_flags rf; +struct rq *rq; + +/* +* Lock the task and the rq where the task is (or was) queued. +* +* We might lock the (previous) rq of a !RUNNABLE task, but that's the +* price to pay to safely serialize util_{min,max} updates with +* enqueues, dequeues and migration operations. +* This is the same locking schema used by __set_cpus_allowed_ptr(). +*/ + rq =task_rq_lock (p, &rf); + +/* +* Setting the clamp bucket is serialized by task_rq_lock(). +* If the task is not yet RUNNABLE and its task_struct is not +* affecting a valid clamp bucket, the next time it's enqueued, +* it will already see the updated clamp bucket value. +*/ +if (!p->uclamp[clamp_id].active) + goto done; + +uclamp_rq_dec_id (rq, p, clamp_id); +uclamp_rq_inc_id (rq, p, clamp_id); + +done: + +task_rq_unlock (rq, p, &rf); +} + +static inline void +uclamp_update_active_tasks (struct cgroup_subsys_state *css, +unsigned int clamps) +{ +struct css_task_iter it; +struct task_struct *p; +unsigned int clamp_id; + +css_task_iter_start (css,0 , &it); +while ((p =css_task_iter_next (&it))) { +for_each_clamp_id (clamp_id) { +if ((0x1 << clamp_id) & clamps) +uclamp_update_active (p, clamp_id); + } + } +css_task_iter_end (&it); +} +#ifdef CONFIG_UCLAMP_TASK_GROUP static void cpu_util_update_eff (struct cgroup_subsys_state *css);static void uclamp_update_root_tg (void )
@@ -7148,8 +7199,13 @@ static void cpu_util_update_eff(struct cgroup_subsys_state *css) uc_se[clamp_id].bucket_id =uclamp_bucket_id (eff[clamp_id]); clamps |= (0x1 << clamp_id); } -if (!clamps) +if (!clamps) { css =css_rightmost_descendant (css); +continue ; + } + +/* Immediately update descendants RUNNABLE tasks */ +uclamp_update_active_tasks (css, clamps); } }
Signature · 1 line
2.22.0
PATCH v13 6/6 sched/core: uclamp: always use enum uclamp_id for clamp_id values
In reply to Patrick BellasiThe supported clamp indexes are defined in enum clamp_id however, because of the code logic in some of the first utilization clamping series version, sometimes we needed to use unsigned int to represent indexes.
This is not more required since the final version of the uclamp_* APIs can always use the proper enum uclamp_id type.
Fix it with a bulk rename now that we have all the bits merged.
--- kernel/sched/core.c | 38 +++++++++++++++++++------------------- kernel/sched/sched.h | 2 +- 2 files changed, 20 insertions(+), 20 deletions(-)
kernel/sched/core.c +19 -19
@@ -810,7 +810,7 @@ static inline unsigned int uclamp_bucket_base_value(unsigned int clamp_value)return UCLAMP_BUCKET_DELTA *uclamp_bucket_id (clamp_value); } -static inline unsigned int uclamp_none (int clamp_id) +static inline enum uclamp_id uclamp_none (enum uclamp_id clamp_id) {if (clamp_id ==UCLAMP_MIN )return 0 ;
@@ -826,7 +826,7 @@ static inline void uclamp_se_set(struct uclamp_se *uc_se, }static inline unsigned int -uclamp_idle_value (struct rq *rq,unsigned int clamp_id, +uclamp_idle_value (struct rq *rq,enum uclamp_id clamp_id,unsigned int clamp_value) {/*
@@ -842,7 +842,7 @@ uclamp_idle_value(struct rq *rq, unsigned int clamp_id,return uclamp_none (UCLAMP_MIN ); } -static inline void uclamp_idle_reset (struct rq *rq,unsigned int clamp_id, +static inline void uclamp_idle_reset (struct rq *rq,enum uclamp_id clamp_id,unsigned int clamp_value) {/* Reset max-clamp retention only on idle exit */
@@ -853,8 +853,8 @@ static inline void uclamp_idle_reset(struct rq *rq, unsigned int clamp_id, }static inline -unsigned int uclamp_rq_max_value (struct rq *rq,unsigned int clamp_id, -unsigned int clamp_value) +enum uclamp_id uclamp_rq_max_value (struct rq *rq,enum uclamp_id clamp_id, +unsigned int clamp_value) {struct uclamp_bucket *bucket = rq->uclamp[clamp_id].bucket;int bucket_id =UCLAMP_BUCKETS -1 ;
@@ -874,7 +874,7 @@ unsigned int uclamp_rq_max_value(struct rq *rq, unsigned int clamp_id, }static inline struct uclamp_se -uclamp_tg_restrict (struct task_struct *p,unsigned int clamp_id) +uclamp_tg_restrict (struct task_struct *p,enum uclamp_id clamp_id) {struct uclamp_se uc_req = p->uclamp_req[clamp_id];#ifdef CONFIG_UCLAMP_TASK_GROUP
@@ -906,7 +906,7 @@ uclamp_tg_restrict(struct task_struct *p, unsigned int clamp_id)* - the system default clamp value, defined by the sysadmin */ static inline struct uclamp_se -uclamp_eff_get (struct task_struct *p,unsigned int clamp_id) +uclamp_eff_get (struct task_struct *p,enum uclamp_id clamp_id) {struct uclamp_se uc_req =uclamp_tg_restrict (p, clamp_id);struct uclamp_se uc_max = uclamp_default[clamp_id];
@@ -918,7 +918,7 @@ uclamp_eff_get(struct task_struct *p, unsigned int clamp_id)return uc_req; } -unsigned int uclamp_eff_value (struct task_struct *p,unsigned int clamp_id) +enum uclamp_id uclamp_eff_value (struct task_struct *p,enum uclamp_id clamp_id) {struct uclamp_se uc_eff;
@@ -942,7 +942,7 @@ unsigned int uclamp_eff_value(struct task_struct *p, unsigned int clamp_id)* for each bucket when all its RUNNABLE tasks require the same clamp. */ static inline void uclamp_rq_inc_id (struct rq *rq,struct task_struct *p, -unsigned int clamp_id) +enum uclamp_id clamp_id) {struct uclamp_rq *uc_rq = &rq->uclamp[clamp_id];struct uclamp_se *uc_se = &p->uclamp[clamp_id];
@@ -980,7 +980,7 @@ static inline void uclamp_rq_inc_id(struct rq *rq, struct task_struct *p,* enforce the expected state and warn. */ static inline void uclamp_rq_dec_id (struct rq *rq,struct task_struct *p, -unsigned int clamp_id) +enum uclamp_id clamp_id) {struct uclamp_rq *uc_rq = &rq->uclamp[clamp_id];struct uclamp_se *uc_se = &p->uclamp[clamp_id];
@@ -1019,7 +1019,7 @@ static inline void uclamp_rq_dec_id(struct rq *rq, struct task_struct *p,static inline void uclamp_rq_inc (struct rq *rq,struct task_struct *p) { -unsigned int clamp_id; +enum uclamp_id clamp_id;if (unlikely (!p->sched_class->uclamp_enabled))return ;
@@ -1034,7 +1034,7 @@ static inline void uclamp_rq_inc(struct rq *rq, struct task_struct *p)static inline void uclamp_rq_dec (struct rq *rq,struct task_struct *p) { -unsigned int clamp_id; +enum uclamp_id clamp_id;if (unlikely (!p->sched_class->uclamp_enabled))return ;
@@ -1044,7 +1044,7 @@ static inline void uclamp_rq_dec(struct rq *rq, struct task_struct *p) }static inline void -uclamp_update_active (struct task_struct *p,unsigned int clamp_id) +uclamp_update_active (struct task_struct *p,enum uclamp_id clamp_id) {struct rq_flags rf;struct rq *rq;
@@ -1080,9 +1080,9 @@ static inline void uclamp_update_active_tasks(struct cgroup_subsys_state *css, unsigned int clamps) { +enum uclamp_id clamp_id;struct css_task_iter it;struct task_struct *p; -unsigned int clamp_id;css_task_iter_start (css,0 , &it);while ((p =css_task_iter_next (&it))) {
@@ -1190,7 +1190,7 @@ static int uclamp_validate(struct task_struct *p,static void __setscheduler_uclamp (struct task_struct *p,const struct sched_attr *attr) { -unsigned int clamp_id; +enum uclamp_id clamp_id;/* * On scheduling class change, reset to default clamps for tasks
@@ -1227,7 +1227,7 @@ static void __setscheduler_uclamp(struct task_struct *p,static void uclamp_fork (struct task_struct *p) { -unsigned int clamp_id; +enum uclamp_id clamp_id;for_each_clamp_id (clamp_id) p->uclamp[clamp_id].active = false;
@@ -1249,7 +1249,7 @@ static void uclamp_fork(struct task_struct *p)static void __init init_uclamp (void ) {struct uclamp_se uc_max = {}; -unsigned int clamp_id; +enum uclamp_id clamp_id;int cpu;mutex_init (&uclamp_mutex);
@@ -6912,7 +6912,7 @@ static inline void alloc_uclamp_sched_group(struct task_group *tg, struct task_group *parent) {#ifdef CONFIG_UCLAMP_TASK_GROUP -int clamp_id; +enum uclamp_id clamp_id;for_each_clamp_id (clamp_id) { uclamp_se_set(&tg->uclamp_req[clamp_id],
@@ -7170,7 +7170,7 @@ static void cpu_util_update_eff(struct cgroup_subsys_state *css)struct uclamp_se *uc_parent =NULL ;struct uclamp_se *uc_se =NULL ;unsigned int eff[UCLAMP_CNT ]; -unsigned int clamp_id; +enum uclamp_id clamp_id;unsigned int clamps;css_for_each_descendant_pre (css, top_css) {
kernel/sched/sched.h +1 -1
@@ -2276,7 +2276,7 @@ static inline void cpufreq_update_util(struct rq *rq, unsigned int flags) {}#endif /* CONFIG_CPU_FREQ */ #ifdef CONFIG_UCLAMP_TASK -unsigned int uclamp_eff_value (struct task_struct *p,unsigned int clamp_id); +enum uclamp_id uclamp_eff_value (struct task_struct *p,enum uclamp_id clamp_id); static __always_inline unsigned int uclamp_util_with(struct rq *rq, unsigned int util,
Signature · 1 line
2.22.0
Michal Koutný
maintainerQuoted 7 lines · Patrick Bellasi
Patrick Bellasi (6):
sched/core: uclamp: Extend CPU's cgroup controller sched/core: uclamp: Propagate parent clamps sched/core: uclamp: Propagate system defaults to root group sched/core: uclamp: Use TG's clamps to restrict TASK's clamps sched/core: uclamp: Update CPU's refcount on TG's clamp changes sched/core: uclamp: always use enumuclamp_idforclamp_idvalues
Thank you Patrick for your patience. I used the time to revisit the series once again and I think the RCU locks can be streamlined a bit. If you find that correct, feel free to add my Reviewed-by to the updated series (for 1/6 and legacy, I'm just asking).
Michal
On Tue, Aug 06, 2019 at 17:12:06 +0100, Michal Koutný wrote...
Quoted 8 lines · Patrick Bellasi
Quoted 7 lines
Patrick Bellasi (6):
sched/core: uclamp: Extend CPU's cgroup controller sched/core: uclamp: Propagate parent clamps sched/core: uclamp: Propagate system defaults to root group sched/core: uclamp: Use TG's clamps to restrict TASK's clamps sched/core: uclamp: Update CPU's refcount on TG's clamp changes sched/core: uclamp: always use enumuclamp_idforclamp_idvalues
Hi Michal!
Quoted 1 line
Thank you Patrick for your patience.
Thanks to you for your reviews.
Quoted 2 lines
I used the time to revisit the series once again and I think the RCU locks can be streamlined a bit.
I'll have a look at those, thanks!
Quoted 2 lines
If you find that correct, feel free to add my Reviewed-by to the updated series (for 1/6 and legacy, I'm just asking).
Sure, actually sorry for not having already added that tag in the current version, it will be there in v14 ;)
Quoted 1 line
Michal
Cheers,
Patrick
--
#include <best/regards.h>
Patrick Bellasi