Skip to contentSkip to conversation

Add utilization clamping support

PATCH Superseded Patrick Bellasi 3 participants 50 messages Oct 29, 2018 – Nov 22, 2018

Subsystems (6)

v5 of 14 Superseded Latest is v14 →

lore
Share & tools
lore
#0 lore raw

v5 cover Add utilization clamping support

Hi all, this is a respin of:

https://lore.kernel.org/lkml/[email protected]/

which has been rebased on v4.19.

Thanks for all the valuable comments collected so far!

This version will be presented and discussed at the upcoming LPC. Meanwhile, any comments and feedbacks are more than welcome!

Cheers Patrick

Main changes in v5
==================

.:: Better split core bits from task groups support
---------------------------------------------------

As per Tejun request:

https://lore.kernel.org/lkml/[email protected]/

we now have _all and only_ the core scheduler bits at the beginning of the series, the first 10 patches of this series provide:
- per task utilization clamping API,
via sched_setattr
- system default clamps for both FAIR and RT tasks,

    via /proc/sys/kernel/sched_uclamp_util_{min,max}

- schedutil integration for both FAIR and RT tasks

cgroups v1 and v2 support comes as an extension of the CPU controller in the last 5 patches of this series. These bits are kept together with the core scheduler ones to give a better view on the overall solution we are proposing. Moreover, it helps to ensure we have core data structure and concepts which properly fits for cgroups usage too.

.:: Spinlock removal in favour of atomic operations
---------------------------------------------------

As suggested by Peter:

https://lore.kernel.org/lkml/[email protected]/

the main data structures, i.e. clamp maps and clamp groups, have been re-defined as bitfields mainly to:
a) compress those data structures to use less memory
b) avoid usage of spinlocks in favor of atomic operations

As an additional bonus, some spare bits can now be dedicated to track and flag special conditions which was previously encoded in a more confusing way. The final code looks (hopefully) much more clean and easy to read.

.:: Improve consistency and enforce invariant conditions
--------------------------------------------------------

We now ensure that every scheduling entity has always a valid clamp group and value assigned. This made it possible to remove the previously confusing usage of the UCLAMP_NOT_VALID=-1 special value and different checks here and there.

Data type for clamp groups and values are consistently defined and always used as "unsigned", while spare bits are used whenever a special condition still need to be tracked.

.:: Use of bucketization since the beginning
--------------------------------------------

In the previous version we added a couple of patches to deal with the limited number of clamp groups. Those patches introduced the idea of using buckets for the per-CPU clamp groups used to refcount RUNNABLE tasks. However, as pointed out by Peter:

https://lore.kernel.org/lkml/[email protected]/

the previous implementation was misleading and it also introduced some checks not really required, e.g. a privileged API to request a clamp which should always be possible.

Since the fundamental idea of bucketization still seems to be sound and acceptable, those bits have been used from the start of this patch-set. This made it possible to simplify the overall series, thanks to the removal of code previously required just to deal with the limited number of clamp groups.

The new implementation should now implement what Peter proposed above, specifically:

a) we don't need to search anymore for all the required groups before actually refcounting them. A clamp group is now granted to be always available for each possible requested clamp value.

b) the userspace APIs to set scheduling entity specific clamp values are no more privileged. Userspace can always ask for a clamp value and the system will always assign it to the most appropriate "effective" clamp group which matches with all the task group or system default clamps constraints.

.:: Series Organization
-----------------------

The series is organized into these main sections:

- Patches [01-08]: Per task (primary) API
- Patches [09-10]: Schedutil integration for CFS and RT tasks
- Patches [11-15]: Per task group (secondary) API

Newcomer's Short Abstract (Updated)
===================================

The Linux scheduler tracks a "utilization" signal for each scheduling entity (SE), e.g. tasks, to know how much CPU time they use. This signal allows the scheduler to know how "big" a task is and, in principle, it can support advanced task placement strategies by selecting the best CPU to run a task. Some of these strategies are represented by the Energy Aware Scheduler [1].

When the schedutil cpufreq governor is in use, the utilization signal allows the Linux scheduler also to drive frequency selection. The CPU utilization signal, which represents the aggregated utilization of tasks scheduled on that CPU, is used to select the frequency which best fits the workload generated by the tasks.

However, the current translation of utilization values into a frequency selection is pretty simple: we just go to max for RT tasks or to the minimum frequency which can accommodate the utilization of DL+FAIR tasks. Instead, utilization is of limited usage for tasks placement since its value alone is not enough to properly describe what the _expected_ power/performance behaviors of each task really is from a userspace standpoint.

In general, for both RT and FAIR tasks we can aim at better tasks placement and frequency selection policies if we take hints coming from user-space into consideration.

Utilization clamping is a mechanism which allows to "clamp" (i.e. filter) the utilization generated by RT and FAIR tasks within a range defined from user-space. The clamped utilization value can then be used, for example, to enforce a minimum and/or maximum frequency depending on which tasks are currently active on a CPU.

The main use-cases for utilization clamping are:

- boosting: better interactive response for small tasks which are affecting the user experience.

Consider for example the case of a small control thread for an external accelerator (e.g. GPU, DSP, other devices). In this case, from the task utilization the scheduler does not have a complete view of what the task requirements are and, if it's a small utilization task, it keep selecting a more energy efficient CPU, with smaller capacity and lower frequency, thus affecting the overall time required to complete task activations.

- capping: increase energy efficiency for background tasks not directly affecting the user experience.

Since running on a lower capacity CPU at a lower frequency is in general more energy efficient, when the completion time is not a main goal, then capping the utilization considered for certain (maybe big) tasks can have positive effects, both on energy consumption and thermal headroom. Moreover, this feature allows also to make RT tasks more energy friendly on mobile systems, where running them on high capacity CPUs and at the maximum frequency is not strictly required.

Quoted 1 line

From these two use-cases, it's worth to notice that frequency selection

biasing, introduced by patches 9 and 10 of this series, is just one possible usage of utilization clamping. Another compelling extension of utilization clamping is in helping the scheduler on tasks placement decisions.

Utilization is (also) a task specific property which is used by the scheduler to know how much CPU bandwidth a task requires, at least as long as there is idle time.
Thus, the utilization clamp values, defined either per-task or per-taskgroup, can be used to represent tasks to the scheduler as being bigger (or smaller) than what they really are.

Utilization clamping thus ultimately enables interesting additional optimizations, especially on asymmetric capacity systems like Arm big.LITTLE and DynamIQ CPUs, where:

- boosting: small/foreground tasks are preferably scheduled on higher-capacity CPUs where, despite being less energy efficient, they are expected to complete faster.

- capping: big/background tasks are preferably scheduled on low-capacity CPUs where, being more energy efficient, they can still run but save power and thermal headroom for more important tasks.

This additional usage of utilization clamping is not presented in this series but it's an integral part of the EAS feature set, where [1] is one of its main components. A solution similar to utilization clamping, namely SchedTune, is already used on Android kernels to bias both 'frequency selection' and 'task placement'.

This series provides the foundation bits to add similar features to mainline while focusing, for the time being, just on schedutil integration.

[1] https://lore.kernel.org/lkml/[email protected]/

Detailed Changelog
==================

Changes in v5:
Message-ID: <[email protected]>
- use bitfields and atomic_long_cmpxchg() operations to both compress the clamp maps and avoid usage of spinlock.
- remove enforced __cacheline_aligned_in_smp on uclamp_map since it's accessed from the slow path only and we don't care about performance
- better describe the usage of uclamp_map::se_lock
Message-ID: <[email protected]>
- remove inline from uclamp_group_{get,put}() and __setscheduler_uclamp()
- set lower/upper bounds at the beginning of __setscheduler_uclamp()
- avoid usage of pr_err from unprivileged syscall paths
in  __setscheduler_uclamp(), replaced by ratelimited version
Message-ID: <20180914134128.GP1413@e110439-lin>
- remove/limit usage of UCLAMP_NOT_VALID whenever not strictly required
Message-ID: <[email protected]>
- allow sched_setattr() syscall to sleep on mutex
- fix return value for successfull uclamp syscalls
Message-ID: <CAJuCfpF36-VZm0JVVNnOnGm-ukVejzxbPhH33X3z9gAQ06t9gQ@mail.gmail.com>
- reorder conditions in uclamp_group_find() loop
- use uc_se->xxx in uclamp_fork()
Message-ID: <20180914134128.GP1413@e110439-lin>
- remove not required check for (group_id == UCLAMP_NOT_VALID)
in uclamp_cpu_put_id
- remove not required uclamp_task_affects() since now all tasks always have a valid clamp group assigned
Message-ID: <20180912174456.GJ1413@e110439-lin>
- use bitfields to compress uclamp_group
Message-ID: <[email protected]>
- added patch 02/15 which allows to change clamp values without affecting current policy
Message-ID: <[email protected]>
- add a comment to justify the assumptions on util clamping for FAIR tasks
Message-ID: <[email protected]>
- removed uclamp_value and use inline access to data structures
Message-ID: <20180914135712.GQ1413@e110439-lin>
- the (unlikely(val == UCLAMP_NOT_VALID)) check is not more required since we now ensure we always have a valid value configured
Message-ID: <20180912125133.GE1413@e110439-lin>
- make more clear the definition of cpu.util.min.effective
- small typos fixed
Others:
- renamed uclamp_round into uclamp_group_value to better represent what this function returns
- reduce usage of alias local variables whenever the global ones can still be used without affecting code readability
- consistently use "unsigned int" for both clamp_id and group_id
- fixup documentation
- reduced usage of inline comments
- use UCLAMP_GROUPS to track (CONFIG_UCLAMP_GROUPS_COUNT+1)
- rebased on v4.19

Changes in v4:
Message-ID: <20180809152313.lewfhufidhxb2qrk@darkstar>
- implements the idea discussed in this thread
Message-ID: <[email protected]>
- remove not required default setting
- fixed some tabs/spaces
Message-ID: <[email protected]>
- replace/rephrase "bandwidth" references to use "capacity"
- better stress that this do not enforce any bandwidth requirement but "just" give hints to the scheduler
- fixed some typos
Message-ID: <[email protected]>
- add uclamp_exit_task() to release clamp refcount from do_exit()
Message-ID: <20180816133249.GA2964@e110439-lin>
- keep the WARN but beautify a bit that code
- keep the WARN in uclamp_cpu_put_id() but beautify a bit that code
- add another WARN on the unexpected condition of releasing a refcount from a CPU which has a lower clamp value active
Message-ID: <[email protected]>
- move uclamp_enabled at the top of sched_class to keep it on the same cache line of other main wakeup time callbacks
Message-ID: <20180816132249.GA2960@e110439-lin>
- inline uclamp_task_active() code into uclamp_task_update_active()
- get rid of the now unused uclamp_task_active()
Message-ID: <20180816172016.GG2960@e110439-lin>
- ensure to always reset clamp holding on wakeup from IDLE
Message-ID: <CAKfTPtC2adLupg7wy1JU9zxKx1466Sza6fSCcr92wcawm1OYkg@mail.gmail.com>
- use *rq instead of cpu for both uclamp_util() and uclamp_value()
Message-ID: <20180816135300.GC2960@e110439-lin>
- remove uclamp_value() which is never used outside CONFIG_UCLAMP_TASK
Message-ID: <20180816140731.GD2960@e110439-lin>
- add ".effective" attributes to the default hierarchy
- reuse already existing:
task_struct::uclamp::effective::group_id
instead of adding:
task_struct::uclamp_group_id
to back annotate the effective clamp group in which a task has been refcounted
Message-ID: <20180820122728.GM2960@e110439-lin>
- fix unwanted reset of clamp values on refcount success
Other:
- by default all tasks have a UCLAMP_NOT_VALID task specific clamp
- always use:

      p->uclamp[clamp_id].effective.value
   to track the actual clamp value the task has been refcounted into.
   This matches with the usage of
      p->uclamp[clamp_id].effective.group_id

- allow to call uclamp_group_get() without a task pointer, which is used to refcount the initial clamp group for all the global objects (init_task, root_task_group and system_defaults)
- ensure (and check) that all tasks have a valid group_id at uclamp_cpu_get_id()
- rework uclamp_cpu layout to better fit into just 2x64B cache lines
- fix some s/SCHED_DEBUG/CONFIG_SCHED_DEBUG/
- init uclamp for the init_task and refcount its clamp groups
- add uclamp specific fork time code into uclamp_fork
- add support for SCHED_FLAG_RESET_ON_FORK
default clamps are now set for init_task and inherited/reset at fork time (when then flag is set for the parent)
- enable uclamp only for FAIR tasks, RT class will be enabled only by a following patch which also integrate the class to schedutil
- define uclamp_maps ____cacheline_aligned_in_smp
- in uclamp_group_get() ensure to include uclamp_group_available() and uclamp_group_init() into the atomic section defined by:
uc_map[next_group_id].se_lock
- do not use mutex_lock(&uclamp_mutex) in uclamp_exit_task which is also not needed since refcounting is already guarded by the uc_map[group_id].se_lock spinlock
- consolidate init_uclamp_sched_group() into init_uclamp()
- refcount root_task_group's clamp groups from init_uclamp()
- small documentation fixes
- rebased on v4.19-rc1

Changes in v3:
Message-ID: <CAJuCfpF6=L=0LrmNnJrTNPazT4dWKqNv+thhN0dwpKCgUzs9sg@mail.gmail.com>
- removed UCLAMP_NONE not used by this patch
- remove not necessary checks in uclamp_group_find()
- add WARN on unlikely un-referenced decrement in uclamp_group_put()
- add WARN on unlikely un-referenced decrement in uclamp_cpu_put_id()
- make __setscheduler_uclamp() able to set just one clamp value
- make __setscheduler_uclamp() failing if both clamps are required but there is no clamp groups available for one of them
- remove uclamp_group_find() from uclamp_group_get() which now takes a group_id as a parameter
- add explicit calls to uclamp_group_find()
which is now not more part of uclamp_group_get()
- fixed a not required override
- fixed some typos in comments and changelog
Message-ID: <CAJuCfpGaKvxKcO=RLcmveHRB9qbMrvFs2yFVrk=k-v_m7JkxwQ@mail.gmail.com>
- few typos fixed
Message-ID: <[email protected]>
- use "." notation for attributes naming
i.e. s/util_{min,max}/util.{min,max}/
- added new patches: 09 and 12
Other changes:
- rebased on tip/sched/core

Changes in v2:
Message-ID: <[email protected]>
- refactored struct rq::uclamp_cpu to be more cache efficient no more holes, re-arranged vectors to match cache lines with expected data locality
Message-ID: <[email protected]>
- use *rq as parameter whenever already available
- add scheduling class's uclamp_enabled marker
- get rid of the "confusing" single callback uclamp_task_update() and use uclamp_cpu_{get,put}() directly from {en,de}queue_task()
- fix/remove "bad" comments
Message-ID: <20180413113337.GU14248@e110439-lin>
- remove inline from init_uclamp, flag it __init
Message-ID: <[email protected]>
- get rid of the group_id back annotation
which is not requires at this stage where we have only per-task clamping support. It will be introduce later when cgroup support is added.
Message-ID: <[email protected]>
- make attributes available only on non-root nodes
a system wide API seems of not immediate interest and thus it's not supported anymore
- remove implicit parent-child constraints and dependencies
Message-ID: <[email protected]>
- add some cgroup-v2 documentation for the new attributes
- (hopefully) better explain intended use-cases
the changelog above has been extended to better justify the naming proposed by the new attributes
Other changes:
- improved documentation to make more explicit some concepts
- set UCLAMP_GROUPS_COUNT=2 by default
which allows to fit all the hot-path CPU clamps data into a single cache line while still supporting up to 2 different {min,max}_utiql clamps.
- use -ERANGE as range violation error
- add attributes to the default hierarchy as well as the legacy one
- implement a "nice" semantics where cgroup clamp values are always used to restrict task specific clamp values,
i.e. tasks running on a TG are only allowed to demote themself.
- patches re-ordering in top-down way
- rebased on v4.18-rc4

Patrick Bellasi (15):
sched/core: uclamp: extend sched_setattr to support utilization
clamping
sched/core: make sched_setattr able to tune the current policy sched/core: uclamp: map TASK's clamp values into CPU's clamp groups sched/core: uclamp: add CPU's clamp groups refcounting sched/core: uclamp: update CPU's refcount on clamp changes sched/core: uclamp: enforce last task UCLAMP_MAX sched/core: uclamp: add clamp group bucketing support sched/core: uclamp: add system default clamps sched/cpufreq: uclamp: add utilization clamping for FAIR tasks sched/cpufreq: uclamp: add utilization clamping for RT tasks sched/core: uclamp: extend CPU's cgroup controller sched/core: uclamp: propagate parent clamps
sched/core: uclamp: map TG's clamp values into CPU's clamp groups sched/core: uclamp: use TG's clamps to restrict TASK's clamps sched/core: uclamp: update CPU's refcount on TG's clamp changes

 Documentation/admin-guide/cgroup-v2.rst |   46 +
 include/linux/sched.h                   |   78 ++
 include/linux/sched/sysctl.h            |   11 +
 include/linux/sched/task.h              |    6 +
 include/linux/sched/topology.h          |    6 -
 include/uapi/linux/sched.h              |   11 +-
 include/uapi/linux/sched/types.h        |   67 +-
 init/Kconfig                            |   63 ++
 init/init_task.c                        |    1 +
 kernel/exit.c                           |    1 +
 kernel/sched/core.c                     | 1112 ++++++++++++++++++++++-
 kernel/sched/cpufreq_schedutil.c        |   31 +-
 kernel/sched/fair.c                     |    4 +
 kernel/sched/features.h                 |    5 +
 kernel/sched/rt.c                       |    4 +
 kernel/sched/sched.h                    |  121 ++-
 kernel/sysctl.c                         |   16 +
 17 files changed, 1553 insertions(+), 30 deletions(-)
Signature · 1 line
2.18.0
#1 lore raw

PATCH v5 1/15 sched/core: uclamp: extend sched_setattr to support utilization clamping

The SCHED_DEADLINE scheduling class provides an advanced and formal model to define tasks requirements which can be translated into proper decisions for both task placements and frequencies selections. Other classes have a more simplified model which is essentially based on the relatively simple concept of POSIX priorities.

Such a simple priority based model however does not allow to exploit some of the most advanced features of the Linux scheduler like, for example, driving frequencies selection via the schedutil cpufreq governor. However, also for non SCHED_DEADLINE tasks, it's still interesting to define tasks properties which can be used to better support certain scheduler decisions.

Utilization clamping aims at exposing to user-space a new set of per-task attributes which can be used to provide the scheduler with some hints about the expected/required utilization for a task. This will allow to implement a more advanced per-task frequency control mechanism which is not based just on a "passive" measured task utilization but on a more "proactive" approach. For example, it could be possible to boost interactive tasks, thus getting better performance, or cap background tasks, thus being more energy/thermal efficient.

Ultimately, such a mechanism can be considered similar to the cpufreq powersave, performance and userspace governors but with a much fine grained and per-task control.

Let's introduce a new API to set utilization clamping values for a specified task by extending sched_setattr, a syscall which already allows to define task specific properties for different scheduling classes. Specifically, a new pair of attributes allows to specify a minimum and maximum utilization which the scheduler should consider for a task.

Cc Ingo Molnar <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Tejun Heo <[email protected]>
Cc Rafael J. Wysocki <[email protected]>
Cc Vincent Guittot <[email protected]>
Cc Viresh Kumar <[email protected]>
Cc Randy Dunlap <[email protected]>
Cc Paul Turner <[email protected]>
Cc Suren Baghdasaryan <[email protected]>
Cc Todd Kjos <[email protected]>
Cc Joel Fernandes <[email protected]>
Cc Steve Muckle <[email protected]>
Cc Juri Lelli <[email protected]>
Cc Quentin Perret <[email protected]>
Cc Dietmar Eggemann <[email protected]>
Cc Morten Rasmussen <[email protected]>

---
Changes in v5:
Others:
- rebased on v4.19

Changes in v4:
Message-ID: <[email protected]>
- remove not required default setting
- fixed some tabs/spaces
Message-ID: <[email protected]>
- replace/rephrase "bandwidth" references to use "capacity"
- better stress that this do not enforce any bandwidth requirement but "just" give hints to the scheduler
- fixed some typos
Others:
- add support for SCHED_FLAG_RESET_ON_FORK
default clamps are now set for init_task and inherited/reset at fork time (when then flag is set for the parent)
- rebased on v4.19-rc1
Changes in v3:
Message-ID: <CAJuCfpF6=L=0LrmNnJrTNPazT4dWKqNv+thhN0dwpKCgUzs9sg@mail.gmail.com>
- removed UCLAMP_NONE not used by this patch
Others:
- rebased on tip/sched/core
Changes in v2:
- rebased on v4.18-rc4
- move at the head of the series

As discussed at OSPM, using a [0..SCHED_CAPACITY_SCALE] range seems to be acceptable. However, an additional patch has been added at the end of the series which introduces a simple abstraction to use a more generic [0..100] range.

At OSPM we also discarded the idea to "recycle" the usage of sched_runtime and sched_period which would have made the API too much complex for limited benefits.

---
 include/linux/sched.h            | 13 +++++++
 include/uapi/linux/sched.h       |  4 +-
 include/uapi/linux/sched/types.h | 67 +++++++++++++++++++++++++++-----
 init/Kconfig                     | 21 ++++++++++
 init/init_task.c                 |  5 +++
 kernel/sched/core.c              | 39 +++++++++++++++++++
 6 files changed, 139 insertions(+), 10 deletions(-)
include/linux/sched.h +13 -0
index 977cb57d7bc9..880a0c5c1f87 100644
@@ -279,6 +279,14 @@ struct vtime {
 	u64			gtime;
 };
 
+enum uclamp_id {
+	UCLAMP_MIN = 0, /* Minimum utilization */
+	UCLAMP_MAX,     /* Maximum utilization */
+
+	/* Utilization clamping constraints count */
+	UCLAMP_CNT
+};
+
 struct sched_info {
 #ifdef CONFIG_SCHED_INFO
 	/* Cumulative counters: */
@@ -649,6 +657,11 @@ struct task_struct {
 #endif
 	struct sched_dl_entity		dl;
 
+#ifdef CONFIG_UCLAMP_TASK
+	/* Utlization clamp values for this task */
+	int				uclamp[UCLAMP_CNT];
+#endif
+
 #ifdef CONFIG_PREEMPT_NOTIFIERS
 	/* List of struct preempt_notifier: */
 	struct hlist_head		preempt_notifiers;
include/uapi/linux/sched.h +3 -1
index 22627f80063e..c27d6e81517b 100644
@@ -50,9 +50,11 @@
 #define SCHED_FLAG_RESET_ON_FORK	0x01
 #define SCHED_FLAG_RECLAIM		0x02
 #define SCHED_FLAG_DL_OVERRUN		0x04
+#define SCHED_FLAG_UTIL_CLAMP		0x08
 
 #define SCHED_FLAG_ALL	(SCHED_FLAG_RESET_ON_FORK	| \
 			 SCHED_FLAG_RECLAIM		| \
-			 SCHED_FLAG_DL_OVERRUN)
+			 SCHED_FLAG_DL_OVERRUN		| \
+			 SCHED_FLAG_UTIL_CLAMP)
 
 #endif /* _UAPI_LINUX_SCHED_H */
include/uapi/linux/sched/types.h +58 -9

98 lines not on this page · raw message

init/Kconfig +21 -0
index 1e234e2f1cba..738974c4f628 100644
@@ -613,6 +613,27 @@ config HAVE_UNSTABLE_SCHED_CLOCK
 config GENERIC_SCHED_CLOCK
 	bool
 
+menu "Scheduler features"
+
+config UCLAMP_TASK
+	bool "Enable utilization clamping for RT/FAIR tasks"
+	depends on CPU_FREQ_GOV_SCHEDUTIL
+	help
+	  This feature enables the scheduler to track the clamped utilization
+	  of each CPU based on RUNNABLE tasks currently scheduled on that CPU.
+
+	  When this option is enabled, the user can specify a min and max CPU
+	  utilization which is allowed for RUNNABLE tasks.
+	  The max utilization allows to request a maximum frequency a task should
+	  use, while the min utilization allows to request a minimum frequency a
+	  task should use.
+	  Both min and max utilization clamp values are hints to the scheduler,
+	  aiming at improving its frequency selection policy, but they do not
+	  enforce or grant any specific bandwidth for tasks.
+
+	  If in doubt, say N.
+
+endmenu
 #
 # For architectures that want to enable the support for NUMA-affine scheduler
 # balancing logic:
init/init_task.c +5 -0
index 5aebe3be4d7c..5bfdcc3fb839 100644
@@ -6,6 +6,7 @@
 #include <linux/sched/sysctl.h>
 #include <linux/sched/rt.h>
 #include <linux/sched/task.h>
+#include <linux/sched/topology.h>
 #include <linux/init.h>
 #include <linux/fs.h>
 #include <linux/mm.h>
@@ -91,6 +92,10 @@ struct task_struct init_task
 #endif
 #ifdef CONFIG_CGROUP_SCHED
 	.sched_task_group = &root_task_group,
+#endif
+#ifdef CONFIG_UCLAMP_TASK
+	.uclamp[UCLAMP_MIN] = 0,
+	.uclamp[UCLAMP_MAX] = SCHED_CAPACITY_SCALE,
 #endif
 	.ptraced	= LIST_HEAD_INIT(init_task.ptraced),
 	.ptrace_entry	= LIST_HEAD_INIT(init_task.ptrace_entry),
kernel/sched/core.c +39 -0
index ad97f3ba5ec5..3701bb1e6698 100644
@@ -716,6 +716,28 @@ static void set_load_weight(struct task_struct *p, bool update_load)
 	}
 }
 
+#ifdef CONFIG_UCLAMP_TASK
+static inline int __setscheduler_uclamp(struct task_struct *p,
+					const struct sched_attr *attr)
+{
+	if (attr->sched_util_min > attr->sched_util_max)
+		return -EINVAL;
+	if (attr->sched_util_max > SCHED_CAPACITY_SCALE)
+		return -EINVAL;
+
+	p->uclamp[UCLAMP_MIN] = attr->sched_util_min;
+	p->uclamp[UCLAMP_MAX] = attr->sched_util_max;
+
+	return 0;
+}
+#else /* CONFIG_UCLAMP_TASK */
+static inline int __setscheduler_uclamp(struct task_struct *p,
+					const struct sched_attr *attr)
+{
+	return -EINVAL;
+}
+#endif /* CONFIG_UCLAMP_TASK */
+
 static inline void enqueue_task(struct rq *rq, struct task_struct *p, int flags)
 {
 	if (!(flags & ENQUEUE_NOCLOCK))
@@ -2320,6 +2342,11 @@ int sched_fork(unsigned long clone_flags, struct task_struct *p)
 		p->prio = p->normal_prio = __normal_prio(p);
 		set_load_weight(p, false);
 
+#ifdef CONFIG_UCLAMP_TASK
+		p->uclamp[UCLAMP_MIN] = 0;
+		p->uclamp[UCLAMP_MAX] = SCHED_CAPACITY_SCALE;
+#endif
+
 		/*
 		 * We don't need the reset flag anymore after the fork. It has
 		 * fulfilled its duty:
@@ -4215,6 +4242,13 @@ static int __sched_setscheduler(struct task_struct *p,
 			return retval;
 	}
 
+	/* Configure utilization clamps for the task */
+	if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP) {
+		retval = __setscheduler_uclamp(p, attr);
+		if (retval)
+			return retval;
+	}
+
 	/*
 	 * Make sure no PI-waiters arrive (or leave) while we are
 	 * changing the priority of the task:
@@ -4721,6 +4755,11 @@ SYSCALL_DEFINE4(sched_getattr, pid_t, pid, struct sched_attr __user *, uattr,
 	else
 		attr.sched_nice = task_nice(p);
 
+#ifdef CONFIG_UCLAMP_TASK
+	attr.sched_util_min = p->uclamp[UCLAMP_MIN];
+	attr.sched_util_max = p->uclamp[UCLAMP_MAX];
+#endif
+
 	rcu_read_unlock();
 
 	retval = sched_read_attr(uattr, &attr, size);
Signature · 1 line
2.18.0
reviewer
#2 lore raw

Re: [PATCH v5 01/15] sched/core: uclamp: extend sched_setattr to support utilization clamping

Quoted 18 lines · Patrick Bellasi
include/uapi/linux/sched/types.h +9 -0
index 10fbb8031930..fde7301ed28c 100644
@@ -53,6 +73,30 @@ struct sched_param {
  * As of now, the SCHED_DEADLINE policy (sched_dl scheduling class) is the
  * only user of this new interface. More information about the algorithm
  * available in the scheduling class file or in Documentation/.
+ *
+ * Task Utilization Attributes
+ * ===========================
+ *
+ * A subset of sched_attr attributes allows to specify the utilization which
+ * should be expected by a task. These attributes allow to inform the
+ * scheduler about the utilization boundaries within which it is expected to
+ * schedule the task. These boundaries are valuable hints to support scheduler
+ * decisions on both task placement and frequencies selection.

include/uapi/linux/sched/types.h struct sched_param { · line 94

Quoted 10 lines
include/uapi/linux/sched/types.h +10 -0
+ *
+ *  @sched_util_min	represents the minimum utilization
+ *  @sched_util_max	represents the maximum utilization
+ *
+ * Utilization is a value in the range [0..SCHED_CAPACITY_SCALE] which
+ * represents the percentage of CPU time used by a task when running at the
+ * maximum frequency on the highest capacity CPU of the system. Thus, for
+ * example, a 20% utilization task is a task running for 2ms every 10ms.
+ *
+ * A task with a min utilization value bigger then 0 is more likely to be

than

include/uapi/linux/sched/types.h struct sched_param { · line 98

Quoted 4 lines
include/uapi/linux/sched/types.h +4 -0
+ * scheduled on a CPU which has a capacity big enough to fit the specified
+ * minimum utilization value.
+ * A task with a max utilization value smaller then 1024 is more likely to be
+ * scheduled on a CPU which do not necessarily have more capacity then the

does than

include/uapi/linux/sched/types.h struct sched_param { · line 102

Quoted 4 lines
include/uapi/linux/sched/types.h +1 -0
+ * specified max utilization value.
  */
 struct sched_attr {
 	__u32 size;

cheers.

Signature · 1 line
~Randy
maintainer
#3 lore raw

Re: [PATCH v5 01/15] sched/core: uclamp: extend sched_setattr to support utilization clamping

In reply to Patrick Bellasi
Quoted 22 lines · Patrick Bellasi
include/uapi/linux/sched.h +3 -1
index 22627f80063e..c27d6e81517b 100644
@@ -50,9 +50,11 @@
 #define SCHED_FLAG_RESET_ON_FORK	0x01
 #define SCHED_FLAG_RECLAIM		0x02
 #define SCHED_FLAG_DL_OVERRUN		0x04
+#define SCHED_FLAG_UTIL_CLAMP		0x08
 
 #define SCHED_FLAG_ALL	(SCHED_FLAG_RESET_ON_FORK	| \
 			 SCHED_FLAG_RECLAIM		| \
-			 SCHED_FLAG_DL_OVERRUN)
+			 SCHED_FLAG_DL_OVERRUN		| \
+			 SCHED_FLAG_UTIL_CLAMP)
 
 #endif /* _UAPI_LINUX_SCHED_H */
include/uapi/linux/sched/types.h +0 -0
index 10fbb8031930..fde7301ed28c 100644

include/uapi/linux/sched/types.h struct sched_param { · line 15

Quoted 7 lines
include/uapi/linux/sched/types.h +1 -0
@@ -9,6 +9,7 @@ struct sched_param {
 };
 
 #define SCHED_ATTR_SIZE_VER0	48	/* sizeof first published struct */
+#define SCHED_ATTR_SIZE_VER1	56	/* add: util_{min,max} */
 
 /*
  * Extended scheduling parameters data structure.
kernel/sched/core.c +4 -0
@@ -4533,6 +4533,10 @@ static int sched_copy_attr(struct sched_
 	if (ret)
 		return -EFAULT;
 
+	if ((attr->sched_flags & SCHED_FLAG_UTIL_CLAMP) &&
+	    size < SCHED_ATTR_SIZE_VER1)
+		return -EINVAL;
+
 	/*
 	 * XXX: Do we want to be lenient like existing syscalls; or do we want
 	 * to be strict and return an error on out-of-bounds values?
#4 lore raw

PATCH v5 2/15 sched/core: make sched_setattr able to tune the current policy

In reply to Patrick Bellasi

Currently, sched_setattr mandates that a policy is always specified. Since utilization clamp attributes could apply across different scheduling policies (i.e. all but SCHED_DEADLINE), this requires also to always know which policy a task has before changing its clamp values.

This is not just cumbersome but it's also racy, indeed we cannot be sure that a task policy has been changed in between its policy read and the actual clamp value change. Sometimes however, this could be the actual use-case, we wanna change the clamps without affecting the policy.

Let's fix this adding an additional attribute flag (SCHED_FLAG_TUNE_POLICY) which, when specified, will ensure to always force the usage of the current policy. This is done by re-using the SETPARAM_POLICY thing we already have for the sched_setparam syscall, thus extending its usage to the non-POSIX sched_setattr while not exposing that internal concept to user-space.

Cc Ingo Molnar <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Suren Baghdasaryan <[email protected]>
Cc Todd Kjos <[email protected]>
Cc Joel Fernandes <[email protected]>
Cc Steve Muckle <[email protected]>
Cc Juri Lelli <[email protected]>

---
Changes in v5:
Message-ID: <[email protected]>
- allow to change clamp values without affecting current policy

---
 include/uapi/linux/sched.h |  6 +++++-
 kernel/sched/core.c        | 11 ++++++++++-
 2 files changed, 15 insertions(+), 2 deletions(-)
include/uapi/linux/sched.h +5 -1
index c27d6e81517b..62498d749bec 100644
@@ -40,6 +40,8 @@
 /* SCHED_ISO: reserved but not implemented yet */
 #define SCHED_IDLE		5
 #define SCHED_DEADLINE		6
+/* Must be the last entry: used check attr.policy values */
+#define SCHED_POLICY_MAX	7
 
 /* Can be ORed in to make sure the process is reverted back to SCHED_NORMAL on fork */
 #define SCHED_RESET_ON_FORK     0x40000000
@@ -50,11 +52,13 @@
 #define SCHED_FLAG_RESET_ON_FORK	0x01
 #define SCHED_FLAG_RECLAIM		0x02
 #define SCHED_FLAG_DL_OVERRUN		0x04
-#define SCHED_FLAG_UTIL_CLAMP		0x08
+#define SCHED_FLAG_TUNE_POLICY		0x08
+#define SCHED_FLAG_UTIL_CLAMP		0x10
 
 #define SCHED_FLAG_ALL	(SCHED_FLAG_RESET_ON_FORK	| \
 			 SCHED_FLAG_RECLAIM		| \
 			 SCHED_FLAG_DL_OVERRUN		| \
+			 SCHED_FLAG_TUNE_POLICY		| \
 			 SCHED_FLAG_UTIL_CLAMP)
 
 #endif /* _UAPI_LINUX_SCHED_H */
kernel/sched/core.c +10 -1
index 3701bb1e6698..9a2e12eaa377 100644
@@ -4595,8 +4595,17 @@ SYSCALL_DEFINE3(sched_setattr, pid_t, pid, struct sched_attr __user *, uattr,
 	if (retval)
 		return retval;
 
-	if ((int)attr.sched_policy < 0)
+	/*
+	 * A valid policy has always to be required from userspace.  Unless
+	 * the SCHED_FLAG_TUNE_POLICY is set, in which case, the current
+	 * policy will be enforced for this call.
+	 */
+	if (attr.sched_policy >= SCHED_POLICY_MAX &&
+	    !(attr.sched_flags & SCHED_FLAG_TUNE_POLICY)) {
 		return -EINVAL;
+	}
+	if (attr.sched_flags & SCHED_FLAG_TUNE_POLICY)
+		attr.sched_policy = SETPARAM_POLICY;
 
 	rcu_read_lock();
 	retval = -ESRCH;
Signature · 1 line
2.18.0
maintainer
#5 lore raw

Re: [PATCH v5 02/15] sched/core: make sched_setattr able to tune the current policy

include/uapi/linux/sched.h line 56

Quoted 7 lines · Patrick Bellasi
include/uapi/linux/sched.h +2 -1
@@ -50,11 +52,13 @@
 #define SCHED_FLAG_RESET_ON_FORK	0x01
 #define SCHED_FLAG_RECLAIM		0x02
 #define SCHED_FLAG_DL_OVERRUN		0x04
-#define SCHED_FLAG_UTIL_CLAMP		0x08
+#define SCHED_FLAG_TUNE_POLICY		0x08
+#define SCHED_FLAG_UTIL_CLAMP		0x10

That seems to suggest you want to do this patch first, but you didn't..

#6 lore raw

Re: [PATCH v5 02/15] sched/core: make sched_setattr able to tune the current policy

Quoted 11 lines · Peter Zijlstra

include/uapi/linux/sched.h line 56

Quoted 7 lines · Patrick Bellasi
include/uapi/linux/sched.h +2 -1
@@ -50,11 +52,13 @@
 #define SCHED_FLAG_RESET_ON_FORK	0x01
 #define SCHED_FLAG_RECLAIM		0x02
 #define SCHED_FLAG_DL_OVERRUN		0x04
-#define SCHED_FLAG_UTIL_CLAMP		0x08
+#define SCHED_FLAG_TUNE_POLICY		0x08
+#define SCHED_FLAG_UTIL_CLAMP		0x10

That seems to suggest you want to do this patch first, but you didn't..

I've kept it here just to better highlight this change, suggested by Juri, since we was not entirely sure you are fine with it...

If you think it's ok adding a SCHED_FLAG_TUNE_POLICY behavior to the sched_setattr syscall, I can certainly squash into the previous patch, which gives more context on why we need it.

Otherwise, if we want to keep these bits better isolated for possible future bisects, I can also move it at the beginning of the series.

What do you like best ?

Since we are at that, are we supposed to document some{where,how} these API changes ?

Signature · 3 lines
#include <best/regards.h>

Patrick Bellasi
maintainer
#7 lore raw

Re: [PATCH v5 02/15] sched/core: make sched_setattr able to tune the current policy

Quoted 13 lines · Patrick Bellasi
Quoted 11 lines · Peter Zijlstra

include/uapi/linux/sched.h line 56

Quoted 7 lines · Patrick Bellasi
include/uapi/linux/sched.h +2 -1
@@ -50,11 +52,13 @@
 #define SCHED_FLAG_RESET_ON_FORK	0x01
 #define SCHED_FLAG_RECLAIM		0x02
 #define SCHED_FLAG_DL_OVERRUN		0x04
-#define SCHED_FLAG_UTIL_CLAMP		0x08
+#define SCHED_FLAG_TUNE_POLICY		0x08
+#define SCHED_FLAG_UTIL_CLAMP		0x10

That seems to suggest you want to do this patch first, but you didn't..

Quoted 6 lines

I've kept it here just to better highlight this change, suggested by Juri, since we was not entirely sure you are fine with it...

If you think it's ok adding a SCHED_FLAG_TUNE_POLICY behavior to the sched_setattr syscall, I can certainly squash into the previous patch, which gives more context on why we need it.

I'm fine with the idea I think. It's the details I worry about. Which fields in particular are not updated with this. Are flags?

Also, I'm not too keen on the name; since it explicitly does not modify the policy and its related parameters, so TUNE_POLICY is actively wrong.

But the thing that confused me most is how fiddled the numbers to fit this before UTIL_CLAMP.

Quoted 2 lines

Since we are at that, are we supposed to document some{where,how} these API changes ?

I'm pretty sure there's a manpage somewhere... SCHED_SETATTR(2) seems to exist on my machine. So that wants updates.

#8 lore raw

PATCH v5 3/15 sched/core: uclamp: map TASK's clamp values into CPU's clamp groups

In reply to Patrick Bellasi

Utilization clamping requires each CPU to know which clamp values are assigned to tasks that are currently RUNNABLE on that CPU: multiple tasks can be assigned the same clamp value and tasks with different clamp values can be concurrently active on the same CPU.

A proper data structure is required to support a fast and efficient aggregation of the clamp values required by the currently RUNNABLE tasks. To this purpose, a per-CPU array of reference counters can be used, where each slot tracks how many tasks, requiring the same clamp value, are currently RUNNABLE on each CPU.

Thus we need a mechanism to map each "clamp value" into a corresponding "clamp index" which identifies the position within a reference counters array used to track RUNNABLE tasks.

Let's introduce the support to map tasks to "clamp groups". Specifically we introduce the required functions to translate a "clamp value" (clamp_value) into a clamp's "group index" (group_id).

                                 :
       (user-space changes)      :      (kernel space / scheduler)
                                 :
             SLOW PATH           :             FAST PATH
                                 :
    task_struct::uclamp::value   :     sched/core::enqueue/dequeue
                                 :         cpufreq_schedutil
                                 :
  +----------------+    +--------------------+     +-------------------+
  |      TASK      |    |     CLAMP GROUP    |     |    CPU CLAMPS     |
  +----------------+    +--------------------+     +-------------------+
  |                |    |   clamp_{min,max}  |     |  clamp_{min,max}  |
  | util_{min,max} |    |      se_count      |     |    tasks count    |
  +----------------+    +--------------------+     +-------------------+
                                 :
           +------------------>  :  +------------------->
    group_id = map(clamp_value)  :  ref_count(group_id)
                                 :
                                 :

Only a limited number of (different) clamp values are supported since:
1. there are usually only few classes of workloads for which it makes sense to boost/limit to different frequencies,
e.g. background vs foreground, interactive vs low-priority
2. it allows a simpler and more memory/time efficient tracking of the per-CPU clamp values in the fast path.

The number of possible different clamp values is currently defined at compile time. Thus, setting a new clamp value for a task can result into the impossibility to map it into a dedicated clamp index. Those tasks will be flagged as "not mapped" and not tracked at enqueue/dequeue time.

Cc Ingo Molnar <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Paul Turner <[email protected]>
Cc Suren Baghdasaryan <[email protected]>
Cc Todd Kjos <[email protected]>
Cc Joel Fernandes <[email protected]>
Cc Juri Lelli <[email protected]>
Cc Quentin Perret <[email protected]>
Cc Dietmar Eggemann <[email protected]>
Cc Morten Rasmussen <[email protected]>

---

A following patch:

sched/core: uclamp: add clamp group bucketing support

will fix the "not mapped" tasks not being tracked.

Changes in v5:
Message-ID: <[email protected]>
- use bitfields and atomic_long_cmpxchg() operations to both compress the clamp maps and avoid usage of spinlock.
- remove enforced __cacheline_aligned_in_smp on uclamp_map since it's accessed from the slow path only and we don't care about performance
- better describe the usage of uclamp_map::se_lock
Message-ID: <[email protected]>
- remove inline from uclamp_group_{get,put}() and __setscheduler_uclamp()
- set lower/upper bounds at the beginning of __setscheduler_uclamp()
- avoid usage of pr_err from unprivileged syscall paths
in  __setscheduler_uclamp(), replaced by ratelimited version
Message-ID: <20180914134128.GP1413@e110439-lin>
- remove/limit usage of UCLAMP_NOT_VALID whenever not strictly required
Message-ID: <[email protected]>
- allow sched_setattr() syscall to sleep on mutex
- fix return value for successfull uclamp syscalls
Message-ID: <CAJuCfpF36-VZm0JVVNnOnGm-ukVejzxbPhH33X3z9gAQ06t9gQ@mail.gmail.com>
- reorder conditions in uclamp_group_find() loop
- use uc_se->xxx in uclamp_fork()
Others:
- use UCLAMP_GROUPS to track (CONFIG_UCLAMP_GROUPS_COUNT+1)
- rebased on v4.19

Changes in v4:
Message-ID: <[email protected]>
- add uclamp_exit_task() to release clamp refcount from do_exit()
Message-ID: <20180816133249.GA2964@e110439-lin>
- keep the WARN but butify a bit that code
Message-ID: <[email protected]>
- move uclamp_enabled at the top of sched_class to keep it on the same cache line of other main wakeup time callbacks
Others:
- init uclamp for the init_task and refcount its clamp groups
- add uclamp specific fork time code into uclamp_fork
- add support for SCHED_FLAG_RESET_ON_FORK
default clamps are now set for init_task and inherited/reset at fork time (when then flag is set for the parent)
- enable uclamp only for FAIR tasks, RT class will be enabled only by a following patch which also integrate the class to schedutil
- define uclamp_maps ____cacheline_aligned_in_smp
- in uclamp_group_get() ensure to include uclamp_group_available() and uclamp_group_init() into the atomic section defined by:
uc_map[next_group_id].se_lock
- do not use mutex_lock(&uclamp_mutex) in uclamp_exit_task
which is also not needed since refcounting is already guarded by the uc_map[group_id].se_lock spinlock
- rebased on v4.19-rc1
Changes in v3:
Message-ID: <CAJuCfpF6=L=0LrmNnJrTNPazT4dWKqNv+thhN0dwpKCgUzs9sg@mail.gmail.com>
- rename UCLAMP_NONE into UCLAMP_NOT_VALID
- remove not necessary checks in uclamp_group_find()
- add WARN on unlikely un-referenced decrement in uclamp_group_put()
- make __setscheduler_uclamp() able to set just one clamp value
- make __setscheduler_uclamp() failing if both clamps are required but there is no clamp groups available for one of them
- remove uclamp_group_find() from uclamp_group_get() which now takes a group_id as a parameter
Others:
- rebased on tip/sched/core
Changes in v2:
- rabased on v4.18-rc4
- set UCLAMP_GROUPS_COUNT=2 by default
which allows to fit all the hot-path CPU clamps data, partially intorduced also by the following patches, into a single cache line while still supporting up to 2 different {min,max}_utiql clamps.

---
 include/linux/sched.h          |  39 ++++-
 include/linux/sched/task.h     |   6 +
 include/linux/sched/topology.h |   6 -
 include/uapi/linux/sched.h     |   5 +-
 init/Kconfig                   |  20 +++
 init/init_task.c               |   4 -
 kernel/exit.c                  |   1 +
 kernel/sched/core.c            | 283 ++++++++++++++++++++++++++++++---
 kernel/sched/fair.c            |   4 +
 kernel/sched/sched.h           |  28 +++-
 10 files changed, 363 insertions(+), 33 deletions(-)
include/linux/sched.h +38 -1
index 880a0c5c1f87..facace271ea1 100644
@@ -318,6 +318,12 @@ struct sched_info {
 # define SCHED_FIXEDPOINT_SHIFT		10
 # define SCHED_FIXEDPOINT_SCALE		(1L << SCHED_FIXEDPOINT_SHIFT)
 
+/*
+ * Increase resolution of cpu_capacity calculations
+ */
+#define SCHED_CAPACITY_SHIFT	SCHED_FIXEDPOINT_SHIFT
+#define SCHED_CAPACITY_SCALE	(1L << SCHED_CAPACITY_SHIFT)
+
 struct load_weight {
 	unsigned long			weight;
 	u32				inv_weight;
@@ -575,6 +581,37 @@ struct sched_dl_entity {
 	struct hrtimer inactive_timer;
 };
 
+#ifdef CONFIG_UCLAMP_TASK
+/*
+ * Number of utiliation clamp groups
+ *
+ * The first clamp group (group_id=0) is used to track non clamped tasks, i.e.
+ * util_{min,max} (0,SCHED_CAPACITY_SCALE). Thus we allocate one more group in
+ * addition to the configured number.
+ */
+#define UCLAMP_GROUPS (CONFIG_UCLAMP_GROUPS_COUNT + 1)
+
+/**
+ * Utilization clamp group
+ *
+ * A utilization clamp group maps a:
+ *   clamp value (value), i.e.
+ *   util_{min,max} value requested from userspace
+ * to a:
+ *   clamp group index (group_id), i.e.
+ *   index of the per-cpu RUNNABLE tasks refcounting array
+ *
+ * The mapped bit is set whenever a task has been mapped on a clamp group for
+ * the first time. When this bit is set, any clamp group get (for a new clamp
+ * value) will be matches by a clamp group put (for the old clamp value).
+ */
+struct uclamp_se {
+	unsigned int value		: SCHED_CAPACITY_SHIFT + 1;
+	unsigned int group_id		: order_base_2(UCLAMP_GROUPS);
+	unsigned int mapped		: 1;
+};
+#endif /* CONFIG_UCLAMP_TASK */
+
 union rcu_special {
 	struct {
 		u8			blocked;
@@ -659,7 +696,7 @@ struct task_struct {
 
 #ifdef CONFIG_UCLAMP_TASK
 	/* Utlization clamp values for this task */
-	int				uclamp[UCLAMP_CNT];
+	struct uclamp_se		uclamp[UCLAMP_CNT];
 #endif
 
 #ifdef CONFIG_PREEMPT_NOTIFIERS
include/linux/sched/task.h +6 -0
index 108ede99e533..36c81c364112 100644
@@ -68,6 +68,12 @@ static inline void exit_thread(struct task_struct *tsk)
 #endif
 extern void do_group_exit(int);
 
+#ifdef CONFIG_UCLAMP_TASK
+extern void uclamp_exit_task(struct task_struct *p);
+#else
+static inline void uclamp_exit_task(struct task_struct *p) { }
+#endif /* CONFIG_UCLAMP_TASK */
+
 extern void exit_files(struct task_struct *);
 extern void exit_itimers(struct signal_struct *);
 
include/linux/sched/topology.h +0 -6
index 26347741ba50..350043d203db 100644
@@ -6,12 +6,6 @@
 
 #include <linux/sched/idle.h>
 
-/*
- * Increase resolution of cpu_capacity calculations
- */
-#define SCHED_CAPACITY_SHIFT	SCHED_FIXEDPOINT_SHIFT
-#define SCHED_CAPACITY_SCALE	(1L << SCHED_CAPACITY_SHIFT)
-
 /*
  * sched-domains (multiprocessor balancing) declarations:
  */
include/uapi/linux/sched.h +4 -1
index 62498d749bec..e6f2453eb5a5 100644
@@ -53,7 +53,10 @@
 #define SCHED_FLAG_RECLAIM		0x02
 #define SCHED_FLAG_DL_OVERRUN		0x04
 #define SCHED_FLAG_TUNE_POLICY		0x08
-#define SCHED_FLAG_UTIL_CLAMP		0x10
+#define SCHED_FLAG_UTIL_CLAMP_MIN	0x10
+#define SCHED_FLAG_UTIL_CLAMP_MAX	0x20
+#define SCHED_FLAG_UTIL_CLAMP	(SCHED_FLAG_UTIL_CLAMP_MIN | \
+				 SCHED_FLAG_UTIL_CLAMP_MAX)
 
 #define SCHED_FLAG_ALL	(SCHED_FLAG_RESET_ON_FORK	| \
 			 SCHED_FLAG_RECLAIM		| \
init/Kconfig +20 -0
index 738974c4f628..4c5475030286 100644
@@ -633,7 +633,27 @@ config UCLAMP_TASK
 
 	  If in doubt, say N.
 
+config UCLAMP_GROUPS_COUNT
+	int "Number of different utilization clamp values supported"
+	range 0 32
+	default 5
+	depends on UCLAMP_TASK
+	help
+	  This defines the maximum number of different utilization clamp
+	  values which can be concurrently enforced for each utilization
+	  clamp index (i.e. minimum and maximum utilization).
+
+	  Only a limited number of clamp values are supported because:
+	    1. there are usually only few classes of workloads for which it
+	       makes sense to boost/cap for different frequencies,
+	       e.g. background vs foreground, interactive vs low-priority.
+	    2. it allows a simpler and more memory/time efficient tracking of
+	       per-CPU clamp values.
+
+	  If in doubt, use the default value.
+
 endmenu
+
 #
 # For architectures that want to enable the support for NUMA-affine scheduler
 # balancing logic:
init/init_task.c +0 -4
index 5bfdcc3fb839..7f77741b6a9b 100644
@@ -92,10 +92,6 @@ struct task_struct init_task
 #endif
 #ifdef CONFIG_CGROUP_SCHED
 	.sched_task_group = &root_task_group,
-#endif
-#ifdef CONFIG_UCLAMP_TASK
-	.uclamp[UCLAMP_MIN] = 0,
-	.uclamp[UCLAMP_MAX] = SCHED_CAPACITY_SCALE,
 #endif
 	.ptraced	= LIST_HEAD_INIT(init_task.ptraced),
 	.ptrace_entry	= LIST_HEAD_INIT(init_task.ptrace_entry),
kernel/exit.c +1 -0
index 0e21e6d21f35..feb540558051 100644
@@ -877,6 +877,7 @@ void __noreturn do_exit(long code)
 
 	sched_autogroup_exit_task(tsk);
 	cgroup_exit(tsk);
+	uclamp_exit_task(tsk);
 
 	/*
 	 * FIXME: do that only when needed, using sched_exit tracepoint
kernel/sched/core.c +265 -18

344 lines not on this page · raw message

kernel/sched/fair.c +4 -0
index 908c9cdae2f0..6c92cd2d637a 100644
@@ -10157,6 +10157,10 @@ const struct sched_class fair_sched_class = {
 #ifdef CONFIG_FAIR_GROUP_SCHED
 	.task_change_group	= task_change_group_fair,
 #endif
+
+#ifdef CONFIG_UCLAMP_TASK
+	.uclamp_enabled		= 1,
+#endif
 };
 
 #ifdef CONFIG_SCHED_DEBUG
kernel/sched/sched.h +25 -3
index 9683f458aec7..947ab14d3d5b 100644
@@ -1504,10 +1504,12 @@ extern const u32		sched_prio_to_wmult[40];
 struct sched_class {
 	const struct sched_class *next;
 
+#ifdef CONFIG_UCLAMP_TASK
+	int uclamp_enabled;
+#endif
+
 	void (*enqueue_task) (struct rq *rq, struct task_struct *p, int flags);
 	void (*dequeue_task) (struct rq *rq, struct task_struct *p, int flags);
-	void (*yield_task)   (struct rq *rq);
-	bool (*yield_to_task)(struct rq *rq, struct task_struct *p, bool preempt);
 
 	void (*check_preempt_curr)(struct rq *rq, struct task_struct *p, int flags);
 
@@ -1540,7 +1542,6 @@ struct sched_class {
 	void (*set_curr_task)(struct rq *rq);
 	void (*task_tick)(struct rq *rq, struct task_struct *p, int queued);
 	void (*task_fork)(struct task_struct *p);
-	void (*task_dead)(struct task_struct *p);
 
 	/*
 	 * The switched_from() call is allowed to drop rq->lock, therefore we
@@ -1557,12 +1558,17 @@ struct sched_class {
 
 	void (*update_curr)(struct rq *rq);
 
+	void (*yield_task)   (struct rq *rq);
+	bool (*yield_to_task)(struct rq *rq, struct task_struct *p, bool preempt);
+
 #define TASK_SET_GROUP		0
 #define TASK_MOVE_GROUP		1
 
 #ifdef CONFIG_FAIR_GROUP_SCHED
 	void (*task_change_group)(struct task_struct *p, int type);
 #endif
+
+	void (*task_dead)(struct task_struct *p);
 };
 
 static inline void put_prev_task(struct rq *rq, struct task_struct *prev)
@@ -2180,6 +2186,22 @@ static inline void cpufreq_update_util(struct rq *rq, unsigned int flags)
 static inline void cpufreq_update_util(struct rq *rq, unsigned int flags) {}
 #endif /* CONFIG_CPU_FREQ */
 
+/**
+ * uclamp_none: default value for a clamp
+ *
+ * This returns the default value for each clamp
+ * - 0 for a min utilization clamp
+ * - SCHED_CAPACITY_SCALE for a max utilization clamp
+ *
+ * Return: the default value for a given utilization clamp
+ */
+static inline unsigned int uclamp_none(int clamp_id)
+{
+	if (clamp_id == UCLAMP_MIN)
+		return 0;
+	return SCHED_CAPACITY_SCALE;
+}
+
 #ifdef arch_scale_freq_capacity
 # ifndef arch_scale_freq_invariant
 #  define arch_scale_freq_invariant()	true
Signature · 1 line
2.18.0
maintainer
#9 lore raw

Re: [PATCH v5 03/15] sched/core: uclamp: map TASK's clamp values into CPU's clamp groups

include/linux/sched.h struct sched_dl_entity { · line 610

Quoted 3 lines · Patrick Bellasi
include/linux/sched.h +3 -0
+struct uclamp_se {
+	unsigned int value		: SCHED_CAPACITY_SHIFT + 1;
+	unsigned int group_id		: order_base_2(UCLAMP_GROUPS);

Are you sure about ob2() ? seems weird we'll end up with 0 for UCLAMP_GROUPS==1.

Quoted 2 lines
include/linux/sched.h +2 -0
+	unsigned int mapped		: 1;
+};
#10 lore raw

Re: [PATCH v5 03/15] sched/core: uclamp: map TASK's clamp values into CPU's clamp groups

Quoted 7 lines · Peter Zijlstra

include/linux/sched.h struct sched_dl_entity { · line 610

Quoted 3 lines · Patrick Bellasi
include/linux/sched.h +3 -0
+struct uclamp_se {
+	unsigned int value		: SCHED_CAPACITY_SHIFT + 1;
+	unsigned int group_id		: order_base_2(UCLAMP_GROUPS);

Are you sure about ob2() ? seems weird we'll end up with 0 for UCLAMP_GROUPS==1.

So, we have:

   #define UCLAMP_GROUPS (CONFIG_UCLAMP_GROUPS_COUNT + 1)

because one clamp group is always reserved for defaults. Thus, ob2(in) is always called with n>=2.

... should be safe no ?

However, will check better again on v6 for possible corner-cases...

Quoted 2 lines
Quoted 2 lines
include/linux/sched.h +2 -0
+	unsigned int mapped		: 1;
+};
Signature · 3 lines
#include <best/regards.h>

Patrick Bellasi
maintainer
#11 lore raw

Re: [PATCH v5 03/15] sched/core: uclamp: map TASK's clamp values into CPU's clamp groups

Quoted 9 lines · Patrick Bellasi
Quoted 7 lines · Peter Zijlstra

include/linux/sched.h struct sched_dl_entity { · line 610

Quoted 3 lines · Patrick Bellasi
include/linux/sched.h +3 -0
+struct uclamp_se {
+	unsigned int value		: SCHED_CAPACITY_SHIFT + 1;
+	unsigned int group_id		: order_base_2(UCLAMP_GROUPS);

Are you sure about ob2() ? seems weird we'll end up with 0 for UCLAMP_GROUPS==1.

Quoted 8 lines

So, we have:

   #define UCLAMP_GROUPS (CONFIG_UCLAMP_GROUPS_COUNT + 1)

because one clamp group is always reserved for defaults. Thus, ob2(in) is always called with n>=2.

... should be safe no ?

+config UCLAMP_GROUPS_COUNT

+       int "Number of different utilization clamp values supported"
+       range 0 32
+       default 5

0+1 == 1

Increase the min range and you should be good I think.

#12 lore raw

Re: [PATCH v5 03/15] sched/core: uclamp: map TASK's clamp values into CPU's clamp groups

Quoted 24 lines · Peter Zijlstra
Quoted 17 lines · Patrick Bellasi
Quoted 7 lines · Peter Zijlstra

include/linux/sched.h struct sched_dl_entity { · line 610

Quoted 3 lines · Patrick Bellasi
include/linux/sched.h +3 -0
+struct uclamp_se {
+	unsigned int value		: SCHED_CAPACITY_SHIFT + 1;
+	unsigned int group_id		: order_base_2(UCLAMP_GROUPS);

Are you sure about ob2() ? seems weird we'll end up with 0 for UCLAMP_GROUPS==1.

So, we have:

   #define UCLAMP_GROUPS (CONFIG_UCLAMP_GROUPS_COUNT + 1)

because one clamp group is always reserved for defaults. Thus, ob2(in) is always called with n>=2.

... should be safe no ?

init/Kconfig +4 -0
+config UCLAMP_GROUPS_COUNT
+       int "Number of different utilization clamp values supported"
+       range 0 32
+       default 5
Quoted 1 line

0+1 == 1

Seems so... :)

Quoted 1 line

Increase the min range and you should be good I think.

... dunno why I was absolutely convinced that was already 1, since 0 group does not make a lot of sense. :/

Signature · 3 lines
#include <best/regards.h>

Patrick Bellasi
maintainer
#13 lore raw

Re: [PATCH v5 03/15] sched/core: uclamp: map TASK's clamp values into CPU's clamp groups

In reply to Patrick Bellasi
Quoted 6 lines · Patrick Bellasi
kernel/sched/core.c +5 -0
+static void uclamp_group_put(unsigned int clamp_id, unsigned int group_id)
 {
+	union uclamp_map *uc_maps = &uclamp_maps[clamp_id][0];
+	union uclamp_map uc_map_old, uc_map_new;
+	long res;
+

kernel/sched/core.c static void set_load_weight(struct task_struct *p, bool update_load) · line 816

Quoted 10 lines
kernel/sched/core.c +10 -0
+retry:
+
+	uc_map_old.data = atomic_long_read(&uc_maps[group_id].adata);
+	uc_map_new = uc_map_old;
+	uc_map_new.se_count -= 1;
+	res = atomic_long_cmpxchg(&uc_maps[group_id].adata,
+				  uc_map_old.data, uc_map_new.data);
+	if (res != uc_map_old.data)
+		goto retry;
+}

Please write cmpxchg loops in the form:

	atomic_long_t *ptr = &uclamp_maps[clamp_id][group_id].adata;
	union uclamp_map old, new;

	old.data = atomic_long_read(ptr);
	do {
		new.data = old.data;
		new.se_cound--;
	} while (!atomic_long_try_cmpxchg(ptr, &old.data, new.data));

(same for all the others of course)

#14 lore raw

Re: [PATCH v5 03/15] sched/core: uclamp: map TASK's clamp values into CPU's clamp groups

Quoted 31 lines · Peter Zijlstra
Quoted 6 lines · Patrick Bellasi
kernel/sched/core.c +5 -0
+static void uclamp_group_put(unsigned int clamp_id, unsigned int group_id)
 {
+	union uclamp_map *uc_maps = &uclamp_maps[clamp_id][0];
+	union uclamp_map uc_map_old, uc_map_new;
+	long res;
+

kernel/sched/core.c static void set_load_weight(struct task_struct *p, bool update_load) · line 816

Quoted 10 lines
kernel/sched/core.c +10 -0
+retry:
+
+	uc_map_old.data = atomic_long_read(&uc_maps[group_id].adata);
+	uc_map_new = uc_map_old;
+	uc_map_new.se_count -= 1;
+	res = atomic_long_cmpxchg(&uc_maps[group_id].adata,
+				  uc_map_old.data, uc_map_new.data);
+	if (res != uc_map_old.data)
+		goto retry;
+}

Please write cmpxchg loops in the form:

	atomic_long_t *ptr = &uclamp_maps[clamp_id][group_id].adata;
	union uclamp_map old, new;

	old.data = atomic_long_read(ptr);
	do {
		new.data = old.data;
		new.se_cound--;
	} while (!atomic_long_try_cmpxchg(ptr, &old.data, new.data));
Quoted 1 line

(same for all the others of course)

Ok, I did that to save some indentation, but actually it's most commonly used in a while loop... will update in v6.

Out of curiosity, apart from code consistency, is that required also specifically for any possible compiler related (mis)behavior ?

Signature · 3 lines
#include <best/regards.h>

Patrick Bellasi
maintainer
#15 lore raw

Re: [PATCH v5 03/15] sched/core: uclamp: map TASK's clamp values into CPU's clamp groups

Quoted 14 lines · Patrick Bellasi
Quoted 12 lines · Peter Zijlstra

Please write cmpxchg loops in the form:

	atomic_long_t *ptr = &uclamp_maps[clamp_id][group_id].adata;
	union uclamp_map old, new;

	old.data = atomic_long_read(ptr);
	do {
		new.data = old.data;
		new.se_cound--;
	} while (!atomic_long_try_cmpxchg(ptr, &old.data, new.data));
Quoted 7 lines
Quoted 1 line

(same for all the others of course)

Ok, I did that to save some indentation, but actually it's most commonly used in a while loop... will update in v6.

Out of curiosity, apart from code consistency, is that required also specifically for any possible compiler related (mis)behavior ?

No; it is just the 'normal' form my brain likes :-)

And the try_cmpxchg() thing is slightly more efficient on x86 vs the traditional form:

	while (cmpxchg(ptr, old, new) != old)
maintainer
#16 lore raw

Re: [PATCH v5 03/15] sched/core: uclamp: map TASK's clamp values into CPU's clamp groups

In reply to Patrick Bellasi
Quoted 33 lines · Patrick Bellasi
kernel/sched/core.c +33 -0
+/**
+ * uclamp_group_get: increase the reference count for a clamp group
+ * @uc_se: the utilization clamp data for the task
+ * @clamp_id: the clamp index affected by the task
+ * @clamp_value: the new clamp value for the task
+ *
+ * Each time a task changes its utilization clamp value, for a specified clamp
+ * index, we need to find an available clamp group which can be used to track
+ * this new clamp value. The corresponding clamp group index will be used to
+ * reference count the corresponding clamp value while the task is enqueued on
+ * a CPU.
+ */
+static void uclamp_group_get(struct uclamp_se *uc_se, unsigned int clamp_id,
+			     unsigned int clamp_value)
+{
+	union uclamp_map *uc_maps = &uclamp_maps[clamp_id][0];
+	unsigned int prev_group_id = uc_se->group_id;
+	union uclamp_map uc_map_old, uc_map_new;
+	unsigned int free_group_id;
+	unsigned int group_id;
+	unsigned long res;
+
+retry:
+
+	free_group_id = UCLAMP_GROUPS;
+	for (group_id = 0; group_id < UCLAMP_GROUPS; ++group_id) {
+		uc_map_old.data = atomic_long_read(&uc_maps[group_id].adata);
+		if (free_group_id == UCLAMP_GROUPS && !uc_map_old.se_count)
+			free_group_id = group_id;
+		if (uc_map_old.value == clamp_value)
+			break;
+	}
+	if (group_id >= UCLAMP_GROUPS) {

kernel/sched/core.c static void set_load_weight(struct task_struct *p, bool update_load) · line 860

Quoted 10 lines
kernel/sched/core.c +10 -0
+#ifdef CONFIG_SCHED_DEBUG
+#define UCLAMP_MAPERR "clamp value [%u] mapping to clamp group failed\n"
+		if (unlikely(free_group_id == UCLAMP_GROUPS)) {
+			pr_err_ratelimited(UCLAMP_MAPERR, clamp_value);
+			return;
+		}
+#endif
+		group_id = free_group_id;
+		uc_map_old.data = atomic_long_read(&uc_maps[group_id].adata);
+	}

You forgot to check for refcount overflow here ;-)

And I'm not really a fan of hiding that error in a define like you keep doing.

What's wrong with something like:

	if (SCHED_WARN(free_group_id == UCLAMP_GROUPS))
		return;

and

kernel/sched/core.c static void set_load_weight(struct task_struct *p, bool update_load) · line 862

Quoted 1 line
kernel/sched/core.c +1 -0
+	uc_map_new.se_count = uc_map_old.se_count + 1;
	if (SCHED_WARN(!new.se_count))
		new.se_count = -1;
Quoted 15 lines
kernel/sched/core.c +15 -0
+	uc_map_new.value = clamp_value;
+	res = atomic_long_cmpxchg(&uc_maps[group_id].adata,
+				  uc_map_old.data, uc_map_new.data);
+	if (res != uc_map_old.data)
+		goto retry;
+
+	/* Update SE's clamp values and attach it to new clamp group */
+	uc_se->value = clamp_value;
+	uc_se->group_id = group_id;
+
+	/* Release the previous clamp group */
+	if (uc_se->mapped)
+		uclamp_group_put(clamp_id, prev_group_id);
+	uc_se->mapped = true;
+}
#17 lore raw

Re: [PATCH v5 03/15] sched/core: uclamp: map TASK's clamp values into CPU's clamp groups

Quoted 40 lines · Peter Zijlstra
Quoted 8 lines · Patrick Bellasi

+/**
+ * uclamp_group_get: increase the reference count for a clamp group
+ * @uc_se: the utilization clamp data for the task
+ * @clamp_id: the clamp index affected by the task
+ * @clamp_value: the new clamp value for the task
+ *
+ * Each time a task changes its utilization clamp value, for a specified clamp
+ * index, we need to find an available clamp group which can be used to track

You mean se_count overflow ?

+ * this new clamp value. The corresponding clamp group index will be used to

Quoted 28 lines

+ * reference count the corresponding clamp value while the task is enqueued on
+ * a CPU.
+ */

kernel/sched/core.c +25 -0
+static void uclamp_group_get(struct uclamp_se *uc_se, unsigned int clamp_id,
+			     unsigned int clamp_value)
+{
+	union uclamp_map *uc_maps = &uclamp_maps[clamp_id][0];
+	unsigned int prev_group_id = uc_se->group_id;
+	union uclamp_map uc_map_old, uc_map_new;
+	unsigned int free_group_id;
+	unsigned int group_id;
+	unsigned long res;
+
+retry:
+
+	free_group_id = UCLAMP_GROUPS;
+	for (group_id = 0; group_id < UCLAMP_GROUPS; ++group_id) {
+		uc_map_old.data = atomic_long_read(&uc_maps[group_id].adata);
+		if (free_group_id == UCLAMP_GROUPS && !uc_map_old.se_count)
+			free_group_id = group_id;
+		if (uc_map_old.value == clamp_value)
+			break;
+	}
+	if (group_id >= UCLAMP_GROUPS) {
+#ifdef CONFIG_SCHED_DEBUG
+#define UCLAMP_MAPERR "clamp value [%u] mapping to clamp group failed\n"
+		if (unlikely(free_group_id == UCLAMP_GROUPS)) {
+			pr_err_ratelimited(UCLAMP_MAPERR, clamp_value);
Quoted 8 lines

kernel/sched/core.c static void set_load_weight(struct task_struct *p, bool update_load) · line 860

Quoted 6 lines
kernel/sched/core.c +6 -0
+			return;
+		}
+#endif
+		group_id = free_group_id;
+		uc_map_old.data = atomic_long_read(&uc_maps[group_id].adata);
+	}

You forgot to check for refcount overflow here ;-)

You mean se_count overflow ?

That se_count is (BITS_PER_LONG - SCHED_CAPACITY_SHIFT - 1) which makes it able to track up to:

+2mln tasks/task_groups on 32bit systems (with SCHED_CAPACITY_SHIFT 10) +10^12 tasks/task_groups on 64bit systems (with SCHED_CAPACITY_SHIFT 20)

I don't expect overflow on 64bit systems, do we ?

It's more likely on 32bit systems, especially if in the future we should increase SCHED_CAPACITY_SHIFT.

Quoted 2 lines

And I'm not really a fan of hiding that error in a define like you keep doing.

The #define is not there to mask an overflow, it's there to catch the case in which the refcount should be corrupted and we end up violating the invariant: "there is always a clamp group available".

NOTE: that invariant is granted once we add

sched/core: uclamp: add clamp group bucketing support

The warning reports the issue only on CONFIG_SCHED_DEBUG, but... it makes sense to keep it always enabled.

While, in case of data corruption, we should just return thus not setting the scheduling entity as "mapped" towards the end of the function... which makes me see that it's actually wrong to conditionally compile the above "return"

Quoted 4 lines

What's wrong with something like:

	if (SCHED_WARN(free_group_id == UCLAMP_GROUPS))
		return;

Right, the flow should be:

1. try to find a valid clamp group
2. if you don't find one, the data structures are corrupted warn once for data corruption
do not map this scheduling entity and return
3. map the scheduling entity

Is that ok ?

Quoted 6 lines

and

kernel/sched/core.c static void set_load_weight(struct task_struct *p, bool update_load) · line 862

Quoted 1 line
kernel/sched/core.c +1 -0
+	uc_map_new.se_count = uc_map_old.se_count + 1;
	if (SCHED_WARN(!new.se_count))
		new.se_count = -1;

Mmm... not sure we can recover from a corrupted refcount or an overflow.

What should we do on these cases, disable uclamp completely ?

Quoted 15 lines
Quoted 15 lines
kernel/sched/core.c +15 -0
+	uc_map_new.value = clamp_value;
+	res = atomic_long_cmpxchg(&uc_maps[group_id].adata,
+				  uc_map_old.data, uc_map_new.data);
+	if (res != uc_map_old.data)
+		goto retry;
+
+	/* Update SE's clamp values and attach it to new clamp group */
+	uc_se->value = clamp_value;
+	uc_se->group_id = group_id;
+
+	/* Release the previous clamp group */
+	if (uc_se->mapped)
+		uclamp_group_put(clamp_id, prev_group_id);
+	uc_se->mapped = true;
+}
Signature · 3 lines
#include <best/regards.h>

Patrick Bellasi
maintainer
#18 lore raw

Re: [PATCH v5 03/15] sched/core: uclamp: map TASK's clamp values into CPU's clamp groups

Quoted 2 lines · Patrick Bellasi

On 07-Nov 14:35, Peter Zijlstra wrote:
You mean se_count overflow ?

Yah..

Quoted 4 lines
Quoted 2 lines

And I'm not really a fan of hiding that error in a define like you keep doing.

The #define is not there to mask an overflow, it's there to catch the

+#define UCLAMP_MAPERR "clamp value [%u] mapping to clamp group failed\n"

Is what I was talking about.

Quoted 7 lines
Quoted 4 lines

What's wrong with something like:

	if (SCHED_WARN(free_group_id == UCLAMP_GROUPS))
		return;

Right, the flow should be:

Quoted 7 lines

1. try to find a valid clamp group
2. if you don't find one, the data structures are corrupted warn once for data corruption
do not map this scheduling entity and return
3. map the scheduling entity

Is that ok ?

That's what the proposed does.

Quoted 11 lines
Quoted 6 lines

and

kernel/sched/core.c static void set_load_weight(struct task_struct *p, bool update_load) · line 862

Quoted 1 line
kernel/sched/core.c +1 -0
+	uc_map_new.se_count = uc_map_old.se_count + 1;
	if (SCHED_WARN(!new.se_count))
		new.se_count = -1;

Mmm... not sure we can recover from a corrupted refcount or an overflow.

What should we do on these cases, disable uclamp completely ?

You can teach put to never decrement -1 (aka. all 1s).

But its all SCHED_DEBUG stuff anyway, so who really cares.

#19 lore raw

Re: [PATCH v5 03/15] sched/core: uclamp: map TASK's clamp values into CPU's clamp groups

Quoted 39 lines · Peter Zijlstra
Quoted 2 lines · Patrick Bellasi

On 07-Nov 14:35, Peter Zijlstra wrote:
You mean se_count overflow ?

Yah..

Quoted 4 lines
Quoted 2 lines

And I'm not really a fan of hiding that error in a define like you keep doing.

The #define is not there to mask an overflow, it's there to catch the

+#define UCLAMP_MAPERR "clamp value [%u] mapping to clamp group failed\n"

Is what I was talking about.

Quoted 7 lines
Quoted 4 lines

What's wrong with something like:

	if (SCHED_WARN(free_group_id == UCLAMP_GROUPS))
		return;

Right, the flow should be:

Quoted 7 lines

1. try to find a valid clamp group
2. if you don't find one, the data structures are corrupted warn once for data corruption
do not map this scheduling entity and return
3. map the scheduling entity

Is that ok ?

That's what the proposed does.

Quoted 7 lines
Quoted 6 lines

and

kernel/sched/core.c static void set_load_weight(struct task_struct *p, bool update_load) · line 862

Quoted 1 line
kernel/sched/core.c +1 -0
+	uc_map_new.se_count = uc_map_old.se_count + 1;
	if (SCHED_WARN(!new.se_count))
		new.se_count = -1;
Quoted 6 lines
Quoted 4 lines

Mmm... not sure we can recover from a corrupted refcount or an overflow.

What should we do on these cases, disable uclamp completely ?

You can teach put to never decrement -1 (aka. all 1s).

Still we don't know when re-enable -1 again...

But, with bucketization, this will eventually turns into a small performance penalty at run-time when a CPU clamp group is going to be empty (since we will end up scanning an array with some holes to find out the new max)... maybe still acceptable...

Will look into that for v6!

Thanks

Quoted 1 line

But its all SCHED_DEBUG stuff anyway, so who really cares.

Signature · 3 lines
#include <best/regards.h>

Patrick Bellasi
maintainer
#20 lore raw

Re: [PATCH v5 03/15] sched/core: uclamp: map TASK's clamp values into CPU's clamp groups

In reply to Patrick Bellasi
Quoted 30 lines · Patrick Bellasi
kernel/sched/core.c +30 -0
+/**
+ * uclamp_group_get: increase the reference count for a clamp group
+ * @uc_se: the utilization clamp data for the task
+ * @clamp_id: the clamp index affected by the task
+ * @clamp_value: the new clamp value for the task
+ *
+ * Each time a task changes its utilization clamp value, for a specified clamp
+ * index, we need to find an available clamp group which can be used to track
+ * this new clamp value. The corresponding clamp group index will be used to
+ * reference count the corresponding clamp value while the task is enqueued on
+ * a CPU.
+ */
+static void uclamp_group_get(struct uclamp_se *uc_se, unsigned int clamp_id,
+			     unsigned int clamp_value)
+{
+	union uclamp_map *uc_maps = &uclamp_maps[clamp_id][0];
+	unsigned int prev_group_id = uc_se->group_id;
+	union uclamp_map uc_map_old, uc_map_new;
+	unsigned int free_group_id;
+	unsigned int group_id;
+	unsigned long res;
+
+retry:
+
+	free_group_id = UCLAMP_GROUPS;
+	for (group_id = 0; group_id < UCLAMP_GROUPS; ++group_id) {
+		uc_map_old.data = atomic_long_read(&uc_maps[group_id].adata);
+		if (free_group_id == UCLAMP_GROUPS && !uc_map_old.se_count)
+			free_group_id = group_id;
+		if (uc_map_old.value == clamp_value)

kernel/sched/core.c static void set_load_weight(struct task_struct *p, bool update_load) · line 857

Quoted 10 lines
kernel/sched/core.c +10 -0
+			break;
+	}
+	if (group_id >= UCLAMP_GROUPS) {
+#ifdef CONFIG_SCHED_DEBUG
+#define UCLAMP_MAPERR "clamp value [%u] mapping to clamp group failed\n"
+		if (unlikely(free_group_id == UCLAMP_GROUPS)) {
+			pr_err_ratelimited(UCLAMP_MAPERR, clamp_value);
+			return;
+		}
+#endif

Can you please put in a comment, either here or on top, on why this can not in fact happen? And we're always guaranteed a free group.

Quoted 20 lines
kernel/sched/core.c +20 -0
+		group_id = free_group_id;
+		uc_map_old.data = atomic_long_read(&uc_maps[group_id].adata);
+	}
+
+	uc_map_new.se_count = uc_map_old.se_count + 1;
+	uc_map_new.value = clamp_value;
+	res = atomic_long_cmpxchg(&uc_maps[group_id].adata,
+				  uc_map_old.data, uc_map_new.data);
+	if (res != uc_map_old.data)
+		goto retry;
+
+	/* Update SE's clamp values and attach it to new clamp group */
+	uc_se->value = clamp_value;
+	uc_se->group_id = group_id;
+
+	/* Release the previous clamp group */
+	if (uc_se->mapped)
+		uclamp_group_put(clamp_id, prev_group_id);
+	uc_se->mapped = true;
+}
#21 lore raw

Re: [PATCH v5 03/15] sched/core: uclamp: map TASK's clamp values into CPU's clamp groups

Quoted 36 lines · Peter Zijlstra
Quoted 35 lines · Patrick Bellasi
kernel/sched/core.c +35 -0
+/**
+ * uclamp_group_get: increase the reference count for a clamp group
+ * @uc_se: the utilization clamp data for the task
+ * @clamp_id: the clamp index affected by the task
+ * @clamp_value: the new clamp value for the task
+ *
+ * Each time a task changes its utilization clamp value, for a specified clamp
+ * index, we need to find an available clamp group which can be used to track
+ * this new clamp value. The corresponding clamp group index will be used to
+ * reference count the corresponding clamp value while the task is enqueued on
+ * a CPU.
+ */
+static void uclamp_group_get(struct uclamp_se *uc_se, unsigned int clamp_id,
+			     unsigned int clamp_value)
+{
+	union uclamp_map *uc_maps = &uclamp_maps[clamp_id][0];
+	unsigned int prev_group_id = uc_se->group_id;
+	union uclamp_map uc_map_old, uc_map_new;
+	unsigned int free_group_id;
+	unsigned int group_id;
+	unsigned long res;
+
+retry:
+
+	free_group_id = UCLAMP_GROUPS;
+	for (group_id = 0; group_id < UCLAMP_GROUPS; ++group_id) {
+		uc_map_old.data = atomic_long_read(&uc_maps[group_id].adata);
+		if (free_group_id == UCLAMP_GROUPS && !uc_map_old.se_count)
+			free_group_id = group_id;
+		if (uc_map_old.value == clamp_value)
+			break;
+	}
+	if (group_id >= UCLAMP_GROUPS) {
+#ifdef CONFIG_SCHED_DEBUG
+#define UCLAMP_MAPERR "clamp value [%u] mapping to clamp group failed\n"
Quoted 8 lines

kernel/sched/core.c static void set_load_weight(struct task_struct *p, bool update_load) · line 857

Quoted 5 lines
kernel/sched/core.c +5 -0
+		if (unlikely(free_group_id == UCLAMP_GROUPS)) {
+			pr_err_ratelimited(UCLAMP_MAPERR, clamp_value);
+			return;
+		}
+#endif

Can you please put in a comment, either here or on top, on why this can not in fact happen? And we're always guaranteed a free group.

You right, that's confusing especially because up to this point we are not granted. We are always granted a free group once we add:

sched/core: uclamp: add clamp group bucketing support

I've kept it separated to better document how we introduce that support.

Is it ok for for you if I better call out in the change log that the guarantee comes from a following patch... and add the comment in that later patch ?

Quoted 21 lines
Quoted 21 lines
kernel/sched/core.c +20 -0
+		group_id = free_group_id;
+		uc_map_old.data = atomic_long_read(&uc_maps[group_id].adata);
+	}
+
+	uc_map_new.se_count = uc_map_old.se_count + 1;
+	uc_map_new.value = clamp_value;
+	res = atomic_long_cmpxchg(&uc_maps[group_id].adata,
+				  uc_map_old.data, uc_map_new.data);
+	if (res != uc_map_old.data)
+		goto retry;
+
+	/* Update SE's clamp values and attach it to new clamp group */
+	uc_se->value = clamp_value;
+	uc_se->group_id = group_id;
+
+	/* Release the previous clamp group */
+	if (uc_se->mapped)
+		uclamp_group_put(clamp_id, prev_group_id);
+	uc_se->mapped = true;
+}
Signature · 3 lines
#include <best/regards.h>

Patrick Bellasi
maintainer
#22 lore raw

Re: [PATCH v5 03/15] sched/core: uclamp: map TASK's clamp values into CPU's clamp groups

Quoted 38 lines · Patrick Bellasi
Quoted 25 lines · Peter Zijlstra
Quoted 23 lines · Patrick Bellasi
kernel/sched/core.c +23 -0
+static void uclamp_group_get(struct uclamp_se *uc_se, unsigned int clamp_id,
+			     unsigned int clamp_value)
+{
+	union uclamp_map *uc_maps = &uclamp_maps[clamp_id][0];
+	unsigned int prev_group_id = uc_se->group_id;
+	union uclamp_map uc_map_old, uc_map_new;
+	unsigned int free_group_id;
+	unsigned int group_id;
+	unsigned long res;
+
+retry:
+
+	free_group_id = UCLAMP_GROUPS;
+	for (group_id = 0; group_id < UCLAMP_GROUPS; ++group_id) {
+		uc_map_old.data = atomic_long_read(&uc_maps[group_id].adata);
+		if (free_group_id == UCLAMP_GROUPS && !uc_map_old.se_count)
+			free_group_id = group_id;
+		if (uc_map_old.value == clamp_value)
+			break;
+	}
+	if (group_id >= UCLAMP_GROUPS) {
+#ifdef CONFIG_SCHED_DEBUG
+#define UCLAMP_MAPERR "clamp value [%u] mapping to clamp group failed\n"
Quoted 8 lines

kernel/sched/core.c static void set_load_weight(struct task_struct *p, bool update_load) · line 857

Quoted 5 lines
kernel/sched/core.c +5 -0
+		if (unlikely(free_group_id == UCLAMP_GROUPS)) {
+			pr_err_ratelimited(UCLAMP_MAPERR, clamp_value);
+			return;
+		}
+#endif

Can you please put in a comment, either here or on top, on why this can not in fact happen? And we're always guaranteed a free group.

You right, that's confusing especially because up to this point we are not granted. We are always granted a free group once we add:

Quoted 8 lines

sched/core: uclamp: add clamp group bucketing support

I've kept it separated to better document how we introduce that support.

Is it ok for for you if I better call out in the change log that the guarantee comes from a following patch... and add the comment in that later patch ?

Urgh.. that is mighty confusing and since this stuff actually 'works' might result in bisection issues too, right?

I would really rather prefer a series that makes sense in the order you read it.

#23 lore raw

Re: [PATCH v5 03/15] sched/core: uclamp: map TASK's clamp values into CPU's clamp groups

Quoted 44 lines · Peter Zijlstra
Quoted 43 lines · Patrick Bellasi
Quoted 25 lines · Peter Zijlstra
Quoted 23 lines · Patrick Bellasi
kernel/sched/core.c +23 -0
+static void uclamp_group_get(struct uclamp_se *uc_se, unsigned int clamp_id,
+			     unsigned int clamp_value)
+{
+	union uclamp_map *uc_maps = &uclamp_maps[clamp_id][0];
+	unsigned int prev_group_id = uc_se->group_id;
+	union uclamp_map uc_map_old, uc_map_new;
+	unsigned int free_group_id;
+	unsigned int group_id;
+	unsigned long res;
+
+retry:
+
+	free_group_id = UCLAMP_GROUPS;
+	for (group_id = 0; group_id < UCLAMP_GROUPS; ++group_id) {
+		uc_map_old.data = atomic_long_read(&uc_maps[group_id].adata);
+		if (free_group_id == UCLAMP_GROUPS && !uc_map_old.se_count)
+			free_group_id = group_id;
+		if (uc_map_old.value == clamp_value)
+			break;
+	}
+	if (group_id >= UCLAMP_GROUPS) {
+#ifdef CONFIG_SCHED_DEBUG
+#define UCLAMP_MAPERR "clamp value [%u] mapping to clamp group failed\n"
Quoted 8 lines

kernel/sched/core.c static void set_load_weight(struct task_struct *p, bool update_load) · line 857

Quoted 5 lines
kernel/sched/core.c +5 -0
+		if (unlikely(free_group_id == UCLAMP_GROUPS)) {
+			pr_err_ratelimited(UCLAMP_MAPERR, clamp_value);
+			return;
+		}
+#endif

Can you please put in a comment, either here or on top, on why this can not in fact happen? And we're always guaranteed a free group.

You right, that's confusing especially because up to this point we are not granted. We are always granted a free group once we add:

sched/core: uclamp: add clamp group bucketing support

I've kept it separated to better document how we introduce that support.

Quoted 6 lines
Quoted 3 lines

Is it ok for for you if I better call out in the change log that the guarantee comes from a following patch... and add the comment in that later patch ?

Urgh.. that is mighty confusing and since this stuff actually 'works' might result in bisection issues too, right?

True...

Quoted 2 lines

I would really rather prefer a series that makes sense in the order you read it.

... yes, bisects can be a problem, if we run functional tests on them.

Ok, let see what you think about the bucketing support and then we can see if it's better to keep them separate by adding here some check to remove after... or just squash them in since the beginning.

Signature · 3 lines
#include <best/regards.h>

Patrick Bellasi
maintainer
#24 lore raw

Re: [PATCH v5 03/15] sched/core: uclamp: map TASK's clamp values into CPU's clamp groups

In reply to Patrick Bellasi
Quoted 23 lines · Patrick Bellasi
kernel/sched/core.c +20 -0
+static int __setscheduler_uclamp(struct task_struct *p,
+				 const struct sched_attr *attr)
+{
+	unsigned int lower_bound = p->uclamp[UCLAMP_MIN].value;
+	unsigned int upper_bound = p->uclamp[UCLAMP_MAX].value;
+	int result = 0;
+
+	if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP_MIN)
+		lower_bound = attr->sched_util_min;
+
+	if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP_MAX)
+		upper_bound = attr->sched_util_max;
+
+	if (lower_bound > upper_bound ||
+	    upper_bound > SCHED_CAPACITY_SCALE)
 		return -EINVAL;
 
+	mutex_lock(&uclamp_mutex);
 
+	/* Update each required clamp group */
+	if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP_MIN) {
+		uclamp_group_get(&p->uclamp[UCLAMP_MIN],
+				 UCLAMP_MIN, lower_bound);

kernel/sched/core.c static void set_load_weight(struct task_struct *p, bool update_load) · line 911

Quoted 10 lines
kernel/sched/core.c +10 -0
+	}
+	if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP_MAX) {
+		uclamp_group_get(&p->uclamp[UCLAMP_MAX],
+				 UCLAMP_MAX, upper_bound);
+	}
+
+	mutex_unlock(&uclamp_mutex);
+
+	return result;
+}

I'm missing where we error due to lack of groups.

#25 lore raw

Re: [PATCH v5 03/15] sched/core: uclamp: map TASK's clamp values into CPU's clamp groups

Quoted 20 lines · Peter Zijlstra
Quoted 19 lines · Patrick Bellasi
kernel/sched/core.c +16 -0
+static int __setscheduler_uclamp(struct task_struct *p,
+				 const struct sched_attr *attr)
+{
+	unsigned int lower_bound = p->uclamp[UCLAMP_MIN].value;
+	unsigned int upper_bound = p->uclamp[UCLAMP_MAX].value;
+	int result = 0;
+
+	if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP_MIN)
+		lower_bound = attr->sched_util_min;
+
+	if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP_MAX)
+		upper_bound = attr->sched_util_max;
+
+	if (lower_bound > upper_bound ||
+	    upper_bound > SCHED_CAPACITY_SCALE)
 		return -EINVAL;
 
+	mutex_lock(&uclamp_mutex);
Quoted 16 lines
Quoted 4 lines
kernel/sched/core.c +4 -0
+	/* Update each required clamp group */
+	if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP_MIN) {
+		uclamp_group_get(&p->uclamp[UCLAMP_MIN],
+				 UCLAMP_MIN, lower_bound);

kernel/sched/core.c static void set_load_weight(struct task_struct *p, bool update_load) · line 911

Quoted 10 lines
kernel/sched/core.c +10 -0
+	}
+	if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP_MAX) {
+		uclamp_group_get(&p->uclamp[UCLAMP_MAX],
+				 UCLAMP_MAX, upper_bound);
+	}
+
+	mutex_unlock(&uclamp_mutex);
+
+	return result;
+}

I'm missing where we error due to lack of groups.

Again, after:

sched/core: uclamp: add clamp group bucketing support

we should never error.

Perhaps I should really squash the bucketing in these first patches directly... lemme see what's your take on those bits, if you like them better not in a separate patch, I'll squash them on v6.

Signature · 3 lines
#include <best/regards.h>

Patrick Bellasi
#26 lore raw

PATCH v5 4/15 sched/core: uclamp: add CPU's clamp groups accounting

In reply to Patrick Bellasi

Utilization clamping allows to clamp the utilization of a CPU within a [util_min, util_max] range which depends on the set of currently RUNNABLE tasks on that CPU.
Each task references two "clamp groups" defining the minimum and maximum utilization clamp values to be considered for that task. These clamp value are mapped by a clamp group which is enforced on a CPU only when there is at least one RUNNABLE task referencing that clamp group.

When tasks are enqueued/dequeued on/from a CPU, the set of clamp groups active on that CPU can change. Since each clamp group enforces a different utilization clamp value, once the set of these groups changes it's required to re-compute what is the new "aggregated" clamp value to apply on that CPU.

Clamp values are always MAX aggregated for both util_min and util_max. This is to ensure that no tasks can affect the performance of other co-scheduled tasks which are either more boosted (i.e. with higher util_min clamp) or less capped (i.e. with higher util_max clamp).

Here we introduce the required support to properly reference count clamp groups at each task enqueue/dequeue time.

Tasks have a:
task_struct::uclamp::group_id[clamp_idx]
indexing, for each clamp index (i.e. util_{min,max}), the clamp group they have to refcount at enqueue time.

CPUs rq have a:
rq::uclamp::group[clamp_idx][group_idx].tasks
which is used to reference count how many tasks are currently RUNNABLE on that CPU for each clamp group of each clamp index.

The clamp value of each clamp group is tracked by
rq::uclamp::group[][].value
thus making rq::uclamp::group[][] an unordered array of clamp values. However, the MAX aggregation of the currently active clamp groups is implemented to minimize the number of times we need to scan the complete (unordered) clamp group array to figure out the new max value. This operation indeed happens only when we dequeue the last task of the clamp group corresponding to the current max clamp, and thus the CPU is either entering IDLE or going to schedule a less boosted or more clamped task. Moreover, the expected number of different clamp values, which can be configured at build time, is usually so small that a more advanced ordering algorithm is not needed. In real use-cases we expect less then 10 different clamp values for each clamp index.

Cc Ingo Molnar <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Paul Turner <[email protected]>
Cc Suren Baghdasaryan <[email protected]>
Cc Todd Kjos <[email protected]>
Cc Joel Fernandes <[email protected]>
Cc Juri Lelli <[email protected]>
Cc Quentin Perret <[email protected]>
Cc Dietmar Eggemann <[email protected]>
Cc Morten Rasmussen <[email protected]>

---
Changes in v5:
Message-ID: <20180914134128.GP1413@e110439-lin>
- remove not required check for (group_id == UCLAMP_NOT_VALID) in uclamp_cpu_put_id
Message-ID: <20180912174456.GJ1413@e110439-lin>
- use bitfields to compress uclamp_group
Others:
- consistently use "unsigned int" for both clamp_id and group_id
- fixup documentation
- reduced usage of inline comments
- rebased on v4.19.0

Changes in v4:
Message-ID: <20180816133249.GA2964@e110439-lin>
- keep the WARN in uclamp_cpu_put_id() but beautify a bit that code
- add another WARN on the unexpected condition of releasing a refcount from a CPU which has a lower clamp value active
Other:
- ensure (and check) that all tasks have a valid group_id at uclamp_cpu_get_id()
- rework uclamp_cpu layout to better fit into just 2x64B cache lines
- fix some s/SCHED_DEBUG/CONFIG_SCHED_DEBUG/
- rebased on v4.19-rc1
Changes in v3:
Message-ID: <CAJuCfpF6=L=0LrmNnJrTNPazT4dWKqNv+thhN0dwpKCgUzs9sg@mail.gmail.com>
- add WARN on unlikely un-referenced decrement in uclamp_cpu_put_id()
- rename UCLAMP_NONE into UCLAMP_NOT_VALID
Message-ID: <CAJuCfpGaKvxKcO=RLcmveHRB9qbMrvFs2yFVrk=k-v_m7JkxwQ@mail.gmail.com>
- few typos fixed
Other:
- rebased on tip/sched/core
Changes in v2:
Message-ID: <[email protected]>
- refactored struct rq::uclamp_cpu to be more cache efficient
no more holes, re-arranged vectors to match cache lines with expected data locality
Message-ID: <[email protected]>
- use *rq as parameter whenever already available
- add scheduling class's uclamp_enabled marker
- get rid of the "confusing" single callback uclamp_task_update() and use uclamp_cpu_{get,put}() directly from {en,de}queue_task()
- fix/remove "bad" comments
Message-ID: <20180413113337.GU14248@e110439-lin>
- remove inline from init_uclamp, flag it __init
Other:
- rabased on v4.18-rc4
- improved documentation to make more explicit some concepts.

---
 include/linux/sched.h |   5 ++
 kernel/sched/core.c   | 185 ++++++++++++++++++++++++++++++++++++++++++
 kernel/sched/sched.h  |  49 +++++++++++
 3 files changed, 239 insertions(+)
include/linux/sched.h +5 -0
index facace271ea1..3ab1cbd4e3b1 100644
@@ -604,11 +604,16 @@ struct sched_dl_entity {
  * The mapped bit is set whenever a task has been mapped on a clamp group for
  * the first time. When this bit is set, any clamp group get (for a new clamp
  * value) will be matches by a clamp group put (for the old clamp value).
+ *
+ * The active bit is set whenever a task has got an effective clamp group
+ * and value assigned, which can be different from the user requested ones.
+ * This allows to know a task is actually refcounting a CPU's clamp group.
  */
 struct uclamp_se {
 	unsigned int value		: SCHED_CAPACITY_SHIFT + 1;
 	unsigned int group_id		: order_base_2(UCLAMP_GROUPS);
 	unsigned int mapped		: 1;
+	unsigned int active		: 1;
 };
 #endif /* CONFIG_UCLAMP_TASK */
 
kernel/sched/core.c +185 -0

236 lines not on this page · raw message

kernel/sched/sched.h +49 -0

61 lines not on this page · raw message

Signature · 1 line
2.18.0
#27 lore raw

Re: [PATCH v5 04/15] sched/core: uclamp: add CPU's clamp groups accounting

Slightly older version posted by error along with the correct one. Please comment on:

Message-ID: <[email protected]>

Sorry for the noise.

Quoted 453 lines · Patrick Bellasi

Utilization clamping allows to clamp the utilization of a CPU within a [util_min, util_max] range which depends on the set of currently RUNNABLE tasks on that CPU.
Each task references two "clamp groups" defining the minimum and maximum utilization clamp values to be considered for that task. These clamp value are mapped by a clamp group which is enforced on a CPU only when there is at least one RUNNABLE task referencing that clamp group.

When tasks are enqueued/dequeued on/from a CPU, the set of clamp groups active on that CPU can change. Since each clamp group enforces a different utilization clamp value, once the set of these groups changes it's required to re-compute what is the new "aggregated" clamp value to apply on that CPU.

Clamp values are always MAX aggregated for both util_min and util_max. This is to ensure that no tasks can affect the performance of other co-scheduled tasks which are either more boosted (i.e. with higher util_min clamp) or less capped (i.e. with higher util_max clamp).

Here we introduce the required support to properly reference count clamp groups at each task enqueue/dequeue time.

Tasks have a:
task_struct::uclamp::group_id[clamp_idx]
indexing, for each clamp index (i.e. util_{min,max}), the clamp group they have to refcount at enqueue time.

CPUs rq have a:
rq::uclamp::group[clamp_idx][group_idx].tasks
which is used to reference count how many tasks are currently RUNNABLE on that CPU for each clamp group of each clamp index.

The clamp value of each clamp group is tracked by
rq::uclamp::group[][].value
thus making rq::uclamp::group[][] an unordered array of clamp values. However, the MAX aggregation of the currently active clamp groups is implemented to minimize the number of times we need to scan the complete (unordered) clamp group array to figure out the new max value. This operation indeed happens only when we dequeue the last task of the clamp group corresponding to the current max clamp, and thus the CPU is either entering IDLE or going to schedule a less boosted or more clamped task. Moreover, the expected number of different clamp values, which can be configured at build time, is usually so small that a more advanced ordering algorithm is not needed. In real use-cases we expect less then 10 different clamp values for each clamp index.

Cc Ingo Molnar <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Paul Turner <[email protected]>
Cc Suren Baghdasaryan <[email protected]>
Cc Todd Kjos <[email protected]>
Cc Joel Fernandes <[email protected]>
Cc Juri Lelli <[email protected]>
Cc Quentin Perret <[email protected]>
Cc Dietmar Eggemann <[email protected]>
Cc Morten Rasmussen <[email protected]>

---
Changes in v5:
Message-ID: <20180914134128.GP1413@e110439-lin>
- remove not required check for (group_id == UCLAMP_NOT_VALID) in uclamp_cpu_put_id
Message-ID: <20180912174456.GJ1413@e110439-lin>
- use bitfields to compress uclamp_group
Others:
- consistently use "unsigned int" for both clamp_id and group_id
- fixup documentation
- reduced usage of inline comments
- rebased on v4.19.0

Changes in v4:
Message-ID: <20180816133249.GA2964@e110439-lin>
- keep the WARN in uclamp_cpu_put_id() but beautify a bit that code
- add another WARN on the unexpected condition of releasing a refcount from a CPU which has a lower clamp value active
Other:
- ensure (and check) that all tasks have a valid group_id at uclamp_cpu_get_id()
- rework uclamp_cpu layout to better fit into just 2x64B cache lines
- fix some s/SCHED_DEBUG/CONFIG_SCHED_DEBUG/
- rebased on v4.19-rc1
Changes in v3:
Message-ID: <CAJuCfpF6=L=0LrmNnJrTNPazT4dWKqNv+thhN0dwpKCgUzs9sg@mail.gmail.com>
- add WARN on unlikely un-referenced decrement in uclamp_cpu_put_id()
- rename UCLAMP_NONE into UCLAMP_NOT_VALID
Message-ID: <CAJuCfpGaKvxKcO=RLcmveHRB9qbMrvFs2yFVrk=k-v_m7JkxwQ@mail.gmail.com>
- few typos fixed
Other:
- rebased on tip/sched/core
Changes in v2:
Message-ID: <[email protected]>
- refactored struct rq::uclamp_cpu to be more cache efficient
no more holes, re-arranged vectors to match cache lines with expected data locality
Message-ID: <[email protected]>
- use *rq as parameter whenever already available
- add scheduling class's uclamp_enabled marker
- get rid of the "confusing" single callback uclamp_task_update() and use uclamp_cpu_{get,put}() directly from {en,de}queue_task()
- fix/remove "bad" comments
Message-ID: <20180413113337.GU14248@e110439-lin>
- remove inline from init_uclamp, flag it __init
Other:
- rabased on v4.18-rc4
- improved documentation to make more explicit some concepts.

---
 include/linux/sched.h |   5 ++
 kernel/sched/core.c   | 185 ++++++++++++++++++++++++++++++++++++++++++
 kernel/sched/sched.h  |  49 +++++++++++
 3 files changed, 239 insertions(+)
include/linux/sched.h +5 -0
index facace271ea1..3ab1cbd4e3b1 100644
@@ -604,11 +604,16 @@ struct sched_dl_entity {
  * The mapped bit is set whenever a task has been mapped on a clamp group for
  * the first time. When this bit is set, any clamp group get (for a new clamp
  * value) will be matches by a clamp group put (for the old clamp value).
+ *
+ * The active bit is set whenever a task has got an effective clamp group
+ * and value assigned, which can be different from the user requested ones.
+ * This allows to know a task is actually refcounting a CPU's clamp group.
  */
 struct uclamp_se {
 	unsigned int value		: SCHED_CAPACITY_SHIFT + 1;
 	unsigned int group_id		: order_base_2(UCLAMP_GROUPS);
 	unsigned int mapped		: 1;
+	unsigned int active		: 1;
 };
 #endif /* CONFIG_UCLAMP_TASK */
 
kernel/sched/core.c +185 -0
index 654327d7f212..a98a96a7d9f1 100644
@@ -783,6 +783,159 @@ union uclamp_map {
  */
 static union uclamp_map uclamp_maps[UCLAMP_CNT][UCLAMP_GROUPS];
 
+/**
+ * uclamp_cpu_update: updates the utilization clamp of a CPU
+ * @rq: the CPU's rq which utilization clamp has to be updated
+ * @clamp_id: the clamp index to update
+ *
+ * When tasks are enqueued/dequeued on/from a CPU, the set of currently active
+ * clamp groups can change. Since each clamp group enforces a different
+ * utilization clamp value, once the set of active groups changes it can be
+ * required to re-compute what is the new clamp value to apply for that CPU.
+ *
+ * For the specified clamp index, this method computes the new CPU utilization
+ * clamp to use until the next change on the set of active clamp groups.
+ */
+static inline void uclamp_cpu_update(struct rq *rq, unsigned int clamp_id)
+{
+	unsigned int group_id;
+	int max_value = 0;
+
+	for (group_id = 0; group_id < UCLAMP_GROUPS; ++group_id) {
+		if (!rq->uclamp.group[clamp_id][group_id].tasks)
+			continue;
+		/* Both min and max clamps are MAX aggregated */
+		if (max_value < rq->uclamp.group[clamp_id][group_id].value)
+			max_value = rq->uclamp.group[clamp_id][group_id].value;
+		if (max_value >= SCHED_CAPACITY_SCALE)
+			break;
+	}
+	rq->uclamp.value[clamp_id] = max_value;
+}
+
+/**
+ * uclamp_cpu_get_id(): increase reference count for a clamp group on a CPU
+ * @p: the task being enqueued on a CPU
+ * @rq: the CPU's rq where the clamp group has to be reference counted
+ * @clamp_id: the clamp index to update
+ *
+ * Once a task is enqueued on a CPU's rq, the clamp group currently defined by
+ * the task's uclamp::group_id is reference counted on that CPU.
+ */
+static inline void uclamp_cpu_get_id(struct task_struct *p, struct rq *rq,
+				     unsigned int clamp_id)
+{
+	unsigned int group_id;
+
+	if (unlikely(!p->uclamp[clamp_id].mapped))
+		return;
+
+	group_id = p->uclamp[clamp_id].group_id;
+	p->uclamp[clamp_id].active = true;
+
+	rq->uclamp.group[clamp_id][group_id].tasks += 1;
+
+	if (rq->uclamp.value[clamp_id] < p->uclamp[clamp_id].value)
+		rq->uclamp.value[clamp_id] = p->uclamp[clamp_id].value;
+}
+
+/**
+ * uclamp_cpu_put_id(): decrease reference count for a clamp group on a CPU
+ * @p: the task being dequeued from a CPU
+ * @rq: the CPU's rq from where the clamp group has to be released
+ * @clamp_id: the clamp index to update
+ *
+ * When a task is dequeued from a CPU's rq, the CPU's clamp group reference
+ * counted by the task is released.
+ * If this was the last task reference coutning the current max clamp group,
+ * then the CPU clamping is updated to find the new max for the specified
+ * clamp index.
+ */
+static inline void uclamp_cpu_put_id(struct task_struct *p, struct rq *rq,
+				     unsigned int clamp_id)
+{
+	unsigned int clamp_value;
+	unsigned int group_id;
+
+	if (unlikely(!p->uclamp[clamp_id].mapped))
+		return;
+
+	group_id = p->uclamp[clamp_id].group_id;
+	p->uclamp[clamp_id].active = false;
+
+	if (likely(rq->uclamp.group[clamp_id][group_id].tasks))
+		rq->uclamp.group[clamp_id][group_id].tasks -= 1;
+#ifdef CONFIG_SCHED_DEBUG
+	else {
+		WARN(1, "invalid CPU[%d] clamp group [%u:%u] refcount\n",
+		     cpu_of(rq), clamp_id, group_id);
+	}
+#endif
+
+	if (likely(rq->uclamp.group[clamp_id][group_id].tasks))
+		return;
+
+	clamp_value = rq->uclamp.group[clamp_id][group_id].value;
+#ifdef CONFIG_SCHED_DEBUG
+	if (unlikely(clamp_value > rq->uclamp.value[clamp_id])) {
+		WARN(1, "invalid CPU[%d] clamp group [%u:%u] value\n",
+		     cpu_of(rq), clamp_id, group_id);
+	}
+#endif
+	if (clamp_value >= rq->uclamp.value[clamp_id])
+		uclamp_cpu_update(rq, clamp_id);
+}
+
+/**
+ * uclamp_cpu_get(): increase CPU's clamp group refcount
+ * @rq: the CPU's rq where the task is enqueued
+ * @p: the task being enqueued
+ *
+ * When a task is enqueued on a CPU's rq, all the clamp groups currently
+ * enforced on a task are reference counted on that rq. Since not all
+ * scheduling classes have utilization clamping support, their tasks will
+ * be silently ignored.
+ *
+ * This method updates the utilization clamp constraints considering the
+ * requirements for the specified task. Thus, this update must be done before
+ * calling into the scheduling classes, which will eventually update schedutil
+ * considering the new task requirements.
+ */
+static inline void uclamp_cpu_get(struct rq *rq, struct task_struct *p)
+{
+	unsigned int clamp_id;
+
+	if (unlikely(!p->sched_class->uclamp_enabled))
+		return;
+
+	for (clamp_id = 0; clamp_id < UCLAMP_CNT; ++clamp_id)
+		uclamp_cpu_get_id(p, rq, clamp_id);
+}
+
+/**
+ * uclamp_cpu_put(): decrease CPU's clamp group refcount
+ * @rq: the CPU's rq from where the task is dequeued
+ * @p: the task being dequeued
+ *
+ * When a task is dequeued from a CPU's rq, all the clamp groups the task has
+ * reference counted at enqueue time are now released.
+ *
+ * This method updates the utilization clamp constraints considering the
+ * requirements for the specified task. Thus, this update must be done before
+ * calling into the scheduling classes, which will eventually update schedutil
+ * considering the new task requirements.
+ */
+static inline void uclamp_cpu_put(struct rq *rq, struct task_struct *p)
+{
+	unsigned int clamp_id;
+
+	if (unlikely(!p->sched_class->uclamp_enabled))
+		return;
+
+	for (clamp_id = 0; clamp_id < UCLAMP_CNT; ++clamp_id)
+		uclamp_cpu_put_id(p, rq, clamp_id);
+}
+
 /**
  * uclamp_group_put: decrease the reference count for a clamp group
  * @clamp_id: the clamp index which was affected by a task group
@@ -836,6 +989,7 @@ static void uclamp_group_get(struct uclamp_se *uc_se, unsigned int clamp_id,
 	unsigned int free_group_id;
 	unsigned int group_id;
 	unsigned long res;
+	int cpu;
 
 retry:
 
@@ -866,6 +1020,28 @@ static void uclamp_group_get(struct uclamp_se *uc_se, unsigned int clamp_id,
 	if (res != uc_map_old.data)
 		goto retry;
 
+	/* Ensure each CPU tracks the correct value for this clamp group */
+	if (likely(uc_map_new.se_count > 1))
+		goto done;
+	for_each_possible_cpu(cpu) {
+		struct uclamp_cpu *uc_cpu = &cpu_rq(cpu)->uclamp;
+
+		/* Refcounting is expected to be always 0 for free groups */
+		if (unlikely(uc_cpu->group[clamp_id][group_id].tasks)) {
+			uc_cpu->group[clamp_id][group_id].tasks = 0;
+#ifdef CONFIG_SCHED_DEBUG
+			WARN(1, "invalid CPU[%d] clamp group [%u:%u] refcount\n",
+			     cpu, clamp_id, group_id);
+#endif
+		}
+
+		if (uc_cpu->group[clamp_id][group_id].value == clamp_value)
+			continue;
+		uc_cpu->group[clamp_id][group_id].value = clamp_value;
+	}
+
+done:
+
 	/* Update SE's clamp values and attach it to new clamp group */
 	uc_se->value = clamp_value;
 	uc_se->group_id = group_id;
@@ -948,6 +1124,7 @@ static void uclamp_fork(struct task_struct *p, bool reset)
 			clamp_value = uclamp_none(clamp_id);
 
 		p->uclamp[clamp_id].mapped = false;
+		p->uclamp[clamp_id].active = false;
 		uclamp_group_get(&p->uclamp[clamp_id], clamp_id, clamp_value);
 	}
 }
@@ -959,9 +1136,13 @@ static void __init init_uclamp(void)
 {
 	struct uclamp_se *uc_se;
 	unsigned int clamp_id;
+	int cpu;
 
 	mutex_init(&uclamp_mutex);
 
+	for_each_possible_cpu(cpu)
+		memset(&cpu_rq(cpu)->uclamp, 0, sizeof(struct uclamp_cpu));
+
 	memset(uclamp_maps, 0, sizeof(uclamp_maps));
 	for (clamp_id = 0; clamp_id < UCLAMP_CNT; ++clamp_id) {
 		uc_se = &init_task.uclamp[clamp_id];
@@ -970,6 +1151,8 @@ static void __init init_uclamp(void)
 }
 
 #else /* CONFIG_UCLAMP_TASK */
+static inline void uclamp_cpu_get(struct rq *rq, struct task_struct *p) { }
+static inline void uclamp_cpu_put(struct rq *rq, struct task_struct *p) { }
 static inline int __setscheduler_uclamp(struct task_struct *p,
 					const struct sched_attr *attr)
 {
@@ -987,6 +1170,7 @@ static inline void enqueue_task(struct rq *rq, struct task_struct *p, int flags)
 	if (!(flags & ENQUEUE_RESTORE))
 		sched_info_queued(rq, p);
 
+	uclamp_cpu_get(rq, p);
 	p->sched_class->enqueue_task(rq, p, flags);
 }
 
@@ -998,6 +1182,7 @@ static inline void dequeue_task(struct rq *rq, struct task_struct *p, int flags)
 	if (!(flags & DEQUEUE_SAVE))
 		sched_info_dequeued(rq, p);
 
+	uclamp_cpu_put(rq, p);
 	p->sched_class->dequeue_task(rq, p, flags);
 }
 
kernel/sched/sched.h +49 -0
index 947ab14d3d5b..1755c9c9f4f0 100644
@@ -766,6 +766,50 @@ extern void rto_push_irq_work_func(struct irq_work *work);
 #endif
 #endif /* CONFIG_SMP */
 
+#ifdef CONFIG_UCLAMP_TASK
+/**
+ * struct uclamp_group - Utilization clamp Group
+ * @value: utilization clamp value for tasks on this clamp group
+ * @tasks: number of RUNNABLE tasks on this clamp group
+ *
+ * Keep track of how many tasks are RUNNABLE for a given utilization
+ * clamp value.
+ */
+struct uclamp_group {
+	unsigned long value : SCHED_CAPACITY_SHIFT + 1;
+	unsigned long tasks : BITS_PER_LONG - SCHED_CAPACITY_SHIFT - 1;
+};
+
+/**
+ * struct uclamp_cpu - CPU's utilization clamp
+ * @value: currently active clamp values for a CPU
+ * @group: utilization clamp groups affecting a CPU
+ *
+ * Keep track of RUNNABLE tasks on a CPUs to aggregate their clamp values.
+ * A clamp value is affecting a CPU where there is at least one task RUNNABLE
+ * (or actually running) with that value.
+ *
+ * We have up to UCLAMP_CNT possible different clamp value, which are
+ * currently only two: minmum utilization and maximum utilization.
+ *
+ * All utilization clamping values are MAX aggregated, since:
+ * - for util_min: we want to run the CPU at least at the max of the minimum
+ *   utilization required by its currently RUNNABLE tasks.
+ * - for util_max: we want to allow the CPU to run up to the max of the
+ *   maximum utilization allowed by its currently RUNNABLE tasks.
+ *
+ * Since on each system we expect only a limited number of different
+ * utilization clamp values (CONFIG_UCLAMP_GROUPS_COUNT), we use a simple
+ * array to track the metrics required to compute all the per-CPU utilization
+ * clamp values. The additional slot is used to track the default clamp
+ * values, i.e. no min/max clamping at all.
+ */
+struct uclamp_cpu {
+	struct uclamp_group group[UCLAMP_CNT][CONFIG_UCLAMP_GROUPS_COUNT + 1];
+	int value[UCLAMP_CNT];
+};
+#endif /* CONFIG_UCLAMP_TASK */
+
 /*
  * This is the main, per-CPU runqueue data structure.
  *
@@ -804,6 +848,11 @@ struct rq {
 	unsigned long		nr_load_updates;
 	u64			nr_switches;
 
+#ifdef CONFIG_UCLAMP_TASK
+	/* Utilization clamp values based on CPU's RUNNABLE tasks */
+	struct uclamp_cpu	uclamp ____cacheline_aligned;
+#endif
+
 	struct cfs_rq		cfs;
 	struct rt_rq		rt;
 	struct dl_rq		dl;
Signature · 1 line
2.18.0
Signature · 3 lines
#include <best/regards.h>

Patrick Bellasi
#28 lore raw

PATCH v5 4/15 sched/core: uclamp: add CPU's clamp groups refcounting

In reply to Patrick Bellasi

Utilization clamping allows to clamp the utilization of a CPU within a [util_min, util_max] range which depends on the set of currently RUNNABLE tasks on that CPU.
Each task references two "clamp groups" defining the minimum and maximum utilization "clamp values" to be considered for that task. Each clamp value is mapped to a clamp group which is enforced on a CPU only when there is at least one RUNNABLE task referencing that clamp group.

When tasks are enqueued/dequeued on/from a CPU, the set of clamp groups active on that CPU can change. Since each clamp group enforces a different utilization clamp value, once the set of these groups changes it's required to re-compute what is the new "aggregated" clamp value to apply on that CPU.

Clamp values are always MAX aggregated for both util_min and util_max. This is to ensure that no tasks can affect the performance of other co-scheduled tasks which are either more boosted (i.e. with higher util_min clamp) or less capped (i.e. with higher util_max clamp).

Here we introduce the required support to properly reference count clamp groups at each task enqueue/dequeue time.

Tasks have a:
task_struct::uclamp[clamp_idx]::group_id
indexing, for each "clamp index" (i.e. util_{min,max}), the "group index" of the clamp group they have to refcount at enqueue time.

CPUs rq have a:
rq::uclamp::group[clamp_idx][group_idx].tasks
which is used to reference count how many tasks are currently RUNNABLE on that CPU for each clamp group of each clamp index.

The clamp value of each clamp group is tracked by
rq::uclamp::group[clamp_idx][group_idx].value
thus making rq::uclamp::group[][] an unordered array of clamp values.

The MAX aggregation of the currently active clamp groups is implemented to minimize the number of times we need to scan the complete (unordered) clamp group array to figure out the new max value.
This operation indeed happens only when we dequeue the last task of the clamp group corresponding to the current max clamp, and thus the CPU is either entering IDLE or going to schedule a less boosted or more clamped task. Moreover, the expected number of different clamp values, which can be configured at build time, is usually so small that a more advanced ordering algorithm is not needed. In real use-cases we expect less then 10 different clamp values for each clamp index.

Cc Ingo Molnar <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Paul Turner <[email protected]>
Cc Suren Baghdasaryan <[email protected]>
Cc Todd Kjos <[email protected]>
Cc Joel Fernandes <[email protected]>
Cc Juri Lelli <[email protected]>
Cc Quentin Perret <[email protected]>
Cc Dietmar Eggemann <[email protected]>
Cc Morten Rasmussen <[email protected]>

---
Changes in v5:
Message-ID: <20180914134128.GP1413@e110439-lin>
- remove not required check for (group_id == UCLAMP_NOT_VALID) in uclamp_cpu_put_id
Message-ID: <20180912174456.GJ1413@e110439-lin>
- use bitfields to compress uclamp_group
Others:
- consistently use "unsigned int" for both clamp_id and group_id
- fixup documentation
- reduced usage of inline comments
- rebased on v4.19

Changes in v4:
Message-ID: <20180816133249.GA2964@e110439-lin>
- keep the WARN in uclamp_cpu_put_id() but beautify a bit that code
- add another WARN on the unexpected condition of releasing a refcount from a CPU which has a lower clamp value active
Other:
- ensure (and check) that all tasks have a valid group_id at uclamp_cpu_get_id()
- rework uclamp_cpu layout to better fit into just 2x64B cache lines
- fix some s/SCHED_DEBUG/CONFIG_SCHED_DEBUG/
- rebased on v4.19-rc1
Changes in v3:
Message-ID: <CAJuCfpF6=L=0LrmNnJrTNPazT4dWKqNv+thhN0dwpKCgUzs9sg@mail.gmail.com>
- add WARN on unlikely un-referenced decrement in uclamp_cpu_put_id()
- rename UCLAMP_NONE into UCLAMP_NOT_VALID
Message-ID: <CAJuCfpGaKvxKcO=RLcmveHRB9qbMrvFs2yFVrk=k-v_m7JkxwQ@mail.gmail.com>
- few typos fixed
Other:
- rebased on tip/sched/core
Changes in v2:
Message-ID: <[email protected]>
- refactored struct rq::uclamp_cpu to be more cache efficient
no more holes, re-arranged vectors to match cache lines with expected data locality
Message-ID: <[email protected]>
- use *rq as parameter whenever already available
- add scheduling class's uclamp_enabled marker
- get rid of the "confusing" single callback uclamp_task_update() and use uclamp_cpu_{get,put}() directly from {en,de}queue_task()
- fix/remove "bad" comments
Message-ID: <20180413113337.GU14248@e110439-lin>
- remove inline from init_uclamp, flag it __init
Other:
- rabased on v4.18-rc4
- improved documentation to make more explicit some concepts.

---
 include/linux/sched.h |   5 ++
 kernel/sched/core.c   | 185 ++++++++++++++++++++++++++++++++++++++++++
 kernel/sched/sched.h  |  49 +++++++++++
 3 files changed, 239 insertions(+)
include/linux/sched.h +5 -0
index facace271ea1..3ab1cbd4e3b1 100644
@@ -604,11 +604,16 @@ struct sched_dl_entity {
  * The mapped bit is set whenever a task has been mapped on a clamp group for
  * the first time. When this bit is set, any clamp group get (for a new clamp
  * value) will be matches by a clamp group put (for the old clamp value).
+ *
+ * The active bit is set whenever a task has got an effective clamp group
+ * and value assigned, which can be different from the user requested ones.
+ * This allows to know a task is actually refcounting a CPU's clamp group.
  */
 struct uclamp_se {
 	unsigned int value		: SCHED_CAPACITY_SHIFT + 1;
 	unsigned int group_id		: order_base_2(UCLAMP_GROUPS);
 	unsigned int mapped		: 1;
+	unsigned int active		: 1;
 };
 #endif /* CONFIG_UCLAMP_TASK */
 
kernel/sched/core.c +185 -0

236 lines not on this page · raw message

kernel/sched/sched.h +49 -0

61 lines not on this page · raw message

Signature · 1 line
2.18.0
maintainer
#29 lore raw

Re: [PATCH v5 04/15] sched/core: uclamp: add CPU's clamp groups refcounting

kernel/sched/core.c union uclamp_map { · line 809

Quoted 11 lines · Patrick Bellasi
kernel/sched/core.c +11 -0
+static inline void uclamp_cpu_update(struct rq *rq, unsigned int clamp_id)
+{
+	unsigned int group_id;
+	int max_value = 0;
+
+	for (group_id = 0; group_id < UCLAMP_GROUPS; ++group_id) {
+		if (!rq->uclamp.group[clamp_id][group_id].tasks)
+			continue;
+		/* Both min and max clamps are MAX aggregated */
+		if (max_value < rq->uclamp.group[clamp_id][group_id].value)
+			max_value = rq->uclamp.group[clamp_id][group_id].value;
		max_value = max(max_value, rq->uclamp.group[clamp_id][group_id].value);
Quoted 17 lines
kernel/sched/core.c +17 -0
+		if (max_value >= SCHED_CAPACITY_SCALE)
+			break;
+	}
+	rq->uclamp.value[clamp_id] = max_value;
+}
+
+/**
+ * uclamp_cpu_get_id(): increase reference count for a clamp group on a CPU
+ * @p: the task being enqueued on a CPU
+ * @rq: the CPU's rq where the clamp group has to be reference counted
+ * @clamp_id: the clamp index to update
+ *
+ * Once a task is enqueued on a CPU's rq, the clamp group currently defined by
+ * the task's uclamp::group_id is reference counted on that CPU.
+ */
+static inline void uclamp_cpu_get_id(struct task_struct *p, struct rq *rq,
+				     unsigned int clamp_id)

kernel/sched/core.c union uclamp_map { · line 836

Quoted 10 lines
kernel/sched/core.c +10 -0
+{
+	unsigned int group_id;
+
+	if (unlikely(!p->uclamp[clamp_id].mapped))
+		return;
+
+	group_id = p->uclamp[clamp_id].group_id;
+	p->uclamp[clamp_id].active = true;
+
+	rq->uclamp.group[clamp_id][group_id].tasks += 1;
	++

kernel/sched/core.c union uclamp_map { · line 839

Quoted 3 lines
kernel/sched/core.c +3 -0
+
+	if (rq->uclamp.value[clamp_id] < p->uclamp[clamp_id].value)
+		rq->uclamp.value[clamp_id] = p->uclamp[clamp_id].value;
	rq->uclamp.value[clamp_id] = max(rq->uclamp.value[clamp_id],
					 p->uclamp[clamp_id].value);
Quoted 16 lines
kernel/sched/core.c +16 -0
+}
+
+/**
+ * uclamp_cpu_put_id(): decrease reference count for a clamp group on a CPU
+ * @p: the task being dequeued from a CPU
+ * @rq: the CPU's rq from where the clamp group has to be released
+ * @clamp_id: the clamp index to update
+ *
+ * When a task is dequeued from a CPU's rq, the CPU's clamp group reference
+ * counted by the task is released.
+ * If this was the last task reference coutning the current max clamp group,
+ * then the CPU clamping is updated to find the new max for the specified
+ * clamp index.
+ */
+static inline void uclamp_cpu_put_id(struct task_struct *p, struct rq *rq,
+				     unsigned int clamp_id)

kernel/sched/core.c union uclamp_map { · line 865

Quoted 10 lines
kernel/sched/core.c +10 -0
+{
+	unsigned int clamp_value;
+	unsigned int group_id;
+
+	if (unlikely(!p->uclamp[clamp_id].mapped))
+		return;
+
+	group_id = p->uclamp[clamp_id].group_id;
+	p->uclamp[clamp_id].active = false;
+
	SCHED_WARN_ON(!rq->uclamp.group[clamp_id][group_id].tasks);

kernel/sched/core.c union uclamp_map { · line 867

Quoted 2 lines
kernel/sched/core.c +2 -0
+	if (likely(rq->uclamp.group[clamp_id][group_id].tasks))
+		rq->uclamp.group[clamp_id][group_id].tasks -= 1;
	--
Quoted 8 lines
kernel/sched/core.c +7 -0
+#ifdef CONFIG_SCHED_DEBUG
+	else {
+		WARN(1, "invalid CPU[%d] clamp group [%u:%u] refcount\n",
+		     cpu_of(rq), clamp_id, group_id);
+	}
+#endif
 
+

kernel/sched/core.c union uclamp_map { · line 884

Quoted 10 lines
kernel/sched/core.c +10 -0
+	if (likely(rq->uclamp.group[clamp_id][group_id].tasks))
+		return;
+
+	clamp_value = rq->uclamp.group[clamp_id][group_id].value;
+#ifdef CONFIG_SCHED_DEBUG
+	if (unlikely(clamp_value > rq->uclamp.value[clamp_id])) {
+		WARN(1, "invalid CPU[%d] clamp group [%u:%u] value\n",
+		     cpu_of(rq), clamp_id, group_id);
+	}
+#endif
	SCHED_WARN_ON(clamp_value > rq->uclamp.value[clamp_id]);

kernel/sched/core.c static void uclamp_group_get(struct uclamp_se *uc_se, unsigned int clamp_id, · line 1026

Quoted 12 lines
kernel/sched/core.c +7 -0
+	if (clamp_value >= rq->uclamp.value[clamp_id])
+		uclamp_cpu_update(rq, clamp_id);
+}
@@ -866,6 +1020,28 @@ static void uclamp_group_get(struct uclamp_se *uc_se, unsigned int clamp_id,
 	if (res != uc_map_old.data)
 		goto retry;
 
+	/* Ensure each CPU tracks the correct value for this clamp group */
+	if (likely(uc_map_new.se_count > 1))
+		goto done;
+	for_each_possible_cpu(cpu) {

yuck yuck yuck.. why!?

kernel/sched/core.c static void uclamp_group_get(struct uclamp_se *uc_se, unsigned int clamp_id, · line 1035

Quoted 9 lines
kernel/sched/core.c +9 -0
+		struct uclamp_cpu *uc_cpu = &cpu_rq(cpu)->uclamp;
+
+		/* Refcounting is expected to be always 0 for free groups */
+		if (unlikely(uc_cpu->group[clamp_id][group_id].tasks)) {
+			uc_cpu->group[clamp_id][group_id].tasks = 0;
+#ifdef CONFIG_SCHED_DEBUG
+			WARN(1, "invalid CPU[%d] clamp group [%u:%u] refcount\n",
+			     cpu, clamp_id, group_id);
+#endif
		SCHED_WARN_ON();
Quoted 12 lines
kernel/sched/core.c +9 -0
+		}
+
+		if (uc_cpu->group[clamp_id][group_id].value == clamp_value)
+			continue;
+		uc_cpu->group[clamp_id][group_id].value = clamp_value;
+	}
+
+done:
+
 	/* Update SE's clamp values and attach it to new clamp group */
 	uc_se->value = clamp_value;
 	uc_se->group_id = group_id;
#30 lore raw

Re: [PATCH v5 04/15] sched/core: uclamp: add CPU's clamp groups refcounting

Quoted 14 lines · Peter Zijlstra

kernel/sched/core.c union uclamp_map { · line 809

Quoted 11 lines · Patrick Bellasi
kernel/sched/core.c +11 -0
+static inline void uclamp_cpu_update(struct rq *rq, unsigned int clamp_id)
+{
+	unsigned int group_id;
+	int max_value = 0;
+
+	for (group_id = 0; group_id < UCLAMP_GROUPS; ++group_id) {
+		if (!rq->uclamp.group[clamp_id][group_id].tasks)
+			continue;
+		/* Both min and max clamps are MAX aggregated */
+		if (max_value < rq->uclamp.group[clamp_id][group_id].value)
+			max_value = rq->uclamp.group[clamp_id][group_id].value;
		max_value = max(max_value, rq->uclamp.group[clamp_id][group_id].value);

Right, I get used to this pattern to avoid write instructions. I guess that here, being just a function local variable, we don't really care much...

Quoted 29 lines
Quoted 17 lines
kernel/sched/core.c +17 -0
+		if (max_value >= SCHED_CAPACITY_SCALE)
+			break;
+	}
+	rq->uclamp.value[clamp_id] = max_value;
+}
+
+/**
+ * uclamp_cpu_get_id(): increase reference count for a clamp group on a CPU
+ * @p: the task being enqueued on a CPU
+ * @rq: the CPU's rq where the clamp group has to be reference counted
+ * @clamp_id: the clamp index to update
+ *
+ * Once a task is enqueued on a CPU's rq, the clamp group currently defined by
+ * the task's uclamp::group_id is reference counted on that CPU.
+ */
+static inline void uclamp_cpu_get_id(struct task_struct *p, struct rq *rq,
+				     unsigned int clamp_id)

kernel/sched/core.c union uclamp_map { · line 836

Quoted 10 lines
kernel/sched/core.c +10 -0
+{
+	unsigned int group_id;
+
+	if (unlikely(!p->uclamp[clamp_id].mapped))
+		return;
+
+	group_id = p->uclamp[clamp_id].group_id;
+	p->uclamp[clamp_id].active = true;
+
+	rq->uclamp.group[clamp_id][group_id].tasks += 1;
	++
Quoted 6 lines

kernel/sched/core.c union uclamp_map { · line 839

Quoted 3 lines
kernel/sched/core.c +3 -0
+
+	if (rq->uclamp.value[clamp_id] < p->uclamp[clamp_id].value)
+		rq->uclamp.value[clamp_id] = p->uclamp[clamp_id].value;
	rq->uclamp.value[clamp_id] = max(rq->uclamp.value[clamp_id],
					 p->uclamp[clamp_id].value);

In this case instead, since we are updating a variable visible from other CPUs, should not be preferred to avoid assignment when not required ?

Is the compiler is smart enough to optimize the code above? ... will check better.

Quoted 62 lines
Quoted 16 lines
kernel/sched/core.c +16 -0
+}
+
+/**
+ * uclamp_cpu_put_id(): decrease reference count for a clamp group on a CPU
+ * @p: the task being dequeued from a CPU
+ * @rq: the CPU's rq from where the clamp group has to be released
+ * @clamp_id: the clamp index to update
+ *
+ * When a task is dequeued from a CPU's rq, the CPU's clamp group reference
+ * counted by the task is released.
+ * If this was the last task reference coutning the current max clamp group,
+ * then the CPU clamping is updated to find the new max for the specified
+ * clamp index.
+ */
+static inline void uclamp_cpu_put_id(struct task_struct *p, struct rq *rq,
+				     unsigned int clamp_id)

kernel/sched/core.c union uclamp_map { · line 865

Quoted 10 lines
kernel/sched/core.c +10 -0
+{
+	unsigned int clamp_value;
+	unsigned int group_id;
+
+	if (unlikely(!p->uclamp[clamp_id].mapped))
+		return;
+
+	group_id = p->uclamp[clamp_id].group_id;
+	p->uclamp[clamp_id].active = false;
+
	SCHED_WARN_ON(!rq->uclamp.group[clamp_id][group_id].tasks);

kernel/sched/core.c union uclamp_map { · line 867

Quoted 2 lines
kernel/sched/core.c +2 -0
+	if (likely(rq->uclamp.group[clamp_id][group_id].tasks))
+		rq->uclamp.group[clamp_id][group_id].tasks -= 1;
	--
Quoted 8 lines
kernel/sched/core.c +7 -0
+#ifdef CONFIG_SCHED_DEBUG
+	else {
+		WARN(1, "invalid CPU[%d] clamp group [%u:%u] refcount\n",
+		     cpu_of(rq), clamp_id, group_id);
+	}
+#endif
 
+

kernel/sched/core.c union uclamp_map { · line 884

Quoted 10 lines
kernel/sched/core.c +10 -0
+	if (likely(rq->uclamp.group[clamp_id][group_id].tasks))
+		return;
+
+	clamp_value = rq->uclamp.group[clamp_id][group_id].value;
+#ifdef CONFIG_SCHED_DEBUG
+	if (unlikely(clamp_value > rq->uclamp.value[clamp_id])) {
+		WARN(1, "invalid CPU[%d] clamp group [%u:%u] value\n",
+		     cpu_of(rq), clamp_id, group_id);
+	}
+#endif
	SCHED_WARN_ON(clamp_value > rq->uclamp.value[clamp_id]);
Quoted 8 lines
kernel/sched/core.c +3 -0
+	if (clamp_value >= rq->uclamp.value[clamp_id])
+		uclamp_cpu_update(rq, clamp_id);
+}
@@ -866,6 +1020,28 @@ static void uclamp_group_get(struct uclamp_se *uc_se, unsigned int clamp_id,
 	if (res != uc_map_old.data)
 		goto retry;
 
Quoted 6 lines

kernel/sched/core.c static void uclamp_group_get(struct uclamp_se *uc_se, unsigned int clamp_id, · line 1026

Quoted 4 lines
kernel/sched/core.c +4 -0
+	/* Ensure each CPU tracks the correct value for this clamp group */
+	if (likely(uc_map_new.se_count > 1))
+		goto done;
+	for_each_possible_cpu(cpu) {

yuck yuck yuck.. why!?

When a clamp group is released, i.e. no more SEs refcount it, that group could be mapped in the future to a different clamp value.

Thus, when this happens, a different clamp value can be assigned to that clamp group and we need to update the value tracked in the CPU side data structures. That's the value actually used to figure out the min/max clamps at enqueue/dequeue times.

However, since here we are in the slow-path, this should not be an issue, isn't it ?

Quoted 25 lines

kernel/sched/core.c static void uclamp_group_get(struct uclamp_se *uc_se, unsigned int clamp_id, · line 1035

Quoted 10 lines
kernel/sched/core.c +9 -0
+		struct uclamp_cpu *uc_cpu = &cpu_rq(cpu)->uclamp;
+
+		/* Refcounting is expected to be always 0 for free groups */
+		if (unlikely(uc_cpu->group[clamp_id][group_id].tasks)) {
+			uc_cpu->group[clamp_id][group_id].tasks = 0;
+#ifdef CONFIG_SCHED_DEBUG
+			WARN(1, "invalid CPU[%d] clamp group [%u:%u] refcount\n",
+			     cpu, clamp_id, group_id);
+#endif
		SCHED_WARN_ON();
Quoted 12 lines
kernel/sched/core.c +9 -0
+		}
+
+		if (uc_cpu->group[clamp_id][group_id].value == clamp_value)
+			continue;
+		uc_cpu->group[clamp_id][group_id].value = clamp_value;
+	}
+
+done:
+
 	/* Update SE's clamp values and attach it to new clamp group */
 	uc_se->value = clamp_value;
 	uc_se->group_id = group_id;
Signature · 3 lines
#include <best/regards.h>

Patrick Bellasi
#31 lore raw

Re: [PATCH v5 04/15] sched/core: uclamp: add CPU's clamp groups refcounting

Quoted 2 lines · Patrick Bellasi
Quoted 1 line · Peter Zijlstra

On Mon, Oct 29, 2018 at 06:32:59PM +0000, Patrick Bellasi wrote:

[...]

Quoted 9 lines
Quoted 5 lines

kernel/sched/core.c union uclamp_map { · line 809

Quoted 3 lines
kernel/sched/core.c +3 -0
+		/* Both min and max clamps are MAX aggregated */
+		if (max_value < rq->uclamp.group[clamp_id][group_id].value)
+			max_value = rq->uclamp.group[clamp_id][group_id].value;
		max_value = max(max_value, rq->uclamp.group[clamp_id][group_id].value);

Right, I get used to this pattern to avoid write instructions. I guess that here, being just a function local variable, we don't really care much...

The above does not work also because we now use bitfields:

In file included from ./include/linux/list.h:9:0,

                 from ./include/linux/rculist.h:10,
                 from ./include/linux/pid.h:5,
                 from ./include/linux/sched.h:14,
                 from kernel/sched/sched.h:5,
                 from kernel/sched/core.c:8:
   kernel/sched/core.c: In function ‘uclamp_cpu_update’:
   kernel/sched/core.c:867:5: error: ‘typeof’ applied to a bit-field
           rq->uclamp.group[clamp_id][group_id].value);
           ^

[...]

Quoted 9 lines
Quoted 5 lines

kernel/sched/core.c union uclamp_map { · line 839

Quoted 2 lines
kernel/sched/core.c +2 -0
+	if (rq->uclamp.value[clamp_id] < p->uclamp[clamp_id].value)
+		rq->uclamp.value[clamp_id] = p->uclamp[clamp_id].value;
	rq->uclamp.value[clamp_id] = max(rq->uclamp.value[clamp_id],
					 p->uclamp[clamp_id].value);

In this case instead, since we are updating a variable visible from other CPUs, should not be preferred to avoid assignment when not required ?

And what about this ?

Quoted 2 lines

Is the compiler is smart enough to optimize the code above? ... will check better.

Did not really checked what the compiler does in the two cases but, given also the above, for consistency I would probably prefer to keep both max aggregation as originally defined.

What do you think ?

Signature · 3 lines
#include <best/regards.h>

Patrick Bellasi
#32 lore raw

PATCH v5 5/15 sched/core: uclamp: update CPU's refcount on clamp changes

In reply to Patrick Bellasi

Utilization clamp values enforced on a CPU by a task can be updated at run-time, for example via a sched_setattr syscall, while a task is currently RUNNABLE on that CPU. In these cases, the task is already refcounting a clamp group for its CPU and thus we need to update this reference to ensure the new constraints are immediately enforced.

Since a clamp value change always implies a clamp group refcount update, this patch hooks into uclamp_group_get() to trigger a CPU refcount syncup, via uclamp_cpu_{put,get}_id(), whenever a task is RUNNABLE.

Cc Ingo Molnar <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Paul Turner <[email protected]>
Cc Suren Baghdasaryan <[email protected]>
Cc Todd Kjos <[email protected]>
Cc Joel Fernandes <[email protected]>
Cc Juri Lelli <[email protected]>
Cc Quentin Perret <[email protected]>
Cc Dietmar Eggemann <[email protected]>
Cc Morten Rasmussen <[email protected]>

---
Changes in v5:
Message-ID: <20180914134128.GP1413@e110439-lin>
- remove not required uclamp_task_affects() since now all tasks always have a valid clamp group assigned
Others:
- consistently use "unsigned int" for both clamp_id and group_id
- fixup documentation
- reduced usage of inline comments
- rebased on v4.19

Changes in v4:
Message-ID: <20180816132249.GA2960@e110439-lin>
- inline uclamp_task_active() code into uclamp_task_update_active()
- get rid of the now unused uclamp_task_active()
Other:
- allow to call uclamp_group_get() without a task pointer, which is used to refcount the initial clamp group for all the global objects (init_task, root_task_group and system_defaults)
- rebased on v4.19-rc1
Changes in v3:
Message-ID: <CAJuCfpF6=L=0LrmNnJrTNPazT4dWKqNv+thhN0dwpKCgUzs9sg@mail.gmail.com>
- rename UCLAMP_NONE into UCLAMP_NOT_VALID
Other:
- rabased on tip/sched/core
Changes in v2:
Message-ID: <[email protected]>
- get rid of the group_id back annotation
which is not requires at this stage where we have only per-task clamping support. It will be introduce later when CGroups support is added.
Other:
- rabased on v4.18-rc4
- this code has been split from a previous patch to simplify the review

---
 kernel/sched/core.c | 60 ++++++++++++++++++++++++++++++++++++++++-----
 1 file changed, 54 insertions(+), 6 deletions(-)
kernel/sched/core.c +54 -6
index a98a96a7d9f1..21f6251a1d44 100644
@@ -936,6 +936,48 @@ static inline void uclamp_cpu_put(struct rq *rq, struct task_struct *p)
 		uclamp_cpu_put_id(p, rq, clamp_id);
 }
 
+/**
+ * uclamp_task_update_active: update the clamp group of a RUNNABLE task
+ * @p: the task which clamp groups must be updated
+ * @clamp_id: the clamp index to consider
+ *
+ * Each time the clamp value of a task group is changed, the old and new clamp
+ * groups must be updated for each CPU containing a RUNNABLE task belonging to
+ * that task group. Sleeping tasks are not updated since they will be enqueued
+ * with the proper clamp group index at their next activation.
+ */
+static inline void
+uclamp_task_update_active(struct task_struct *p, unsigned int clamp_id)
+{
+	struct rq_flags rf;
+	struct rq *rq;
+
+	/*
+	 * Lock the task and the CPU where the task is (or was) queued.
+	 *
+	 * We might lock the (previous) rq of a !RUNNABLE task, but that's the
+	 * price to pay to safely serialize util_{min,max} updates with
+	 * enqueues, dequeues and migration operations.
+	 * This is the same locking schema used by __set_cpus_allowed_ptr().
+	 */
+	rq = task_rq_lock(p, &rf);
+
+	/*
+	 * The setting of the clamp group is serialized by task_rq_lock().
+	 * Thus, if the task is not yet RUNNABLE and its task_struct is not
+	 * affecting a valid clamp group, then the next time it's going to be
+	 * enqueued it will already see the updated clamp group value.
+	 */
+	if (!p->uclamp[clamp_id].active)
+		goto done;
+
+	uclamp_cpu_put_id(p, rq, clamp_id);
+	uclamp_cpu_get_id(p, rq, clamp_id);
+
+done:
+	task_rq_unlock(rq, p, &rf);
+}
+
 /**
  * uclamp_group_put: decrease the reference count for a clamp group
  * @clamp_id: the clamp index which was affected by a task group
@@ -970,6 +1012,7 @@ static void uclamp_group_put(unsigned int clamp_id, unsigned int group_id)
 
 /**
  * uclamp_group_get: increase the reference count for a clamp group
+ * @p: the task which clamp value must be tracked
  * @uc_se: the utilization clamp data for the task
  * @clamp_id: the clamp index affected by the task
  * @clamp_value: the new clamp value for the task
@@ -980,8 +1023,8 @@ static void uclamp_group_put(unsigned int clamp_id, unsigned int group_id)
  * reference count the corresponding clamp value while the task is enqueued on
  * a CPU.
  */
-static void uclamp_group_get(struct uclamp_se *uc_se, unsigned int clamp_id,
-			     unsigned int clamp_value)
+static void uclamp_group_get(struct task_struct *p, struct uclamp_se *uc_se,
+			     unsigned int clamp_id, unsigned int clamp_value)
 {
 	union uclamp_map *uc_maps = &uclamp_maps[clamp_id][0];
 	unsigned int prev_group_id = uc_se->group_id;
@@ -1046,6 +1089,10 @@ static void uclamp_group_get(struct uclamp_se *uc_se, unsigned int clamp_id,
 	uc_se->value = clamp_value;
 	uc_se->group_id = group_id;
 
+	/* Update CPU's clamp group refcounts of RUNNABLE task */
+	if (p)
+		uclamp_task_update_active(p, clamp_id);
+
 	/* Release the previous clamp group */
 	if (uc_se->mapped)
 		uclamp_group_put(clamp_id, prev_group_id);
@@ -1073,11 +1120,11 @@ static int __setscheduler_uclamp(struct task_struct *p,
 
 	/* Update each required clamp group */
 	if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP_MIN) {
-		uclamp_group_get(&p->uclamp[UCLAMP_MIN],
+		uclamp_group_get(p, &p->uclamp[UCLAMP_MIN],
 				 UCLAMP_MIN, lower_bound);
 	}
 	if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP_MAX) {
-		uclamp_group_get(&p->uclamp[UCLAMP_MAX],
+		uclamp_group_get(p, &p->uclamp[UCLAMP_MAX],
 				 UCLAMP_MAX, upper_bound);
 	}
 
@@ -1125,7 +1172,8 @@ static void uclamp_fork(struct task_struct *p, bool reset)
 
 		p->uclamp[clamp_id].mapped = false;
 		p->uclamp[clamp_id].active = false;
-		uclamp_group_get(&p->uclamp[clamp_id], clamp_id, clamp_value);
+		uclamp_group_get(NULL, &p->uclamp[clamp_id],
+				 clamp_id, clamp_value);
 	}
 }
 
@@ -1146,7 +1194,7 @@ static void __init init_uclamp(void)
 	memset(uclamp_maps, 0, sizeof(uclamp_maps));
 	for (clamp_id = 0; clamp_id < UCLAMP_CNT; ++clamp_id) {
 		uc_se = &init_task.uclamp[clamp_id];
-		uclamp_group_get(uc_se, clamp_id, uclamp_none(clamp_id));
+		uclamp_group_get(NULL, uc_se, clamp_id, uclamp_none(clamp_id));
 	}
 }
 
Signature · 1 line
2.18.0
#33 lore raw

PATCH v5 6/15 sched/core: uclamp: enforce last task UCLAMP_MAX

In reply to Patrick Bellasi

When a util_max clamped task sleeps, its clamp constraints are removed from the CPU. However, the blocked utilization on that CPU can still be higher than the max clamp value enforced while that task was running.

The release of a util_max clamp when a CPU is going to be idle could thus allow unwanted CPU frequency increases while tasks are not running. This can happen, for example, when a frequency update is triggered from another CPU of the same frequency domain.
In this case, when we aggregate the utilization of all the CPUs in a shared frequency domain, schedutil can still see the full not clamped blocked utilization of all the CPUs and thus, eventually, increase the frequency.

Let's fix this by using:

uclamp_cpu_put_id(UCLAMP_MAX)
uclamp_cpu_update(last_clamp_value)

to detect when a CPU has no more RUNNABLE clamped tasks and to flag this condition. Thus, while a CPU is idle, we can still enforce the last used clamp value for it.

To the contrary, we do not track any UCLAMP_MIN since, while a CPU is idle, we don't want to enforce any minimum frequency. Indeed, in this case, we rely just on the decay of the blocked utilization to smoothly reduce the CPU frequency.

Cc Ingo Molnar <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Rafael J. Wysocki <[email protected]>
Cc Viresh Kumar <[email protected]>
Cc Suren Baghdasaryan <[email protected]>
Cc Todd Kjos <[email protected]>
Cc Joel Fernandes <[email protected]>
Cc Juri Lelli <[email protected]>
Cc Quentin Perret <[email protected]>
Cc Dietmar Eggemann <[email protected]>
Cc Morten Rasmussen <[email protected]>

---
Changes in v5:
Others:
- reduce usage of alias local variables whenever the global ones can still be used without affecting code readability
- reduced usage of inline comments
- rebased on v4.19

Changes in v4:
Message-ID: <20180816172016.GG2960@e110439-lin>
- ensure to always reset clamp holding on wakeup from IDLE
Others:
- rebased on v4.19-rc1
Changes in v3:
Message-ID: <CAJuCfpFnj2g3+ZpR4fP4yqfxs0zd=c-Zehr2XM7m_C+WdL9jNA@mail.gmail.com>
- rename UCLAMP_NONE into UCLAMP_NOT_VALID
Changes in v2:
- rabased on v4.18-rc4
- new patch to improve a specific issue

---
 kernel/sched/core.c  | 41 ++++++++++++++++++++++++++++++++++++++---
 kernel/sched/sched.h | 11 +++++++++++
 2 files changed, 49 insertions(+), 3 deletions(-)
kernel/sched/core.c +38 -3
index 21f6251a1d44..b23f80c07be9 100644
@@ -796,10 +796,11 @@ static union uclamp_map uclamp_maps[UCLAMP_CNT][UCLAMP_GROUPS];
  * For the specified clamp index, this method computes the new CPU utilization
  * clamp to use until the next change on the set of active clamp groups.
  */
-static inline void uclamp_cpu_update(struct rq *rq, unsigned int clamp_id)
+static inline void uclamp_cpu_update(struct rq *rq, unsigned int clamp_id,
+				     unsigned int last_clamp_value)
 {
 	unsigned int group_id;
-	int max_value = 0;
+	int max_value = -1;
 
 	for (group_id = 0; group_id < UCLAMP_GROUPS; ++group_id) {
 		if (!rq->uclamp.group[clamp_id][group_id].tasks)
@@ -810,6 +811,28 @@ static inline void uclamp_cpu_update(struct rq *rq, unsigned int clamp_id)
 		if (max_value >= SCHED_CAPACITY_SCALE)
 			break;
 	}
+
+	/*
+	 * Just for the UCLAMP_MAX value, in case there are no RUNNABLE
+	 * task, we want to keep the CPU clamped to the last task's clamp
+	 * value. This is to avoid frequency spikes to MAX when one CPU, with
+	 * an high blocked utilization, sleeps and another CPU, in the same
+	 * frequency domain, do not see anymore the clamp on the first CPU.
+	 *
+	 * The UCLAMP_FLAG_IDLE is set whenever we detect, from the above
+	 * loop, that there are no more RUNNABLE taks on that CPU.
+	 * In this case we enforce the CPU util_max to that of the last
+	 * dequeued task.
+	 */
+	if (max_value < 0) {
+		if (clamp_id == UCLAMP_MAX) {
+			rq->uclamp.flags |= UCLAMP_FLAG_IDLE;
+			max_value = last_clamp_value;
+		} else {
+			max_value = uclamp_none(UCLAMP_MIN);
+		}
+	}
+
 	rq->uclamp.value[clamp_id] = max_value;
 }
 
@@ -835,6 +858,18 @@ static inline void uclamp_cpu_get_id(struct task_struct *p, struct rq *rq,
 
 	rq->uclamp.group[clamp_id][group_id].tasks += 1;
 
+	if (unlikely(rq->uclamp.flags & UCLAMP_FLAG_IDLE)) {
+		/*
+		 * Reset clamp holds on idle exit.
+		 * This function is called for both UCLAMP_MIN (before) and
+		 * UCLAMP_MAX (after). Let's reset the flag only the second
+		 * once we know that UCLAMP_MIN has been already updated.
+		 */
+		if (clamp_id == UCLAMP_MAX)
+			rq->uclamp.flags &= ~UCLAMP_FLAG_IDLE;
+		rq->uclamp.value[clamp_id] = p->uclamp[clamp_id].value;
+	}
+
 	if (rq->uclamp.value[clamp_id] < p->uclamp[clamp_id].value)
 		rq->uclamp.value[clamp_id] = p->uclamp[clamp_id].value;
 }
@@ -883,7 +918,7 @@ static inline void uclamp_cpu_put_id(struct task_struct *p, struct rq *rq,
 	}
 #endif
 	if (clamp_value >= rq->uclamp.value[clamp_id])
-		uclamp_cpu_update(rq, clamp_id);
+		uclamp_cpu_update(rq, clamp_id, clamp_value);
 }
 
 /**
kernel/sched/sched.h +11 -0
index 94c4f2f410ad..859192ec492c 100644
@@ -807,6 +807,17 @@ struct uclamp_group {
 struct uclamp_cpu {
 	struct uclamp_group group[UCLAMP_CNT][UCLAMP_GROUPS];
 	int value[UCLAMP_CNT];
+/*
+ * Idle clamp holding
+ * Whenever a CPU is idle, we enforce the util_max clamp value of the last
+ * task running on that CPU. This bit is used to flag a clamp holding
+ * currently active for a CPU. This flag is:
+ * - set when we update the clamp value of a CPU at the time of dequeuing the
+ *   last before entering idle
+ * - reset when we enqueue the first task after a CPU wakeup from IDLE
+ */
+#define UCLAMP_FLAG_IDLE 0x01
+	int flags;
 };
 #endif /* CONFIG_UCLAMP_TASK */
 
Signature · 1 line
2.18.0
maintainer
#34 lore raw

Re: [PATCH v5 06/15] sched/core: uclamp: enforce last task UCLAMP_MAX

Quoted 31 lines · Patrick Bellasi

When a util_max clamped task sleeps, its clamp constraints are removed from the CPU. However, the blocked utilization on that CPU can still be higher than the max clamp value enforced while that task was running.

The release of a util_max clamp when a CPU is going to be idle could thus allow unwanted CPU frequency increases while tasks are not running. This can happen, for example, when a frequency update is triggered from another CPU of the same frequency domain.
In this case, when we aggregate the utilization of all the CPUs in a shared frequency domain, schedutil can still see the full not clamped blocked utilization of all the CPUs and thus, eventually, increase the frequency.

kernel/sched/core.c +14 -0
@@ -810,6 +811,28 @@ static inline void uclamp_cpu_update(struct rq *rq, unsigned int clamp_id)
 		if (max_value >= SCHED_CAPACITY_SCALE)
 			break;
 	}
+
+	/*
+	 * Just for the UCLAMP_MAX value, in case there are no RUNNABLE
+	 * task, we want to keep the CPU clamped to the last task's clamp
+	 * value. This is to avoid frequency spikes to MAX when one CPU, with
+	 * an high blocked utilization, sleeps and another CPU, in the same
+	 * frequency domain, do not see anymore the clamp on the first CPU.
+	 *
+	 * The UCLAMP_FLAG_IDLE is set whenever we detect, from the above
+	 * loop, that there are no more RUNNABLE taks on that CPU.
+	 * In this case we enforce the CPU util_max to that of the last
+	 * dequeued task.
+	 */
+	if (max_value < 0) {

kernel/sched/core.c static inline void uclamp_cpu_update(struct rq *rq, unsigned int clamp_id) · line 837

Quoted 10 lines
kernel/sched/core.c +8 -0
+		if (clamp_id == UCLAMP_MAX) {
+			rq->uclamp.flags |= UCLAMP_FLAG_IDLE;
+			max_value = last_clamp_value;
+		} else {
+			max_value = uclamp_none(UCLAMP_MIN);
+		}
+	}
+
 	rq->uclamp.value[clamp_id] = max_value;
 }

*groan*, so it could be jet-lag, but I find the comment really hard to understand.

Would not something like:

/*
* Avoid blocked utilization pushing up the frequency when we go
* idle (which drops the max-clamp) by retaining the last known
* max-clamp.
*/

Be more clear?

#35 lore raw

Re: [PATCH v5 06/15] sched/core: uclamp: enforce last task UCLAMP_MAX

Quoted 48 lines · Peter Zijlstra
Quoted 31 lines · Patrick Bellasi

When a util_max clamped task sleeps, its clamp constraints are removed from the CPU. However, the blocked utilization on that CPU can still be higher than the max clamp value enforced while that task was running.

The release of a util_max clamp when a CPU is going to be idle could thus allow unwanted CPU frequency increases while tasks are not running. This can happen, for example, when a frequency update is triggered from another CPU of the same frequency domain.
In this case, when we aggregate the utilization of all the CPUs in a shared frequency domain, schedutil can still see the full not clamped blocked utilization of all the CPUs and thus, eventually, increase the frequency.

kernel/sched/core.c +14 -0
@@ -810,6 +811,28 @@ static inline void uclamp_cpu_update(struct rq *rq, unsigned int clamp_id)
 		if (max_value >= SCHED_CAPACITY_SCALE)
 			break;
 	}
+
+	/*
+	 * Just for the UCLAMP_MAX value, in case there are no RUNNABLE
+	 * task, we want to keep the CPU clamped to the last task's clamp
+	 * value. This is to avoid frequency spikes to MAX when one CPU, with
+	 * an high blocked utilization, sleeps and another CPU, in the same
+	 * frequency domain, do not see anymore the clamp on the first CPU.
+	 *
+	 * The UCLAMP_FLAG_IDLE is set whenever we detect, from the above
+	 * loop, that there are no more RUNNABLE taks on that CPU.
+	 * In this case we enforce the CPU util_max to that of the last
+	 * dequeued task.
+	 */
+	if (max_value < 0) {

kernel/sched/core.c static inline void uclamp_cpu_update(struct rq *rq, unsigned int clamp_id) · line 837

Quoted 10 lines
kernel/sched/core.c +8 -0
+		if (clamp_id == UCLAMP_MAX) {
+			rq->uclamp.flags |= UCLAMP_FLAG_IDLE;
+			max_value = last_clamp_value;
+		} else {
+			max_value = uclamp_none(UCLAMP_MIN);
+		}
+	}
+
 	rq->uclamp.value[clamp_id] = max_value;
 }

*groan*, so it could be jet-lag, but I find the comment really hard to understand.

Would not something like:

Quoted 7 lines

/*
* Avoid blocked utilization pushing up the frequency when we go
* idle (which drops the max-clamp) by retaining the last known
* max-clamp.
*/

Be more clear?

It works: short and effective... will update in v6.

Thanks.

Signature · 3 lines
#include <best/regards.h>

Patrick Bellasi
#36 lore raw

PATCH v5 7/15 sched/core: uclamp: add clamp group bucketing support

In reply to Patrick Bellasi

The limited number of clamp groups is required to have an effective and efficient run-time tracking of the clamp groups required by RUNNABLE tasks. However, we must ensure we can always know which clamp group to use in the fast path (task enqueue/dequeue time) to refcount each tasks, whatever its clamp value is.

To this purpose we can trade-off CPU clamping precision for efficiency by turning CPU's clamp groups into buckets, each one representing a range of possible clamp values.

The number of clamp groups configured at compile time defines the range of utilization clamp values tracked by each CPU clamp group. For example, with the default configuration:
CONFIG_UCLAMP_GROUPS_COUNT 5
we will have 5 clamp groups tracking 20% utilization each. In this case, a task with util_min=25% will have group_id=1.

This bucketing mechanisms applies only on the fast-path, when tasks are refcounted into a per-CPU clamp groups at enqueue/dequeue time, while tasks keep tracking their task-specific clamp value requested from user-space. This allows to track, within each bucket, the maximum task-specific clamp value for tasks refcounted in that bucket.

In the example above, a 25% boosted tasks will be refcounted in the [20..39]% bucket and will set the bucket clamp effective value to 25%. If a second 30% boosted task should be co-scheduled on the same CPU, that task will be refcounted in the same bucket of the first task and it will boost the bucket clamp effective value to 30%.
The clamp effective value of a bucket will be reset to its nominal value (named the "group_value", 20% in the example above) when there are anymore tasks refcounted in that bucket.

On a real system we expect a limited number of sufficiently different clamp values thus, this simple bucketing mechanism is still effective to track tasks clamp effective values quite closely.
An additional boost/capping margin can be added to some tasks, in the example above the 25% task will be boosted to 30% until it exits the CPU.

If that should be considered not acceptable on certain systems, it's always possible to reduce the margin by increasing the bucketing resolution. Indeed, by properly setting the number of CONFIG_UCLAMP_GROUPS_COUNT, we can trade off memory efficiency for resolution.

The already existing mechanism to map "clamp values" into "clamp groups" ensures to use only the minimal set of clamp groups actually required. For example, if we have only 20% and 25% clamped tasks, by setting:
CONFIG_UCLAMP_GROUPS_COUNT 20
we will allocate 20 5% resolution buckets, however we will use only 2 two of them in the fast-path, since their 5% resolution will be enough to always distinguish them.

Cc Ingo Molnar <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Rafael J. Wysocki <[email protected]>
Cc Paul Turner <[email protected]>
Cc Suren Baghdasaryan <[email protected]>
Cc Todd Kjos <[email protected]>
Cc Joel Fernandes <[email protected]>
Cc Steve Muckle <[email protected]>
Cc Juri Lelli <[email protected]>
Cc Quentin Perret <[email protected]>
Cc Dietmar Eggemann <[email protected]>
Cc Morten Rasmussen <[email protected]>

---
Changes in v5:
Others:
- renamed uclamp_round into uclamp_group_value to better represent what this function returns
- rebased on v4.19

Changes in v4:
Message-ID: <20180809152313.lewfhufidhxb2qrk@darkstar>
- implements the idea discussed in this thread Others:
- new patch added in this version
- rebased on v4.19-rc1

---
 kernel/sched/core.c | 48 ++++++++++++++++++++++++++++++++++++++++-----
 1 file changed, 43 insertions(+), 5 deletions(-)
kernel/sched/core.c +43 -5
index b23f80c07be9..9b49062439f3 100644
@@ -783,6 +783,27 @@ union uclamp_map {
  */
 static union uclamp_map uclamp_maps[UCLAMP_CNT][UCLAMP_GROUPS];
 
+/*
+ * uclamp_group_value: get the "group value" for a given "clamp value"
+ * @value: the utiliation "clamp value" to translate
+ *
+ * The number of clamp group, which is defined at compile time, allows to
+ * track a finite number of different clamp values. Thus clamp values are
+ * grouped into bins each one representing a different "group value".
+ * This method returns the "group value" corresponding to the specified
+ * "clamp value".
+ */
+static inline unsigned int uclamp_group_value(unsigned int clamp_value)
+{
+#define UCLAMP_GROUP_DELTA (SCHED_CAPACITY_SCALE / CONFIG_UCLAMP_GROUPS_COUNT)
+#define UCLAMP_GROUP_UPPER (UCLAMP_GROUP_DELTA * CONFIG_UCLAMP_GROUPS_COUNT)
+
+	if (clamp_value >= UCLAMP_GROUP_UPPER)
+		return SCHED_CAPACITY_SCALE;
+
+	return UCLAMP_GROUP_DELTA * (clamp_value / UCLAMP_GROUP_DELTA);
+}
+
 /**
  * uclamp_cpu_update: updates the utilization clamp of a CPU
  * @rq: the CPU's rq which utilization clamp has to be updated
@@ -848,6 +869,7 @@ static inline void uclamp_cpu_update(struct rq *rq, unsigned int clamp_id,
 static inline void uclamp_cpu_get_id(struct task_struct *p, struct rq *rq,
 				     unsigned int clamp_id)
 {
+	unsigned int clamp_value;
 	unsigned int group_id;
 
 	if (unlikely(!p->uclamp[clamp_id].mapped))
@@ -870,6 +892,11 @@ static inline void uclamp_cpu_get_id(struct task_struct *p, struct rq *rq,
 		rq->uclamp.value[clamp_id] = p->uclamp[clamp_id].value;
 	}
 
+	/* CPU's clamp groups track the max effective clamp value */
+	clamp_value = p->uclamp[clamp_id].value;
+	if (clamp_value > rq->uclamp.group[clamp_id][group_id].value)
+		rq->uclamp.group[clamp_id][group_id].value = clamp_value;
+
 	if (rq->uclamp.value[clamp_id] < p->uclamp[clamp_id].value)
 		rq->uclamp.value[clamp_id] = p->uclamp[clamp_id].value;
 }
@@ -917,8 +944,16 @@ static inline void uclamp_cpu_put_id(struct task_struct *p, struct rq *rq,
 		     cpu_of(rq), clamp_id, group_id);
 	}
 #endif
-	if (clamp_value >= rq->uclamp.value[clamp_id])
+	if (clamp_value >= rq->uclamp.value[clamp_id]) {
+		/*
+		 * Each CPU's clamp group value is reset to its nominal group
+		 * value whenever there are anymore RUNNABLE tasks refcounting
+		 * that clamp group.
+		 */
+		rq->uclamp.group[clamp_id][group_id].value =
+			uclamp_maps[clamp_id][group_id].value;
 		uclamp_cpu_update(rq, clamp_id, clamp_value);
+	}
 }
 
 /**
@@ -1065,10 +1100,13 @@ static void uclamp_group_get(struct task_struct *p, struct uclamp_se *uc_se,
 	unsigned int prev_group_id = uc_se->group_id;
 	union uclamp_map uc_map_old, uc_map_new;
 	unsigned int free_group_id;
+	unsigned int group_value;
 	unsigned int group_id;
 	unsigned long res;
 	int cpu;
 
+	group_value = uclamp_group_value(clamp_value);
+
 retry:
 
 	free_group_id = UCLAMP_GROUPS;
@@ -1076,7 +1114,7 @@ static void uclamp_group_get(struct task_struct *p, struct uclamp_se *uc_se,
 		uc_map_old.data = atomic_long_read(&uc_maps[group_id].adata);
 		if (free_group_id == UCLAMP_GROUPS && !uc_map_old.se_count)
 			free_group_id = group_id;
-		if (uc_map_old.value == clamp_value)
+		if (uc_map_old.value == group_value)
 			break;
 	}
 	if (group_id >= UCLAMP_GROUPS) {
@@ -1092,7 +1130,7 @@ static void uclamp_group_get(struct task_struct *p, struct uclamp_se *uc_se,
 	}
 
 	uc_map_new.se_count = uc_map_old.se_count + 1;
-	uc_map_new.value = clamp_value;
+	uc_map_new.value = group_value;
 	res = atomic_long_cmpxchg(&uc_maps[group_id].adata,
 				  uc_map_old.data, uc_map_new.data);
 	if (res != uc_map_old.data)
@@ -1113,9 +1151,9 @@ static void uclamp_group_get(struct task_struct *p, struct uclamp_se *uc_se,
 #endif
 		}
 
-		if (uc_cpu->group[clamp_id][group_id].value == clamp_value)
+		if (uc_cpu->group[clamp_id][group_id].value == group_value)
 			continue;
-		uc_cpu->group[clamp_id][group_id].value = clamp_value;
+		uc_cpu->group[clamp_id][group_id].value = group_value;
 	}
 
 done:
Signature · 1 line
2.18.0
maintainer
#37 lore raw

Re: [PATCH v5 07/15] sched/core: uclamp: add clamp group bucketing support

Quoted 6 lines · Patrick Bellasi

The number of clamp groups configured at compile time defines the range of utilization clamp values tracked by each CPU clamp group. For example, with the default configuration:
CONFIG_UCLAMP_GROUPS_COUNT 5
we will have 5 clamp groups tracking 20% utilization each. In this case, a task with util_min=25% will have group_id=1.

OK I suppose; but should we not do a wholesale s/group/bucket/ at this point?

We should probably raise the minimum number of buckets from 1 though :-)

Quoted 10 lines
kernel/sched/core.c +10 -0
+/*
+ * uclamp_group_value: get the "group value" for a given "clamp value"
+ * @value: the utiliation "clamp value" to translate
+ *
+ * The number of clamp group, which is defined at compile time, allows to
+ * track a finite number of different clamp values. Thus clamp values are
+ * grouped into bins each one representing a different "group value".
+ * This method returns the "group value" corresponding to the specified
+ * "clamp value".
+ */

kernel/sched/core.c union uclamp_map { · line 805

Quoted 10 lines
kernel/sched/core.c +10 -0
+static inline unsigned int uclamp_group_value(unsigned int clamp_value)
+{
+#define UCLAMP_GROUP_DELTA (SCHED_CAPACITY_SCALE / CONFIG_UCLAMP_GROUPS_COUNT)
+#define UCLAMP_GROUP_UPPER (UCLAMP_GROUP_DELTA * CONFIG_UCLAMP_GROUPS_COUNT)
+
+	if (clamp_value >= UCLAMP_GROUP_UPPER)
+		return SCHED_CAPACITY_SCALE;
+
+	return UCLAMP_GROUP_DELTA * (clamp_value / UCLAMP_GROUP_DELTA);
+}

Can't we further simplify; I mean, at this point all we really need to know is the rq's highest group_id that is in use. We don't need to actually track the value anymore.

#38 lore raw

Re: [PATCH v5 07/15] sched/core: uclamp: add clamp group bucketing support

Quoted 10 lines · Peter Zijlstra
Quoted 6 lines · Patrick Bellasi

The number of clamp groups configured at compile time defines the range of utilization clamp values tracked by each CPU clamp group. For example, with the default configuration:
CONFIG_UCLAMP_GROUPS_COUNT 5
we will have 5 clamp groups tracking 20% utilization each. In this case, a task with util_min=25% will have group_id=1.

OK I suppose; but should we not do a wholesale s/group/bucket/ at this point?

Yes, if bucketization is acceptable, we should probably rename.

Question is: are you ok for a renaming in this patch. or you better prefer I use that naming since the beginning ?

If we wanna use "bucket" since the beginning, then we should also probably squash the entire patch into the previous ones and drop this one.

I personally prefer to keep this concept into a separate patch, but at the same time I don't very like the idea of a massive renaming in this patch.

Quoted 2 lines

We should probably raise the minimum number of buckets from 1 though :-)

Mmm... the default is already set to what fits into a single cache line... perhaps we can use that as a minimum too ?

But. technically we can (partially) track different clamp values also with just one bucket... (explanation in the following comment).

Quoted 16 lines
Quoted 16 lines
kernel/sched/core.c +16 -0
+/*
+ * uclamp_group_value: get the "group value" for a given "clamp value"
+ * @value: the utiliation "clamp value" to translate
+ *
+ * The number of clamp group, which is defined at compile time, allows to
+ * track a finite number of different clamp values. Thus clamp values are
+ * grouped into bins each one representing a different "group value".
+ * This method returns the "group value" corresponding to the specified
+ * "clamp value".
+ */
+static inline unsigned int uclamp_group_value(unsigned int clamp_value)
+{
+#define UCLAMP_GROUP_DELTA (SCHED_CAPACITY_SCALE / CONFIG_UCLAMP_GROUPS_COUNT)
+#define UCLAMP_GROUP_UPPER (UCLAMP_GROUP_DELTA * CONFIG_UCLAMP_GROUPS_COUNT)
+
+	if (clamp_value >= UCLAMP_GROUP_UPPER)
Quoted 8 lines

kernel/sched/core.c union uclamp_map { · line 805

Quoted 4 lines
kernel/sched/core.c +4 -0
+		return SCHED_CAPACITY_SCALE;
+
+	return UCLAMP_GROUP_DELTA * (clamp_value / UCLAMP_GROUP_DELTA);
+}

Can't we further simplify; I mean, at this point all we really need to know is the rq's highest group_id that is in use. We don't need to actually track the value anymore.

This will force to track each clamp value with the exact bucket value.

Instead, by tracking the actual clamp value within a bucket, we have the chance to updte the bucket value to the actual (max) clamp value of the RUNNABLE tasks in that bucket.

In a properly configured system, this allows to track exact clamp values with a minimum number of buckets.

Signature · 3 lines
#include <best/regards.h>

Patrick Bellasi
#39 lore raw

PATCH v5 8/15 sched/core: uclamp: add system default clamps

In reply to Patrick Bellasi

Tasks without a user-defined clamp value are considered not clamped and by default their utilization can be any value in the [0..SCHED_CAPACITY_SCALE] range. Tasks with a user-defined clamp value are allowed to request any value in that range, and we currently unconditionally enforce the required clamps.
However, a "System Management Software" could be interested in unconditionally limiting the range of clamp values allowed for all tasks.

Let's fix this by explicitly adding a privileged interface to define a system default configuration via:

/proc/sys/kernel/sched_uclamp_util_{min,max}

which works as an unconditional clamp range restriction for all tasks.

If a task specific value is not compliant with the system default range, it will be forced to the corresponding system default value.

Cc Ingo Molnar <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Tejun Heo <[email protected]>
Cc Paul Turner <[email protected]>
Cc Suren Baghdasaryan <[email protected]>
Cc Todd Kjos <[email protected]>
Cc Joel Fernandes <[email protected]>
Cc Steve Muckle <[email protected]>
Cc Juri Lelli <[email protected]>
Cc Quentin Perret <[email protected]>
Cc Dietmar Eggemann <[email protected]>
Cc Morten Rasmussen <[email protected]>

---
The current restriction could be too aggressive since, for example if a task has a util_min which is higher then the system default max, it will be forced to the system default min unconditionally.

We should probably better restrict util_min to the maximum system default value, but that whould make the code more complex and we keep it for a future update.

Changes in v5:
Other:
- rebased on v4.19

Changes in v4:
Message-ID: <20180820122728.GM2960@e110439-lin>
- fix unwanted reset of clamp values on refcount success Others:
- by default all tasks have a UCLAMP_NOT_VALID task specific clamp
- always use:

      p->uclamp[clamp_id].effective.value
   to track the actual clamp value the task has been refcounted into.
   This matches with the usage of
      p->uclamp[clamp_id].effective.group_id
 - rebased on v4.19-rc1
---
 include/linux/sched.h        |   5 ++
 include/linux/sched/sysctl.h |  11 +++
 kernel/sched/core.c          | 131 ++++++++++++++++++++++++++++++++---
 kernel/sysctl.c              |  16 +++++
 4 files changed, 154 insertions(+), 9 deletions(-)
include/linux/sched.h +5 -0
index 3ab1cbd4e3b1..ec6783ea4e7d 100644
@@ -614,6 +614,11 @@ struct uclamp_se {
 	unsigned int group_id		: order_base_2(UCLAMP_GROUPS);
 	unsigned int mapped		: 1;
 	unsigned int active		: 1;
+	/* Clamp group and value actually used by a RUNNABLE task */
+	struct {
+		unsigned int value	: SCHED_CAPACITY_SHIFT + 1;
+		unsigned int group_id	: order_base_2(UCLAMP_GROUPS);
+	} effective;
 };
 #endif /* CONFIG_UCLAMP_TASK */
 
include/linux/sched/sysctl.h +11 -0
index a9c32daeb9d8..445fb54eaeff 100644
@@ -56,6 +56,11 @@ int sched_proc_update_handler(struct ctl_table *table, int write,
 extern unsigned int sysctl_sched_rt_period;
 extern int sysctl_sched_rt_runtime;
 
+#ifdef CONFIG_UCLAMP_TASK
+extern unsigned int sysctl_sched_uclamp_util_min;
+extern unsigned int sysctl_sched_uclamp_util_max;
+#endif
+
 #ifdef CONFIG_CFS_BANDWIDTH
 extern unsigned int sysctl_sched_cfs_bandwidth_slice;
 #endif
@@ -75,6 +80,12 @@ extern int sched_rt_handler(struct ctl_table *table, int write,
 		void __user *buffer, size_t *lenp,
 		loff_t *ppos);
 
+#ifdef CONFIG_UCLAMP_TASK
+extern int sched_uclamp_handler(struct ctl_table *table, int write,
+				void __user *buffer, size_t *lenp,
+				loff_t *ppos);
+#endif
+
 extern int sysctl_numa_balancing(struct ctl_table *table, int write,
 				 void __user *buffer, size_t *lenp,
 				 loff_t *ppos);
kernel/sched/core.c +122 -9

185 lines not on this page · raw message

kernel/sysctl.c +16 -0
index cc02050fd0c4..378ea57e5fc5 100644
@@ -445,6 +445,22 @@ static struct ctl_table kern_table[] = {
 		.mode		= 0644,
 		.proc_handler	= sched_rr_handler,
 	},
+#ifdef CONFIG_UCLAMP_TASK
+	{
+		.procname	= "sched_uclamp_util_min",
+		.data		= &sysctl_sched_uclamp_util_min,
+		.maxlen		= sizeof(unsigned int),
+		.mode		= 0644,
+		.proc_handler	= sched_uclamp_handler,
+	},
+	{
+		.procname	= "sched_uclamp_util_max",
+		.data		= &sysctl_sched_uclamp_util_max,
+		.maxlen		= sizeof(unsigned int),
+		.mode		= 0644,
+		.proc_handler	= sched_uclamp_handler,
+	},
+#endif
 #ifdef CONFIG_SCHED_AUTOGROUP
 	{
 		.procname	= "sched_autogroup_enabled",
Signature · 1 line
2.18.0
#40 lore raw

PATCH v5 9/15 sched/cpufreq: uclamp: add utilization clamping for FAIR tasks

In reply to Patrick Bellasi

Each time a frequency update is required via schedutil, a frequency is selected to (possibly) satisfy the utilization reported by the CFS class. However, when utilization clamping is in use, the frequency selection should consider userspace utilization clamping hints. This will allow, for example, to:

- boost tasks which are directly affecting the user experience by running them at least at a minimum "requested" frequency

- cap low priority tasks not directly affecting the user experience by running them only up to a maximum "allowed" frequency

These constraints are meant to support a per-task based tuning of the frequency selection thus supporting a fine grained definition of performance boosting vs energy saving strategies in kernel space.

This adds the required support to clamp the utilization generated by RUNNABLE FAIR tasks within the boundaries defined by their aggregated utilization clamp constraints.

On each CPU the aggregated clamp values are obtained by considering the maximum of the {min,max}_util values for each task. This max aggregation responds to the goal of not penalizing, for example, high boosted (i.e. more important for the user-experience) CFS tasks which happens to be co-scheduled with high capped (i.e. less important for the user-experience) CFS tasks.

For FAIR tasks both the utilization and the IOWait boost values are clamped according to the CPU aggregated utilization clamp constraints.

The default values for boosting and capping are defined to be:
- util_min: 0
- util_max: SCHED_CAPACITY_SCALE
which means that by default no boosting/capping is enforced on FAIR tasks, and thus the frequency will be selected considering the actual utilization value of each CPU.

Cc Ingo Molnar <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Rafael J. Wysocki <[email protected]>
Cc Viresh Kumar <[email protected]>
Cc Suren Baghdasaryan <[email protected]>
Cc Todd Kjos <[email protected]>
Cc Joel Fernandes <[email protected]>
Cc Juri Lelli <[email protected]>
Cc Quentin Perret <[email protected]>
Cc Dietmar Eggemann <[email protected]>
Cc Morten Rasmussen <[email protected]>

---
Changes in v5:
Message-ID: <[email protected]>
- add a comment to justify the assumptions on util clamping for FAIR tasks
Message-ID: <[email protected]>
- removed uclamp_value and use inline access to data structures
Message-ID: <20180914135712.GQ1413@e110439-lin>
- the (unlikely(val == UCLAMP_NOT_VALID)) check is not more required since we now ensure we always have a valid value configured
Others:
- rebased on v4.19

Changes in v4:
Message-ID: <CAKfTPtC2adLupg7wy1JU9zxKx1466Sza6fSCcr92wcawm1OYkg@mail.gmail.com>
- use *rq instead of cpu for both uclamp_util() and uclamp_value()
Message-ID: <20180816135300.GC2960@e110439-lin>
- remove uclamp_value() which is never used outside CONFIG_UCLAMP_TASK Others:
- rebased on v4.19-rc1
Changes in v3:
Message-ID: <CAJuCfpF6=L=0LrmNnJrTNPazT4dWKqNv+thhN0dwpKCgUzs9sg@mail.gmail.com>
- rename UCLAMP_NONE into UCLAMP_NOT_VALID
Others:
- rebased on tip/sched/core
Changes in v2:
- rebased on v4.18-rc4

---
 kernel/sched/cpufreq_schedutil.c | 25 ++++++++++++++++++++++---
 kernel/sched/sched.h             | 28 ++++++++++++++++++++++++++++
 2 files changed, 50 insertions(+), 3 deletions(-)
kernel/sched/cpufreq_schedutil.c +22 -3
index 3fffad3bc8a8..fd3fe55d605b 100644
@@ -222,8 +222,13 @@ static unsigned long sugov_get_util(struct sugov_cpu *sg_cpu)
 	 * CFS tasks and we use the same metric to track the effective
 	 * utilization (PELT windows are synchronized) we can directly add them
 	 * to obtain the CPU's actual utilization.
+	 *
+	 * CFS utilization can be boosted or capped, depending on utilization
+	 * clamp constraints requested by currently RUNNABLE tasks.
+	 * When there are no CFS RUNNABLE tasks, clamps are released and OPPs
+	 * will be gracefully reduced with the utilization decay.
 	 */
-	util = cpu_util_cfs(rq);
+	util  = uclamp_util(rq, cpu_util_cfs(rq));
 	util += cpu_util_rt(rq);
 
 	/*
@@ -307,6 +312,7 @@ static void sugov_iowait_boost(struct sugov_cpu *sg_cpu, u64 time,
 			       unsigned int flags)
 {
 	bool set_iowait_boost = flags & SCHED_CPUFREQ_IOWAIT;
+	unsigned int max_boost;
 
 	/* Reset boost if the CPU appears to have been idle enough */
 	if (sg_cpu->iowait_boost &&
@@ -322,11 +328,24 @@ static void sugov_iowait_boost(struct sugov_cpu *sg_cpu, u64 time,
 		return;
 	sg_cpu->iowait_boost_pending = true;
 
+	/*
+	 * Boost FAIR tasks only up to the CPU clamped utilization.
+	 *
+	 * Since DL tasks have a much more advanced bandwidth control, it's
+	 * safe to assume that IO boost does not apply to those tasks.
+	 * Instead, since RT tasks are not utiliation clamped, we don't want
+	 * to apply clamping on IO boost while there is blocked RT
+	 * utilization.
+	 */
+	max_boost = sg_cpu->iowait_boost_max;
+	if (!cpu_util_rt(cpu_rq(sg_cpu->cpu)))
+		max_boost = uclamp_util(cpu_rq(sg_cpu->cpu), max_boost);
+
 	/* Double the boost at each request */
 	if (sg_cpu->iowait_boost) {
 		sg_cpu->iowait_boost <<= 1;
-		if (sg_cpu->iowait_boost > sg_cpu->iowait_boost_max)
-			sg_cpu->iowait_boost = sg_cpu->iowait_boost_max;
+		if (sg_cpu->iowait_boost > max_boost)
+			sg_cpu->iowait_boost = max_boost;
 		return;
 	}
 
kernel/sched/sched.h +28 -0
index 859192ec492c..a7e9b7041ea5 100644
@@ -2262,6 +2262,34 @@ static inline unsigned int uclamp_none(int clamp_id)
 	return SCHED_CAPACITY_SCALE;
 }
 
+#ifdef CONFIG_UCLAMP_TASK
+/**
+ * clamp_util: clamp a utilization value for a specified CPU
+ * @rq: the CPU's RQ to get the clamp values from
+ * @util: the utilization signal to clamp
+ *
+ * Each CPU tracks util_{min,max} clamp values depending on the set of its
+ * currently RUNNABLE tasks. Given a utilization signal, i.e a signal in
+ * the [0..SCHED_CAPACITY_SCALE] range, this function returns a clamped
+ * utilization signal considering the current clamp values for the
+ * specified CPU.
+ *
+ * Return: a clamped utilization signal for a given CPU.
+ */
+static inline unsigned int uclamp_util(struct rq *rq, unsigned int util)
+{
+	unsigned int min_util = rq->uclamp.value[UCLAMP_MIN];
+	unsigned int max_util = rq->uclamp.value[UCLAMP_MAX];
+
+	return clamp(util, min_util, max_util);
+}
+#else /* CONFIG_UCLAMP_TASK */
+static inline unsigned int uclamp_util(struct rq *rq, unsigned int util)
+{
+	return util;
+}
+#endif /* CONFIG_UCLAMP_TASK */
+
 #ifdef arch_scale_freq_capacity
 # ifndef arch_scale_freq_invariant
 #  define arch_scale_freq_invariant()	true
Signature · 1 line
2.18.0
#41 lore raw

Re: [PATCH v5 09/15] sched/cpufreq: uclamp: add utilization clamping for FAIR tasks

On 29-Oct 18:33, Patrick Bellasi wrote:

[...]

Quoted 8 lines
kernel/sched/sched.h +8 -0
+#ifdef CONFIG_UCLAMP_TASK
+/**
+ * clamp_util: clamp a utilization value for a specified CPU
+ * @rq: the CPU's RQ to get the clamp values from
+ * @util: the utilization signal to clamp
+ *
+ * Each CPU tracks util_{min,max} clamp values depending on the set of its
+ * currently RUNNABLE tasks. Given a utilization signal, i.e a signal in

kernel/sched/sched.h static inline unsigned int uclamp_none(int clamp_id) · line 2282

Quoted 10 lines
kernel/sched/sched.h +10 -0
+ * the [0..SCHED_CAPACITY_SCALE] range, this function returns a clamped
+ * utilization signal considering the current clamp values for the
+ * specified CPU.
+ *
+ * Return: a clamped utilization signal for a given CPU.
+ */
+static inline unsigned int uclamp_util(struct rq *rq, unsigned int util)
+{
+	unsigned int min_util = rq->uclamp.value[UCLAMP_MIN];
+	unsigned int max_util = rq->uclamp.value[UCLAMP_MAX];

Just notice here we can have an issue.

For each scheduling entity, we always ensure that:

util_min <= util_max

However, since CPU's {min,max}_util clamps are always MAX aggregated considering the corresponding clamps of RUNNABLE tasks with _different_ clamps, we can end up with CPU clamps where:

util_min > util_max

Thus, we need to add the following sanity check here:

+ if (unlikely(min_util > max_util))
+ return min_util;

Quoted 3 lines
kernel/sched/sched.h +3 -0
+
+	return clamp(util, min_util, max_util);
+}
Signature · 3 lines
#include <best/regards.h>

Patrick Bellasi
#42 lore raw

PATCH v5 10/15 sched/cpufreq: uclamp: add utilization clamping for RT tasks

In reply to Patrick Bellasi

Currently schedutil enforces a maximum frequency when RT tasks are RUNNABLE. Such a mandatory policy can be made more tunable from userspace thus allowing for example to define a max frequency which is still reasonable for the execution of a specific RT workload. This will contribute to make the RT class more friendly for power/energy sensitive use-cases.

This patch extends the usage of util_{min,max} to the RT scheduling class. Whenever a task in this class is RUNNABLE, the util required is defined by its task specific clamp value. However, we still want to run at maximum capacity RT tasks which do not have task specific clamp values.

Let's add uclamp_default_perf, a special set of clamp value to be used for tasks that require maximum performance. This set of clamps are then used whenever the above conditions matches for an RT task being enqueued on a CPU.

Since utilization clamping applies now to both CFS and RT tasks, we clamp the combined utilization of these two classes.
This approach, contrary to combining individually clamped utilizations, is more power efficient. Indeed, it selects lower frequencies when we have both RT and CFS clamped tasks.
However, it could also affect performance of the lower priority CFS class, since the CFS's minimum utilization clamp could be completely eclipsed by the RT workloads.

The IOWait boost value also is subject to clamping for RT tasks. This is to ensure that RT tasks as well as CFS ones are always subject to the set of current utilization clamping constraints.

Cc Ingo Molnar <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Rafael J. Wysocki <[email protected]>
Cc Viresh Kumar <[email protected]>
Cc Suren Baghdasaryan <[email protected]>
Cc Todd Kjos <[email protected]>
Cc Joel Fernandes <[email protected]>
Cc Juri Lelli <[email protected]>
Cc Quentin Perret <[email protected]>
Cc Dietmar Eggemann <[email protected]>
Cc Morten Rasmussen <[email protected]>

---
Changes in v5:
Others:
- rebased on v4.19

Changes in v4:
Message-ID: <20180813150112.GE2605@e110439-lin>
- remove UCLAMP_SCHED_CLASS policy since we do not have in the current implementation a proper per-sched_class clamp tracking support
Message-ID: <20180809155551.bp46sixk4u3ilcnh@queper01-lin>
- add default boost for not clamped RT tasks
Others:
- rebased on v4.19-rc1
Changes in v3:
- rebased on tip/sched/core
Changes in v2:
- rebased on v4.18-rc4

---
 kernel/sched/core.c              | 19 +++++++++++++++----
 kernel/sched/cpufreq_schedutil.c | 22 ++++++++++------------
 kernel/sched/rt.c                |  4 ++++
 3 files changed, 29 insertions(+), 16 deletions(-)
kernel/sched/core.c +15 -4
index 8421ef96ec97..b9dd1980ec93 100644
@@ -745,6 +745,7 @@ unsigned int sysctl_sched_uclamp_util_max = SCHED_CAPACITY_SCALE;
  * Tasks's clamp values are required to be within this range
  */
 static struct uclamp_se uclamp_default[UCLAMP_CNT];
+static struct uclamp_se uclamp_default_perf[UCLAMP_CNT];
 
 /**
  * uclamp_map: reference count utilization clamp groups
@@ -895,6 +896,7 @@ static inline void uclamp_cpu_update(struct rq *rq, unsigned int clamp_id,
 static inline unsigned int uclamp_effective_group_id(struct task_struct *p,
 						     unsigned int clamp_id)
 {
+	struct uclamp_se *default_clamp;
 	unsigned int clamp_value;
 	unsigned int group_id;
 
@@ -906,15 +908,20 @@ static inline unsigned int uclamp_effective_group_id(struct task_struct *p,
 	clamp_value = p->uclamp[clamp_id].value;
 	group_id = p->uclamp[clamp_id].group_id;
 
+	/* RT tasks have different default values */
+	default_clamp = task_has_rt_policy(p)
+		? uclamp_default_perf
+		: uclamp_default;
+
 	/* System default restriction */
-	if (unlikely(clamp_value < uclamp_default[UCLAMP_MIN].value ||
-		     clamp_value > uclamp_default[UCLAMP_MAX].value)) {
+	if (unlikely(clamp_value < default_clamp[UCLAMP_MIN].value ||
+		     clamp_value > default_clamp[UCLAMP_MAX].value)) {
 		/*
 		 * Unconditionally enforce system defaults, which is a simpler
 		 * solution compared to a proper clamping.
 		 */
-		clamp_value = uclamp_default[clamp_id].value;
-		group_id = uclamp_default[clamp_id].group_id;
+		clamp_value = default_clamp[clamp_id].value;
+		group_id = default_clamp[clamp_id].group_id;
 	}
 
 	p->uclamp[clamp_id].effective.value = clamp_value;
@@ -1381,6 +1388,10 @@ static void __init init_uclamp(void)
 
 		uc_se = &uclamp_default[clamp_id];
 		uclamp_group_get(NULL, uc_se, clamp_id, uclamp_none(clamp_id));
+
+		/* RT tasks by default will go to max frequency */
+		uc_se = &uclamp_default_perf[clamp_id];
+		uclamp_group_get(NULL, uc_se, clamp_id, uclamp_none(UCLAMP_MAX));
 	}
 }
 
kernel/sched/cpufreq_schedutil.c +10 -12
index fd3fe55d605b..1156c7117fc2 100644
@@ -205,9 +205,6 @@ static unsigned long sugov_get_util(struct sugov_cpu *sg_cpu)
 	sg_cpu->max = max = arch_scale_cpu_capacity(NULL, sg_cpu->cpu);
 	sg_cpu->bw_dl = cpu_bw_dl(rq);
 
-	if (rt_rq_is_runnable(&rq->rt))
-		return max;
-
 	/*
 	 * Early check to see if IRQ/steal time saturates the CPU, can be
 	 * because of inaccuracies in how we track these -- see
@@ -223,13 +220,16 @@ static unsigned long sugov_get_util(struct sugov_cpu *sg_cpu)
 	 * utilization (PELT windows are synchronized) we can directly add them
 	 * to obtain the CPU's actual utilization.
 	 *
-	 * CFS utilization can be boosted or capped, depending on utilization
-	 * clamp constraints requested by currently RUNNABLE tasks.
+	 * CFS and RT utilization can be boosted or capped, depending on
+	 * utilization clamp constraints requested by currently RUNNABLE
+	 * tasks.
 	 * When there are no CFS RUNNABLE tasks, clamps are released and OPPs
 	 * will be gracefully reduced with the utilization decay.
 	 */
-	util  = uclamp_util(rq, cpu_util_cfs(rq));
-	util += cpu_util_rt(rq);
+	util = cpu_util_rt(rq) + cpu_util_cfs(rq);
+	util = uclamp_util(rq, util);
+	if (unlikely(util >= max))
+		return max;
 
 	/*
 	 * We do not make cpu_util_dl() a permanent part of this sum because we
@@ -333,13 +333,11 @@ static void sugov_iowait_boost(struct sugov_cpu *sg_cpu, u64 time,
 	 *
 	 * Since DL tasks have a much more advanced bandwidth control, it's
 	 * safe to assume that IO boost does not apply to those tasks.
-	 * Instead, since RT tasks are not utiliation clamped, we don't want
-	 * to apply clamping on IO boost while there is blocked RT
-	 * utilization.
+	 * Instead, for CFS and RT tasks we clamp the IO boost max value
+	 * considering the current constraints for the CPU.
 	 */
 	max_boost = sg_cpu->iowait_boost_max;
-	if (!cpu_util_rt(cpu_rq(sg_cpu->cpu)))
-		max_boost = uclamp_util(cpu_rq(sg_cpu->cpu), max_boost);
+	max_boost = uclamp_util(cpu_rq(sg_cpu->cpu), max_boost);
 
 	/* Double the boost at each request */
 	if (sg_cpu->iowait_boost) {
kernel/sched/rt.c +4 -0
index 2e2955a8cf8f..06ec33467dd9 100644
@@ -2404,6 +2404,10 @@ const struct sched_class rt_sched_class = {
 	.switched_to		= switched_to_rt,
 
 	.update_curr		= update_curr_rt,
+
+#ifdef CONFIG_UCLAMP_TASK
+	.uclamp_enabled		= 1,
+#endif
 };
 
 #ifdef CONFIG_RT_GROUP_SCHED
Signature · 1 line
2.18.0
#43 lore raw

PATCH v5 11/15 sched/core: uclamp: extend CPU's cgroup controller

In reply to Patrick Bellasi

The cgroup's CPU bandwidth controller allows to assign a specified (maximum) bandwidth to the tasks of a group. However this bandwidth is defined and enforced only on a temporal base, without considering the actual frequency a CPU is running on. Thus, the amount of computation completed by a task within an allocated bandwidth can be very different depending on the actual frequency the CPU is running that task. The amount of computation can be affected also by the specific CPU a task is running on, especially when running on asymmetric capacity systems like Arm's big.LITTLE.

With the availability of schedutil, the scheduler is now able to drive frequency selections based on actual task utilization. Moreover, the utilization clamping support provides a mechanism to bias the frequency selection operated by schedutil depending on constraints assigned to the tasks currently RUNNABLE on a CPU.

Give the above mechanisms, it is now possible to extend the cpu controller to specify what is the minimum (or maximum) utilization which a task is expected (or allowed) to generate.
Constraints on minimum and maximum utilization allowed for tasks in a CPU cgroup can improve the control on the actual amount of CPU bandwidth consumed by tasks.

Utilization clamping constraints are useful not only to bias frequency selection, when a task is running, but also to better support certain scheduler decisions regarding task placement. For example, on asymmetric capacity systems, a utilization clamp value can be conveniently used to enforce important interactive tasks on more capable CPUs or to run low priority and background tasks on more energy efficient CPUs.

The ultimate goal of utilization clamping is thus to enable:

- boosting: by selecting an higher capacity CPU and/or higher execution frequency for small tasks which are affecting the user interactive experience.

- capping: by selecting more energy efficiency CPUs or lower execution frequency, for big tasks which are mainly related to background activities, and thus without a direct impact on the user experience.

Thus, a proper extension of the cpu controller with utilization clamping support will make this controller even more suitable for integration with advanced system management software (e.g. Android). Indeed, an informed user-space can provide rich information hints to the scheduler regarding the tasks it's going to schedule.

This patch extends the CPU controller by adding a couple of new attributes, util.min and util.max, which can be used to enforce task's utilization boosting and capping. Specifically:

- util.min: defines the minimum utilization which should be considered, e.g. when schedutil selects the frequency for a CPU while a task in this group is RUNNABLE.
i.e. the task will run at least at a minimum frequency which
corresponds to the min_util utilization

- util.max: defines the maximum utilization which should be considered, e.g. when schedutil selects the frequency for a CPU while a task in this group is RUNNABLE.
i.e. the task will run up to a maximum frequency which
corresponds to the max_util utilization

These attributes:

a) are available only for non-root nodes, both on default and legacy hierarchies, while system wide clamps are defined by a generic interface which does not depends on cgroups

b) do not enforce any constraints and/or dependency between the parent and its child nodes, thus relying:
- on permission settings defined by the system management software, to define if subgroups can configure their clamp values
- on the delegation model, to ensure that effective clamps are updated to consider both subgroup requests and parent group constraints

c) allow to (eventually) further restrict task-specific clamps defined via sched_setattr(2)

This patch provides the basic support to expose the two new attributes and to validate their run-time updates, while we do not actually allocated clamp groups.

Cc Ingo Molnar <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Tejun Heo <[email protected]>
Cc Rafael J. Wysocki <[email protected]>
Cc Viresh Kumar <[email protected]>
Cc Suren Baghdasaryan <[email protected]>
Cc Todd Kjos <[email protected]>
Cc Joel Fernandes <[email protected]>
Cc Juri Lelli <[email protected]>
Cc Quentin Perret <[email protected]>
Cc Dietmar Eggemann <[email protected]>
Cc Morten Rasmussen <[email protected]>

---
Changes in v5:
Others:
- rebased on v4.19

Changes in v4:
Others:
- consolidate init_uclamp_sched_group() into init_uclamp()
- refcount root_task_group's clamp groups from init_uclamp()
- small documentation fixes
Changes in v3:
Message-ID: <CAJuCfpFnj2g3+ZpR4fP4yqfxs0zd=c-Zehr2XM7m_C+WdL9jNA@mail.gmail.com>
- rename UCLAMP_NONE into UCLAMP_NOT_VALID
Message-ID: <[email protected]>
- use "." notation for attributes naming
i.e. s/util_{min,max}/util.{min,max}/
Others
- rebased on v4.19-rc1
Changes in v2:
Message-ID: <[email protected]>
- make attributes available only on non-root nodes
a system wide API seems of not immediate interest and thus it's not supported anymore
- remove implicit parent-child constraints and dependencies
Message-ID: <[email protected]>
- add some cgroup-v2 documentation for the new attributes
- (hopefully) better explain intended use-cases
the changelog above has been extended to better justify the naming proposed by the new attributes
Others:
- rebased on v4.18-rc4
- reduced code to simplify the review of this patch
which now provides just the basic code for CGroups integration
- add attributes to the default hierarchy as well as the legacy one
- use -ERANGE as range violation error

These additional bits:
- refcounting of clamp groups
- RUNNABLE tasks refcount updates
- aggregation of per-task and per-task_group utilization constraints
are provided in separate and following patches to make it more clear and documented how they are performed.

---
 Documentation/admin-guide/cgroup-v2.rst |  25 ++++
 init/Kconfig                            |  22 ++++
 kernel/sched/core.c                     | 149 ++++++++++++++++++++++++
 kernel/sched/sched.h                    |   5 +
 4 files changed, 201 insertions(+)
Documentation/admin-guide/cgroup-v2.rst +25 -0
index 184193bcb262..a6907266e52f 100644
@@ -907,6 +907,12 @@ controller implements weight and absolute bandwidth limit models for
 normal scheduling policy and absolute bandwidth allocation model for
 realtime scheduling policy.
 
+Cycles distribution is based, by default, on a temporal base and it
+does not account for the frequency at which tasks are executed.
+The (optional) utilization clamping support allows to enforce a minimum
+bandwidth, which should always be provided by a CPU, and a maximum bandwidth,
+which should never be exceeded by a CPU.
+
 WARNING: cgroup2 doesn't yet support control of realtime processes and
 the cpu controller can only be enabled when all RT processes are in
 the root cgroup.  Be aware that system management software may already
@@ -966,6 +972,25 @@ All time durations are in microseconds.
 	$PERIOD duration.  "max" for $MAX indicates no limit.  If only
 	one number is written, $MAX is updated.
 
+  cpu.util.min
+        A read-write single value file which exists on non-root cgroups.
+        The default is "0", i.e. no bandwidth boosting.
+
+        The minimum utilization in the range [0, 1024].
+
+        This interface allows reading and setting minimum utilization clamp
+        values similar to the sched_setattr(2). This minimum utilization
+        value is used to clamp the task specific minimum utilization clamp.
+
+  cpu.util.max
+        A read-write single value file which exists on non-root cgroups.
+        The default is "1024". i.e. no bandwidth capping
+
+        The maximum utilization in the range [0, 1024].
+
+        This interface allows reading and setting maximum utilization clamp
+        values similar to the sched_setattr(2). This maximum utilization
+        value is used to clamp the task specific maximum utilization clamp.
 
 Memory
 ------
init/Kconfig +22 -0
index 4c5475030286..83e4c987641e 100644
@@ -827,6 +827,28 @@ config RT_GROUP_SCHED
 
 endif #CGROUP_SCHED
 
+config UCLAMP_TASK_GROUP
+	bool "Utilization clamping per group of tasks"
+	depends on CGROUP_SCHED
+	depends on UCLAMP_TASK
+	default n
+	help
+	  This feature enables the scheduler to track the clamped utilization
+	  of each CPU based on RUNNABLE tasks currently scheduled on that CPU.
+
+	  When this option is enabled, the user can specify a min and max
+	  CPU bandwidth which is allowed for each single task in a group.
+	  The max bandwidth allows to clamp the maximum frequency a task
+	  can use, while the min bandwidth allows to define a minimum
+	  frequency a task will always use.
+
+	  When task group based utilization clamping is enabled, an eventually
+	  specified task-specific clamp value is constrained by the cgroup
+	  specified clamp value. Both minimum and maximum task clamping cannot
+	  be bigger than the corresponding clamping defined at task group level.
+
+	  If in doubt, say N.
+
 config CGROUP_PIDS
 	bool "PIDs controller"
 	help
kernel/sched/core.c +149 -0

185 lines not on this page · raw message

kernel/sched/sched.h +5 -0
index a7e9b7041ea5..c766abf0721d 100644
@@ -391,6 +391,11 @@ struct task_group {
 #endif
 
 	struct cfs_bandwidth	cfs_bandwidth;
+
+#ifdef CONFIG_UCLAMP_TASK_GROUP
+	struct			uclamp_se uclamp[UCLAMP_CNT];
+#endif
+
 };
 
 #ifdef CONFIG_FAIR_GROUP_SCHED
Signature · 1 line
2.18.0
#44 lore raw

PATCH v5 12/15 sched/core: uclamp: propagate parent clamps

In reply to Patrick Bellasi

In order to properly support hierarchical resources control, the cgroup delegation model requires that attribute writes from a child group never fail but still are (potentially) constrained based on parent's assigned resources. This requires to properly propagate and aggregate parent attributes down to its descendants.

Let's implement this mechanism by adding a new "effective" clamp value for each task group. The effective clamp value is defined as the smaller value between the clamp value of a group and the effective clamp value of its parent. This represent also the clamp value which is actually used to clamp tasks in each task group.

Since it can be interesting for tasks in a cgroup to know exactly what is the currently propagated/enforced configuration, the effective clamp values are exposed to user-space by means of a new pair of read-only
attributes: cpu.util.{min,max}.effective.

Cc Ingo Molnar <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Tejun Heo <[email protected]>
Cc Rafael J. Wysocki <[email protected]>
Cc Viresh Kumar <[email protected]>
Cc Suren Baghdasaryan <[email protected]>
Cc Todd Kjos <[email protected]>
Cc Joel Fernandes <[email protected]>
Cc Juri Lelli <[email protected]>
Cc Quentin Perret <[email protected]>
Cc Dietmar Eggemann <[email protected]>
Cc Morten Rasmussen <[email protected]>

---
Changes in v5:
Message-ID: <20180912125133.GE1413@e110439-lin>
- make more clear the definition of cpu.util.min.effective
- small typos fixed
Others:
- rebased on v4.19

Changes in v4:
Message-ID: <20180816140731.GD2960@e110439-lin>
- add ".effective" attributes to the default hierarchy Others:
- small documentation fixes
- rebased on v4.19-rc1
Changes in v3:
Message-ID: <[email protected]>
- new patch in v3, to implement a suggestion from v1 review

---
 Documentation/admin-guide/cgroup-v2.rst |  25 +++++-
 include/linux/sched.h                   |  10 ++-
 kernel/sched/core.c                     | 113 +++++++++++++++++++++++-
 3 files changed, 141 insertions(+), 7 deletions(-)
Documentation/admin-guide/cgroup-v2.rst +23 -2
index a6907266e52f..56bc56513721 100644
@@ -976,22 +976,43 @@ All time durations are in microseconds.
         A read-write single value file which exists on non-root cgroups.
         The default is "0", i.e. no bandwidth boosting.
 
-        The minimum utilization in the range [0, 1024].
+        The requested minimum utilization in the range [0, 1024].
 
         This interface allows reading and setting minimum utilization clamp
         values similar to the sched_setattr(2). This minimum utilization
         value is used to clamp the task specific minimum utilization clamp.
 
+  cpu.util.min.effective
+        A read-only single value file which exists on non-root cgroups and
+        reports minimum utilization clamp value currently enforced on a task
+        group.
+
+        The actual minimum utilization in the range [0, 1024].
+
+        This value can be lower then cpu.util.min in case a parent cgroup
+        allows only smaller minimum utilization values.
+
   cpu.util.max
         A read-write single value file which exists on non-root cgroups.
         The default is "1024". i.e. no bandwidth capping
 
-        The maximum utilization in the range [0, 1024].
+        The requested maximum utilization in the range [0, 1024].
 
         This interface allows reading and setting maximum utilization clamp
         values similar to the sched_setattr(2). This maximum utilization
         value is used to clamp the task specific maximum utilization clamp.
 
+  cpu.util.max.effective
+        A read-only single value file which exists on non-root cgroups and
+        reports maximum utilization clamp value currently enforced on a task
+        group.
+
+        The actual maximum utilization in the range [0, 1024].
+
+        This value can be lower then cpu.util.max in case a parent cgroup
+        is enforcing a more restrictive clamping on max utilization.
+
+
 Memory
 ------
 
include/linux/sched.h +9 -1
index ec6783ea4e7d..d3f6bf62ab3f 100644
@@ -614,7 +614,15 @@ struct uclamp_se {
 	unsigned int group_id		: order_base_2(UCLAMP_GROUPS);
 	unsigned int mapped		: 1;
 	unsigned int active		: 1;
-	/* Clamp group and value actually used by a RUNNABLE task */
+	/*
+	 * Clamp group and value actually used by a scheduling entity,
+	 * i.e. a (RUNNABLE) task or a task group.
+	 * For task groups, this is the value (eventually) enforced by a
+	 * parent task group.
+	 * For a task, this is the value (eventually) enforced by the
+	 * task group the task is currently part of or by the system
+	 * default clamp values, whichever is the most restrictive.
+	 */
 	struct {
 		unsigned int value	: SCHED_CAPACITY_SHIFT + 1;
 		unsigned int group_id	: order_base_2(UCLAMP_GROUPS);
kernel/sched/core.c +109 -4

189 lines not on this page · raw message

Signature · 1 line
2.18.0
#45 lore raw

PATCH v5 13/15 sched/core: uclamp: map TG's clamp values into CPU's clamp groups

In reply to Patrick Bellasi

Utilization clamping requires to map each different clamp value into one of the available clamp groups used in the scheduler fast-path to account for RUNNABLE tasks. Thus, each time a TG's clamp value sysfs attribute is updated via:
cpu_util_{min,max}_write_u64()
we need to update the task group reference to the new value's clamp group and release the reference to the previous one.

Let's ensure that, whenever a task group is assigned a specific clamp_value, this is properly translated into a unique clamp group to be used in the fast-path (i.e. at enqueue/dequeue time).
We do that by slightly refactoring uclamp_group_get() to make the *task_struct parameter optional. This allows to re-use the code already available for the support of the per-task API.

Cc Ingo Molnar <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Tejun Heo <[email protected]>
Cc Rafael J. Wysocki <[email protected]>
Cc Viresh Kumar <[email protected]>
Cc Suren Baghdasaryan <[email protected]>
Cc Todd Kjos <[email protected]>
Cc Joel Fernandes <[email protected]>
Cc Juri Lelli <[email protected]>
Cc Quentin Perret <[email protected]>
Cc Dietmar Eggemann <[email protected]>
Cc Morten Rasmussen <[email protected]>

---
Changes in v5:
Others:
- rebased on v4.19

Changes in v4:
Others:
- rebased on v4.19-rc1
Changes in v3:
Message-ID: <CAJuCfpF6=L=0LrmNnJrTNPazT4dWKqNv+thhN0dwpKCgUzs9sg@mail.gmail.com>
- add explicit calls to uclamp_group_find(), which is now not more part of uclamp_group_get()
Others:
- rebased on tip/sched/core
Changes in v2:
- rebased on v4.18-rc4
- this code has been split from a previous patch to simplify the review

---
 include/linux/sched.h |  7 +++--
 kernel/sched/core.c   | 64 +++++++++++++++++++++++++++++++++++--------
 2 files changed, 56 insertions(+), 15 deletions(-)
include/linux/sched.h +4 -3
index d3f6bf62ab3f..7698e7554892 100644
@@ -601,9 +601,10 @@ struct sched_dl_entity {
  *   clamp group index (group_id), i.e.
  *   index of the per-cpu RUNNABLE tasks refcounting array
  *
- * The mapped bit is set whenever a task has been mapped on a clamp group for
- * the first time. When this bit is set, any clamp group get (for a new clamp
- * value) will be matches by a clamp group put (for the old clamp value).
+ * The mapped bit is set whenever a scheduling entity has been mapped on a
+ * clamp group for the first time. When this bit is set, any clamp group get
+ * (for a new clamp value) will be matches by a clamp group put (for the old
+ * clamp value).
  *
  * The active bit is set whenever a task has got an effective clamp group
  * and value assigned, which can be different from the user requested ones.
kernel/sched/core.c +52 -12

159 lines not on this page · raw message

Signature · 1 line
2.18.0
#46 lore raw

PATCH v5 14/15 sched/core: uclamp: use TG's clamps to restrict Task's clamps

In reply to Patrick Bellasi

When a task's util_clamp value is configured via sched_setattr(2), this value has to be properly accounted in the corresponding clamp group every time the task is enqueued and dequeued. When cgroups are also in use, per-task clamp values have to be aggregated to those of the CPU's controller's Task Group (TG) in which the task is currently living.

Let's update uclamp_cpu_get() to provide aggregation between the task and the TG clamp values. Every time a task is enqueued, it will be accounted in the clamp_group which defines the smaller clamp between the task specific value and its TG effective value.

This also mimics what already happen for a task's CPU affinity mask when the task is also living in a cpuset. The overall idea is that cgroup attributes are always used to restrict the per-task attributes.

Thus, this implementation allows to:

1. ensure cgroup clamps are always used to restrict task specific requests, i.e. boosted only up to the effective granted value or clamped at least to a certain value
2. implements a "nice-like" policy, where tasks are still allowed to request less then what enforced by their current TG

For this mechanisms to work properly, we exploit the concept of "effective" clamp, which is already used by a TG to track parent enforced restrictions.
In this patch we re-use the same variable:
task_struct::uclamp::effective::group_id
to track the currently most restrictive clamp group each task is subject to and thus it's also currently refcounted into.

This solution allows also to better decouple the slow-path, where task and task group clamp values are updated, from the fast-path, where the most appropriate clamp value is tracked by refcounting clamp groups.

For consistency purposes, as well as to properly inform userspace, the sched_getattr(2) call is updated to always return the properly aggregated constrains as described above. This will also make sched_getattr(2) a convenient userspace API to know the utilization constraints enforced on a task by the cgroup's CPU controller.

Cc Ingo Molnar <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Tejun Heo <[email protected]>
Cc Paul Turner <[email protected]>
Cc Suren Baghdasaryan <[email protected]>
Cc Todd Kjos <[email protected]>
Cc Joel Fernandes <[email protected]>
Cc Steve Muckle <[email protected]>
Cc Juri Lelli <[email protected]>
Cc Quentin Perret <[email protected]>
Cc Dietmar Eggemann <[email protected]>
Cc Morten Rasmussen <[email protected]>

---
Changes in v4:
Message-ID: <20180816140731.GD2960@e110439-lin>
- reuse already existing:
task_struct::uclamp::effective::group_id
instead of adding:
task_struct::uclamp_group_id
to back annotate the effective clamp group in which a task has been refcounted
Others:
- small documentation fixes
- rebased on v4.19-rc1

Changes in v3:
Message-ID: <CAJuCfpFnj2g3+ZpR4fP4yqfxs0zd=c-Zehr2XM7m_C+WdL9jNA@mail.gmail.com>
- rename UCLAMP_NONE into UCLAMP_NOT_VALID
- fix not required override
- fix typos in changelog
Others:
- clean up uclamp_cpu_get_id()/sched_getattr() code by moving task's clamp group_id/value code into dedicated getter functions:
uclamp_task_group_id(), uclamp_group_value() and uclamp_task_value()
- rebased on tip/sched/core
Changes in v2:
OSPM discussion:
- implement a "nice" semantics where cgroup clamp values are always used to restrict task specific clamp values, i.e. tasks running on a TG are only allowed to demote themself.
Other:
- rabased on v4.18-rc4
- this code has been split from a previous patch to simplify the review

---
 include/linux/sched.h |  9 +++++++
 kernel/sched/core.c   | 58 +++++++++++++++++++++++++++++++++++++++----
 2 files changed, 62 insertions(+), 5 deletions(-)
include/linux/sched.h +9 -0
index 7698e7554892..4b61fbcb0797 100644
@@ -609,12 +609,21 @@ struct sched_dl_entity {
  * The active bit is set whenever a task has got an effective clamp group
  * and value assigned, which can be different from the user requested ones.
  * This allows to know a task is actually refcounting a CPU's clamp group.
+ *
+ * The user_defined bit is set whenever a task has got a task-specific clamp
+ * value requested from userspace, i.e. the system defaults applies to this
+ * task just as a restriction. This allows to relax TG's clamps when a less
+ * restrictive task specific value has been defined, thus allowing to
+ * implement a "nice" semantic when both task group and task specific values
+ * are used. For example, a task running on a 20% boosted TG can still drop
+ * its own boosting to 0%.
  */
 struct uclamp_se {
 	unsigned int value		: SCHED_CAPACITY_SHIFT + 1;
 	unsigned int group_id		: order_base_2(UCLAMP_GROUPS);
 	unsigned int mapped		: 1;
 	unsigned int active		: 1;
+	unsigned int user_defined	: 1;
 	/*
 	 * Clamp group and value actually used by a scheduling entity,
 	 * i.e. a (RUNNABLE) task or a task group.
kernel/sched/core.c +53 -5

106 lines not on this page · raw message

Signature · 1 line
2.18.0
#47 lore raw

Re: [PATCH v5 14/15] sched/core: uclamp: use TG's clamps to restrict Task's clamps

Slightly older version posted by error along with the correct one. Please comment on:

Message-ID: <[email protected]>

Sorry for the noise.

Quoted 239 lines · Patrick Bellasi

When a task's util_clamp value is configured via sched_setattr(2), this value has to be properly accounted in the corresponding clamp group every time the task is enqueued and dequeued. When cgroups are also in use, per-task clamp values have to be aggregated to those of the CPU's controller's Task Group (TG) in which the task is currently living.

Let's update uclamp_cpu_get() to provide aggregation between the task and the TG clamp values. Every time a task is enqueued, it will be accounted in the clamp_group which defines the smaller clamp between the task specific value and its TG effective value.

This also mimics what already happen for a task's CPU affinity mask when the task is also living in a cpuset. The overall idea is that cgroup attributes are always used to restrict the per-task attributes.

Thus, this implementation allows to:

1. ensure cgroup clamps are always used to restrict task specific requests, i.e. boosted only up to the effective granted value or clamped at least to a certain value
2. implements a "nice-like" policy, where tasks are still allowed to request less then what enforced by their current TG

For this mechanisms to work properly, we exploit the concept of "effective" clamp, which is already used by a TG to track parent enforced restrictions.
In this patch we re-use the same variable:
task_struct::uclamp::effective::group_id
to track the currently most restrictive clamp group each task is subject to and thus it's also currently refcounted into.

This solution allows also to better decouple the slow-path, where task and task group clamp values are updated, from the fast-path, where the most appropriate clamp value is tracked by refcounting clamp groups.

For consistency purposes, as well as to properly inform userspace, the sched_getattr(2) call is updated to always return the properly aggregated constrains as described above. This will also make sched_getattr(2) a convenient userspace API to know the utilization constraints enforced on a task by the cgroup's CPU controller.

Cc Ingo Molnar <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Tejun Heo <[email protected]>
Cc Paul Turner <[email protected]>
Cc Suren Baghdasaryan <[email protected]>
Cc Todd Kjos <[email protected]>
Cc Joel Fernandes <[email protected]>
Cc Steve Muckle <[email protected]>
Cc Juri Lelli <[email protected]>
Cc Quentin Perret <[email protected]>
Cc Dietmar Eggemann <[email protected]>
Cc Morten Rasmussen <[email protected]>

---
Changes in v4:
Message-ID: <20180816140731.GD2960@e110439-lin>
- reuse already existing:
task_struct::uclamp::effective::group_id
instead of adding:
task_struct::uclamp_group_id
to back annotate the effective clamp group in which a task has been refcounted
Others:
- small documentation fixes
- rebased on v4.19-rc1

Changes in v3:
Message-ID: <CAJuCfpFnj2g3+ZpR4fP4yqfxs0zd=c-Zehr2XM7m_C+WdL9jNA@mail.gmail.com>
- rename UCLAMP_NONE into UCLAMP_NOT_VALID
- fix not required override
- fix typos in changelog
Others:
- clean up uclamp_cpu_get_id()/sched_getattr() code by moving task's clamp group_id/value code into dedicated getter functions:
uclamp_task_group_id(), uclamp_group_value() and uclamp_task_value()
- rebased on tip/sched/core
Changes in v2:
OSPM discussion:
- implement a "nice" semantics where cgroup clamp values are always used to restrict task specific clamp values, i.e. tasks running on a TG are only allowed to demote themself.
Other:
- rabased on v4.18-rc4
- this code has been split from a previous patch to simplify the review

---
 include/linux/sched.h |  9 +++++++
 kernel/sched/core.c   | 58 +++++++++++++++++++++++++++++++++++++++----
 2 files changed, 62 insertions(+), 5 deletions(-)
include/linux/sched.h +9 -0
index 7698e7554892..4b61fbcb0797 100644
@@ -609,12 +609,21 @@ struct sched_dl_entity {
  * The active bit is set whenever a task has got an effective clamp group
  * and value assigned, which can be different from the user requested ones.
  * This allows to know a task is actually refcounting a CPU's clamp group.
+ *
+ * The user_defined bit is set whenever a task has got a task-specific clamp
+ * value requested from userspace, i.e. the system defaults applies to this
+ * task just as a restriction. This allows to relax TG's clamps when a less
+ * restrictive task specific value has been defined, thus allowing to
+ * implement a "nice" semantic when both task group and task specific values
+ * are used. For example, a task running on a 20% boosted TG can still drop
+ * its own boosting to 0%.
  */
 struct uclamp_se {
 	unsigned int value		: SCHED_CAPACITY_SHIFT + 1;
 	unsigned int group_id		: order_base_2(UCLAMP_GROUPS);
 	unsigned int mapped		: 1;
 	unsigned int active		: 1;
+	unsigned int user_defined	: 1;
 	/*
 	 * Clamp group and value actually used by a scheduling entity,
 	 * i.e. a (RUNNABLE) task or a task group.
kernel/sched/core.c +53 -5
index e2292c698e3b..2ce84d22ab17 100644
@@ -875,6 +875,28 @@ static inline void uclamp_cpu_update(struct rq *rq, unsigned int clamp_id,
 	rq->uclamp.value[clamp_id] = max_value;
 }
 
+/**
+ * uclamp_apply_defaults: check if p is subject to system default clamps
+ * @p: the task to check
+ *
+ * Tasks in the root group or autogroups are always and only limited by system
+ * defaults. All others instead are limited by their TG's specific value.
+ * This method checks the conditions under witch a task is subject to system
+ * default clamps.
+ */
+#ifdef CONFIG_UCLAMP_TASK_GROUP
+static inline bool uclamp_apply_defaults(struct task_struct *p)
+{
+	if (task_group_is_autogroup(task_group(p)))
+		return true;
+	if (task_group(p) == &root_task_group)
+		return true;
+	return false;
+}
+#else
+#define uclamp_apply_defaults(p) true
+#endif
+
 /**
  * uclamp_effective_group_id: get the effective clamp group index of a task
  * @p: the task to get the effective clamp value for
@@ -882,9 +904,11 @@ static inline void uclamp_cpu_update(struct rq *rq, unsigned int clamp_id,
  *
  * The effective clamp group index of a task depends on:
  * - the task specific clamp value, explicitly requested from userspace
+ * - the task group effective clamp value, for tasks not in the root group or
+ *   in an autogroup
  * - the system default clamp value, defined by the sysadmin
- * and tasks specific's clamp values are always restricted by system
- * defaults clamp values.
+ * and tasks specific's clamp values are always restricted, with increasing
+ * priority, by their task group first and the system defaults after.
  *
  * This method returns the effective group index for a task, depending on its
  * status and a proper aggregation of the clamp values listed above.
@@ -908,6 +932,22 @@ static inline unsigned int uclamp_effective_group_id(struct task_struct *p,
 	clamp_value = p->uclamp[clamp_id].value;
 	group_id = p->uclamp[clamp_id].group_id;
 
+	if (!uclamp_apply_defaults(p)) {
+#ifdef CONFIG_UCLAMP_TASK_GROUP
+		unsigned int clamp_max =
+			task_group(p)->uclamp[clamp_id].effective.value;
+		unsigned int group_max =
+			task_group(p)->uclamp[clamp_id].effective.group_id;
+
+		if (!p->uclamp[clamp_id].user_defined ||
+		    clamp_value > clamp_max) {
+			clamp_value = clamp_max;
+			group_id = group_max;
+		}
+#endif
+		goto done;
+	}
+
 	/* RT tasks have different default values */
 	default_clamp = task_has_rt_policy(p)
 		? uclamp_default_perf
@@ -924,6 +964,8 @@ static inline unsigned int uclamp_effective_group_id(struct task_struct *p,
 		group_id = default_clamp[clamp_id].group_id;
 	}
 
+done:
+
 	p->uclamp[clamp_id].effective.value = clamp_value;
 	p->uclamp[clamp_id].effective.group_id = group_id;
 
@@ -936,8 +978,10 @@ static inline unsigned int uclamp_effective_group_id(struct task_struct *p,
  * @rq: the CPU's rq where the clamp group has to be reference counted
  * @clamp_id: the clamp index to update
  *
- * Once a task is enqueued on a CPU's rq, the clamp group currently defined by
- * the task's uclamp::group_id is reference counted on that CPU.
+ * Once a task is enqueued on a CPU's rq, with increasing priority, we
+ * reference count the most restrictive clamp group between the task specific
+ * clamp value, the clamp value of its task group and the system default clamp
+ * value.
  */
 static inline void uclamp_cpu_get_id(struct task_struct *p, struct rq *rq,
 				     unsigned int clamp_id)
@@ -1312,10 +1356,12 @@ static int __setscheduler_uclamp(struct task_struct *p,
 
 	/* Update each required clamp group */
 	if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP_MIN) {
+		p->uclamp[UCLAMP_MIN].user_defined = true;
 		uclamp_group_get(p, &p->uclamp[UCLAMP_MIN],
 				 UCLAMP_MIN, lower_bound);
 	}
 	if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP_MAX) {
+		p->uclamp[UCLAMP_MAX].user_defined = true;
 		uclamp_group_get(p, &p->uclamp[UCLAMP_MAX],
 				 UCLAMP_MAX, upper_bound);
 	}
@@ -1359,8 +1405,10 @@ static void uclamp_fork(struct task_struct *p, bool reset)
 	for (clamp_id = 0; clamp_id < UCLAMP_CNT; ++clamp_id) {
 		unsigned int clamp_value = p->uclamp[clamp_id].value;
 
-		if (unlikely(reset))
+		if (unlikely(reset)) {
 			clamp_value = uclamp_none(clamp_id);
+			p->uclamp[clamp_id].user_defined = false;
+		}
 
 		p->uclamp[clamp_id].mapped = false;
 		p->uclamp[clamp_id].active = false;
Signature · 1 line
2.18.0
Signature · 3 lines
#include <best/regards.h>

Patrick Bellasi
#48 lore raw

PATCH v5 14/15 sched/core: uclamp: use TG's clamps to restrict TASK's clamps

In reply to Patrick Bellasi

When a task's util_clamp value is configured via sched_setattr(2), this value has to be properly accounted in the corresponding clamp group every time the task is enqueued and dequeued. However, when cgroups are also in use, the per-task clamp values could be restricted by the TG in which the task is currently living.

Let's update uclamp_cpu_get() to provide aggregation between the task and the TG clamp values. Every time a task is enqueued, it will be accounted in the clamp_group which defines the smaller clamp between the task specific value and its TG effective value.

This also mimics what already happen for a task's CPU affinity mask when the task is also living in a cpuset. The overall idea is that cgroup attributes are always used to restrict per-task attributes.

Thus, this implementation allows to:

1. ensure cgroup clamps are always used to restrict task specific requests, i.e. boosted only up to the effective granted value or clamped at least to a certain value

2. implements a "nice-like" policy, where tasks are still allowed to request less then what enforced by their current TG

For this mechanisms to work properly, we exploit the concept of "effective" clamp, which is already used by a TG to track parent enforced restrictions. Here we re-use the same variable:

task_struct::uclamp::effective::group_id

to track the currently most restrictive clamp group each task is subject to and thus currently refcounted into.

This solution allows also to better decouple the slow-path, where task and task group clamp values are updated, from the fast-path, where the most appropriate clamp value is tracked by refcounting clamp groups.

Cc Ingo Molnar <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Tejun Heo <[email protected]>
Cc Paul Turner <[email protected]>
Cc Suren Baghdasaryan <[email protected]>
Cc Todd Kjos <[email protected]>
Cc Joel Fernandes <[email protected]>
Cc Steve Muckle <[email protected]>
Cc Juri Lelli <[email protected]>
Cc Quentin Perret <[email protected]>
Cc Dietmar Eggemann <[email protected]>
Cc Morten Rasmussen <[email protected]>

---
Changes in v5:
Others:
- rebased on v4.19

Changes in v4:
Message-ID: <20180816140731.GD2960@e110439-lin>
- reuse already existing:
task_struct::uclamp::effective::group_id
instead of adding:
task_struct::uclamp_group_id
to back annotate the effective clamp group in which a task has been refcounted
Others:
- small documentation fixes
- rebased on v4.19-rc1
Changes in v3:
Message-ID: <CAJuCfpFnj2g3+ZpR4fP4yqfxs0zd=c-Zehr2XM7m_C+WdL9jNA@mail.gmail.com>
- rename UCLAMP_NONE into UCLAMP_NOT_VALID
- fix not required override
- fix typos in changelog
Others:
- clean up uclamp_cpu_get_id()/sched_getattr() code by moving task's clamp group_id/value code into dedicated getter functions:
uclamp_task_group_id(), uclamp_group_value() and uclamp_task_value()
- rebased on tip/sched/core
Changes in v2:
OSPM discussion:
- implement a "nice" semantics where cgroup clamp values are always used to restrict task specific clamp values, i.e. tasks running on a TG are only allowed to demote themself.
Other:
- rabased on v4.18-rc4
- this code has been split from a previous patch to simplify the review

---
 include/linux/sched.h |  9 +++++++
 kernel/sched/core.c   | 58 +++++++++++++++++++++++++++++++++++++++----
 2 files changed, 62 insertions(+), 5 deletions(-)
include/linux/sched.h +9 -0
index 7698e7554892..4b61fbcb0797 100644
@@ -609,12 +609,21 @@ struct sched_dl_entity {
  * The active bit is set whenever a task has got an effective clamp group
  * and value assigned, which can be different from the user requested ones.
  * This allows to know a task is actually refcounting a CPU's clamp group.
+ *
+ * The user_defined bit is set whenever a task has got a task-specific clamp
+ * value requested from userspace, i.e. the system defaults applies to this
+ * task just as a restriction. This allows to relax TG's clamps when a less
+ * restrictive task specific value has been defined, thus allowing to
+ * implement a "nice" semantic when both task group and task specific values
+ * are used. For example, a task running on a 20% boosted TG can still drop
+ * its own boosting to 0%.
  */
 struct uclamp_se {
 	unsigned int value		: SCHED_CAPACITY_SHIFT + 1;
 	unsigned int group_id		: order_base_2(UCLAMP_GROUPS);
 	unsigned int mapped		: 1;
 	unsigned int active		: 1;
+	unsigned int user_defined	: 1;
 	/*
 	 * Clamp group and value actually used by a scheduling entity,
 	 * i.e. a (RUNNABLE) task or a task group.
kernel/sched/core.c +53 -5

106 lines not on this page · raw message

Signature · 1 line
2.18.0
#49 lore raw

PATCH v5 15/15 sched/core: uclamp: update CPU's refcount on TG's clamp changes

In reply to Patrick Bellasi

When a task group refcounts a new clamp group, we need to ensure that the new clamp values are immediately enforced to all its tasks which are currently RUNNABLE. This is to ensure that all currently RUNNABLE tasks are boosted and/or clamped as requested as soon as possible.

Let's ensure that, whenever a new clamp group is refcounted by a task group, all its RUNNABLE tasks are correctly accounted in their respective CPUs. We do that by slightly refactoring uclamp_group_get() to get an additional parameter *cgroup_subsys_state which, when provided, it's used to walk the list of tasks in the corresponding TGs and update the RUNNABLE ones.

Cc Ingo Molnar <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Tejun Heo <[email protected]>
Cc Paul Turner <[email protected]>
Cc Suren Baghdasaryan <[email protected]>
Cc Todd Kjos <[email protected]>
Cc Joel Fernandes <[email protected]>
Cc Steve Muckle <[email protected]>
Cc Juri Lelli <[email protected]>
Cc Quentin Perret <[email protected]>
Cc Dietmar Eggemann <[email protected]>
Cc Morten Rasmussen <[email protected]>

---
Changes in v5:
Others:
- rebased in v4.19

Changes in v4:
Others:
- rebased on v4.19-rc1
Changes in v3:
- rebased on tip/sched/core
- fixed some typos
Changes in v2:
- rebased on v4.18-rc4
- this code has been split from a previous patch to simplify the review

---
 kernel/sched/core.c     | 65 ++++++++++++++++++++++++++++++++---------
 kernel/sched/features.h |  5 ++++
 2 files changed, 56 insertions(+), 14 deletions(-)
kernel/sched/core.c +51 -14
index f2e35b0a1f0c..06f0c98a1b32 100644
@@ -1197,9 +1197,30 @@ static void uclamp_group_put(unsigned int clamp_id, unsigned int group_id)
 		goto retry;
 }
 
+static inline void uclamp_group_get_tg(struct cgroup_subsys_state *css,
+				       int clamp_id, unsigned int group_id)
+{
+	struct css_task_iter it;
+	struct task_struct *p;
+
+	/*
+	 * In lazy update mode, tasks will be accounted into the right clamp
+	 * group the next time they will be requeued.
+	 */
+	if (unlikely(sched_feat(UCLAMP_LAZY_UPDATE)))
+		return;
+
+	/* Update clamp groups for RUNNABLE tasks in this TG */
+	css_task_iter_start(css, 0, &it);
+	while ((p = css_task_iter_next(&it)))
+		uclamp_task_update_active(p, clamp_id);
+	css_task_iter_end(&it);
+}
+
 /**
  * uclamp_group_get: increase the reference count for a clamp group
  * @p: the task which clamp value must be tracked
+ * @css: the task group which clamp value must be tracked
  * @uc_se: the utilization clamp data for the task
  * @clamp_id: the clamp index affected by the task
  * @clamp_value: the new clamp value for the task
@@ -1210,7 +1231,9 @@ static void uclamp_group_put(unsigned int clamp_id, unsigned int group_id)
  * reference count the corresponding clamp value while the task is enqueued on
  * a CPU.
  */
-static void uclamp_group_get(struct task_struct *p, struct uclamp_se *uc_se,
+static void uclamp_group_get(struct task_struct *p,
+			     struct cgroup_subsys_state *css,
+			     struct uclamp_se *uc_se,
 			     unsigned int clamp_id, unsigned int clamp_value)
 {
 	union uclamp_map *uc_maps = &uclamp_maps[clamp_id][0];
@@ -1279,6 +1302,10 @@ static void uclamp_group_get(struct task_struct *p, struct uclamp_se *uc_se,
 	uc_se->value = clamp_value;
 	uc_se->group_id = group_id;
 
+	/* Newly created TG don't have tasks assigned */
+	if (css)
+		uclamp_group_get_tg(css, clamp_id, group_id);
+
 	/* Update CPU's clamp group refcounts of RUNNABLE task */
 	if (p)
 		uclamp_task_update_active(p, clamp_id);
@@ -1314,11 +1341,11 @@ int sched_uclamp_handler(struct ctl_table *table, int write,
 	}
 
 	if (old_min != sysctl_sched_uclamp_util_min) {
-		uclamp_group_get(NULL, &uclamp_default[UCLAMP_MIN],
+		uclamp_group_get(NULL, NULL, &uclamp_default[UCLAMP_MIN],
 				 UCLAMP_MIN, sysctl_sched_uclamp_util_min);
 	}
 	if (old_max != sysctl_sched_uclamp_util_max) {
-		uclamp_group_get(NULL, &uclamp_default[UCLAMP_MAX],
+		uclamp_group_get(NULL, NULL, &uclamp_default[UCLAMP_MAX],
 				 UCLAMP_MAX, sysctl_sched_uclamp_util_max);
 	}
 	goto done;
@@ -1355,12 +1382,12 @@ static int __setscheduler_uclamp(struct task_struct *p,
 	/* Update each required clamp group */
 	if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP_MIN) {
 		p->uclamp[UCLAMP_MIN].user_defined = true;
-		uclamp_group_get(p, &p->uclamp[UCLAMP_MIN],
+		uclamp_group_get(p, NULL, &p->uclamp[UCLAMP_MIN],
 				 UCLAMP_MIN, lower_bound);
 	}
 	if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP_MAX) {
 		p->uclamp[UCLAMP_MAX].user_defined = true;
-		uclamp_group_get(p, &p->uclamp[UCLAMP_MAX],
+		uclamp_group_get(p, NULL, &p->uclamp[UCLAMP_MAX],
 				 UCLAMP_MAX, upper_bound);
 	}
 
@@ -1410,7 +1437,7 @@ static void uclamp_fork(struct task_struct *p, bool reset)
 
 		p->uclamp[clamp_id].mapped = false;
 		p->uclamp[clamp_id].active = false;
-		uclamp_group_get(NULL, &p->uclamp[clamp_id],
+		uclamp_group_get(NULL, NULL, &p->uclamp[clamp_id],
 				 clamp_id, clamp_value);
 	}
 }
@@ -1432,19 +1459,23 @@ static void __init init_uclamp(void)
 	memset(uclamp_maps, 0, sizeof(uclamp_maps));
 	for (clamp_id = 0; clamp_id < UCLAMP_CNT; ++clamp_id) {
 		uc_se = &init_task.uclamp[clamp_id];
-		uclamp_group_get(NULL, uc_se, clamp_id, uclamp_none(clamp_id));
+		uclamp_group_get(NULL, NULL, uc_se, clamp_id,
+				 uclamp_none(clamp_id));
 
 		uc_se = &uclamp_default[clamp_id];
-		uclamp_group_get(NULL, uc_se, clamp_id, uclamp_none(clamp_id));
+		uclamp_group_get(NULL, NULL, uc_se, clamp_id,
+				 uclamp_none(clamp_id));
 
 		/* RT tasks by default will go to max frequency */
 		uc_se = &uclamp_default_perf[clamp_id];
-		uclamp_group_get(NULL, uc_se, clamp_id, uclamp_none(UCLAMP_MAX));
+		uclamp_group_get(NULL, NULL, uc_se, clamp_id,
+				 uclamp_none(UCLAMP_MAX));
 
 #ifdef CONFIG_UCLAMP_TASK_GROUP
 		/* Init root TG's clamp group */
 		uc_se = &root_task_group.uclamp[clamp_id];
-		uclamp_group_get(NULL, uc_se, clamp_id, uclamp_none(UCLAMP_MAX));
+		uclamp_group_get(NULL, NULL, uc_se, clamp_id,
+				 uclamp_none(UCLAMP_MAX));
 		uc_se->effective.group_id = uc_se->group_id;
 		uc_se->effective.value = uc_se->value;
 #endif
@@ -7053,8 +7084,8 @@ static inline int alloc_uclamp_sched_group(struct task_group *tg,
 	int clamp_id;
 
 	for (clamp_id = 0; clamp_id < UCLAMP_CNT; ++clamp_id) {
-		uclamp_group_get(NULL, &tg->uclamp[clamp_id], clamp_id,
-				 parent->uclamp[clamp_id].value);
+		uclamp_group_get(NULL, NULL, &tg->uclamp[clamp_id],
+				 clamp_id, parent->uclamp[clamp_id].value);
 		tg->uclamp[clamp_id].effective.value =
 			parent->uclamp[clamp_id].effective.value;
 		tg->uclamp[clamp_id].effective.group_id =
@@ -7390,6 +7421,10 @@ static void cpu_util_update_hier(struct cgroup_subsys_state *css,
 
 		uc_se->effective.value = value;
 		uc_se->effective.group_id = group_id;
+
+		/* Immediately updated descendants active tasks */
+		if (css != top_css)
+			uclamp_group_get_tg(css, clamp_id, group_id);
 	}
 }
 
@@ -7414,7 +7449,8 @@ static int cpu_util_min_write_u64(struct cgroup_subsys_state *css,
 	}
 
 	/* Update TG's reference count */
-	uclamp_group_get(NULL, &tg->uclamp[UCLAMP_MIN], UCLAMP_MIN, min_value);
+	uclamp_group_get(NULL, css, &tg->uclamp[UCLAMP_MIN],
+			 UCLAMP_MIN, min_value);
 
 	/* Update effective clamps to track the most restrictive value */
 	cpu_util_update_hier(css, UCLAMP_MIN, tg->uclamp[UCLAMP_MIN].group_id,
@@ -7448,7 +7484,8 @@ static int cpu_util_max_write_u64(struct cgroup_subsys_state *css,
 	}
 
 	/* Update TG's reference count */
-	uclamp_group_get(NULL, &tg->uclamp[UCLAMP_MAX], UCLAMP_MAX, max_value);
+	uclamp_group_get(NULL, css, &tg->uclamp[UCLAMP_MAX],
+			 UCLAMP_MAX, max_value);
 
 	/* Update effective clamps to track the most restrictive value */
 	cpu_util_update_hier(css, UCLAMP_MAX, tg->uclamp[UCLAMP_MAX].group_id,
kernel/sched/features.h +5 -0
index 85ae8488039c..aad826aa55f8 100644
@@ -90,3 +90,8 @@ SCHED_FEAT(WA_BIAS, true)
  * UtilEstimation. Use estimated CPU utilization.
  */
 SCHED_FEAT(UTIL_EST, true)
+
+/*
+ * Utilization clamping lazy update.
+ */
+SCHED_FEAT(UCLAMP_LAZY_UPDATE, false)
Signature · 1 line
2.18.0