SCHED_DEADLINE cgroups support
Subsystems (2)
Juri Lelli
maintainercover SCHED_DEADLINE cgroups support
Hi,
A long time ago there was a patch [1] (written by Dario) adding DEADLINE bandwidth management control for task groups. That was then removed from the set of patches that made to mainline because outside of the bare minimum of features to possibly start playing with SCHED_DEADLINE, and because quite some discussion points remained open.
Fast forward to present day and more features have been added, DEADLINE usage is however still reserved to root only. Several things are still missing before we can comfortably relax privilegies, bandwidth management for group of tasks being one of the most important (together with a better/safer PI mechanism I'd say).
Another (different) attempt to add cgroup support was proposed last year [2]. The set was implementing hierachical scheduling support (RT entities running inside DEADLINE servers). Complexity (and maybe not enough documentation? :) made discussion around that proposal difficult to happen.
Even though hierachical scheduling is still what we want in the end, this set tries to start getting there by adding cgroup based bandwidth management for SCHED_DEADLINE. The following design choices have been made (also detailed in changelog/doc):
- implementation _is not_ hierarchical: only single/plain DEADLINE entities can be handled, and they get scheduled at root rq level
- DEADLINE_GROUP_SCHED requires RT_GROUP_SCHED (because of the points below)
- DEADLINE and RT share bandwidth; therefore, DEADLINE tasks will eat RT bandwidth, as they do today at root level; support for RT_RUNTIME_ SHARE is however missing, an RT task might be able to exceed its group bandwidth constrain if such feature is enabled (more thinking required)
- and therefore cpu.rt_runtime_us and cpu.rt_period_us are still controlling a group bandwidth; however, two additional (read only) knobs are added
# cpu.dl_bw : maximum bandwidth available for the group on each CPU
(rt_runtime_us/rt_period_us)
# cpu.dl_total_bw : current total (across CPUs) amount of bandwidth
allocated by the group (sum of tasks bandwidth)- father/children/siblings rules are the same as for RT
Adding this kind of support should be useful to be able to let normal users use DEADLINE, as the sys admin (with root privilegies) could reserve a fraction of the total available bandwidth to users and let them allocate what needed inside such space.
I'm more than sure that there are problems lurking in this set (e.g., too much ifdeffery) and many discussion points are still open, but I wanted to share what I have early and see what people thinks about it (possibly understaning how to move forward).
First patch might actually be a standalone cleanup change.
The set (based on tip/sched/core as of today) is available at:
https://github.com/jlelli/linux.git upstream/deadline/cgroup-rfc-v1
Comments and feedback are the purpose of this RFC. Thanks in advance!
Best,
- Juri
[1] https://lkml.org/lkml/2010/2/28/119
[2] https://lwn.net/Articles/718645/
Juri Lelli (3):
sched/deadline: merge dl_bw into dl_bandwidth sched/deadline: add task groups bandwidth management support Documentation/scheduler/sched-deadline: add info about cgroup support
Documentation/scheduler/sched-deadline.txt | 36 +++-- init/Kconfig | 12 ++ kernel/sched/autogroup.c | 7 + kernel/sched/core.c | 56 ++++++- kernel/sched/deadline.c | 241 +++++++++++++++++++++++------ kernel/sched/debug.c | 6 +- kernel/sched/rt.c | 52 ++++++- kernel/sched/sched.h | 68 ++++---- kernel/sched/topology.c | 2 +- 9 files changed, 381 insertions(+), 99 deletions(-)
Signature · 1 line
2.14.3
Juri Lelli
maintainerRFC PATCH 1/3 sched/deadline: merge dl_bw into dl_bandwidth
Both dl_bandwidth and dl_bw hold information about DEADLINE bandwidth admitted to the system (at different levels). However, they are separate and threated as two different beasts.
Merge them as it makes more sense, it's easier to manage and to better align with RT (that already has a single rt_bandwidth).
--- kernel/sched/core.c | 2 +- kernel/sched/deadline.c | 84 +++++++++++++++++++++++-------------------------- kernel/sched/debug.c | 6 ++-- kernel/sched/sched.h | 48 +++++++++++----------------- kernel/sched/topology.c | 2 +- 5 files changed, 63 insertions(+), 79 deletions(-)
kernel/sched/core.c +1 -1
@@ -4234,7 +4234,7 @@ static int __sched_setscheduler(struct task_struct *p,* will also fail if there's no bandwidth available. */ if (!cpumask_subset (span, &p->cpus_allowed) || - rq->rd->dl_bw.bw ==0 ) { + rq->rd->dl_bw.dl_bw ==0 ) {task_rq_unlock (rq, p, &rf);return -EPERM ; }
kernel/sched/deadline.c +39 -45
233 lines not on this page · raw message
kernel/sched/debug.c +3 -3
@@ -622,7 +622,7 @@ void print_rt_rq(struct seq_file *m, int cpu, struct rt_rq *rt_rq)void print_dl_rq (struct seq_file *m,int cpu,struct dl_rq *dl_rq) { -struct dl_bw *dl_bw; +struct dl_bandwidth *dl_bw;SEQ_printf (m,"\ndl_rq[%d]:\n" , cpu);
@@ -636,8 +636,8 @@ void print_dl_rq(struct seq_file *m, int cpu, struct dl_rq *dl_rq)#else dl_bw = &dl_rq->dl_bw;#endif -SEQ_printf (m," .%-30s: %lld\n" ,"dl_bw->bw" , dl_bw->bw); -SEQ_printf (m," .%-30s: %lld\n" ,"dl_bw->total_bw" , dl_bw->total_bw); +SEQ_printf (m," .%-30s: %lld\n" ,"dl_bw->dl_bw" , dl_bw->dl_bw); +SEQ_printf (m," .%-30s: %lld\n" ,"dl_bw->dl_total_bw" , dl_bw->dl_total_bw);#undef PU }
kernel/sched/sched.h +19 -29
110 lines not on this page · raw message
kernel/sched/topology.c +1 -1
@@ -276,7 +276,7 @@ static int init_rootdomain(struct root_domain *rd)init_irq_work (&rd->rto_push_work, rto_push_irq_work_func);#endif -init_dl_bw (&rd->dl_bw); +init_dl_bandwidth (&rd->dl_bw,global_rt_period (),global_rt_runtime ());if (cpudl_init (&rd->cpudl) !=0 ) goto free_rto_mask;
Signature · 1 line
2.14.3
Steven Rostedt
reviewerRe: [RFC PATCH 1/3] sched/deadline: merge dl_bw into dl_bandwidth
Quoted 3 lines · On Mon, 12 Feb 2018 14:40:28 +0100 Juri Lelli <[email protected]> wrote:
+ * -
dl_bw(< 100%) is the bandwidth of the system (domain) on each CPU;
+ * -dl_total_bwarray contains the currently allocated bandwidth on the
+ * i-eth CPU.
The comment for dl_total_bw doesn't make sense. You mean that dl_total_bw is the cpu's bandwidth? If so, let's not call it total, because that would suggest it's the bandwidth of all CPUs. What about dl_cpu_bw?
-- Steve
Juri Lelli
maintainerRe: [RFC PATCH 1/3] sched/deadline: merge dl_bw into dl_bandwidth
Quoted 11 lines · Steven Rostedt
Quoted 3 lines · On Mon, 12 Feb 2018 14:40:28 +0100 Juri Lelli <[email protected]> wrote:
+ * -
dl_bw(< 100%) is the bandwidth of the system (domain) on each CPU;
+ * -dl_total_bwarray contains the currently allocated bandwidth on the
+ * i-eth CPU.The comment for
dl_total_bwdoesn't make sense. You mean thatdl_total_bwis the cpu's bandwidth? If so, let's not call it total, because that would suggest it's the bandwidth of all CPUs. What aboutdl_cpu_bw?
Huh, I meant to properly fix this (broken already in mainline) comment, but I only managed to do that (hopefully) in next patch. :/
However, this surely needs to be fixed here. It's tracking the sum of all tasks' (across CPUs) bandwidth admitted on the system, so that's why it's called dl_total_bw. Incremented when a task passes sched_setattr() and decremented when it leaves the system or changes scheduling class.
Does it make a bit more sense? Would you still prefer a different name?
Thanks,
- Juri
Steven Rostedt
reviewerRe: [RFC PATCH 1/3] sched/deadline: merge dl_bw into dl_bandwidth
Quoted 6 lines · On Mon, 12 Feb 2018 18:43:12 +0100 Juri Lelli <[email protected]> wrote:
However, this surely needs to be fixed here. It's tracking the sum of all tasks' (across CPUs) bandwidth admitted on the system, so that's why it's called
dl_total_bw. Incremented when a task passessched_setattr()and decremented when it leaves the system or changes scheduling class.Does it make a bit more sense? Would you still prefer a different name?
No the name is fine, the comment needs to change.
- dl_total_bw - tracks the sum of all tasks' bandwidth across CPUs.
How's that?
-- Steve
Juri Lelli
maintainerRe: [RFC PATCH 1/3] sched/deadline: merge dl_bw into dl_bandwidth
Quoted 8 lines · Steven Rostedt
Quoted 5 lines · On Mon, 12 Feb 2018 18:43:12 +0100 Juri Lelli <[email protected]> wrote:
However, this surely needs to be fixed here. It's tracking the sum of all tasks' (across CPUs) bandwidth admitted on the system, so that's why it's called
dl_total_bw. Incremented when a task passessched_setattr()and decremented when it leaves the system or changes scheduling class.
Quoted 7 lines
Quoted 1 line
Does it make a bit more sense? Would you still prefer a different name?
No the name is fine, the comment needs to change.
-
dl_total_bw- tracks the sum of all tasks' bandwidth across CPUs.How's that?
LGTM. I'll fix in next version.
Thanks!
- Juri
Juri Lelli
maintainerRFC PATCH 2/3 sched/deadline: add task groups bandwidth management support
In reply to Juri LelliOne of the missing features of DEADLINE (w.r.t. RT) is some way of controlling CPU bandwidth allocation for task groups. Such feature would be especially useful to be able to let normal users use DEADLINE, as the sys admin (with root privilegies) could reserve a fraction of the total available bandwidth to users and let them allocate what needed inside such space.
This patch implements cgroup support for DEADLINE, with the following design choices:
- implementation _is not_ hierarchical: only single/plain DEADLINE entities can be handled, and they get scheduled at root rq level
- DEADLINE_GROUP_SCHED requires RT_GROUP_SCHED (because of the points below)
- DEADLINE and RT share bandwidth; therefore, DEADLINE tasks will eat RT bandwidth, as they do today at root level; support for RT_RUNTIME_ SHARE is however missing, an RT task might be able to exceed its group bandwidth constrain if such feature is enabled (more thinking required)
- and therefore cpu.rt_runtime_us and cpu.rt_period_us are still controlling a group bandwidth; however, two additional knobs are added
# cpu.dl_bw : maximum bandwidth available for the group on each CPU
(rt_runtime_us/rt_period_us)
# cpu.dl_total_bw : current total (across CPUs) amount of bandwidth
allocated by the group (sum of tasks bandwidth)- father/children/siblings rules are the same as for RT
--- init/Kconfig | 12 ++++ kernel/sched/autogroup.c | 7 +++ kernel/sched/core.c | 54 +++++++++++++++- kernel/sched/deadline.c | 159 ++++++++++++++++++++++++++++++++++++++++++++++- kernel/sched/rt.c | 52 ++++++++++++++-- kernel/sched/sched.h | 20 +++++- 6 files changed, 292 insertions(+), 12 deletions(-)
init/Kconfig +12 -0
@@ -751,6 +751,18 @@ config RT_GROUP_SCHED realtime bandwidth for them. See Documentation/scheduler/sched-rt-group.txt for more information. +config DEADLINE_GROUP_SCHED + bool "Group scheduling for SCHED_DEADLINE" + depends on CGROUP_SCHED + select RT_GROUP_SCHED + default n + help + This feature lets you explicitly specify, in terms of runtime + and period, the bandwidth of each task control group. This means + tasks (and other groups) can be added to it only up to such + "bandwidth cap", which might be useful for avoiding or + controlling oversubscription. + endif #CGROUP_SCHED config CGROUP_PIDS
kernel/sched/autogroup.c +7 -0
@@ -90,6 +90,13 @@ static inline struct autogroup *autogroup_create(void)free_rt_sched_group (tg); tg->rt_se = root_task_group.rt_se; tg->rt_rq = root_task_group.rt_rq; +#endif +#ifdef CONFIG_DEADLINE_GROUP_SCHED +/* +* Similarly to what above we do for DEADLINE tasks. +*/ +free_dl_sched_group (tg); + tg->dl_rq = root_task_group.dl_rq;#endif tg->autogroup = ag;
kernel/sched/core.c +52 -2
121 lines not on this page · raw message
kernel/sched/deadline.c +156 -3
208 lines not on this page · raw message
kernel/sched/rt.c +46 -6
109 lines not on this page · raw message
kernel/sched/sched.h +19 -1
@@ -285,6 +285,7 @@ extern bool dl_cpu_busy(unsigned int cpu);struct cfs_rq ;struct rt_rq ; +struct dl_rq ;extern struct list_head task_groups;
@@ -333,6 +334,10 @@ struct task_group {struct rt_bandwidth rt_bandwidth;#endif +#ifdef CONFIG_DEADLINE_GROUP_SCHED +struct dl_rq **dl_rq; +struct dl_bandwidth dl_bandwidth; +#endif struct rcu_head rcu;struct list_head list;
@@ -404,6 +409,19 @@ extern int sched_group_set_rt_period(struct task_group *tg, u64 rt_period_us);extern long sched_group_rt_runtime (struct task_group *tg);extern long sched_group_rt_period (struct task_group *tg);extern int sched_rt_can_attach (struct task_group *tg,struct task_struct *tsk); +extern struct rt_rq *sched_rt_period_rt_rq (struct rt_bandwidth *rt_b,int cpu); +extern struct rt_bandwidth *sched_rt_bandwidth_tg (struct task_group *tg); + +extern void free_dl_sched_group (struct task_group *tg); +extern int alloc_dl_sched_group (struct task_group *tg,struct task_group *parent); +extern void init_tg_dl_entry (struct task_group *tg,struct dl_rq *dl_rq, +struct sched_dl_entity *dl_se,int cpu, +struct sched_dl_entity *parent); +extern int tg_has_dl_tasks (struct task_group *tg); +extern u64 sched_group_dl_bw (struct task_group *tg); +extern u64 sched_group_dl_total_bw (struct task_group *tg); +extern int sched_dl_can_attach (struct task_group *tg,struct task_struct *tsk); +extern struct task_group *sched_create_group (struct task_group *parent); extern void sched_online_group(struct task_group *tg,
@@ -1194,7 +1212,7 @@ static inline struct task_group *task_group(struct task_struct *p)/* Change a task's cfs_rq and parent entity if it moves across CPUs/groups */ static inline void set_task_rq (struct task_struct *p,unsigned int cpu) { -#if defined(CONFIG_FAIR_GROUP_SCHED ) || defined(CONFIG_RT_GROUP_SCHED ) +#if defined(CONFIG_FAIR_GROUP_SCHED ) || defined(CONFIG_RT_GROUP_SCHED ) || defined(CONFIG_DEADLINE_GROUP_SCHED )struct task_group *tg =task_group (p);#endif
Signature · 1 line
2.14.3
Re: [RFC PATCH 2/3] sched/deadline: add task groups bandwidth management support
Hello,
Quoted 2 lines · Juri Lelli
- implementation
_isnot_hierarchical: only single/plain DEADLINE entities can be handled, and they get scheduled at root rq level
This usually is a deal breaker and often indicates that the cgroup filesystem is not the right interface for the feature. Can you please elaborate the interface with some details?
Thanks.
Signature · 1 line
tejun
Juri Lelli
maintainerRe: [RFC PATCH 2/3] sched/deadline: add task groups bandwidth management support
Hi,
Quoted 9 lines · Tejun Heo
Hello,
Quoted 2 lines · Juri Lelli
- implementation
_isnot_hierarchical: only single/plain DEADLINE entities can be handled, and they get scheduled at root rq levelThis usually is a deal breaker and often indicates that the cgroup filesystem is not the right interface for the feature. Can you please elaborate the interface with some details?
The interface is the same as what we have today for groups of RT tasks, and same rules apply. The difference is that when using RT <group>/cpu.rt_runtime_us and <group>/cpu.rt_period_us control RT-Throttling behaviour (fraction of CPU time and granularity), while for DEADLINE the same interface would be used only at admission control time (while servicing a sched_setattr(), attaching tasks to a group or changing group's parameters) since DEADLINE task have their own throttling mechanism already.
Intended usage should be very similar. For example, a sys admin that wants to reserve and guarantee CPU bandwidth for a group of tasks would create a group, configure its rt_runtime_us, rt_period_us and put DEADLINE tasks inside it (e.g. video/audio pipeline). Related to what I was saying in the cover letter (i.e., non root access to DEADLINE scheduling) might be a different situation, where sys admin wants to grant a user a certain percentage of CPU time (by creating a group and putting user session inside it) and also control that user doesn't exceed what granted. User would then be free to spawn DEADLINE tasks to service her/his needs up to the maximum bandwidth cap set by sys admin.
Does this make any sense and provide a bit more information?
Thanks a lot for looking at this!
Best,
- Juri
Juri Lelli
maintainerRFC PATCH 3/3 Documentation/scheduler/sched-deadline: add info about cgroup support
In reply to Juri LelliAdd documentation for SCHED_DEADLINE cgroup support (CONFIG_DEADLINE_ GROUP_SCHED config option).
--- Documentation/scheduler/sched-deadline.txt | 36 ++++++++++++++++++++++-------- 1 file changed, 27 insertions(+), 9 deletions(-)
Documentation/scheduler/sched-deadline.txt +27 -9
@@ -528,11 +528,8 @@ CONTENTS to -deadline tasks is similar to the one already used for -rt tasks with real-time group scheduling (a.k.a. RT-throttling - see Documentation/scheduler/sched-rt-group.txt), and is based on readable/ - writable control files located in procfs (for system wide settings). - Notice that per-group settings (controlled through cgroupfs) are still not - defined for -deadline tasks, because more discussion is needed in order to - figure out how we want to manage SCHED_DEADLINE bandwidth at the task group - level. + writable control files located in procfs (for system wide settings) and in + cgroupfs (per-group settings). A main difference between deadline bandwidth management and RT-throttling is that -deadline tasks have bandwidth on their own (while -rt ones don't!),
@@ -553,9 +550,9 @@ CONTENTS For now the -rt knobs are used for -deadline admission control and the -deadline runtime is accounted against the -rt runtime. We realize that this isn't entirely desirable; however, it is better to have a small interface for - now, and be able to change it easily later. The ideal situation (see 5.) is to - run -rt tasks from a -deadline server; in which case the -rt bandwidth is a - direct subset of dl_bw. + now, and be able to change it easily later. The ideal situation (see 6.) is to + run -rt tasks from a -deadline server (H-CBS); in which case the -rt bandwidth + is a direct subset of dl_bw. This means that, for a root_domain comprising M CPUs, -deadline tasks can be created while the sum of their bandwidths stays below:
@@ -623,6 +620,27 @@ CONTENTS make the leftoever runtime available for reclamation by other SCHED_DEADLINE tasks. +4.4 Grouping tasks +------------------ + +CONFIG_DEADLINE_GROUP_SCHED depends on CONFIG_RT_GROUP_SCHED, so go on and +read Documentation/scheduler/sched-rt-group.txt first. + +Enabling CONFIG_DEADLINE_GROUP_SCHED lets you explicitly manage CPU bandwidth +for task groups. + +This uses the cgroup virtual file system and "<cgroup>/cpu.rt_runtime_us +<cgroup>/cpu.rt_period_us" to control the CPU time reserved for each control +group. Yes, they are the same of CONFIG_RT_GROUP_SCHED since RT and DEADLINE +share the same bandwidth. In addition to these CONFIG_DEADLINE_GROUP_SCHED +adds "<cgroup>/cpu.dl_bw" (maximum bandwidth on each CPU available to the +group, corresponds to cpu.rt_runtime_us/cpu.rt_period_us) and +"<cgroup>/cpu.dl_total_bw" (a group's current allocated bandwidth); both are +non-writable. + +Group settings are checked against same limits of CONFIG_RT_GROUP_SCHED: + + \Sum_{i} runtime_{i} / global_period <= global_runtime / global_period 5. Tasks CPU affinity =====================
@@ -661,7 +679,7 @@ CONTENTS of retaining bandwidth isolation among non-interacting tasks. This is being studied from both theoretical and practical points of view, and hopefully we should be able to produce some demonstrative code soon; - - (c)group based bandwidth management, and maybe scheduling; + - (c)group based scheduling (Hierachical-CBS); - access control for non-root users (and related security concerns to address), which is the best way to allow unprivileged use of the mechanisms and how to prevent non-root users "cheat" the system?
Signature · 1 line
2.14.3