Skip to contentSkip to conversation

SCHED_DEADLINE cgroups support

RFC PATCH In review Juri Lelli 3 participants 10 messages Feb 12, 2018

Subsystems (2)
lore
Share & tools
lore

Juri Lelli

author maintainer
#0 lore raw

cover SCHED_DEADLINE cgroups support

Hi,

A long time ago there was a patch [1] (written by Dario) adding DEADLINE bandwidth management control for task groups. That was then removed from the set of patches that made to mainline because outside of the bare minimum of features to possibly start playing with SCHED_DEADLINE, and because quite some discussion points remained open.

Fast forward to present day and more features have been added, DEADLINE usage is however still reserved to root only. Several things are still missing before we can comfortably relax privilegies, bandwidth management for group of tasks being one of the most important (together with a better/safer PI mechanism I'd say).

Another (different) attempt to add cgroup support was proposed last year [2]. The set was implementing hierachical scheduling support (RT entities running inside DEADLINE servers). Complexity (and maybe not enough documentation? :) made discussion around that proposal difficult to happen.

Even though hierachical scheduling is still what we want in the end, this set tries to start getting there by adding cgroup based bandwidth management for SCHED_DEADLINE. The following design choices have been made (also detailed in changelog/doc):

- implementation _is not_ hierarchical: only single/plain DEADLINE entities can be handled, and they get scheduled at root rq level

- DEADLINE_GROUP_SCHED requires RT_GROUP_SCHED (because of the points below)

- DEADLINE and RT share bandwidth; therefore, DEADLINE tasks will eat RT bandwidth, as they do today at root level; support for RT_RUNTIME_ SHARE is however missing, an RT task might be able to exceed its group bandwidth constrain if such feature is enabled (more thinking required)

- and therefore cpu.rt_runtime_us and cpu.rt_period_us are still controlling a group bandwidth; however, two additional (read only) knobs are added

     # cpu.dl_bw : maximum bandwidth available for the group on each CPU
                   (rt_runtime_us/rt_period_us)
     # cpu.dl_total_bw : current total (across CPUs) amount of bandwidth
                         allocated by the group (sum of tasks bandwidth)

- father/children/siblings rules are the same as for RT

Adding this kind of support should be useful to be able to let normal users use DEADLINE, as the sys admin (with root privilegies) could reserve a fraction of the total available bandwidth to users and let them allocate what needed inside such space.

I'm more than sure that there are problems lurking in this set (e.g., too much ifdeffery) and many discussion points are still open, but I wanted to share what I have early and see what people thinks about it (possibly understaning how to move forward).

First patch might actually be a standalone cleanup change.

The set (based on tip/sched/core as of today) is available at:

https://github.com/jlelli/linux.git upstream/deadline/cgroup-rfc-v1

Comments and feedback are the purpose of this RFC. Thanks in advance!

Best,

- Juri

[1] https://lkml.org/lkml/2010/2/28/119
[2] https://lwn.net/Articles/718645/

Juri Lelli (3):
sched/deadline: merge dl_bw into dl_bandwidth sched/deadline: add task groups bandwidth management support Documentation/scheduler/sched-deadline: add info about cgroup support

 Documentation/scheduler/sched-deadline.txt |  36 +++--
 init/Kconfig                               |  12 ++
 kernel/sched/autogroup.c                   |   7 +
 kernel/sched/core.c                        |  56 ++++++-
 kernel/sched/deadline.c                    | 241 +++++++++++++++++++++++------
 kernel/sched/debug.c                       |   6 +-
 kernel/sched/rt.c                          |  52 ++++++-
 kernel/sched/sched.h                       |  68 ++++----
 kernel/sched/topology.c                    |   2 +-
 9 files changed, 381 insertions(+), 99 deletions(-)
Signature · 1 line
2.14.3
author maintainer
#1 lore raw

RFC PATCH 1/3 sched/deadline: merge dl_bw into dl_bandwidth

Both dl_bandwidth and dl_bw hold information about DEADLINE bandwidth admitted to the system (at different levels). However, they are separate and threated as two different beasts.

Merge them as it makes more sense, it's easier to manage and to better align with RT (that already has a single rt_bandwidth).

Cc Ingo Molnar <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Luca Abeni <[email protected]>
---
 kernel/sched/core.c     |  2 +-
 kernel/sched/deadline.c | 84 +++++++++++++++++++++++--------------------------
 kernel/sched/debug.c    |  6 ++--
 kernel/sched/sched.h    | 48 +++++++++++-----------------
 kernel/sched/topology.c |  2 +-
 5 files changed, 63 insertions(+), 79 deletions(-)
kernel/sched/core.c +1 -1
index ee420d78e674..772a6b3239eb 100644
@@ -4234,7 +4234,7 @@ static int __sched_setscheduler(struct task_struct *p,
 			 * will also fail if there's no bandwidth available.
 			 */
 			if (!cpumask_subset(span, &p->cpus_allowed) ||
-			    rq->rd->dl_bw.bw == 0) {
+			    rq->rd->dl_bw.dl_bw == 0) {
 				task_rq_unlock(rq, p, &rf);
 				return -EPERM;
 			}
kernel/sched/deadline.c +39 -45

233 lines not on this page · raw message

kernel/sched/debug.c +3 -3
index 1ca0130ed4f9..cf736a30350e 100644
@@ -622,7 +622,7 @@ void print_rt_rq(struct seq_file *m, int cpu, struct rt_rq *rt_rq)
 
 void print_dl_rq(struct seq_file *m, int cpu, struct dl_rq *dl_rq)
 {
-	struct dl_bw *dl_bw;
+	struct dl_bandwidth *dl_bw;
 
 	SEQ_printf(m, "\ndl_rq[%d]:\n", cpu);
 
@@ -636,8 +636,8 @@ void print_dl_rq(struct seq_file *m, int cpu, struct dl_rq *dl_rq)
 #else
 	dl_bw = &dl_rq->dl_bw;
 #endif
-	SEQ_printf(m, "  .%-30s: %lld\n", "dl_bw->bw", dl_bw->bw);
-	SEQ_printf(m, "  .%-30s: %lld\n", "dl_bw->total_bw", dl_bw->total_bw);
+	SEQ_printf(m, "  .%-30s: %lld\n", "dl_bw->dl_bw", dl_bw->dl_bw);
+	SEQ_printf(m, "  .%-30s: %lld\n", "dl_bw->dl_total_bw", dl_bw->dl_total_bw);
 
 #undef PU
 }
kernel/sched/sched.h +19 -29

110 lines not on this page · raw message

kernel/sched/topology.c +1 -1
index 034cbed7f88b..0700f3f40445 100644
@@ -276,7 +276,7 @@ static int init_rootdomain(struct root_domain *rd)
 	init_irq_work(&rd->rto_push_work, rto_push_irq_work_func);
 #endif
 
-	init_dl_bw(&rd->dl_bw);
+	init_dl_bandwidth(&rd->dl_bw, global_rt_period(), global_rt_runtime());
 	if (cpudl_init(&rd->cpudl) != 0)
 		goto free_rto_mask;
 
Signature · 1 line
2.14.3
#2 lore raw

Re: [RFC PATCH 1/3] sched/deadline: merge dl_bw into dl_bandwidth

Quoted 3 lines · On Mon, 12 Feb 2018 14:40:28 +0100 Juri Lelli <[email protected]> wrote:

+ * - dl_bw (< 100%) is the bandwidth of the system (domain) on each CPU;
+ * - dl_total_bw array contains the currently allocated bandwidth on the
+ * i-eth CPU.

The comment for dl_total_bw doesn't make sense. You mean that dl_total_bw is the cpu's bandwidth? If so, let's not call it total, because that would suggest it's the bandwidth of all CPUs. What about dl_cpu_bw?

-- Steve

author maintainer
#3 lore raw

Re: [RFC PATCH 1/3] sched/deadline: merge dl_bw into dl_bandwidth

Quoted 11 lines · Steven Rostedt
Quoted 3 lines · On Mon, 12 Feb 2018 14:40:28 +0100 Juri Lelli <[email protected]> wrote:

+ * - dl_bw (< 100%) is the bandwidth of the system (domain) on each CPU;
+ * - dl_total_bw array contains the currently allocated bandwidth on the
+ * i-eth CPU.

The comment for dl_total_bw doesn't make sense. You mean that dl_total_bw is the cpu's bandwidth? If so, let's not call it total, because that would suggest it's the bandwidth of all CPUs. What about dl_cpu_bw?

Huh, I meant to properly fix this (broken already in mainline) comment, but I only managed to do that (hopefully) in next patch. :/

However, this surely needs to be fixed here. It's tracking the sum of all tasks' (across CPUs) bandwidth admitted on the system, so that's why it's called dl_total_bw. Incremented when a task passes sched_setattr() and decremented when it leaves the system or changes scheduling class.

Does it make a bit more sense? Would you still prefer a different name?

Thanks,

- Juri

#4 lore raw

Re: [RFC PATCH 1/3] sched/deadline: merge dl_bw into dl_bandwidth

Quoted 6 lines · On Mon, 12 Feb 2018 18:43:12 +0100 Juri Lelli <[email protected]> wrote:

However, this surely needs to be fixed here. It's tracking the sum of all tasks' (across CPUs) bandwidth admitted on the system, so that's why it's called dl_total_bw. Incremented when a task passes sched_setattr() and decremented when it leaves the system or changes scheduling class.

Does it make a bit more sense? Would you still prefer a different name?

No the name is fine, the comment needs to change.

- dl_total_bw - tracks the sum of all tasks' bandwidth across CPUs.

How's that?

-- Steve

author maintainer
#5 lore raw

Re: [RFC PATCH 1/3] sched/deadline: merge dl_bw into dl_bandwidth

Quoted 8 lines · Steven Rostedt
Quoted 5 lines · On Mon, 12 Feb 2018 18:43:12 +0100 Juri Lelli <[email protected]> wrote:

However, this surely needs to be fixed here. It's tracking the sum of all tasks' (across CPUs) bandwidth admitted on the system, so that's why it's called dl_total_bw. Incremented when a task passes sched_setattr() and decremented when it leaves the system or changes scheduling class.

Quoted 7 lines
Quoted 1 line

Does it make a bit more sense? Would you still prefer a different name?

No the name is fine, the comment needs to change.

- dl_total_bw - tracks the sum of all tasks' bandwidth across CPUs.

How's that?

LGTM. I'll fix in next version.

Thanks!

- Juri

author maintainer
#6 lore raw

RFC PATCH 2/3 sched/deadline: add task groups bandwidth management support

In reply to Juri Lelli

One of the missing features of DEADLINE (w.r.t. RT) is some way of controlling CPU bandwidth allocation for task groups. Such feature would be especially useful to be able to let normal users use DEADLINE, as the sys admin (with root privilegies) could reserve a fraction of the total available bandwidth to users and let them allocate what needed inside such space.

This patch implements cgroup support for DEADLINE, with the following design choices:

- implementation _is not_ hierarchical: only single/plain DEADLINE entities can be handled, and they get scheduled at root rq level

- DEADLINE_GROUP_SCHED requires RT_GROUP_SCHED (because of the points below)

- DEADLINE and RT share bandwidth; therefore, DEADLINE tasks will eat RT bandwidth, as they do today at root level; support for RT_RUNTIME_ SHARE is however missing, an RT task might be able to exceed its group bandwidth constrain if such feature is enabled (more thinking required)

- and therefore cpu.rt_runtime_us and cpu.rt_period_us are still controlling a group bandwidth; however, two additional knobs are added

     # cpu.dl_bw : maximum bandwidth available for the group on each CPU
                   (rt_runtime_us/rt_period_us)
     # cpu.dl_total_bw : current total (across CPUs) amount of bandwidth
                         allocated by the group (sum of tasks bandwidth)

- father/children/siblings rules are the same as for RT

Cc Ingo Molnar <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Tejun Heo <[email protected]>
Cc Luca Abeni <[email protected]>
---
 init/Kconfig             |  12 ++++
 kernel/sched/autogroup.c |   7 +++
 kernel/sched/core.c      |  54 +++++++++++++++-
 kernel/sched/deadline.c  | 159 ++++++++++++++++++++++++++++++++++++++++++++++-
 kernel/sched/rt.c        |  52 ++++++++++++++--
 kernel/sched/sched.h     |  20 +++++-
 6 files changed, 292 insertions(+), 12 deletions(-)
init/Kconfig +12 -0
index e37f4b2a6445..c6ddda90d51f 100644
@@ -751,6 +751,18 @@ config RT_GROUP_SCHED
 	  realtime bandwidth for them.
 	  See Documentation/scheduler/sched-rt-group.txt for more information.
 
+config DEADLINE_GROUP_SCHED
+	bool "Group scheduling for SCHED_DEADLINE"
+	depends on CGROUP_SCHED
+	select RT_GROUP_SCHED
+	default n
+	help
+	  This feature lets you explicitly specify, in terms of runtime
+	  and period, the bandwidth of each task control group. This means
+	  tasks (and other groups) can be added to it only up to such
+	  "bandwidth cap", which might be useful for avoiding or
+	  controlling oversubscription.
+
 endif #CGROUP_SCHED
 
 config CGROUP_PIDS
kernel/sched/autogroup.c +7 -0
index a43df5193538..7cba2e132ac7 100644
@@ -90,6 +90,13 @@ static inline struct autogroup *autogroup_create(void)
 	free_rt_sched_group(tg);
 	tg->rt_se = root_task_group.rt_se;
 	tg->rt_rq = root_task_group.rt_rq;
+#endif
+#ifdef CONFIG_DEADLINE_GROUP_SCHED
+	/*
+	 * Similarly to what above we do for DEADLINE tasks.
+	 */
+	free_dl_sched_group(tg);
+	tg->dl_rq = root_task_group.dl_rq;
 #endif
 	tg->autogroup = ag;
 
kernel/sched/core.c +52 -2

121 lines not on this page · raw message

kernel/sched/deadline.c +156 -3

208 lines not on this page · raw message

kernel/sched/rt.c +46 -6

109 lines not on this page · raw message

kernel/sched/sched.h +19 -1
index 7c44c8baa98c..850aacc8f241 100644
@@ -285,6 +285,7 @@ extern bool dl_cpu_busy(unsigned int cpu);
 
 struct cfs_rq;
 struct rt_rq;
+struct dl_rq;
 
 extern struct list_head task_groups;
 
@@ -333,6 +334,10 @@ struct task_group {
 
 	struct rt_bandwidth rt_bandwidth;
 #endif
+#ifdef CONFIG_DEADLINE_GROUP_SCHED
+	struct dl_rq **dl_rq;
+	struct dl_bandwidth dl_bandwidth;
+#endif
 
 	struct rcu_head rcu;
 	struct list_head list;
@@ -404,6 +409,19 @@ extern int sched_group_set_rt_period(struct task_group *tg, u64 rt_period_us);
 extern long sched_group_rt_runtime(struct task_group *tg);
 extern long sched_group_rt_period(struct task_group *tg);
 extern int sched_rt_can_attach(struct task_group *tg, struct task_struct *tsk);
+extern struct rt_rq *sched_rt_period_rt_rq(struct rt_bandwidth *rt_b, int cpu);
+extern struct rt_bandwidth *sched_rt_bandwidth_tg(struct task_group *tg);
+
+extern void free_dl_sched_group(struct task_group *tg);
+extern int alloc_dl_sched_group(struct task_group *tg, struct task_group *parent);
+extern void init_tg_dl_entry(struct task_group *tg, struct dl_rq *dl_rq,
+		struct sched_dl_entity *dl_se, int cpu,
+		struct sched_dl_entity *parent);
+extern int tg_has_dl_tasks(struct task_group *tg);
+extern u64 sched_group_dl_bw(struct task_group *tg);
+extern u64 sched_group_dl_total_bw(struct task_group *tg);
+extern int sched_dl_can_attach(struct task_group *tg, struct task_struct *tsk);
+
 
 extern struct task_group *sched_create_group(struct task_group *parent);
 extern void sched_online_group(struct task_group *tg,
@@ -1194,7 +1212,7 @@ static inline struct task_group *task_group(struct task_struct *p)
 /* Change a task's cfs_rq and parent entity if it moves across CPUs/groups */
 static inline void set_task_rq(struct task_struct *p, unsigned int cpu)
 {
-#if defined(CONFIG_FAIR_GROUP_SCHED) || defined(CONFIG_RT_GROUP_SCHED)
+#if defined(CONFIG_FAIR_GROUP_SCHED) || defined(CONFIG_RT_GROUP_SCHED) || defined(CONFIG_DEADLINE_GROUP_SCHED)
 	struct task_group *tg = task_group(p);
 #endif
 
Signature · 1 line
2.14.3
#7 lore raw

Re: [RFC PATCH 2/3] sched/deadline: add task groups bandwidth management support

Hello,

Quoted 2 lines · Juri Lelli

- implementation _is not_ hierarchical: only single/plain DEADLINE entities can be handled, and they get scheduled at root rq level

This usually is a deal breaker and often indicates that the cgroup filesystem is not the right interface for the feature. Can you please elaborate the interface with some details?

Thanks.

Signature · 1 line
tejun
author maintainer
#8 lore raw

Re: [RFC PATCH 2/3] sched/deadline: add task groups bandwidth management support

Hi,

Quoted 9 lines · Tejun Heo

Hello,

Quoted 2 lines · Juri Lelli

- implementation _is not_ hierarchical: only single/plain DEADLINE entities can be handled, and they get scheduled at root rq level

This usually is a deal breaker and often indicates that the cgroup filesystem is not the right interface for the feature. Can you please elaborate the interface with some details?

The interface is the same as what we have today for groups of RT tasks, and same rules apply. The difference is that when using RT <group>/cpu.rt_runtime_us and <group>/cpu.rt_period_us control RT-Throttling behaviour (fraction of CPU time and granularity), while for DEADLINE the same interface would be used only at admission control time (while servicing a sched_setattr(), attaching tasks to a group or changing group's parameters) since DEADLINE task have their own throttling mechanism already.

Intended usage should be very similar. For example, a sys admin that wants to reserve and guarantee CPU bandwidth for a group of tasks would create a group, configure its rt_runtime_us, rt_period_us and put DEADLINE tasks inside it (e.g. video/audio pipeline). Related to what I was saying in the cover letter (i.e., non root access to DEADLINE scheduling) might be a different situation, where sys admin wants to grant a user a certain percentage of CPU time (by creating a group and putting user session inside it) and also control that user doesn't exceed what granted. User would then be free to spawn DEADLINE tasks to service her/his needs up to the maximum bandwidth cap set by sys admin.

Does this make any sense and provide a bit more information?

Thanks a lot for looking at this!

Best,

- Juri

author maintainer
#9 lore raw

RFC PATCH 3/3 Documentation/scheduler/sched-deadline: add info about cgroup support

In reply to Juri Lelli

Add documentation for SCHED_DEADLINE cgroup support (CONFIG_DEADLINE_ GROUP_SCHED config option).

Cc Ingo Molnar <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Tejun Heo <[email protected]>
Cc Jonathan Corbet <[email protected]>
Cc Luca Abeni <[email protected]>
---
 Documentation/scheduler/sched-deadline.txt | 36 ++++++++++++++++++++++--------
 1 file changed, 27 insertions(+), 9 deletions(-)
Documentation/scheduler/sched-deadline.txt +27 -9
index 8ce78f82ae23..65d55c778976 100644
@@ -528,11 +528,8 @@ CONTENTS
  to -deadline tasks is similar to the one already used for -rt
  tasks with real-time group scheduling (a.k.a. RT-throttling - see
  Documentation/scheduler/sched-rt-group.txt), and is based on readable/
- writable control files located in procfs (for system wide settings).
- Notice that per-group settings (controlled through cgroupfs) are still not
- defined for -deadline tasks, because more discussion is needed in order to
- figure out how we want to manage SCHED_DEADLINE bandwidth at the task group
- level.
+ writable control files located in procfs (for system wide settings) and in
+ cgroupfs (per-group settings).
 
  A main difference between deadline bandwidth management and RT-throttling
  is that -deadline tasks have bandwidth on their own (while -rt ones don't!),
@@ -553,9 +550,9 @@ CONTENTS
  For now the -rt knobs are used for -deadline admission control and the
  -deadline runtime is accounted against the -rt runtime. We realize that this
  isn't entirely desirable; however, it is better to have a small interface for
- now, and be able to change it easily later. The ideal situation (see 5.) is to
- run -rt tasks from a -deadline server; in which case the -rt bandwidth is a
- direct subset of dl_bw.
+ now, and be able to change it easily later. The ideal situation (see 6.) is to
+ run -rt tasks from a -deadline server (H-CBS); in which case the -rt bandwidth
+ is a direct subset of dl_bw.
 
  This means that, for a root_domain comprising M CPUs, -deadline tasks
  can be created while the sum of their bandwidths stays below:
@@ -623,6 +620,27 @@ CONTENTS
  make the leftoever runtime available for reclamation by other
  SCHED_DEADLINE tasks.
 
+4.4 Grouping tasks
+------------------
+
+CONFIG_DEADLINE_GROUP_SCHED depends on CONFIG_RT_GROUP_SCHED, so go on and
+read Documentation/scheduler/sched-rt-group.txt first.
+
+Enabling CONFIG_DEADLINE_GROUP_SCHED lets you explicitly manage CPU bandwidth
+for task groups.
+
+This uses the cgroup virtual file system and "<cgroup>/cpu.rt_runtime_us
+<cgroup>/cpu.rt_period_us" to control the CPU time reserved for each control
+group. Yes, they are the same of CONFIG_RT_GROUP_SCHED since RT and DEADLINE
+share the same bandwidth. In addition to these CONFIG_DEADLINE_GROUP_SCHED
+adds "<cgroup>/cpu.dl_bw" (maximum bandwidth on each CPU available to the
+group, corresponds to cpu.rt_runtime_us/cpu.rt_period_us) and
+"<cgroup>/cpu.dl_total_bw" (a group's current allocated bandwidth); both are
+non-writable.
+
+Group settings are checked against same limits of CONFIG_RT_GROUP_SCHED:
+
+   \Sum_{i} runtime_{i} / global_period <= global_runtime / global_period
 
 5. Tasks CPU affinity
 =====================
@@ -661,7 +679,7 @@ CONTENTS
     of retaining bandwidth isolation among non-interacting tasks. This is
     being studied from both theoretical and practical points of view, and
     hopefully we should be able to produce some demonstrative code soon;
-  - (c)group based bandwidth management, and maybe scheduling;
+  - (c)group based scheduling (Hierachical-CBS);
   - access control for non-root users (and related security concerns to
     address), which is the best way to allow unprivileged use of the mechanisms
     and how to prevent non-root users "cheat" the system?
Signature · 1 line
2.14.3