Skip to contentSkip to conversation

sched,fair: flatten CPU controller runqueues

RFC PATCH Superseded Rik van Riel 3 participants 27 messages Jul 22, 2019 – Aug 12, 2019 Scheduler

v3 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

v3 cover sched,fair: flatten CPU controller runqueues

The current implementation of the CPU controller uses hierarchical runqueues, where on wakeup a task is enqueued on its group's runqueue, the group is enqueued on the runqueue of the group above it, etc.

This increases a fairly large amount of overhead for workloads that do a lot of wakeups a second, especially given that the default systemd hierarchy is 2 or 3 levels deep.

This patch series is an attempt at reducing that overhead, by placing all the tasks on the same runqueue, and scaling the task priority by the priority of the group, which is calculated periodically.

My main TODO items for the next period of time are likely going to be testing, testing, and testing. I hope to find and flush out any corner case I can find, and make sure performance does not regress with any workloads, and hopefully improves some.

Other TODO items:
- More code cleanups.
- Remove some more now unused code.
- Reimplement CONFIG_CFS_BANDWIDTH.

Plan for the CONFIG_CFS_BANDWIDTH reimplementation:
- When a cgroup gets throttled, mark the cgroup and its children as throttled.
- When pick_next_entity finds a task that is on a throttled cgroup, stash it on the cgroup runqueue (which is not used for runnable tasks any more). Leave the vruntime unchanged, and adjust that runqueue's vruntime to be that of the left-most task.
- When a cgroup gets unthrottled, and has tasks on it, place it on a vruntime ordered heap separate from the main runqueue.
- Have pick_next_task_fair grab one task off that heap every time it is called, and the min vruntime of that heap is lower than the vruntime of the CPU's cfs_rq (or the CPU has no other runnable tasks).
- Place that selected task on the CPU's cfs_rq, renormalizing its vruntime with the GENTLE_FAIR_SLEEPERS logic. That should help interleave the already runnable tasks with the recently unthrottled group, and prevent thundering herd issues.
- If the group gets throttled again before all of its task had a chance to run, vruntime sorting ensures all the tasks in the throttled cgroup get a chance to run over time.

Changes from v2:
- fixed the web server performance regression, in a way vaguely similar to what Josef Bacik suggested (blame me for the implementation)
- removed some code duplication so the diffstat is redder than before
- propagate sum_exec_runtime up the tree, in preparation for CFS_BANDWIDTH
- small cleanups left and right

Changes from v1:
- use task_se_h_weight instead of task_se_h_load in calc_delta_fair and sched_slice, this seems to improve performance a little, but I still have some remaining regression to chase with our web server workload
- implement a number of the changes suggested by Dietmar Eggemann (still holding out for a better name for group_cfs_rq_of_parent)

This series applies on top of 5.2

 include/linux/sched.h |    7 
 kernel/sched/core.c   |    3 
 kernel/sched/debug.c  |   17 -
 kernel/sched/fair.c   |  780 ++++++++++++++++++++------------------------------
 kernel/sched/pelt.c   |   69 ++--
 kernel/sched/pelt.h   |    3 
 kernel/sched/sched.h  |   11 
 7 files changed, 372 insertions(+), 518 deletions(-)
#1 lore raw

PATCH v3 1/14 sched: introduce task_se_h_load helper

Sometimes the hierarchical load of a sched_entity needs to be calculated. Rename task_h_load to task_se_h_load, and directly pass a sched_entity to that function.

Move the function declaration up above where it will be used later.

No functional changes.

---
 kernel/sched/fair.c | 28 ++++++++++++++--------------
 1 file changed, 14 insertions(+), 14 deletions(-)
kernel/sched/fair.c +14 -14
index f35930f5e528..eadf9a96b3e1 100644
@@ -242,6 +242,7 @@ static u64 __calc_delta(u64 delta_exec, unsigned long weight, struct load_weight
 
 
 const struct sched_class fair_sched_class;
+static unsigned long task_se_h_load(struct sched_entity *se);
 
 /**************************************************************
  * CFS operations on generic schedulable entities:
@@ -706,7 +707,6 @@ static u64 sched_vslice(struct cfs_rq *cfs_rq, struct sched_entity *se)
 #ifdef CONFIG_SMP
 
 static int select_idle_sibling(struct task_struct *p, int prev_cpu, int cpu);
-static unsigned long task_h_load(struct task_struct *p);
 static unsigned long capacity_of(int cpu);
 
 /* Give new sched_entity start runnable values to heavy its load in infant time */
@@ -1668,7 +1668,7 @@ static void task_numa_compare(struct task_numa_env *env,
 	/*
 	 * In the overloaded case, try and keep the load balanced.
 	 */
-	load = task_h_load(env->p) - task_h_load(cur);
+	load = task_se_h_load(env->p->se) - task_se_h_load(cur->se);
 	if (!load)
 		goto assign;
 
@@ -1706,7 +1706,7 @@ static void task_numa_find_cpu(struct task_numa_env *env,
 	bool maymove = false;
 	int cpu;
 
-	load = task_h_load(env->p);
+	load = task_se_h_load(env->p->se);
 	dst_load = env->dst_stats.load + load;
 	src_load = env->src_stats.load - load;
 
@@ -3389,7 +3389,7 @@ static inline void add_tg_cfs_propagate(struct cfs_rq *cfs_rq, long runnable_sum
  * avg. The immediate corollary is that all (fair) tasks must be attached, see
  * post_init_entity_util_avg().
  *
- * cfs_rq->avg is used for task_h_load() and update_cfs_share() for example.
+ * cfs_rq->avg is used for task_se_h_load() and update_cfs_share() for example.
  *
  * Returns true if the load decayed or we removed load.
  *
@@ -3522,7 +3522,7 @@ static inline void update_load_avg(struct cfs_rq *cfs_rq, struct sched_entity *s
 
 	/*
 	 * Track task load average for carrying it to new CPU after migrated, and
-	 * track group sched_entity load average for task_h_load calc in migration
+	 * track group sched_entity load average for task_se_h_load calc in migration
 	 */
 	if (se->avg.last_update_time && !(flags & SKIP_AGE_LOAD))
 		__update_load_avg_se(now, cfs_rq, se);
@@ -3751,7 +3751,7 @@ static inline void update_misfit_status(struct task_struct *p, struct rq *rq)
 		return;
 	}
 
-	rq->misfit_task_load = task_h_load(p);
+	rq->misfit_task_load = task_se_h_load(&p->se);
 }
 
 #else /* CONFIG_SMP */
@@ -5739,7 +5739,7 @@ wake_affine_weight(struct sched_domain *sd, struct task_struct *p,
 	this_eff_load = target_load(this_cpu, sd->wake_idx);
 
 	if (sync) {
-		unsigned long current_load = task_h_load(current);
+		unsigned long current_load = task_se_h_load(&current->se);
 
 		if (current_load > this_eff_load)
 			return this_cpu;
@@ -5747,7 +5747,7 @@ wake_affine_weight(struct sched_domain *sd, struct task_struct *p,
 		this_eff_load -= current_load;
 	}
 
-	task_load = task_h_load(p);
+	task_load = task_se_h_load(&p->se);
 
 	this_eff_load += task_load;
 	if (sched_feat(WA_BIAS))
@@ -7600,7 +7600,7 @@ static int detach_tasks(struct lb_env *env)
 		if (!can_migrate_task(p, env))
 			goto next;
 
-		load = task_h_load(p);
+		load = task_se_h_load(&p->se);
 
 		if (sched_feat(LB_MIN) && load < 16 && !env->sd->nr_balance_failed)
 			goto next;
@@ -7833,12 +7833,12 @@ static void update_cfs_rq_h_load(struct cfs_rq *cfs_rq)
 	}
 }
 
-static unsigned long task_h_load(struct task_struct *p)
+static unsigned long task_se_h_load(struct sched_entity *se)
 {
-	struct cfs_rq *cfs_rq = task_cfs_rq(p);
+	struct cfs_rq *cfs_rq = cfs_rq_of(se);
 
 	update_cfs_rq_h_load(cfs_rq);
-	return div64_ul(p->se.avg.load_avg * cfs_rq->h_load,
+	return div64_ul(se->avg.load_avg * cfs_rq->h_load,
 			cfs_rq_load_avg(cfs_rq) + 1);
 }
 #else
@@ -7865,9 +7865,9 @@ static inline void update_blocked_averages(int cpu)
 	rq_unlock_irqrestore(rq, &rf);
 }
 
-static unsigned long task_h_load(struct task_struct *p)
+static unsigned long task_se_h_load(struct sched_entity *se)
 {
-	return p->se.avg.load_avg;
+	return se->avg.load_avg;
 }
 #endif
 
Signature · 1 line
2.20.1
#2 lore raw

Re: [PATCH 01/14] sched: introduce task_se_h_load helper

Quoted 10 lines · Rik van Riel

Sometimes the hierarchical load of a sched_entity needs to be calculated. Rename task_h_load to task_se_h_load, and directly pass a sched_entity to that function.

Move the function declaration up above where it will be used later.

No functional changes.

[...]

kernel/sched/fair.c static void task_numa_compare(struct task_numa_env *env, · line 1671

Quoted 7 lines

/* Give new sched_entity start runnable values to heavy its load in infant time */

kernel/sched/fair.c +1 -1
@@ -1668,7 +1668,7 @@ static void task_numa_compare(struct task_numa_env *env,
 	/*
 	 * In the overloaded case, try and keep the load balanced.
 	 */
-	load = task_h_load(env->p) - task_h_load(cur);
+	load = task_se_h_load(env->p->se) - task_se_h_load(cur->se);

Shouldn't this be:

load = task_se_h_load(&env->p->se) - task_se_h_load(&cur->se);

kernel/sched/fair.c static void task_numa_find_cpu(struct task_numa_env *env, · line 1709

Quoted 9 lines
kernel/sched/fair.c +1 -1
 	if (!load)
 		goto assign;
 
@@ -1706,7 +1706,7 @@ static void task_numa_find_cpu(struct task_numa_env *env,
 	bool maymove = false;
 	int cpu;
 
-	load = task_h_load(env->p);
+	load = task_se_h_load(env->p->se);

load = task_se_h_load(&env->p->se);

[...]

#3 lore raw

PATCH v3 2/14 sched: change /proc/sched_debug fields

In reply to Rik van Riel

Remove some fields from /proc/sched_debug that are removed from sched_entity in a subsequent patch, and add h_load, which comes in very handy to debug CPU controller weight distribution.

---
 kernel/sched/debug.c | 11 ++---------
 1 file changed, 2 insertions(+), 9 deletions(-)
kernel/sched/debug.c +2 -9
index 14c6a8716ba1..f6beaca97a09 100644
@@ -416,11 +416,9 @@ static void print_cfs_group_stats(struct seq_file *m, int cpu, struct task_group
 	}
 
 	P(se->load.weight);
-	P(se->runnable_weight);
 #ifdef CONFIG_SMP
 	P(se->avg.load_avg);
 	P(se->avg.util_avg);
-	P(se->avg.runnable_load_avg);
 #endif
 
 #undef PN_SCHEDSTAT
@@ -538,7 +536,6 @@ void print_cfs_rq(struct seq_file *m, int cpu, struct cfs_rq *cfs_rq)
 	SEQ_printf(m, "  .%-30s: %d\n", "nr_running", cfs_rq->nr_running);
 	SEQ_printf(m, "  .%-30s: %ld\n", "load", cfs_rq->load.weight);
 #ifdef CONFIG_SMP
-	SEQ_printf(m, "  .%-30s: %ld\n", "runnable_weight", cfs_rq->runnable_weight);
 	SEQ_printf(m, "  .%-30s: %lu\n", "load_avg",
 			cfs_rq->avg.load_avg);
 	SEQ_printf(m, "  .%-30s: %lu\n", "runnable_load_avg",
@@ -547,17 +544,15 @@ void print_cfs_rq(struct seq_file *m, int cpu, struct cfs_rq *cfs_rq)
 			cfs_rq->avg.util_avg);
 	SEQ_printf(m, "  .%-30s: %u\n", "util_est_enqueued",
 			cfs_rq->avg.util_est.enqueued);
-	SEQ_printf(m, "  .%-30s: %ld\n", "removed.load_avg",
-			cfs_rq->removed.load_avg);
 	SEQ_printf(m, "  .%-30s: %ld\n", "removed.util_avg",
 			cfs_rq->removed.util_avg);
-	SEQ_printf(m, "  .%-30s: %ld\n", "removed.runnable_sum",
-			cfs_rq->removed.runnable_sum);
 #ifdef CONFIG_FAIR_GROUP_SCHED
 	SEQ_printf(m, "  .%-30s: %lu\n", "tg_load_avg_contrib",
 			cfs_rq->tg_load_avg_contrib);
 	SEQ_printf(m, "  .%-30s: %ld\n", "tg_load_avg",
 			atomic_long_read(&cfs_rq->tg->load_avg));
+	SEQ_printf(m, "  .%-30s: %lu\n", "h_load",
+			cfs_rq->h_load);
 #endif
 #endif
 #ifdef CONFIG_CFS_BANDWIDTH
@@ -961,10 +956,8 @@ void proc_sched_show_task(struct task_struct *p, struct pid_namespace *ns,
 		   "nr_involuntary_switches", (long long)p->nivcsw);
 
 	P(se.load.weight);
-	P(se.runnable_weight);
 #ifdef CONFIG_SMP
 	P(se.avg.load_sum);
-	P(se.avg.runnable_load_sum);
 	P(se.avg.util_sum);
 	P(se.avg.load_avg);
 	P(se.avg.runnable_load_avg);
Signature · 1 line
2.20.1
#4 lore raw

PATCH v3 3/14 sched,fair: redefine runnable_load_avg as the sum of task_h_load

In reply to Rik van Riel

The runnable_load magic is used to quickly propagate information about runnable tasks up the hierarchy of runqueues. The runnable_load_avg is mostly used for the load balancing code, which only examines the value at the root cfs_rq.

Redefine the root cfs_rq runnable_load_avg to be the sum of task_h_loads of the runnable tasks. This works because the hierarchical runnable_load of a task is already equal to the task_se_h_load today. This provides enough information to the load balancer.

The runnable_load_avg of the cgroup cfs_rqs does not appear to be used for anything, so don't bother calculating those.

This removes one of the things that the code currently traverses the cgroup hierarchy for, and getting rid of it brings us one step closer to a flat runqueue for the CPU controller.

---
 include/linux/sched.h |   3 +-
 kernel/sched/core.c   |   2 -
 kernel/sched/debug.c  |   1 +
 kernel/sched/fair.c   | 125 +++++++++++++-----------------------------
 kernel/sched/pelt.c   |  49 ++++++-----------
 kernel/sched/sched.h  |   6 --
 6 files changed, 55 insertions(+), 131 deletions(-)
include/linux/sched.h +1 -2
index 11837410690f..f5bb6948e40c 100644
@@ -391,7 +391,6 @@ struct util_est {
 struct sched_avg {
 	u64				last_update_time;
 	u64				load_sum;
-	u64				runnable_load_sum;
 	u32				util_sum;
 	u32				period_contrib;
 	unsigned long			load_avg;
@@ -439,7 +438,6 @@ struct sched_statistics {
 struct sched_entity {
 	/* For load-balancing: */
 	struct load_weight		load;
-	unsigned long			runnable_weight;
 	struct rb_node			run_node;
 	struct list_head		group_node;
 	unsigned int			on_rq;
@@ -455,6 +453,7 @@ struct sched_entity {
 
 #ifdef CONFIG_FAIR_GROUP_SCHED
 	int				depth;
+	unsigned long			enqueued_h_load;
 	struct sched_entity		*parent;
 	/* rq on which this entity is (to be) queued: */
 	struct cfs_rq			*cfs_rq;
kernel/sched/core.c +0 -2
index 874c427742a9..fbd96900f715 100644
@@ -744,7 +744,6 @@ static void set_load_weight(struct task_struct *p, bool update_load)
 	if (task_has_idle_policy(p)) {
 		load->weight = scale_load(WEIGHT_IDLEPRIO);
 		load->inv_weight = WMULT_IDLEPRIO;
-		p->se.runnable_weight = load->weight;
 		return;
 	}
 
@@ -757,7 +756,6 @@ static void set_load_weight(struct task_struct *p, bool update_load)
 	} else {
 		load->weight = scale_load(sched_prio_to_weight[prio]);
 		load->inv_weight = sched_prio_to_wmult[prio];
-		p->se.runnable_weight = load->weight;
 	}
 }
 
kernel/sched/debug.c +1 -0
index f6beaca97a09..cefc1b171c0b 100644
@@ -962,6 +962,7 @@ void proc_sched_show_task(struct task_struct *p, struct pid_namespace *ns,
 	P(se.avg.load_avg);
 	P(se.avg.runnable_load_avg);
 	P(se.avg.util_avg);
+	P(se.enqueued_h_load);
 	P(se.avg.last_update_time);
 	P(se.avg.util_est.ewma);
 	P(se.avg.util_est.enqueued);
kernel/sched/fair.c +37 -88

235 lines not on this page · raw message

kernel/sched/pelt.c +16 -33

164 lines not on this page · raw message

kernel/sched/sched.h +0 -6
index b52ed1ada0be..5be14cee61f9 100644
@@ -487,7 +487,6 @@ struct cfs_bandwidth { };
 /* CFS-related fields in a runqueue */
 struct cfs_rq {
 	struct load_weight	load;
-	unsigned long		runnable_weight;
 	unsigned int		nr_running;
 	unsigned int		h_nr_running;
 
@@ -700,11 +699,6 @@ static inline long se_weight(struct sched_entity *se)
 	return scale_load_down(se->load.weight);
 }
 
-static inline long se_runnable(struct sched_entity *se)
-{
-	return scale_load_down(se->runnable_weight);
-}
-
 static inline bool sched_asym_prefer(int a, int b)
 {
 	return arch_asym_cpu_priority(a) > arch_asym_cpu_priority(b);
Signature · 1 line
2.20.1
maintainer
#5 lore raw

Re: [PATCH 03/14] sched,fair: redefine runnable_load_avg as the sum of task_h_load

Quoted 10 lines · Rik van Riel
kernel/sched/fair.c +3 -2
@@ -3012,25 +2983,24 @@ static inline int throttled_hierarchy(struct cfs_rq *cfs_rq);
 static void update_cfs_group(struct sched_entity *se)
 {
 	struct cfs_rq *gcfs_rq = group_cfs_rq(se);
-	long shares, runnable;
+	long shares;
 
-	if (!gcfs_rq)
+	if (!gcfs_rq) {
+		update_runnable_load_avg(se);

kernel/sched/fair.c static inline int throttled_hierarchy(struct cfs_rq *cfs_rq); · line 2997

Quoted 10 lines
kernel/sched/fair.c +1 -2
 		return;
+	}
 
 	if (throttled_hierarchy(gcfs_rq))
 		return;
 
 #ifndef CONFIG_SMP
-	runnable = shares = READ_ONCE(gcfs_rq->tg->shares);
-
 	if (likely(se->load.weight == shares))

I'm thinking this uses @shares uninitialized...

Quoted 9 lines
kernel/sched/fair.c +1 -2
 		return;
 #else
 	shares   = calc_group_shares(gcfs_rq);
-	runnable = calc_group_runnable(gcfs_rq, shares);
 #endif
 
-	reweight_entity(cfs_rq_of(se), se, shares, runnable);
+	reweight_entity(cfs_rq_of(se), se, shares);
 }
#6 lore raw

Re: [PATCH 03/14] sched,fair: redefine runnable_load_avg as the sum of task_h_load

Quoted 18 lines · Peter Zijlstra
Quoted 17 lines · Rik van Riel
kernel/sched/fair.c +4 -2
@@ -3012,25 +2983,24 @@ static inline int
 throttled_hierarchy(struct cfs_rq *cfs_rq);
 static void update_cfs_group(struct sched_entity *se)
 {
 	struct cfs_rq *gcfs_rq = group_cfs_rq(se);
-	long shares, runnable;
+	long shares;
 
-	if (!gcfs_rq)
+	if (!gcfs_rq) {
+		update_runnable_load_avg(se);
 		return;
+	}
 
 	if (throttled_hierarchy(gcfs_rq))
 		return;
 
Quoted 6 lines

kernel/sched/fair.c static inline int throttled_hierarchy(struct cfs_rq *cfs_rq); · line 2997

Quoted 4 lines
kernel/sched/fair.c +0 -2
 #ifndef CONFIG_SMP
-	runnable = shares = READ_ONCE(gcfs_rq->tg->shares);
-
 	if (likely(se->load.weight == shares))

I'm thinking this uses @shares uninitialized...

Oops indeed. Let me put the shares = assignment
back for the !SMP case, and edit that comment.

Quoted 9 lines
Quoted 9 lines
kernel/sched/fair.c +1 -2
 		return;
 #else
 	shares   = calc_group_shares(gcfs_rq);
-	runnable = calc_group_runnable(gcfs_rq, shares);
 #endif
 
-	reweight_entity(cfs_rq_of(se), se, shares, runnable);
+	reweight_entity(cfs_rq_of(se), se, shares);
 }
Signature · 1 line
All Rights Reversed.
maintainer
#7 lore raw

Re: [PATCH 03/14] sched,fair: redefine runnable_load_avg as the sum of task_h_load

In reply to Rik van Riel
Quoted 10 lines · Rik van Riel

The runnable_load magic is used to quickly propagate information about runnable tasks up the hierarchy of runqueues. The runnable_load_avg is mostly used for the load balancing code, which only examines the value at the root cfs_rq.

Redefine the root cfs_rq runnable_load_avg to be the sum of task_h_loads of the runnable tasks. This works because the hierarchical runnable_load of a task is already equal to the task_se_h_load today. This provides enough information to the load balancer.

Quoted 6 lines

The runnable_load_avg of the cgroup cfs_rqs does not appear to be used for anything, so don't bother calculating those.

This removes one of the things that the code currently traverses the cgroup hierarchy for, and getting rid of it brings us one step closer to a flat runqueue for the CPU controller.

Nice!

maintainer
#8 lore raw

Re: [PATCH 03/14] sched,fair: redefine runnable_load_avg as the sum of task_h_load

In reply to Rik van Riel

Review ___update_load_avg(struct sched_avg *sa, unsigned long load, unsigned long runna · line 265

Quoted 11 lines · Rik van Riel
(quoted diff) +2 -2
@@ -263,8 +258,8 @@ ___update_load_avg(struct sched_avg *sa, unsigned long load, unsigned long runna
 
 int __update_load_avg_blocked_se(u64 now, struct sched_entity *se)
 {
-	if (___update_load_sum(now, &se->avg, 0, 0, 0)) {
-		___update_load_avg(&se->avg, se_weight(se), se_runnable(se));
+	if (___update_load_sum(now, &se->avg, 0, 0)) {
+		___update_load_avg(&se->avg, se_weight(se));
 		return 1;
 	}
 

The comment above that needs adjustment too, I think.

kernel/sched/pelt.c +0 -15
@@ -234,28 +234,13 @@ ___update_load_avg(struct sched_avg *sa,
 /*
  * sched_entity:
  *
- *   task:
- *     se_runnable() == se_weight()
- *
- *   group: [ see update_cfs_group() ]
- *     se_weight()   = tg->weight * grq->load_avg / tg->load_avg
- *     se_runnable() = se_weight(se) * grq->runnable_load_avg / grq->load_avg
- *
  *   load_sum := runnable_sum
  *   load_avg = se_weight(se) * runnable_avg
  *
- *   runnable_load_sum := runnable_sum
- *   runnable_load_avg = se_runnable(se) * runnable_avg
- *
- * XXX collapse load_sum and runnable_load_sum
- *
  * cfq_rq:
  *
  *   load_sum = \Sum se_weight(se) * se->avg.load_sum
  *   load_avg = \Sum se->avg.load_avg
- *
- *   runnable_load_sum = \Sum se_runnable(se) * se->avg.runnable_load_sum
- *   runnable_load_avg = \Sum se->avg.runable_load_avg
  */
 
 int __update_load_avg_blocked_se(u64 now, struct sched_entity *se)
#9 lore raw

PATCH v3 4/14 sched,fair: move runnable_load_avg to cfs_rq

In reply to Rik van Riel

Since only the root cfs_rq runnable_load_avg field is used any more, we can move the field from struct sched_avg, which every sched_entity has one of, directly into the struct cfs_rq, of which we have way fewer.

No functional changes.

---
 include/linux/sched.h | 1 -
 kernel/sched/debug.c  | 3 +--
 kernel/sched/fair.c   | 8 ++++----
 kernel/sched/sched.h  | 1 +
 4 files changed, 6 insertions(+), 7 deletions(-)
include/linux/sched.h +0 -1
index f5bb6948e40c..84a6cc6f5c47 100644
@@ -394,7 +394,6 @@ struct sched_avg {
 	u32				util_sum;
 	u32				period_contrib;
 	unsigned long			load_avg;
-	unsigned long			runnable_load_avg;
 	unsigned long			util_avg;
 	struct util_est			util_est;
 } ____cacheline_aligned;
kernel/sched/debug.c +1 -2
index cefc1b171c0b..6e7c8ff210a8 100644
@@ -539,7 +539,7 @@ void print_cfs_rq(struct seq_file *m, int cpu, struct cfs_rq *cfs_rq)
 	SEQ_printf(m, "  .%-30s: %lu\n", "load_avg",
 			cfs_rq->avg.load_avg);
 	SEQ_printf(m, "  .%-30s: %lu\n", "runnable_load_avg",
-			cfs_rq->avg.runnable_load_avg);
+			cfs_rq->runnable_load_avg);
 	SEQ_printf(m, "  .%-30s: %lu\n", "util_avg",
 			cfs_rq->avg.util_avg);
 	SEQ_printf(m, "  .%-30s: %u\n", "util_est_enqueued",
@@ -960,7 +960,6 @@ void proc_sched_show_task(struct task_struct *p, struct pid_namespace *ns,
 	P(se.avg.load_sum);
 	P(se.avg.util_sum);
 	P(se.avg.load_avg);
-	P(se.avg.runnable_load_avg);
 	P(se.avg.util_avg);
 	P(se.enqueued_h_load);
 	P(se.avg.last_update_time);
kernel/sched/fair.c +4 -4
index 860708b687a7..63cb40253b26 100644
@@ -2768,7 +2768,7 @@ enqueue_runnable_load_avg(struct cfs_rq *cfs_rq, struct sched_entity *se)
 		struct cfs_rq *root_cfs_rq = &cfs_rq->rq->cfs;
 		se->enqueued_h_load = task_se_h_load(se);
 
-		root_cfs_rq->avg.runnable_load_avg += se->enqueued_h_load;
+		root_cfs_rq->runnable_load_avg += se->enqueued_h_load;
 	}
 }
 
@@ -2777,7 +2777,7 @@ dequeue_runnable_load_avg(struct cfs_rq *cfs_rq, struct sched_entity *se)
 {
 	if (entity_is_task(se)) {
 		struct cfs_rq *root_cfs_rq = &cfs_rq->rq->cfs;
-		sub_positive(&root_cfs_rq->avg.runnable_load_avg,
+		sub_positive(&root_cfs_rq->runnable_load_avg,
 					se->enqueued_h_load);
 	}
 }
@@ -2795,7 +2795,7 @@ update_runnable_load_avg(struct sched_entity *se)
 
 	new_h_load = task_se_h_load(se);
 	delta = new_h_load - se->enqueued_h_load;
-	root_cfs_rq->avg.runnable_load_avg += delta;
+	root_cfs_rq->runnable_load_avg += delta;
 	se->enqueued_h_load = new_h_load;
 }
 
@@ -3559,7 +3559,7 @@ static void remove_entity_load_avg(struct sched_entity *se)
 
 static inline unsigned long cfs_rq_runnable_load_avg(struct cfs_rq *cfs_rq)
 {
-	return cfs_rq->avg.runnable_load_avg;
+	return cfs_rq->runnable_load_avg;
 }
 
 static inline unsigned long cfs_rq_load_avg(struct cfs_rq *cfs_rq)
kernel/sched/sched.h +1 -0
index 5be14cee61f9..32978a8de8ce 100644
@@ -516,6 +516,7 @@ struct cfs_rq {
 	 * CFS load tracking
 	 */
 	struct sched_avg	avg;
+	unsigned long		runnable_load_avg;
 #ifndef CONFIG_64BIT
 	u64			load_last_update_time_copy;
 #endif
Signature · 1 line
2.20.1
maintainer
#10 lore raw

Re: [PATCH 04/14] sched,fair: move runnable_load_avg to cfs_rq

Quoted 7 lines · Rik van Riel

Since only the root cfs_rq runnable_load_avg field is used any more, we can move the field from struct sched_avg, which every sched_entity has one of, directly into the struct cfs_rq, of which we have way fewer.

No functional changes.

I think he's praying on space in that cacheline ;-)

#11 lore raw

PATCH v3 6/14 sched,cfs: use explicit cfs_rq of parent se helper

In reply to Rik van Riel

Use an explicit "cfs_rq of parent sched_entity" helper in a few strategic places, where cfs_rq_of(se) may no longer point at the right runqueue once we flatten the hierarchical cgroup runqueues.

No functional change.

---
 kernel/sched/fair.c | 17 +++++++++++++----
 1 file changed, 13 insertions(+), 4 deletions(-)
kernel/sched/fair.c +13 -4
index d6a8aa78a948..39f7a2d810e1 100644
@@ -276,6 +276,15 @@ static inline struct cfs_rq *group_cfs_rq(struct sched_entity *grp)
 	return grp->my_q;
 }
 
+/* runqueue owned by the parent entity; the root cfs_rq for a top level se */
+static inline struct cfs_rq *group_cfs_rq_of_parent(struct sched_entity *se)
+{
+	if (se->parent)
+		return group_cfs_rq(se->parent);
+
+	return cfs_rq_of(se);
+}
+
 static inline bool list_add_leaf_cfs_rq(struct cfs_rq *cfs_rq)
 {
 	struct rq *rq = rq_of(cfs_rq);
@@ -3317,7 +3326,7 @@ static inline int propagate_entity_load_avg(struct sched_entity *se)
 
 	gcfs_rq->propagate = 0;
 
-	cfs_rq = cfs_rq_of(se);
+	cfs_rq = group_cfs_rq_of_parent(se);
 
 	add_tg_cfs_propagate(cfs_rq, gcfs_rq->prop_runnable_sum);
 
@@ -7794,7 +7803,7 @@ static void update_cfs_rq_h_load(struct cfs_rq *cfs_rq)
 
 	WRITE_ONCE(cfs_rq->h_load_next, NULL);
 	for_each_sched_entity(se) {
-		cfs_rq = cfs_rq_of(se);
+		cfs_rq = group_cfs_rq_of_parent(se);
 		WRITE_ONCE(cfs_rq->h_load_next, se);
 		if (cfs_rq->last_h_load_update == now)
 			break;
@@ -7817,7 +7826,7 @@ static void update_cfs_rq_h_load(struct cfs_rq *cfs_rq)
 
 static unsigned long task_se_h_load(struct sched_entity *se)
 {
-	struct cfs_rq *cfs_rq = cfs_rq_of(se);
+	struct cfs_rq *cfs_rq = group_cfs_rq_of_parent(se);
 
 	update_cfs_rq_h_load(cfs_rq);
 	return div64_ul(se->avg.load_avg * cfs_rq->h_load,
@@ -10164,7 +10173,7 @@ static void task_tick_fair(struct rq *rq, struct task_struct *curr, int queued)
 	struct sched_entity *se = &curr->se;
 
 	for_each_sched_entity(se) {
-		cfs_rq = cfs_rq_of(se);
+		cfs_rq = group_cfs_rq_of_parent(se);
 		entity_tick(cfs_rq, se, queued);
 	}
 
Signature · 1 line
2.20.1
#12 lore raw

PATCH v3 7/14 sched,cfs: fix zero length timeslice calculation

In reply to Rik van Riel

The way the time slice length is currently calculated, not only do high priority tasks get longer time slices than low priority tasks, but due to fixed point math, low priority tasks could end up with a zero length time slice. This can lead to cache thrashing and other inefficiencies.

Cap the minimum time slice length to sysctl_sched_min_granularity.

Tasks that end up getting a time slice length too long for their relative priority will simply end up having their vruntime advanced much faster than other tasks, resulting in them receiving time slices less frequently.

---
 kernel/sched/fair.c | 7 +++++++
 1 file changed, 7 insertions(+)
kernel/sched/fair.c +7 -0
index 39f7a2d810e1..9ff69b927a3c 100644
@@ -732,6 +732,13 @@ static u64 sched_slice(struct cfs_rq *cfs_rq, struct sched_entity *se)
 		}
 		slice = __calc_delta(slice, se->load.weight, load);
 	}
+
+	/*
+	 * To avoid cache thrashing, run at least sysctl_sched_min_granularity.
+	 * The vruntime of a low priority task advances faster; those tasks
+	 * will simply get time slices less frequently.
+	 */
+	slice = max_t(u64, slice, sysctl_sched_min_granularity);
 	return slice;
 }
 
Signature · 1 line
2.20.1
#13 lore raw

PATCH v3 8/14 sched,fair: simplify timeslice length code

In reply to Rik van Riel

The idea behind __sched_period makes sense, but the results do not always.

When a CPU has one high priority task and a large number of low priority tasks, __sched_period will return a value larger than sysctl_sched_latency, and the one high priority task may end up getting a timeslice all for itself that is also much larger than sysctl_sched_latency.

The low priority tasks will have their time slices rounded up to sysctl_sched_min_granularity, resulting in an even larger scheduling latency than targeted by __sched_period.

Simplify the code by simply ripping out __sched_period and always taking fractions of sysctl_sched_latency.

If a high priority task ends up getting a "too small" time slice compared to low priority tasks, the vruntime scaling ensures that it will simply get scheduled more frequently than low priority tasks.

---
 kernel/sched/fair.c | 18 +-----------------
 1 file changed, 1 insertion(+), 17 deletions(-)
kernel/sched/fair.c +1 -17
index 9ff69b927a3c..b4fc328032e6 100644
@@ -691,22 +691,6 @@ static inline u64 calc_delta_fair(u64 delta, struct sched_entity *se)
 	return delta;
 }
 
-/*
- * The idea is to set a period in which each task runs once.
- *
- * When there are too many tasks (sched_nr_latency) we have to stretch
- * this period because otherwise the slices get too small.
- *
- * p = (nr <= nl) ? l : l*nr/nl
- */
-static u64 __sched_period(unsigned long nr_running)
-{
-	if (unlikely(nr_running > sched_nr_latency))
-		return nr_running * sysctl_sched_min_granularity;
-	else
-		return sysctl_sched_latency;
-}
-
 /*
  * We calculate the wall-time slice from the period by taking a part
  * proportional to the weight.
@@ -715,7 +699,7 @@ static u64 __sched_period(unsigned long nr_running)
  */
 static u64 sched_slice(struct cfs_rq *cfs_rq, struct sched_entity *se)
 {
-	u64 slice = __sched_period(cfs_rq->nr_running + !se->on_rq);
+	u64 slice = sysctl_sched_latency;
 
 	for_each_sched_entity(se) {
 		struct load_weight *load;
Signature · 1 line
2.20.1
#14 lore raw

PATCH v3 9/14 sched,fair: refactor enqueue/dequeue_entity

In reply to Rik van Riel

Refactor enqueue_entity, dequeue_entity, and update_load_avg, in order to split out the things we still want to happen at every level in the cgroup hierarchy with a flat runqueue from the things we only need to happen once.

No functional changes.

---
 kernel/sched/fair.c | 64 +++++++++++++++++++++++++++++----------------
 1 file changed, 42 insertions(+), 22 deletions(-)
kernel/sched/fair.c +42 -22
index b4fc328032e6..c944729423bd 100644
@@ -3500,7 +3500,7 @@ static void detach_entity_load_avg(struct cfs_rq *cfs_rq, struct sched_entity *s
 #define DO_ATTACH	0x4
 
 /* Update task and its cfs_rq load average */
-static inline void update_load_avg(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
+static inline bool update_load_avg(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
 {
 	u64 now = cfs_rq_clock_pelt(cfs_rq);
 	int decayed;
@@ -3529,6 +3529,8 @@ static inline void update_load_avg(struct cfs_rq *cfs_rq, struct sched_entity *s
 
 	} else if (decayed && (flags & UPDATE_TG))
 		update_tg_load_avg(cfs_rq, 0);
+
+	return decayed;
 }
 
 #ifndef CONFIG_64BIT
@@ -3745,9 +3747,10 @@ static inline void update_misfit_status(struct task_struct *p, struct rq *rq)
 #define SKIP_AGE_LOAD	0x0
 #define DO_ATTACH	0x0
 
-static inline void update_load_avg(struct cfs_rq *cfs_rq, struct sched_entity *se, int not_used1)
+static inline bool update_load_avg(struct cfs_rq *cfs_rq, struct sched_entity *se, int not_used1)
 {
 	cfs_rq_util_change(cfs_rq, 0);
+	return false;
 }
 
 static inline void remove_entity_load_avg(struct sched_entity *se) {}
@@ -3870,6 +3873,24 @@ static inline void check_schedstat_required(void)
  * CPU and an up-to-date min_vruntime on the destination CPU.
  */
 
+static bool
+enqueue_entity_groups(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
+{
+	/*
+	 * When enqueuing a sched_entity, we must:
+	 *   - Update loads to have both entity and cfs_rq synced with now.
+	 *   - Add its load to cfs_rq->runnable_avg
+	 *   - For group_entity, update its weight to reflect the new share of
+	 *     its group cfs_rq
+	 *   - Add its new weight to cfs_rq->load.weight
+	 */
+	if (!update_load_avg(cfs_rq, se, UPDATE_TG | DO_ATTACH))
+		return false;
+
+	update_cfs_group(se);
+	return true;
+}
+
 static void
 enqueue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
 {
@@ -3894,16 +3915,6 @@ enqueue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
 	if (renorm && !curr)
 		se->vruntime += cfs_rq->min_vruntime;
 
-	/*
-	 * When enqueuing a sched_entity, we must:
-	 *   - Update loads to have both entity and cfs_rq synced with now.
-	 *   - Add its load to cfs_rq->runnable_avg
-	 *   - For group_entity, update its weight to reflect the new share of
-	 *     its group cfs_rq
-	 *   - Add its new weight to cfs_rq->load.weight
-	 */
-	update_load_avg(cfs_rq, se, UPDATE_TG | DO_ATTACH);
-	update_cfs_group(se);
 	enqueue_runnable_load_avg(cfs_rq, se);
 	account_entity_enqueue(cfs_rq, se);
 
@@ -3970,14 +3981,9 @@ static void clear_buddies(struct cfs_rq *cfs_rq, struct sched_entity *se)
 
 static __always_inline void return_cfs_rq_runtime(struct cfs_rq *cfs_rq);
 
-static void
-dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
+static bool
+dequeue_entity_groups(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
 {
-	/*
-	 * Update run-time statistics of the 'current'.
-	 */
-	update_curr(cfs_rq);
-
 	/*
 	 * When dequeuing a sched_entity, we must:
 	 *   - Update loads to have both entity and cfs_rq synced with now.
@@ -3986,7 +3992,21 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
 	 *   - For group entity, update its weight to reflect the new share
 	 *     of its group cfs_rq.
 	 */
-	update_load_avg(cfs_rq, se, UPDATE_TG);
+	if (!update_load_avg(cfs_rq, se, UPDATE_TG))
+		return false;
+	update_cfs_group(se);
+
+	return true;
+}
+
+static void
+dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
+{
+	/*
+	 * Update run-time statistics of the 'current'.
+	 */
+	update_curr(cfs_rq);
+
 	dequeue_runnable_load_avg(cfs_rq, se);
 
 	update_stats_dequeue(cfs_rq, se, flags);
@@ -4010,8 +4030,6 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
 	/* return excess runtime on last dequeue */
 	return_cfs_rq_runtime(cfs_rq);
 
-	update_cfs_group(se);
-
 	/*
 	 * Now advance min_vruntime if @se was the entity holding it back,
 	 * except when: DEQUEUE_SAVE && !DEQUEUE_MOVE, in this case we'll be
@@ -5176,6 +5194,7 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags)
 		if (se->on_rq)
 			break;
 		cfs_rq = cfs_rq_of(se);
+		enqueue_entity_groups(cfs_rq, se, flags);
 		enqueue_entity(cfs_rq, se, flags);
 
 		/*
@@ -5258,6 +5277,7 @@ static void dequeue_task_fair(struct rq *rq, struct task_struct *p, int flags)
 
 	for_each_sched_entity(se) {
 		cfs_rq = cfs_rq_of(se);
+		dequeue_entity_groups(cfs_rq, se, flags);
 		dequeue_entity(cfs_rq, se, flags);
 
 		/*
Signature · 1 line
2.20.1
maintainer
#15 lore raw

Re: [PATCH 09/14] sched,fair: refactor enqueue/dequeue_entity

Quoted 69 lines · Rik van Riel

Refactor enqueue_entity, dequeue_entity, and update_load_avg, in order to split out the things we still want to happen at every level in the cgroup hierarchy with a flat runqueue from the things we only need to happen once.

No functional changes.

kernel/sched/fair.c +23 -5
@@ -3500,7 +3500,7 @@ static void detach_entity_load_avg(struct cfs_rq *cfs_rq, struct sched_entity *s
 #define DO_ATTACH	0x4
 
 /* Update task and its cfs_rq load average */
-static inline void update_load_avg(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
+static inline bool update_load_avg(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
 {
 	u64 now = cfs_rq_clock_pelt(cfs_rq);
 	int decayed;
@@ -3529,6 +3529,8 @@ static inline void update_load_avg(struct cfs_rq *cfs_rq, struct sched_entity *s
 
 	} else if (decayed && (flags & UPDATE_TG))
 		update_tg_load_avg(cfs_rq, 0);
+
+	return decayed;
 }
 
 #ifndef CONFIG_64BIT
@@ -3745,9 +3747,10 @@ static inline void update_misfit_status(struct task_struct *p, struct rq *rq)
 #define SKIP_AGE_LOAD	0x0
 #define DO_ATTACH	0x0
 
-static inline void update_load_avg(struct cfs_rq *cfs_rq, struct sched_entity *se, int not_used1)
+static inline bool update_load_avg(struct cfs_rq *cfs_rq, struct sched_entity *se, int not_used1)
 {
 	cfs_rq_util_change(cfs_rq, 0);
+	return false;
 }
 
 static inline void remove_entity_load_avg(struct sched_entity *se) {}
@@ -3870,6 +3873,24 @@ static inline void check_schedstat_required(void)
  * CPU and an up-to-date min_vruntime on the destination CPU.
  */
 
+static bool
+enqueue_entity_groups(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
+{
+	/*
+	 * When enqueuing a sched_entity, we must:
+	 *   - Update loads to have both entity and cfs_rq synced with now.
+	 *   - Add its load to cfs_rq->runnable_avg
+	 *   - For group_entity, update its weight to reflect the new share of
+	 *     its group cfs_rq
+	 *   - Add its new weight to cfs_rq->load.weight
+	 */
+	if (!update_load_avg(cfs_rq, se, UPDATE_TG | DO_ATTACH))
+		return false;
+
+	update_cfs_group(se);
+	return true;
+}
+
 static void
 enqueue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
 {
@@ -3894,16 +3915,6 @@ enqueue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
 	if (renorm && !curr)
 		se->vruntime += cfs_rq->min_vruntime;
 
-	/*
-	 * When enqueuing a sched_entity, we must:
-	 *   - Update loads to have both entity and cfs_rq synced with now.

kernel/sched/fair.c enqueue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags) · line 3920

Quoted 10 lines
kernel/sched/fair.c +0 -7
-	 *   - Add its load to cfs_rq->runnable_avg
-	 *   - For group_entity, update its weight to reflect the new share of
-	 *     its group cfs_rq
-	 *   - Add its new weight to cfs_rq->load.weight
-	 */
-	update_load_avg(cfs_rq, se, UPDATE_TG | DO_ATTACH);
-	update_cfs_group(se);
 	enqueue_runnable_load_avg(cfs_rq, se);
 	account_entity_enqueue(cfs_rq, se);
 

No functional, but you did make update_cfs_group() conditional. Now that looks OK, but maybe you can do that part in a separate patch with a little justification of its own.

#16 lore raw

Re: [PATCH 09/14] sched,fair: refactor enqueue/dequeue_entity

Quoted 24 lines · Peter Zijlstra
Quoted 23 lines · Rik van Riel
kernel/sched/fair.c +6 -0
@@ -3870,6 +3873,24 @@ static inline void
 check_schedstat_required(void)
  * CPU and an up-to-date min_vruntime on the destination CPU.
  */
 
+static bool
+enqueue_entity_groups(struct cfs_rq *cfs_rq, struct sched_entity
 *se, int flags)
+{
+	/*
+	 * When enqueuing a sched_entity, we must:
+	 *   - Update loads to have both entity and cfs_rq synced with

now.

(quoted diff) +2 -0
+	 *   - Add its load to cfs_rq->runnable_avg
+	 *   - For group_entity, update its weight to reflect the new

share of

kernel/sched/fair.c +7 -0
+	 *     its group cfs_rq
+	 *   - Add its new weight to cfs_rq->load.weight
+	 */
+	if (!update_load_avg(cfs_rq, se, UPDATE_TG | DO_ATTACH))
+		return false;
+
+	update_cfs_group(se);
Quoted 8 lines

Review

Quoted 3 lines
(quoted diff) +2 -0
+	return true;
+}

No functional, but you did make update_cfs_group() conditional. Now that
looks OK, but maybe you can do that part in a separate patch with a little justification of its own.

Good idea, I will split that out.

Signature · 1 line
All Rights Reversed.
maintainer
#17 lore raw

Re: [PATCH 09/14] sched,fair: refactor enqueue/dequeue_entity

In reply to Peter Zijlstra
Quoted 15 lines · Peter Zijlstra
Quoted 13 lines · Rik van Riel
kernel/sched/fair.c +13 -0
+static bool
+enqueue_entity_groups(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
+{
+	/*
+	 * When enqueuing a sched_entity, we must:
+	 *   - Update loads to have both entity and cfs_rq synced with now.
+	 *   - Add its load to cfs_rq->runnable_avg
+	 *   - For group_entity, update its weight to reflect the new share of
+	 *     its group cfs_rq
+	 *   - Add its new weight to cfs_rq->load.weight
+	 */
+	if (!update_load_avg(cfs_rq, se, UPDATE_TG | DO_ATTACH))
+		return false;
Quoted 8 lines

kernel/sched/fair.c static inline void check_schedstat_required(void) · line 3892

Quoted 4 lines
kernel/sched/fair.c +4 -0
+
+	update_cfs_group(se);
+	return true;
+}

No functional, but you did make update_cfs_group() conditional. Now that looks OK, but maybe you can do that part in a separate patch with a little justification of its own.

To record (and extend) our discussion from IRC yesterday; I now do think the above is in fact a problem.

The thing is that update_cfs_group() does not soly rely on the tg state; it also contains magic to deal with ramp up; for which you later introduce that max_h_load thing.

Specifically (re)read the second part of the comment describing calc_group_shares() where it explains the ramp up:

* The problem with it is that because the average is slow -- it was designed
* to be exactly that of course -- this leads to transients in boundary
* conditions. In specific, the case where the group was idle and we start the
* one task. It takes time for our CPU's grq->avg.load_avg to build up,
* yielding bad latency etc..

(and further)

So by not always calling this (and not updating h_load) you fail to take advantage of this.

So I would suggest keeping update_cfs_group() unconditional, and recomputing the h_load for the entire hierarchy on enqueue.

#18 lore raw

Re: [PATCH 09/14] sched,fair: refactor enqueue/dequeue_entity

Quoted 49 lines · Peter Zijlstra
Quoted 17 lines · Peter Zijlstra
Quoted 16 lines · Rik van Riel
(quoted diff) +2 -0
+static bool
+enqueue_entity_groups(struct cfs_rq *cfs_rq, struct sched_entity

*se, int flags)

kernel/sched/fair.c +4 -0
+{
+	/*
+	 * When enqueuing a sched_entity, we must:
+	 *   - Update loads to have both entity and cfs_rq synced with

now.

(quoted diff) +2 -0
+	 *   - Add its load to cfs_rq->runnable_avg
+	 *   - For group_entity, update its weight to reflect the new

share of

kernel/sched/fair.c +5 -0
+	 *     its group cfs_rq
+	 *   - Add its new weight to cfs_rq->load.weight
+	 */
+	if (!update_load_avg(cfs_rq, se, UPDATE_TG | DO_ATTACH))
+		return false;
Quoted 8 lines

kernel/sched/fair.c static inline void check_schedstat_required(void) · line 3892

Quoted 4 lines
kernel/sched/fair.c +4 -0
+
+	update_cfs_group(se);
+	return true;
+}

No functional, but you did make update_cfs_group() conditional. Now that
looks OK, but maybe you can do that part in a separate patch with a little justification of its own.

To record (and extend) our discussion from IRC yesterday; I now do think
the above is in fact a problem.

The thing is that update_cfs_group() does not soly rely on the tg state;
it also contains magic to deal with ramp up; for which you later introduce that max_h_load thing.

Specifically (re)read the second part of the comment describing calc_group_shares() where it explains the ramp up:

* The problem with it is that because the average is slow -- it was
designed
* to be exactly that of course -- this leads to transients in
boundary
* conditions. In specific, the case where the group was idle and we
start the
* one task. It takes time for our CPU's grq->avg.load_avg to build
up,
* yielding bad latency etc..

Quoted 8 lines

(and further)

So by not always calling this (and not updating h_load) you fail to take
advantage of this.

So I would suggest keeping update_cfs_group() unconditional, and recomputing the h_load for the entire hierarchy on enqueue.

I think I understand the problem you are pointing out, but if update_load_avg() keeps the load average for the runqueue unchanged (because that is rate limited to once a jiffy, and has been like that for a while), why would calc_group_shares() result in a different value than what it returned the last time?

What am I overlooking?

Signature · 1 line
All Rights Reversed.
maintainer
#19 lore raw

Re: [PATCH 09/14] sched,fair: refactor enqueue/dequeue_entity

Quoted 7 lines · Rik van Riel

I think I understand the problem you are pointing out, but if update_load_avg() keeps the load average for the runqueue unchanged (because that is rate limited to once a jiffy, and has been like that for a while), why would calc_group_shares() result in a different value than what it returned the last time?

What am I overlooking?

I'm thinking you're thinking (3):

           tg->weight * grq->avg.load_avg
  shares = ------------------------------
                 tg->load_avg

Where: tg->load_avg ~= \Sum grq->avg.load_avg

Which is the straight forward shares calculation, and purely depends on the load averages (which haven't been changed etc..)

But what we actually do is (6):

                                    tg->weight * grq->avg.load_avg
  shares = ---------------------------------------------------------------------------
           tg->load_avg - grq->avg.load_avg + max(grq->load.weight, grq->avg.load_avg)

And even if tg->load_avg and grq->avg.load_avg haven't changed, grq->load.weight most certainly has.

#20 lore raw

PATCH v3 10/14 sched,fair: add helper functions for flattened runqueue

In reply to Rik van Riel

Add helper functions to make the flattened runqueue patch a little smaller.

The task_se_h_weight function is similar to task_se_h_load, but scales the task weight by the group weight, without taking the task's duty cycle into account.

The task_se_in_cgroup helper is functionally identical to parent_entity, but directly calling a function with that name obscures what the other code is trying to use it for, and would make the code harder to understand.

---
 kernel/sched/fair.c | 31 +++++++++++++++++++++++++++++++
 1 file changed, 31 insertions(+)
kernel/sched/fair.c +31 -0
index c944729423bd..6073fabb171b 100644
@@ -243,6 +243,7 @@ static u64 __calc_delta(u64 delta_exec, unsigned long weight, struct load_weight
 
 const struct sched_class fair_sched_class;
 static unsigned long task_se_h_load(struct sched_entity *se);
+static unsigned long task_se_h_weight(struct sched_entity *se);
 
 /**************************************************************
  * CFS operations on generic schedulable entities:
@@ -431,6 +432,12 @@ static inline struct sched_entity *parent_entity(struct sched_entity *se)
 	return se->parent;
 }
 
+/* Is this (task) sched_entity in a non-root cgroup? */
+static inline bool task_se_in_cgroup(struct sched_entity *se)
+{
+	return parent_entity(se);
+}
+
 static void
 find_matching_se(struct sched_entity **se, struct sched_entity **pse)
 {
@@ -513,6 +520,11 @@ static inline struct sched_entity *parent_entity(struct sched_entity *se)
 	return NULL;
 }
 
+static inline bool task_se_in_cgroup(struct sched_entity *se)
+{
+	return false;
+}
+
 static inline void
 find_matching_se(struct sched_entity **se, struct sched_entity **pse)
 {
@@ -7835,6 +7847,20 @@ static void update_cfs_rq_h_load(struct cfs_rq *cfs_rq)
 	}
 }
 
+static unsigned long task_se_h_weight(struct sched_entity *se)
+{
+	struct cfs_rq *cfs_rq;
+
+	if (!task_se_in_cgroup(se))
+		return se->load.weight;
+
+	cfs_rq = group_cfs_rq_of_parent(se);
+	update_cfs_rq_h_load(cfs_rq);
+
+	/* Reduce the load.weight by the h_load of the group the task is in. */
+	return (cfs_rq->h_load * se->load.weight) >> SCHED_FIXEDPOINT_SHIFT;
+}
+
 static unsigned long task_se_h_load(struct sched_entity *se)
 {
 	struct cfs_rq *cfs_rq = group_cfs_rq_of_parent(se);
@@ -7871,6 +7897,11 @@ static unsigned long task_se_h_load(struct sched_entity *se)
 {
 	return se->avg.load_avg;
 }
+
+static unsigned long task_se_h_weight(struct sched_entity *se)
+{
+	return se->load.weight;
+}
 #endif
 
 /********** Helpers for find_busiest_group ************************/
Signature · 1 line
2.20.1
#21 lore raw

PATCH v3 11/14 sched,fair: flatten hierarchical runqueues

In reply to Rik van Riel

Flatten the hierarchical runqueues into just the per CPU rq.cfs runqueue.

Iteration of the sched_entity hierarchy is rate limited to once per jiffy per sched_entity, which is a smaller change than it seems, because load average adjustments were already rate limited to once per jiffy before this patch series.

This patch breaks CONFIG_CFS_BANDWIDTH. The plan for that is to park tasks from throttled cgroups onto their cgroup runqueues, and slowly (using the GENTLE_FAIR_SLEEPERS) wake them back up, in vruntime order, once the cgroup gets unthrottled, to prevent thundering herd issues.

Header from folded patch 'fix-attach-detach_enticy_cfs_rq.patch~':

Subject: sched,fair: fix attach/detach_entity_cfs_rq

While attach_entity_cfs_rq and detach_entity_cfs_rq should iterate over the hierarchy, they do not need to so that twice.

Passing flags into propagate_entity_cfs_rq allows us to reuse that same loop from other functions.

---
 include/linux/sched.h |   2 +
 kernel/sched/fair.c   | 478 +++++++++++++-----------------------------
 kernel/sched/pelt.c   |   6 +-
 kernel/sched/pelt.h   |   2 +-
 kernel/sched/sched.h  |   2 +-
 5 files changed, 150 insertions(+), 340 deletions(-)
include/linux/sched.h +2 -0
index 84a6cc6f5c47..901c710363e7 100644
@@ -453,6 +453,8 @@ struct sched_entity {
 #ifdef CONFIG_FAIR_GROUP_SCHED
 	int				depth;
 	unsigned long			enqueued_h_load;
+	unsigned long			enqueued_h_weight;
+	struct load_weight		h_load;
 	struct sched_entity		*parent;
 	/* rq on which this entity is (to be) queued: */
 	struct cfs_rq			*cfs_rq;
kernel/sched/fair.c +143 -335

806 lines not on this page · raw message

kernel/sched/pelt.c +3 -3
index 190797e421dd..2c901e0f9fbd 100644
@@ -266,10 +266,10 @@ int __update_load_avg_blocked_se(u64 now, struct sched_entity *se)
 	return 0;
 }
 
-int __update_load_avg_se(u64 now, struct cfs_rq *cfs_rq, struct sched_entity *se)
+int __update_load_avg_se(u64 now, struct cfs_rq *cfs_rq, struct sched_entity *se, bool load, bool running)
 {
-	if (___update_load_sum(now, &se->avg, !!se->on_rq,
-				cfs_rq->curr == se)) {
+	if (___update_load_sum(now, &se->avg, (!!se->on_rq || load),
+				(cfs_rq->curr == se) || running)) {
 
 		___update_load_avg(&se->avg, se_weight(se));
 		cfs_se_util_change(&se->avg);
kernel/sched/pelt.h +1 -1
index 7489d5f56960..1152c4ebf314 100644
@@ -2,7 +2,7 @@
 #include "sched-pelt.h"
 
 int __update_load_avg_blocked_se(u64 now, struct sched_entity *se);
-int __update_load_avg_se(u64 now, struct cfs_rq *cfs_rq, struct sched_entity *se);
+int __update_load_avg_se(u64 now, struct cfs_rq *cfs_rq, struct sched_entity *se, bool load, bool running);
 int __update_load_avg_cfs_rq(u64 now, struct cfs_rq *cfs_rq);
 int update_rt_rq_load_avg(u64 now, struct rq *rq, int running);
 int update_dl_rq_load_avg(u64 now, struct rq *rq, int running);
kernel/sched/sched.h +1 -1
index 32978a8de8ce..b05fd87cf8b5 100644
@@ -1443,7 +1443,7 @@ static inline void set_task_rq(struct task_struct *p, unsigned int cpu)
 
 #ifdef CONFIG_FAIR_GROUP_SCHED
 	set_task_rq_fair(&p->se, p->se.cfs_rq, tg->cfs_rq[cpu]);
-	p->se.cfs_rq = tg->cfs_rq[cpu];
+	p->se.cfs_rq = &cpu_rq(cpu)->cfs;
 	p->se.parent = tg->se[cpu];
 #endif
 
Signature · 1 line
2.20.1
#22 lore raw

PATCH v3 12/14 sched,fair: track cfs_rq->max_h_load for more legitimate h_weight

In reply to Rik van Riel

The cfs_rq->h_load decays to zero quite rapidly, which can easily lead to tasks getting enqueued with an h_weight and h_load of zero, confusing the load balancer, the preemption code, and other parts of the kernel.

Add a slow moving cfs_rq->max_h_load, with a half life of just over a minute, to help tasks get enqueued with more legitimate h_weight values.

---
 kernel/sched/debug.c |  2 ++
 kernel/sched/fair.c  |  6 +++++-
 kernel/sched/pelt.c  | 16 ++++++++++++++++
 kernel/sched/pelt.h  |  1 +
 kernel/sched/sched.h |  2 ++
 5 files changed, 26 insertions(+), 1 deletion(-)
kernel/sched/debug.c +2 -0
index 6e7c8ff210a8..88ed6d4429e6 100644
@@ -553,6 +553,8 @@ void print_cfs_rq(struct seq_file *m, int cpu, struct cfs_rq *cfs_rq)
 			atomic_long_read(&cfs_rq->tg->load_avg));
 	SEQ_printf(m, "  .%-30s: %lu\n", "h_load",
 			cfs_rq->h_load);
+	SEQ_printf(m, "  .%-30s: %lu\n", "max_h_load",
+			cfs_rq->max_h_load);
 #endif
 #endif
 #ifdef CONFIG_CFS_BANDWIDTH
kernel/sched/fair.c +5 -1
index 1d5145cb6096..224cd9b20887 100644
@@ -7636,6 +7636,8 @@ static void update_cfs_rq_h_load(struct cfs_rq *cfs_rq)
 
 	if (!se) {
 		cfs_rq->h_load = cfs_rq_load_avg(cfs_rq);
+		slow_decay(&cfs_rq->max_h_load, &cfs_rq->last_max_h_decay);
+		cfs_rq->max_h_load = max(cfs_rq->max_h_load, cfs_rq->h_load);
 		cfs_rq->last_h_load_update = now;
 	}
 
@@ -7645,6 +7647,8 @@ static void update_cfs_rq_h_load(struct cfs_rq *cfs_rq)
 			cfs_rq_load_avg(cfs_rq) + 1);
 		cfs_rq = group_cfs_rq(se);
 		cfs_rq->h_load = load;
+		slow_decay(&cfs_rq->max_h_load, &cfs_rq->last_max_h_decay);
+		cfs_rq->max_h_load = max(load, cfs_rq->max_h_load);
 		cfs_rq->last_h_load_update = now;
 	}
 }
@@ -7660,7 +7664,7 @@ static unsigned long task_se_h_weight(struct sched_entity *se)
 	update_cfs_rq_h_load(cfs_rq);
 
 	/* Reduce the load.weight by the h_load of the group the task is in. */
-	return (cfs_rq->h_load * se->load.weight) >> SCHED_FIXEDPOINT_SHIFT;
+	return (cfs_rq->max_h_load * se->load.weight) >> SCHED_FIXEDPOINT_SHIFT;
 }
 
 static unsigned long task_se_h_load(struct sched_entity *se)
kernel/sched/pelt.c +16 -0
index 2c901e0f9fbd..2173db8c6209 100644
@@ -58,6 +58,22 @@ static u64 decay_load(u64 val, u64 n)
 	return val;
 }
 
+/*
+ * Like above, with a period of close to a minute for very slow decay.
+ * At 2 seconds per period, LOAD_AVG_PERIOD periods is just over a minute.
+ */
+void slow_decay(unsigned long *load, unsigned long *time)
+{
+	unsigned long periods = (jiffies - *time) / 2;
+
+	if (!periods)
+		return;
+
+	*time += periods * 2;
+
+	*load = decay_load(*load, periods);
+}
+
 static u32 __accumulate_pelt_segments(u64 periods, u32 d1, u32 d3)
 {
 	u32 c1, c2, c3 = d3; /* y^0 == 1 */
kernel/sched/pelt.h +1 -0
index 1152c4ebf314..f16042e2aa06 100644
@@ -6,6 +6,7 @@ int __update_load_avg_se(u64 now, struct cfs_rq *cfs_rq, struct sched_entity *se
 int __update_load_avg_cfs_rq(u64 now, struct cfs_rq *cfs_rq);
 int update_rt_rq_load_avg(u64 now, struct rq *rq, int running);
 int update_dl_rq_load_avg(u64 now, struct rq *rq, int running);
+void slow_decay(unsigned long *load, unsigned long *time);
 
 #ifdef CONFIG_HAVE_SCHED_AVG_IRQ
 int update_irq_load_avg(struct rq *rq, u64 running);
kernel/sched/sched.h +2 -0
index b05fd87cf8b5..b627ca594f97 100644
@@ -540,6 +540,8 @@ struct cfs_rq {
 	 * this group.
 	 */
 	unsigned long		h_load;
+	unsigned long		max_h_load;
+	unsigned long		last_max_h_decay;
 	u64			last_h_load_update;
 	struct sched_entity	*h_load_next;
 #endif /* CONFIG_FAIR_GROUP_SCHED */
Signature · 1 line
2.20.1
#23 lore raw

PATCH v3 13/14 sched,fair: flatten update_curr functionality

In reply to Rik van Riel

Make it clear that update_curr only works on tasks any more.

There is no need for task_tick_fair to call it on every sched entity up the hierarchy, so move the call out of entity_tick.

Header from folded patch 'fix-attach-detach_enticy_cfs_rq.patch':

Subject: sched,fair: fix attach/detach_entity_cfs_rq

While attach_entity_cfs_rq and detach_entity_cfs_rq should iterate over the hierarchy, they do not need to so that twice.

Passing flags into propagate_entity_cfs_rq allows us to reuse that same loop from other functions.

---
 kernel/sched/fair.c | 24 +++++++++++-------------
 1 file changed, 11 insertions(+), 13 deletions(-)
kernel/sched/fair.c +11 -13
index 224cd9b20887..4c7e1818efba 100644
@@ -872,10 +872,11 @@ static void update_tg_load_avg(struct cfs_rq *cfs_rq, int force)
 static void update_curr(struct cfs_rq *cfs_rq)
 {
 	struct sched_entity *curr = cfs_rq->curr;
+	struct task_struct *curtask;
 	u64 now = rq_clock_task(rq_of(cfs_rq));
 	u64 delta_exec;
 
-	if (unlikely(!curr))
+	if (unlikely(!curr) || !entity_is_task(curr))
 		return;
 
 	delta_exec = now - curr->exec_start;
@@ -893,13 +894,10 @@ static void update_curr(struct cfs_rq *cfs_rq)
 	curr->vruntime += calc_delta_fair(delta_exec, curr);
 	update_min_vruntime(cfs_rq);
 
-	if (entity_is_task(curr)) {
-		struct task_struct *curtask = task_of(curr);
-
-		trace_sched_stat_runtime(curtask, delta_exec, curr->vruntime);
-		cgroup_account_cputime(curtask, delta_exec);
-		account_group_exec_runtime(curtask, delta_exec);
-	}
+	curtask = task_of(curr);
+	trace_sched_stat_runtime(curtask, delta_exec, curr->vruntime);
+	cgroup_account_cputime(curtask, delta_exec);
+	account_group_exec_runtime(curtask, delta_exec);
 
 	account_cfs_rq_runtime(cfs_rq, delta_exec);
 }
@@ -4192,11 +4190,6 @@ static void put_prev_entity(struct cfs_rq *cfs_rq, struct sched_entity *prev)
 static void
 entity_tick(struct cfs_rq *cfs_rq, struct sched_entity *curr, int queued)
 {
-	/*
-	 * Update run-time statistics of the 'current'.
-	 */
-	update_curr(cfs_rq);
-
 	/*
 	 * Ensure that runnable average is periodically updated.
 	 */
@@ -10025,6 +10018,11 @@ static void task_tick_fair(struct rq *rq, struct task_struct *curr, int queued)
 	struct cfs_rq *cfs_rq;
 	struct sched_entity *se = &curr->se;
 
+	/*
+	 * Update run-time statistics of the 'current'.
+	 */
+	update_curr(&rq->cfs);
+
 	for_each_sched_entity(se) {
 		cfs_rq = group_cfs_rq_of_parent(se);
 		entity_tick(cfs_rq, se, queued);
Signature · 1 line
2.20.1
#24 lore raw

PATCH v3 14/14 sched,fair: propagate sum_exec_runtime up the hierarchy

In reply to Rik van Riel

Now that enqueue_task_fair and dequeue_task_fair no longer iterate up the hierarchy all the time, a method to lazily propagate sum_exec_runtime up the hierarchy is necessary.

Once a tick, propagate the newly accumulated exec_runtime up the hierarchy, and feed it into CFS bandwidth control.

Remove the pointless call to account_cfs_rq_runtime from update_curr, which is always called with a root cfs_rq.

---
 include/linux/sched.h |  1 +
 kernel/sched/core.c   |  1 +
 kernel/sched/fair.c   | 23 +++++++++++++++++++++--
 3 files changed, 23 insertions(+), 2 deletions(-)
include/linux/sched.h +1 -0
index 901c710363e7..bdca15b3afe7 100644
@@ -454,6 +454,7 @@ struct sched_entity {
 	int				depth;
 	unsigned long			enqueued_h_load;
 	unsigned long			enqueued_h_weight;
+	u64				propagated_exec_runtime;
 	struct load_weight		h_load;
 	struct sched_entity		*parent;
 	/* rq on which this entity is (to be) queued: */
kernel/sched/core.c +1 -0
index fbd96900f715..9915d20e84a9 100644
@@ -2137,6 +2137,7 @@ static void __sched_fork(unsigned long clone_flags, struct task_struct *p)
 	INIT_LIST_HEAD(&p->se.group_node);
 
 #ifdef CONFIG_FAIR_GROUP_SCHED
+	p->se.propagated_exec_runtime	= 0;
 	p->se.cfs_rq			= NULL;
 #endif
 
kernel/sched/fair.c +21 -2
index 4c7e1818efba..0bb8a7e92f07 100644
@@ -898,8 +898,6 @@ static void update_curr(struct cfs_rq *cfs_rq)
 	trace_sched_stat_runtime(curtask, delta_exec, curr->vruntime);
 	cgroup_account_cputime(curtask, delta_exec);
 	account_group_exec_runtime(curtask, delta_exec);
-
-	account_cfs_rq_runtime(cfs_rq, delta_exec);
 }
 
 static void update_curr_fair(struct rq *rq)
@@ -3405,6 +3403,20 @@ static inline bool skip_blocked_update(struct sched_entity *se)
 	return true;
 }
 
+static void propagate_exec_runtime(struct cfs_rq *cfs_rq,
+				struct sched_entity *se)
+{
+	struct sched_entity *parent = se->parent;
+	u64 diff = se->sum_exec_runtime - se->propagated_exec_runtime;
+
+	if (parent) {
+		parent->sum_exec_runtime += diff;
+		account_cfs_rq_runtime(cfs_rq, diff);
+	}
+
+	se->propagated_exec_runtime = se->sum_exec_runtime;
+}
+
 #else /* CONFIG_FAIR_GROUP_SCHED */
 
 static inline void update_tg_load_avg(struct cfs_rq *cfs_rq, int force) {}
@@ -3416,6 +3428,11 @@ static inline int propagate_entity_load_avg(struct sched_entity *se)
 
 static inline void add_tg_cfs_propagate(struct cfs_rq *cfs_rq, long runnable_sum) {}
 
+static void propagate_exec_runtime(struct cfs_rq *cfs_rq,
+				struct sched_entity *se);
+{
+}
+
 #endif /* CONFIG_FAIR_GROUP_SCHED */
 
 /**
@@ -10140,9 +10157,11 @@ static void propagate_entity_cfs_rq(struct sched_entity *se, int flags)
 		if (cfs_rq_throttled(cfs_rq))
 			break;
 
+		/* Walk the hierarchy while PELT says there is work to do.  */
 		if (!update_load_avg(cfs_rq, se, flags))
 			break;
 
+		propagate_exec_runtime(cfs_rq, se);
 		update_cfs_group(se);
 	}
 }
Signature · 1 line
2.20.1
maintainer
#25 lore raw
In reply to Rik van Riel
Quoted 7 lines · Rik van Riel

Plan for the CONFIG_CFS_BANDWIDTH reimplementation:
- When a cgroup gets throttled, mark the cgroup and its children as throttled.
- When pick_next_entity finds a task that is on a throttled cgroup, stash it on the cgroup runqueue (which is not used for runnable tasks any more). Leave the vruntime unchanged, and adjust that runqueue's vruntime to be that of the left-most task.

and ignore such tasks for the load-balancer; I suppose

Quoted 2 lines

- When a cgroup gets unthrottled, and has tasks on it, place it on a vruntime ordered heap separate from the main runqueue.

and expose said heap to the load-balancer..

Now, I suppose you do this because merging heaps is easier than merging RB trees? (not in complexity iirc, but probably in code)

Quoted 3 lines

- Have pick_next_task_fair grab one task off that heap every time it is called, and the min vruntime of that heap is lower than the vruntime of the CPU's cfs_rq (or the CPU has no other runnable tasks).

That's like a smeared out merge :-) But since the other tasks kept on running, this CPUs vruntime will be (much) advanced vs those throttled tasks and we'll most likely end up picking them all before we pick a 'normal' task.

Quoted 7 lines

- Place that selected task on the CPU's cfs_rq, renormalizing its vruntime with the GENTLE_FAIR_SLEEPERS logic. That should help interleave the already runnable tasks with the recently unthrottled group, and prevent thundering herd issues.
- If the group gets throttled again before all of its task had a chance to run, vruntime sorting ensures all the tasks in the throttled cgroup get a chance to run over time.

#26 lore raw
Quoted 10 lines · Peter Zijlstra
Quoted 7 lines · Rik van Riel

Plan for the CONFIG_CFS_BANDWIDTH reimplementation:
- When a cgroup gets throttled, mark the cgroup and its children as throttled.
- When pick_next_entity finds a task that is on a throttled cgroup, stash it on the cgroup runqueue (which is not used for runnable tasks any more). Leave the vruntime unchanged, and adjust that runqueue's vruntime to be that of the left-most task.

and ignore such tasks for the load-balancer; I suppose

Good point. I suppose we need to find a lazy way of doing this, too...

Quoted 15 lines
Quoted 2 lines

- When a cgroup gets unthrottled, and has tasks on it, place it on a vruntime ordered heap separate from the main runqueue.

and expose said heap to the load-balancer..

Now, I suppose you do this because merging heaps is easier than merging
RB trees? (not in complexity iirc, but probably in code)

Quoted 5 lines

- Have pick_next_task_fair grab one task off that heap every time it
is called, and the min vruntime of that heap is lower than the vruntime of the CPU's cfs_rq (or the CPU has no other runnable
tasks).

Quoted 5 lines

That's like a smeared out merge :-) But since the other tasks kept on running, this CPUs vruntime will be (much) advanced vs those throttled
tasks and we'll most likely end up picking them all before we pick a 'normal' task.

The GENTLE_FAIR_SLEEPERS code should place the vruntime of newly unthrottled tasks to the right of that of the current top task on the runqueue, to prevent thundering herd effects (and the throttled group immediately going over its quota again, while causing bad latency for others).

Quoted 12 lines
Quoted 10 lines

- Place that selected task on the CPU's cfs_rq, renormalizing its vruntime with the GENTLE_FAIR_SLEEPERS logic. That should help interleave the already runnable tasks with the recently
unthrottled
group, and prevent thundering herd issues.
- If the group gets throttled again before all of its task had a chance
to run, vruntime sorting ensures all the tasks in the throttled
cgroup
get a chance to run over time.

Signature · 1 line
All Rights Reversed.