Skip to contentSkip to conversation

sched: Rewrite per entity runnable load average tracking

PATCH RESEND Superseded Yuyang Du 3 participants 18 messages Oct 10, 2014 – Oct 24, 2014 Scheduler

v5 of 10 Superseded Latest is v10 →

lore
Share & tools
lore

Yuyang Du

author
#0 lore raw

v5 cover sched: Rewrite per entity runnable load average tracking

Rebase to v3.17-rc7.

v5 changes:

Thank Peter intensively for reviewing this patchset in detail and all his comments. And Mike for general and cgroup pipe-test. Morten, Ben, and Vincent in the discussion.

- Remove dead task and task group load_avg
- Do not update trivial delta to task_group load_avg (threshold 1/64 old_contrib)
- mul_u64_u32_shr() is used in decay_load, so on 64bit, load_sum can afford about 4353082796 (=2^64/47742/88761) entities with the highest weight (=88761) always runnable, greater than previous theoretical maximum 132845
- Various code efficiency and style changes

We carried out some performance tests (thanks to Fengguang and his LKP). The results are shown as follows. The patchset (including threepatches) is on top of mainline v3.16-rc5. We may report more perf numbers later.

Overall, this rewrite has better performance, and reduced net overhead in load average tracking, flat efficiency in multi-layer cgroup pipe-test.

v4 changes:

Thanks to Morten, Ben, and Fengguang for v4 revision.

- Insert memory barrier before writing cfs_rq->load_last_update_copy.
- Fix typos.

v3 changes:

Many thanks to Ben for v3 revision.

Regarding the overflow issue, we now have for both entity and cfs_rq:

struct sched_avg {
    .....
    u64 load_sum;
    unsigned long load_avg;
    .....
};

Given the weight for both entity and cfs_rq is:

struct load_weight {
    unsigned long weight;
    .....
};

So, load_sum's max is 47742 * load.weight (which is unsigned long), then on 32bit, it is absolutly safe. On 64bit, with unsigned long being 64bit, but we can afford about 4353082796 (=2^64/47742/88761) entities with the highest weight (=88761) always runnable, even considering we may multiply 1<<15 in decay_load64, we can still support 132845 (=4353082796/2^15) always runnable, which should be acceptible.

load_avg = load_sum / 47742 = load.weight (which is unsigned long), so it should be perfectly safe for both entity (even with arbitrary user group share) and cfs_rq on both 32bit and 64bit. Originally, we saved this division, but have to get it back because of the overflow issue on 32bit (actually load average itself is safe from overflow, but the rest of the code referencing it always uses long, such as cpu_load, etc., which prevents it from saving).

- Fix overflow issue both for entity and cfs_rq on both 32bit and 64bit.
- Track all entities (both task and group entity) due to group entity's clock issue. This actually improves code simplicity.
- Make a copy of cfs_rq sched_avg's last_update_time, to read an intact 64bit variable on 32bit machine when in data race (hope I did it right).
- Minor fixes and code improvement.

v2 changes:

Thanks to PeterZ and Ben for their help in fixing the issues and improving the quality, and Fengguang and his 0Day in finding compile errors in different configurations for version 2.

- Batch update the tg->load_avg, making sure it is up-to-date before update_cfs_shares
- Remove migrating task from the old CPU/cfs_rq, and do so with atomic operations

Yuyang Du (3):
sched: Remove update_rq_runnable_avg
sched: Rewrite per entity runnable load average tracking
sched: Remove task and group entity load_avg when they are dead

 include/linux/sched.h |   21 +-
 kernel/sched/debug.c  |   30 +--
 kernel/sched/fair.c   |  594 ++++++++++++++++---------------------------------
 kernel/sched/proc.c   |    2 +-
 kernel/sched/sched.h  |   22 +-
 5 files changed, 218 insertions(+), 451 deletions(-)
Signature · 16 lines
1.7.9.5

Yuyang Du (3):
  sched: Remove update_rq_runnable_avg
  sched: Rewrite per entity runnable load average tracking
  sched: Remove task and group entity load_avg when they are dead

 include/linux/sched.h |   21 +-
 kernel/sched/debug.c  |   30 +--
 kernel/sched/fair.c   |  594 ++++++++++++++++---------------------------------
 kernel/sched/proc.c   |    2 +-
 kernel/sched/sched.h  |   22 +-
 5 files changed, 218 insertions(+), 451 deletions(-)

-- 
1.7.9.5
author
#1 lore raw

PATCH RESEND v5 1/3 sched: Remove update_rq_runnable_avg

The current rq->avg is not made use of anywhere, and the code is in fair scheduler's critical path, so remove it.

---
 kernel/sched/debug.c |    8 --------
 kernel/sched/fair.c  |   24 ++++--------------------
 kernel/sched/sched.h |    2 --
 3 files changed, 4 insertions(+), 30 deletions(-)
kernel/sched/debug.c +0 -8
index 627b3c3..da0b7d5 100644
@@ -68,14 +68,6 @@ static void print_cfs_group_stats(struct seq_file *m, int cpu, struct task_group
 #define PN(F) \
 	SEQ_printf(m, "  .%-30s: %lld.%06ld\n", #F, SPLIT_NS((long long)F))
 
-	if (!se) {
-		struct sched_avg *avg = &cpu_rq(cpu)->avg;
-		P(avg->runnable_avg_sum);
-		P(avg->runnable_avg_period);
-		return;
-	}
-
-
 	PN(se->exec_start);
 	PN(se->vruntime);
 	PN(se->sum_exec_runtime);
kernel/sched/fair.c +4 -20
index bfa3c86..1eb6ccd 100644
@@ -2446,18 +2446,12 @@ static inline void __update_group_entity_contrib(struct sched_entity *se)
 	}
 }
 
-static inline void update_rq_runnable_avg(struct rq *rq, int runnable)
-{
-	__update_entity_runnable_avg(rq_clock_task(rq), &rq->avg, runnable);
-	__update_tg_runnable_avg(&rq->avg, &rq->cfs);
-}
 #else /* CONFIG_FAIR_GROUP_SCHED */
 static inline void __update_cfs_rq_tg_load_contrib(struct cfs_rq *cfs_rq,
 						 int force_update) {}
 static inline void __update_tg_runnable_avg(struct sched_avg *sa,
 						  struct cfs_rq *cfs_rq) {}
 static inline void __update_group_entity_contrib(struct sched_entity *se) {}
-static inline void update_rq_runnable_avg(struct rq *rq, int runnable) {}
 #endif /* CONFIG_FAIR_GROUP_SCHED */
 
 static inline void __update_task_entity_contrib(struct sched_entity *se)
@@ -2630,7 +2624,6 @@ static inline void dequeue_entity_load_avg(struct cfs_rq *cfs_rq,
  */
 void idle_enter_fair(struct rq *this_rq)
 {
-	update_rq_runnable_avg(this_rq, 1);
 }
 
 /*
@@ -2640,7 +2633,6 @@ void idle_enter_fair(struct rq *this_rq)
  */
 void idle_exit_fair(struct rq *this_rq)
 {
-	update_rq_runnable_avg(this_rq, 0);
 }
 
 static int idle_balance(struct rq *this_rq);
@@ -2649,7 +2641,6 @@ static int idle_balance(struct rq *this_rq);
 
 static inline void update_entity_load_avg(struct sched_entity *se,
 					  int update_cfs_rq) {}
-static inline void update_rq_runnable_avg(struct rq *rq, int runnable) {}
 static inline void enqueue_entity_load_avg(struct cfs_rq *cfs_rq,
 					   struct sched_entity *se,
 					   int wakeup) {}
@@ -3971,10 +3962,9 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags)
 		update_entity_load_avg(se, 1);
 	}
 
-	if (!se) {
-		update_rq_runnable_avg(rq, rq->nr_running);
+	if (!se)
 		add_nr_running(rq, 1);
-	}
+
 	hrtick_update(rq);
 }
 
@@ -4032,10 +4022,9 @@ static void dequeue_task_fair(struct rq *rq, struct task_struct *p, int flags)
 		update_entity_load_avg(se, 1);
 	}
 
-	if (!se) {
+	if (!se)
 		sub_nr_running(rq, 1);
-		update_rq_runnable_avg(rq, 1);
-	}
+
 	hrtick_update(rq);
 }
 
@@ -5471,9 +5460,6 @@ static void __update_blocked_averages_cpu(struct task_group *tg, int cpu)
 		 */
 		if (!se->avg.runnable_avg_sum && !cfs_rq->nr_running)
 			list_del_leaf_cfs_rq(cfs_rq);
-	} else {
-		struct rq *rq = rq_of(cfs_rq);
-		update_rq_runnable_avg(rq, rq->nr_running);
 	}
 }
 
@@ -7404,8 +7390,6 @@ static void task_tick_fair(struct rq *rq, struct task_struct *curr, int queued)
 
 	if (numabalancing_enabled)
 		task_tick_numa(rq, curr);
-
-	update_rq_runnable_avg(rq, 1);
 }
 
 /*
kernel/sched/sched.h +0 -2
index 579712f..fc26556 100644
@@ -545,8 +545,6 @@ struct rq {
 #ifdef CONFIG_FAIR_GROUP_SCHED
 	/* list of leaf cfs_rq on this cpu: */
 	struct list_head leaf_cfs_rq_list;
-
-	struct sched_avg avg;
 #endif /* CONFIG_FAIR_GROUP_SCHED */
 
 	/*
Signature · 1 line
1.7.9.5
author
#2 lore raw

Re: [RESEND PATCH 1/3 v5] sched: Remove update_rq_runnable_avg

kernel/sched/debug.c static void print_cfs_group_stats(struct seq_file *m, int cpu, struct task_group · line 76

Quoted 11 lines · Yuyang Du
kernel/sched/debug.c +0 -6
 
-	if (!se) {
-		struct sched_avg *avg = &cpu_rq(cpu)->avg;
-		P(avg->runnable_avg_sum);
-		P(avg->runnable_avg_period);
-		return;
-	}

This is a bug. Should be:

- if (!se)
- return;

Thanks to 0Day, and good improvement as previously this bug was not found. Also thanks to Chuck Ebbert.

author
#3 lore raw

PATCH RESEND v5 2/3 sched: Rewrite per entity runnable load average tracking

In reply to Yuyang Du

The idea of per entity runnable load average (let runnable time contribute to load weight) was proposed by Paul Turner, and it is still followed by this rewrite. This rewrite is done due to the following ends:

1. cfs_rq's load average (namely runnable_load_avg and blocked_load_avg) is updated at the granularity of one entity at one time, which results in the cfs_rq load average is partially updated or asynchronous across its entities: at any time, only one entity is up to date and contributes to the cfs_rq, all other entities are effectively lagging behind.

2. cfs_rq load average is different between top rq->cfs_rq and other task_group's per CPU cfs_rqs in whether or not blocked_load_average contributes to the load.

3. How task_group's load is calculated is complex.

This rewrite tackles these by:

1. Combine runnable and blocked load averages for cfs_rq. And track cfs_rq's load average as a whole and is used as such.

2. Track task entity load average for carrying it between CPUs in migration, group cfs_rq and its own entity load averages are tracked for update_cfs_shares and task_h_load calc. task_group's load_avg is aggregated from its per CPU cfs_rq's load_avg, which is aggregated from its sched_entities (both task and group entity). Group entity's weight is proportional to its own cfs_rq's load_avg / task_group's load_avg.

3. All task, cfs_rq/group_entity, and task_group have simple, consistent, up-to-date, and synchronized load_avg.

This rewrite in principle is equivalent to the previous in functionality, but significantly reduces code coplexity and hence increases efficiency and clarity. In addition, the new load_avg is much more smooth/continuous (no abrupt jumping ups and downs) and decayed/updated more quickly and synchronously to reflect the load dynamic. As a result, we have less load tracking overhead and better performance.

---
 include/linux/sched.h |   21 +-
 kernel/sched/debug.c  |   22 +-
 kernel/sched/fair.c   |  559 ++++++++++++++++---------------------------------
 kernel/sched/proc.c   |    2 +-
 kernel/sched/sched.h  |   20 +-
 5 files changed, 204 insertions(+), 420 deletions(-)
include/linux/sched.h +13 -8
index b867a4d..af330a0 100644
@@ -1067,16 +1067,21 @@ struct load_weight {
 	u32 inv_weight;
 };
 
+/*
+ * The load_avg represents an infinite geometric series. The 64 bit
+ * load_sum can:
+ * 1) for cfs_rq, afford 4353082796 (=2^64/47742/88761) entities with
+ * the highest weight (=88761) always runnable, we should not overflow
+ * 2) for entity, support any load.weight always runnable
+ */
 struct sched_avg {
 	/*
-	 * These sums represent an infinite geometric series and so are bound
-	 * above by 1024/(1-y).  Thus we only need a u32 to store them for all
-	 * choices of y < 1-2^(-32)*1024.
+	 * The load_avg represents an infinite geometric series.
 	 */
-	u32 runnable_avg_sum, runnable_avg_period;
-	u64 last_runnable_update;
-	s64 decay_count;
-	unsigned long load_avg_contrib;
+	u64 last_update_time;
+	u64 load_sum;
+	unsigned long load_avg;
+	u32 period_contrib;
 };
 
 #ifdef CONFIG_SCHEDSTATS
@@ -1142,7 +1147,7 @@ struct sched_entity {
 #endif
 
 #ifdef CONFIG_SMP
-	/* Per-entity load-tracking */
+	/* Per entity load average tracking */
 	struct sched_avg	avg;
 #endif
 };
kernel/sched/debug.c +4 -18
index da0b7d5..2830342 100644
@@ -85,10 +85,7 @@ static void print_cfs_group_stats(struct seq_file *m, int cpu, struct task_group
 #endif
 	P(se->load.weight);
 #ifdef CONFIG_SMP
-	P(se->avg.runnable_avg_sum);
-	P(se->avg.runnable_avg_period);
-	P(se->avg.load_avg_contrib);
-	P(se->avg.decay_count);
+	P(se->my_q->avg.load_avg);
 #endif
 #undef PN
 #undef P
@@ -205,19 +202,11 @@ void print_cfs_rq(struct seq_file *m, int cpu, struct cfs_rq *cfs_rq)
 	SEQ_printf(m, "  .%-30s: %d\n", "nr_running", cfs_rq->nr_running);
 	SEQ_printf(m, "  .%-30s: %ld\n", "load", cfs_rq->load.weight);
 #ifdef CONFIG_SMP
-	SEQ_printf(m, "  .%-30s: %ld\n", "runnable_load_avg",
-			cfs_rq->runnable_load_avg);
-	SEQ_printf(m, "  .%-30s: %ld\n", "blocked_load_avg",
-			cfs_rq->blocked_load_avg);
+	SEQ_printf(m, "  .%-30s: %lu\n", "load_avg",
+			cfs_rq->avg.load_avg);
 #ifdef CONFIG_FAIR_GROUP_SCHED
-	SEQ_printf(m, "  .%-30s: %ld\n", "tg_load_contrib",
-			cfs_rq->tg_load_contrib);
-	SEQ_printf(m, "  .%-30s: %d\n", "tg_runnable_contrib",
-			cfs_rq->tg_runnable_contrib);
 	SEQ_printf(m, "  .%-30s: %ld\n", "tg_load_avg",
 			atomic_long_read(&cfs_rq->tg->load_avg));
-	SEQ_printf(m, "  .%-30s: %d\n", "tg->runnable_avg",
-			atomic_read(&cfs_rq->tg->runnable_avg));
 #endif
 #endif
 #ifdef CONFIG_CFS_BANDWIDTH
@@ -624,10 +613,7 @@ void proc_sched_show_task(struct task_struct *p, struct seq_file *m)
 
 	P(se.load.weight);
 #ifdef CONFIG_SMP
-	P(se.avg.runnable_avg_sum);
-	P(se.avg.runnable_avg_period);
-	P(se.avg.load_avg_contrib);
-	P(se.avg.decay_count);
+	P(se.avg.load_avg);
 #endif
 	P(policy);
 	P(prio);
kernel/sched/fair.c +179 -380

865 lines not on this page · raw message

kernel/sched/proc.c +1 -1
index 8ecd552..a3a312a 100644
@@ -497,7 +497,7 @@ static void __update_cpu_load(struct rq *this_rq, unsigned long this_load,
 #ifdef CONFIG_SMP
 static inline unsigned long get_rq_runnable_load(struct rq *rq)
 {
-	return rq->cfs.runnable_load_avg;
+	return rq->cfs.avg.load_avg;
 }
 #else
 static inline unsigned long get_rq_runnable_load(struct rq *rq)
kernel/sched/sched.h +7 -13
index fc26556..f823081 100644
@@ -210,7 +210,6 @@ struct task_group {
 
 #ifdef	CONFIG_SMP
 	atomic_long_t load_avg;
-	atomic_t runnable_avg;
 #endif
 #endif
 
@@ -331,21 +330,16 @@ struct cfs_rq {
 
 #ifdef CONFIG_SMP
 	/*
-	 * CFS Load tracking
-	 * Under CFS, load is tracked on a per-entity basis and aggregated up.
-	 * This allows for the description of both thread and group usage (in
-	 * the FAIR_GROUP_SCHED case).
+	 * CFS load tracking
 	 */
-	unsigned long runnable_load_avg, blocked_load_avg;
-	atomic64_t decay_counter;
-	u64 last_decay;
-	atomic_long_t removed_load;
+	struct sched_avg avg;
+	unsigned long tg_load_avg_contrib;
+	atomic_long_t removed_load_avg;
+#ifndef CONFIG_64BIT
+	u64 load_last_update_time_copy;
+#endif
 
 #ifdef CONFIG_FAIR_GROUP_SCHED
-	/* Required to track per-cpu representation of a task_group */
-	u32 tg_runnable_contrib;
-	unsigned long tg_load_contrib;
-
 	/*
 	 *   h_load = weight * f(tg)
 	 *
Signature · 1 line
1.7.9.5
maintainer
#4 lore raw

Review

Quoted 7 lines · Yuyang Du
(quoted diff) +7 -0
+static __always_inline u64 decay_load(u64 val, u64 n)
+{
+	if (likely(val <= UINT_MAX))
+		return decay_load32(val, n);
+
+	return mul_u64_u32_shr(val, decay_load32(1 << 15, n), 15);
+}

This still doesn't make any sense, why not put that mul_u64_u32_shr() into decay_load() and be done with it?

---
 kernel/sched/fair.c | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)
kernel/sched/fair.c +2 -3
index b78280c59b46..67c08d4a3df8 100644
@@ -2231,9 +2231,8 @@ static __always_inline u64 decay_load(u64 val, u64 n)
 		local_n %= LOAD_AVG_PERIOD;
 	}
 
-	val *= runnable_avg_yN_inv[local_n];
-	/* We don't use SRR here since we always want to round down. */
-	return val >> 32;
+	val = mul_u64_u32_shr(val, runnable_avg_yN_inv[local_n], 32);
+	return val;
 }
 
 /*
author
#5 lore raw

kernel/sched/fair.c static __always_inline u64 decay_load(u64 val, u64 n) · line 2235

Quoted 2 lines · Peter Zijlstra
kernel/sched/fair.c +2 -0
+	val = mul_u64_u32_shr(val, runnable_avg_yN_inv[local_n], 32);
+	return val;

Brilliant!

maintainer
#6 lore raw
In reply to Yuyang Du

kernel/sched/fair.c static __always_inline int __update_entity_runnable_avg(u64 now, · line 2363

Quoted 13 lines · Yuyang Du

/*
+ * Updating tg's load_avg is necessary before update_cfs_share (which is done)
+ * and effective_load (which is not done because it is too costly). */

kernel/sched/fair.c +5 -0
+static inline void update_tg_load_avg(struct cfs_rq *cfs_rq)
 {
+	long delta = cfs_rq->avg.load_avg - cfs_rq->tg_load_avg_contrib;
 
+	if (abs(delta) > cfs_rq->tg_load_avg_contrib / 64) {
+		atomic_long_add(delta, &cfs_rq->tg->load_avg);
+		cfs_rq->tg_load_avg_contrib = cfs_rq->avg.load_avg;
 	}
 }

In the thread here: lkml.kernel.org/r/1409094682.29189.23.camel@j-VirtualBox there are concerns about the error bounds of such constructs. We can basically 'leak' nr_cpus * threshold, which is potentially a very large number.

Do we want to introduce the force updated to combat this?

author
#7 lore raw
Quoted 7 lines · Peter Zijlstra

In the thread here: lkml.kernel.org/r/1409094682.29189.23.camel@j-VirtualBox there are concerns about the error bounds of such constructs. We can basically 'leak' nr_cpus * threshold, which is potentially a very large number.

Do we want to introduce the force updated to combat this?

So introduce a force update here like:

+static inline void update_tg_load_avg(struct cfs_rq *cfs_rq, int force)
...
+ if (force || abs(delta) > cfs_rq->tg_load_avg_contrib / 64) {

That is good. In general, I have been lacking a theory about what threshold should be concerning overhead vs. accuracy. But I think adding a force here provides us an option to better comply with the theory if we have it.

Thanks,
Yuyang

maintainer
#8 lore raw
In reply to Yuyang Du
Quoted 5 lines · Yuyang Du

static inline long calc_tg_weight(struct task_group *tg, struct cfs_rq *cfs_rq) {

kernel/sched/fair.c +0 -3
-	long tg_weight;
-
-	/*

kernel/sched/fair.c account_entity_dequeue(struct cfs_rq *cfs_rq, struct sched_entity *se) · line 2094

Quoted 10 lines
kernel/sched/fair.c +1 -9
-	 * Use this CPU's actual weight instead of the last load_contribution
-	 * to gain a more accurate current total weight. See
-	 * update_cfs_rq_load_contribution().
-	 */
-	tg_weight = atomic_long_read(&tg->load_avg);
-	tg_weight -= cfs_rq->tg_load_contrib;
-	tg_weight += cfs_rq->load.weight;
-
-	return tg_weight;
+	return atomic_long_read(&tg->load_avg);

Since you're now also delaying updating load_avg, why not retain this slightly better approximation?

author
#9 lore raw
Quoted 11 lines · Peter Zijlstra
Quoted 10 lines · Yuyang Du

static inline long calc_tg_weight(struct task_group *tg, struct cfs_rq *cfs_rq) {

kernel/sched/fair.c +0 -8
-	long tg_weight;
-
-	/*
-	 * Use this CPU's actual weight instead of the last load_contribution
-	 * to gain a more accurate current total weight. See
-	 * update_cfs_rq_load_contribution().
-	 */
-	tg_weight = atomic_long_read(&tg->load_avg);
Quoted 8 lines

kernel/sched/fair.c account_entity_dequeue(struct cfs_rq *cfs_rq, struct sched_entity *se) · line 2094

Quoted 5 lines
kernel/sched/fair.c +1 -4
-	tg_weight -= cfs_rq->tg_load_contrib;
-	tg_weight += cfs_rq->load.weight;
-
-	return tg_weight;
+	return atomic_long_read(&tg->load_avg);

Since you're now also delaying updating load_avg, why not retain this slightly better approximation?

Oh, yeah, I know load_avg is delayed, but I don't know why this is slightly better approximation, in which "Use this CPU's actual weight instead of the last load_contribution".

In addition to that, if we use this approximation, should we worry that the child cfs and group's added weight is toward likely to be underrated? No?

or we do this instead:

	tg_weight = atomic_long_read(&tg->load_avg);
	tg_weight -= cfs_rq->tg_load_contrib;
	tg_weight += cfs_rq->load_avg;
                         ~~~~~~~~

Thanks,
Yuyang

maintainer
#10 lore raw
In reply to Yuyang Du
Quoted 53 lines · Yuyang Du
(quoted diff) +37 -1
+/* Group cfs_rq's load_avg is used for task_h_load and update_cfs_share */
+static inline int update_cfs_rq_load_avg(u64 now, struct cfs_rq *cfs_rq)
 {
+	int decayed;
 
+	if (atomic_long_read(&cfs_rq->removed_load_avg)) {
+		long r = atomic_long_xchg(&cfs_rq->removed_load_avg, 0);
+		cfs_rq->avg.load_avg = max_t(long, cfs_rq->avg.load_avg - r, 0);
+		cfs_rq->avg.load_sum =
+			max_t(s64, cfs_rq->avg.load_sum - r * LOAD_AVG_MAX, 0);
 	}
 
+	decayed = __update_load_avg(now, &cfs_rq->avg, cfs_rq->load.weight);
 
+#ifndef CONFIG_64BIT
+	smp_wmb();
+	cfs_rq->load_last_update_time_copy = cfs_rq->avg.last_update_time;
+#endif
 
-static inline u64 cfs_rq_clock_task(struct cfs_rq *cfs_rq);
+	return decayed;
+}
 
 
+void remove_entity_load_avg(struct sched_entity *se)
 {
+	struct cfs_rq *cfs_rq = cfs_rq_of(se);
+	u64 last_update_time;
+
+#ifndef CONFIG_64BIT
+	u64 last_update_time_copy;
+
+	do {
+		last_update_time_copy = cfs_rq->load_last_update_time_copy;
+		smp_rmb();
+		last_update_time = cfs_rq->avg.last_update_time;
+	} while (last_update_time != last_update_time_copy);
+#else
+	last_update_time = cfs_rq->avg.last_update_time;
+#endif
 
+	__update_load_avg(last_update_time, &se->avg, 0);
+	atomic_long_add(se->avg.load_avg, &cfs_rq->removed_load_avg);
 }
 
 
+static void migrate_task_rq_fair(struct task_struct *p, int next_cpu)
 {
 	/*
+	 * We are supposed to update the task to "current" time, then its up to date
+	 * and ready to go to new CPU/cfs_rq. But we have difficulty in getting
+	 * what current time is, so simply throw away the out-of-date time. This
+	 * will result in the wakee task is less decayed, but giving the wakee more

Review

Quoted 10 lines
(quoted diff) +6 -0
+	 * load sounds not bad.
 	 */
+	remove_entity_load_avg(&p->se);
+
+	/* Tell new CPU we are migrated */
+	p->se.avg.last_update_time = 0;
 
 	/* We have migrated, no longer consider this task hot */
+	p->se.exec_start = 0;
 }

Because of:

entity_tick()
update_load_avg()
update_cfs_rq_load_avg()

we're likely to only lag TICK_NSEC behind, right? And thus the truncation we do in migrate_task_rq_fair() is of equal size.

Hmm,. one problem, cgroup cfs_rq can be idle for a long while and not get get any ticks at all, so those can lag unbounded. Then again, this appears to be a problem in the current code too, hmm..

Anybody?

maintainer
#11 lore raw
In reply to Yuyang Du

So I would really like to hear from others on this, as it stands I tend to rather like this code -- and it removes a fair amount of icky lines of code.

That said, it does change semantics slightly in that it removes the blocked/runnable split and last time we added the blocked component to the load balancer we had regressions -- now Yuyand and Wu ran this through some testing (AIM7 iirc) and didn't see horrid, but I can't quite remember where we saw the biggest fail back when.

Also, Morten is a little worried about changing all this code seeing how the existing stuff has proven itself over the past few years and feels its somewhat understood. Then again, going by that we'd never change anything.

Ben, Paul, can you guys give this a go?

#12 lore raw
In reply to Yuyang Du

On 10/10/14 04:21, Yuyang Du wrote:

[...]

Quoted 6 lines
kernel/sched/sched.h +0 -2
@@ -331,21 +330,16 @@ struct cfs_rq {
 
  #ifdef CONFIG_SMP
         /*
-        * CFS Load tracking
-        * Under CFS, load is tracked on a per-entity basis and aggregated up.

kernel/sched/sched.h struct cfs_rq { · line 336

Quoted 10 lines
kernel/sched/sched.h +3 -6
-        * This allows for the description of both thread and group usage (in
-        * the FAIR_GROUP_SCHED case).
+        * CFS load tracking
          */
-       unsigned long runnable_load_avg, blocked_load_avg;
-       atomic64_t decay_counter;
-       u64 last_decay;
-       atomic_long_t removed_load;
+       struct sched_avg avg;
+       unsigned long tg_load_avg_contrib;

IMHO, unsigned long tg_load_avg_contrib (former unsigned long tg_load_contrib) can stay under #ifdef CONFIG_FAIR_GROUP_SCHED.

Quoted 13 lines
kernel/sched/sched.h +4 -4
+       atomic_long_t removed_load_avg;
+#ifndef CONFIG_64BIT
+       u64 load_last_update_time_copy;
+#endif
 
  #ifdef CONFIG_FAIR_GROUP_SCHED
-       /* Required to track per-cpu representation of a task_group */
-       u32 tg_runnable_contrib;
-       unsigned long tg_load_contrib;
-
         /*
          *   h_load = weight * f(tg)
          *

[...]

#13 lore raw
In reply to Yuyang Du

On 10/10/14 04:21, Yuyang Du wrote:

[...]

Quoted 6 lines
kernel/sched/sched.h +0 -2
@@ -331,21 +330,16 @@ struct cfs_rq {
 
  #ifdef CONFIG_SMP
         /*
-        * CFS Load tracking
-        * Under CFS, load is tracked on a per-entity basis and aggregated up.

kernel/sched/sched.h struct cfs_rq { · line 336

Quoted 10 lines
kernel/sched/sched.h +3 -6
-        * This allows for the description of both thread and group usage (in
-        * the FAIR_GROUP_SCHED case).
+        * CFS load tracking
          */
-       unsigned long runnable_load_avg, blocked_load_avg;
-       atomic64_t decay_counter;
-       u64 last_decay;
-       atomic_long_t removed_load;
+       struct sched_avg avg;
+       unsigned long tg_load_avg_contrib;

IMHO, unsigned long tg_load_avg_contrib (former unsigned long tg_load_contrib) can stay under #ifdef CONFIG_FAIR_GROUP_SCHED.

Quoted 13 lines
kernel/sched/sched.h +4 -4
+       atomic_long_t removed_load_avg;
+#ifndef CONFIG_64BIT
+       u64 load_last_update_time_copy;
+#endif
 
  #ifdef CONFIG_FAIR_GROUP_SCHED
-       /* Required to track per-cpu representation of a task_group */
-       u32 tg_runnable_contrib;
-       unsigned long tg_load_contrib;
-
         /*
          *   h_load = weight * f(tg)
          *

[...]

author
#14 lore raw
Quoted 5 lines · Dietmar Eggemann
Quoted 1 line

+ unsigned long tg_load_avg_contrib;

IMHO, unsigned long tg_load_avg_contrib (former unsigned long tg_load_contrib) can stay under #ifdef CONFIG_FAIR_GROUP_SCHED.

Indeed it is. Thanks.

author
#15 lore raw

PATCH RESEND v5 3/3 sched: Remove task and group entity load_avg when they are dead

In reply to Yuyang Du

When task exits or group is destroyed, the entity's load_avg should be removed from its parent cfs_rq's load_avg. Otherwise, it will take time for the parent cfs_rq to decay the dead entity's load_avg to 0, which is not desired.

---
 kernel/sched/fair.c |   11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)
kernel/sched/fair.c +10 -1
index 6555536..1e7b26f 100644
@@ -4435,6 +4435,11 @@ static void migrate_task_rq_fair(struct task_struct *p, int next_cpu)
 	/* We have migrated, no longer consider this task hot */
 	p->se.exec_start = 0;
 }
+
+static void task_dead_fair(struct task_struct *p)
+{
+	remove_entity_load_avg(&p->se);
+}
 #endif /* CONFIG_SMP */
 
 static unsigned long
@@ -7419,8 +7424,11 @@ void free_fair_sched_group(struct task_group *tg)
 	for_each_possible_cpu(i) {
 		if (tg->cfs_rq)
 			kfree(tg->cfs_rq[i]);
-		if (tg->se)
+		if (tg->se) {
+			if (tg->se[i])
+				remove_entity_load_avg(tg->se[i]);
 			kfree(tg->se[i]);
+		}
 	}
 
 	kfree(tg->cfs_rq);
@@ -7606,6 +7614,7 @@ const struct sched_class fair_sched_class = {
 	.rq_offline		= rq_offline_fair,
 
 	.task_waking		= task_waking_fair,
+	.task_dead			= task_dead_fair,
 #endif
 
 	.set_curr_task          = set_curr_task_fair,
Signature · 1 line
1.7.9.5
maintainer
#16 lore raw

Re: [RESEND PATCH 3/3 v5] sched: Remove task and group entity load_avg when they are dead

Quoted 4 lines · Yuyang Du

When task exits or group is destroyed, the entity's load_avg should be removed from its parent cfs_rq's load_avg. Otherwise, it will take time for the parent cfs_rq to decay the dead entity's load_avg to 0, which is not desired.

Right, this is something pjt wanted to do ever since we got that callback in for dl. Nice.

maintainer
#17 lore raw
In reply to Yuyang Du
Quoted 1 line · Yuyang Du

Rebase to v3.17-rc7.

Ah an entire new posting, that works too :-) I'll try and have a look soon, but it might not be until after next week (RTLWS/Linuxcon/LPC and all that stuff over in Dusseldorf).