Skip to contentSkip to conversation

sched: Rewrite per entity runnable load average tracking

PATCH Superseded Yuyang Du 3 participants 6 messages Jul 16, 2014 – Jul 17, 2014 Scheduler

v3 of 10 Superseded Latest is v10 →

lore
Share & tools
lore

Yuyang Du

author
#0 lore raw

v3 cover sched: Rewrite per entity runnable load average tracking

Regarding the overflow issue, we now have for both entity and cfs_rq:

struct sched_avg {
    .....
    u64 load_sum;
    unsigned long load_avg;
    .....
};

Given the weight for both entity and cfs_rq is:

struct load_weight {
    unsigned long weight;
    .....
};

So, load_sum's max is 47742 * load.weight (which is unsigned long), then on 32bit, it is absolutly safe. On 64bit, with unsigned long being 64bit, but we can afford about 4353082796 (=2^64/47742/88761) entities with the highest weight (=88761) always runnable, even considering we may multiply 1<<15 in decay_load64, we can still support 132845 (=4353082796/2^15) always runnable, which should be acceptible.

load_avg = load_sum / 47742 = load.weight (which is unsigned long), so it should be perfectly safe for both entity (even with arbitrary user group share) and cfs_rq on both 32bit and 64bit. Originally, we saved this division, but have to get it back because of the overflow issue on 32bit (actually load average itself is safe from overflow, but the rest of the code referencing it always uses long, such as cpu_load, etc., which prevents it from saving).

Many thanks to Ben for this revision.

v3 changes:

- Fix overflow issue both for entity and cfs_rq on both 32bit and 64bit.
- Track all entities (both task and group entity) due to group entity's clock issue. This actually improves code simplicity.
- Make a copy of cfs_rq sched_avg's last_update_time, to read an intact 64bit variable on 64bit machine when in data race (hope I did it right).
- Minor fixes and code improvement.

Thanks to PeterZ and Ben for their help in fixing the issues and improving the quality, and Fengguang and his 0Day in finding compile errors in different configurations for version 2.

v2 changes:

- Batch update the tg->load_avg, making sure it is up-to-date before update_cfs_shares
- Remove migrating task from the old CPU/cfs_rq, and do so with atomic operations
- Retrack lod_avg of group's entities (if any), since we need it in task_h_load calc, and do it along with its own cfs_rq's update
- Fix 32bit overflow issue of cfs_rq's load_avg, now it is 64bit, should be safe
- Change load.weight in effective_load which uses runnable load_avg consistently

Yuyang Du (2):
sched: Remove update_rq_runnable_avg
sched: Rewrite per entity runnable load average tracking

 include/linux/sched.h |   21 +-
 kernel/sched/debug.c  |   30 +--
 kernel/sched/fair.c   |  565 ++++++++++++++++---------------------------------
 kernel/sched/proc.c   |    2 +-
 kernel/sched/sched.h  |   22 +-
 5 files changed, 206 insertions(+), 434 deletions(-)
Signature · 1 line
1.7.9.5
author
#1 lore raw

PATCH v3 1/2 sched: Remove update_rq_runnable_avg

The current rq->avg is not made use of anywhere, and the code is in fair scheduler's critical path, so remove it.

---
 kernel/sched/debug.c |    8 --------
 kernel/sched/fair.c  |   24 ++++--------------------
 kernel/sched/sched.h |    2 --
 3 files changed, 4 insertions(+), 30 deletions(-)
kernel/sched/debug.c +0 -8
index 695f977..4b864c7 100644
@@ -68,14 +68,6 @@ static void print_cfs_group_stats(struct seq_file *m, int cpu, struct task_group
 #define PN(F) \
 	SEQ_printf(m, "  .%-30s: %lld.%06ld\n", #F, SPLIT_NS((long long)F))
 
-	if (!se) {
-		struct sched_avg *avg = &cpu_rq(cpu)->avg;
-		P(avg->runnable_avg_sum);
-		P(avg->runnable_avg_period);
-		return;
-	}
-
-
 	PN(se->exec_start);
 	PN(se->vruntime);
 	PN(se->sum_exec_runtime);
kernel/sched/fair.c +4 -20
index fea7d33..1a2d04f 100644
@@ -2430,18 +2430,12 @@ static inline void __update_group_entity_contrib(struct sched_entity *se)
 	}
 }
 
-static inline void update_rq_runnable_avg(struct rq *rq, int runnable)
-{
-	__update_entity_runnable_avg(rq_clock_task(rq), &rq->avg, runnable);
-	__update_tg_runnable_avg(&rq->avg, &rq->cfs);
-}
 #else /* CONFIG_FAIR_GROUP_SCHED */
 static inline void __update_cfs_rq_tg_load_contrib(struct cfs_rq *cfs_rq,
 						 int force_update) {}
 static inline void __update_tg_runnable_avg(struct sched_avg *sa,
 						  struct cfs_rq *cfs_rq) {}
 static inline void __update_group_entity_contrib(struct sched_entity *se) {}
-static inline void update_rq_runnable_avg(struct rq *rq, int runnable) {}
 #endif /* CONFIG_FAIR_GROUP_SCHED */
 
 static inline void __update_task_entity_contrib(struct sched_entity *se)
@@ -2614,7 +2608,6 @@ static inline void dequeue_entity_load_avg(struct cfs_rq *cfs_rq,
  */
 void idle_enter_fair(struct rq *this_rq)
 {
-	update_rq_runnable_avg(this_rq, 1);
 }
 
 /*
@@ -2624,7 +2617,6 @@ void idle_enter_fair(struct rq *this_rq)
  */
 void idle_exit_fair(struct rq *this_rq)
 {
-	update_rq_runnable_avg(this_rq, 0);
 }
 
 static int idle_balance(struct rq *this_rq);
@@ -2633,7 +2625,6 @@ static int idle_balance(struct rq *this_rq);
 
 static inline void update_entity_load_avg(struct sched_entity *se,
 					  int update_cfs_rq) {}
-static inline void update_rq_runnable_avg(struct rq *rq, int runnable) {}
 static inline void enqueue_entity_load_avg(struct cfs_rq *cfs_rq,
 					   struct sched_entity *se,
 					   int wakeup) {}
@@ -3936,10 +3927,9 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags)
 		update_entity_load_avg(se, 1);
 	}
 
-	if (!se) {
-		update_rq_runnable_avg(rq, rq->nr_running);
+	if (!se)
 		add_nr_running(rq, 1);
-	}
+
 	hrtick_update(rq);
 }
 
@@ -3997,10 +3987,9 @@ static void dequeue_task_fair(struct rq *rq, struct task_struct *p, int flags)
 		update_entity_load_avg(se, 1);
 	}
 
-	if (!se) {
+	if (!se)
 		sub_nr_running(rq, 1);
-		update_rq_runnable_avg(rq, 1);
-	}
+
 	hrtick_update(rq);
 }
 
@@ -5437,9 +5426,6 @@ static void __update_blocked_averages_cpu(struct task_group *tg, int cpu)
 		 */
 		if (!se->avg.runnable_avg_sum && !cfs_rq->nr_running)
 			list_del_leaf_cfs_rq(cfs_rq);
-	} else {
-		struct rq *rq = rq_of(cfs_rq);
-		update_rq_runnable_avg(rq, rq->nr_running);
 	}
 }
 
@@ -7352,8 +7338,6 @@ static void task_tick_fair(struct rq *rq, struct task_struct *curr, int queued)
 
 	if (numabalancing_enabled)
 		task_tick_numa(rq, curr);
-
-	update_rq_runnable_avg(rq, 1);
 }
 
 /*
kernel/sched/sched.h +0 -2
index 31cc02e..a147571 100644
@@ -542,8 +542,6 @@ struct rq {
 #ifdef CONFIG_FAIR_GROUP_SCHED
 	/* list of leaf cfs_rq on this cpu: */
 	struct list_head leaf_cfs_rq_list;
-
-	struct sched_avg avg;
 #endif /* CONFIG_FAIR_GROUP_SCHED */
 
 	/*
Signature · 1 line
1.7.9.5
author
#2 lore raw

PATCH v3 2/2 sched: Rewrite per entity runnable load average tracking

In reply to Yuyang Du

The idea of per entity runnable load average (let runnable time contribute to load weight) was proposed by Paul Turner, and it is still followed by this rewrite. This rewrite is done due to the following ends:

1. cfs_rq's load average (namely runnable_load_avg and blocked_load_avg) is updated at the granularity of one entity at one time, which results in the cfs_rq load average is partially updated or asynchronous across its entities: at any time, only one entity is up to date and contributes to the cfs_rq, all other entities are effectively lagging behind.

2. cfs_rq load average is different between top rq->cfs_rq and other task_group's per CPU cfs_rqs in whether or not blocked_load_average contributes to the load.

3. How task_group's load is calculated is complex.

This rewrite tackles these by:

1. Combine runnable and blocked load averages for cfs_rq. And track cfs_rq's load average as a whole.

2. Track task entity load average for carrying it between CPUs in migration, cfs_rq and its own entity load averages are tracked together.

3. All task, cfs_rq/group_entity, and task_group have a consistent and synchronized load_avg.

This rewrite in principle is equivalent to the previous in functionality, but significantly reduces code coplexity and hence increases efficiency and clarity.

In addition, the new load_avg is much more smooth/continuous (no abrupt jumping ups and downs) and decayed/updated more quickly and synchronously to reflect the load dynamic.

---
 include/linux/sched.h |   21 +-
 kernel/sched/debug.c  |   22 +-
 kernel/sched/fair.c   |  541 ++++++++++++++++---------------------------------
 kernel/sched/proc.c   |    2 +-
 kernel/sched/sched.h  |   20 +-
 5 files changed, 202 insertions(+), 404 deletions(-)
include/linux/sched.h +13 -8
index 306f4f0..c981f26 100644
@@ -1067,16 +1067,21 @@ struct load_weight {
 	u32 inv_weight;
 };
 
+/*
+ * The load_avg represents an infinite geometric series. The 64 bit
+ * load_sum can:
+ * 1) for cfs_rq, afford 4353082796 (=2^64/47742/88761) entities with
+ * the highest weight (=88761) always runnable, we should not overflow
+ * 2) for entity, support any load.weight always runnable
+ */
 struct sched_avg {
 	/*
-	 * These sums represent an infinite geometric series and so are bound
-	 * above by 1024/(1-y).  Thus we only need a u32 to store them for all
-	 * choices of y < 1-2^(-32)*1024.
+	 * The load_avg represents an infinite geometric series.
 	 */
-	u32 runnable_avg_sum, runnable_avg_period;
-	u64 last_runnable_update;
-	s64 decay_count;
-	unsigned long load_avg_contrib;
+	u64 last_update_time;
+	u64 load_sum;
+	unsigned long load_avg;
+	u32 period_contrib;
 };
 
 #ifdef CONFIG_SCHEDSTATS
@@ -1142,7 +1147,7 @@ struct sched_entity {
 #endif
 
 #ifdef CONFIG_SMP
-	/* Per-entity load-tracking */
+	/* Per entity load average tracking */
 	struct sched_avg	avg;
 #endif
 };
kernel/sched/debug.c +4 -18
index 4b864c7..34a3f26 100644
@@ -85,10 +85,7 @@ static void print_cfs_group_stats(struct seq_file *m, int cpu, struct task_group
 #endif
 	P(se->load.weight);
 #ifdef CONFIG_SMP
-	P(se->avg.runnable_avg_sum);
-	P(se->avg.runnable_avg_period);
-	P(se->avg.load_avg_contrib);
-	P(se->avg.decay_count);
+	P(se->my_q->avg.load_avg);
 #endif
 #undef PN
 #undef P
@@ -205,19 +202,11 @@ void print_cfs_rq(struct seq_file *m, int cpu, struct cfs_rq *cfs_rq)
 	SEQ_printf(m, "  .%-30s: %d\n", "nr_running", cfs_rq->nr_running);
 	SEQ_printf(m, "  .%-30s: %ld\n", "load", cfs_rq->load.weight);
 #ifdef CONFIG_SMP
-	SEQ_printf(m, "  .%-30s: %ld\n", "runnable_load_avg",
-			cfs_rq->runnable_load_avg);
-	SEQ_printf(m, "  .%-30s: %ld\n", "blocked_load_avg",
-			cfs_rq->blocked_load_avg);
+	SEQ_printf(m, "  .%-30s: %lu\n", "load_avg",
+			cfs_rq->avg.load_avg);
 #ifdef CONFIG_FAIR_GROUP_SCHED
-	SEQ_printf(m, "  .%-30s: %ld\n", "tg_load_contrib",
-			cfs_rq->tg_load_contrib);
-	SEQ_printf(m, "  .%-30s: %d\n", "tg_runnable_contrib",
-			cfs_rq->tg_runnable_contrib);
 	SEQ_printf(m, "  .%-30s: %ld\n", "tg_load_avg",
 			atomic_long_read(&cfs_rq->tg->load_avg));
-	SEQ_printf(m, "  .%-30s: %d\n", "tg->runnable_avg",
-			atomic_read(&cfs_rq->tg->runnable_avg));
 #endif
 #endif
 #ifdef CONFIG_CFS_BANDWIDTH
@@ -624,10 +613,7 @@ void proc_sched_show_task(struct task_struct *p, struct seq_file *m)
 
 	P(se.load.weight);
 #ifdef CONFIG_SMP
-	P(se.avg.runnable_avg_sum);
-	P(se.avg.runnable_avg_period);
-	P(se.avg.load_avg_contrib);
-	P(se.avg.decay_count);
+	P(se.avg.load_avg);
 #endif
 	P(policy);
 	P(prio);
kernel/sched/fair.c +177 -364

842 lines not on this page · raw message

kernel/sched/proc.c +1 -1
index 16f5a30..8f547fe 100644
@@ -504,7 +504,7 @@ static void __update_cpu_load(struct rq *this_rq, unsigned long this_load,
 #ifdef CONFIG_SMP
 static inline unsigned long get_rq_runnable_load(struct rq *rq)
 {
-	return rq->cfs.runnable_load_avg;
+	return rq->cfs.avg.load_avg;
 }
 #else
 static inline unsigned long get_rq_runnable_load(struct rq *rq)
kernel/sched/sched.h +7 -13
index a147571..7c8c2a9 100644
@@ -210,7 +210,6 @@ struct task_group {
 
 #ifdef	CONFIG_SMP
 	atomic_long_t load_avg;
-	atomic_t runnable_avg;
 #endif
 #endif
 
@@ -331,21 +330,16 @@ struct cfs_rq {
 
 #ifdef CONFIG_SMP
 	/*
-	 * CFS Load tracking
-	 * Under CFS, load is tracked on a per-entity basis and aggregated up.
-	 * This allows for the description of both thread and group usage (in
-	 * the FAIR_GROUP_SCHED case).
+	 * CFS load tracking
 	 */
-	unsigned long runnable_load_avg, blocked_load_avg;
-	atomic64_t decay_counter;
-	u64 last_decay;
-	atomic_long_t removed_load;
+	struct sched_avg avg;
+	unsigned long tg_load_avg_contrib;
+	atomic_long_t removed_load_avg;
+#ifndef CONFIG_64BIT
+	u64 load_last_update_time_copy;
+#endif
 
 #ifdef CONFIG_FAIR_GROUP_SCHED
-	/* Required to track per-cpu representation of a task_group */
-	u32 tg_runnable_contrib;
-	unsigned long tg_load_contrib;
-
 	/*
 	 *   h_load = weight * f(tg)
 	 *
Signature · 1 line
1.7.9.5
#3 lore raw

On Wed, Jul 16, 2014 at 02:50:47AM +0100, Yuyang Du wrote:

[...]

Quoted 19 lines
kernel/sched/fair.c +11 -6
+/*
+ * Update load_avg of the cfs_rq along with its own se. They should get
+ * synchronized: group se's load_avg is used for task_h_load calc, and
+ * group cfs_rq's load_avg is used for task_h_load (and update_cfs_share
+ * calc).
+ */
+static inline int update_cfs_rq_load_avg(u64 now, struct cfs_rq *cfs_rq)
 {
-       long old_contrib = se->avg.load_avg_contrib;
+       int decayed;
 
-       if (entity_is_task(se)) {
-               __update_task_entity_contrib(se);
-       } else {
-               __update_tg_runnable_avg(&se->avg, group_cfs_rq(se));
-               __update_group_entity_contrib(se);
+       if (atomic_long_read(&cfs_rq->removed_load_avg)) {
+               long r = atomic_long_xchg(&cfs_rq->removed_load_avg, 0);
+               cfs_rq->avg.load_avg = subtract_until_zero(cfs_rq->avg.load_avg, r);

kernel/sched/fair.c static __always_inline int __update_entity_runnable_avg(u64 now, · line 2392

Quoted 10 lines
kernel/sched/fair.c +6 -2
+               r *= LOAD_AVG_MAX;
+               cfs_rq->avg.load_sum = subtract_until_zero(cfs_rq->avg.load_sum, r);
        }
 
-       return se->avg.load_avg_contrib - old_contrib;
-}
+       decayed = __update_load_avg(now, &cfs_rq->avg, cfs_rq->load.weight);
+#ifndef CONFIG_64BIT
+       if (cfs_rq->avg.last_update_time != cfs_rq->load_last_update_time_copy)
+               sa_q->last_update_time_copy = sa_q->last_update_time;

This doesn't build on 32 bit. You need:

- sa_q->last_update_time_copy = sa_q->last_update_time;
+ cfs_rq->load_last_update_time_copy = cfs_rq->avg.last_update_time;

to make it build. But I'm not convinced that this synchronization is right.

First let me say that I'm not an expert on synchronization. It seems to me that there is nothing preventing reordering of the writes in __update_load_avg() which sets cfs_rq->avg.last_update_time and the update of cfs_rq->avg.load_last_update_time_copy.

Quoted 11 lines
kernel/sched/fair.c +2 -7
+#endif
 
-static inline void subtract_blocked_load_contrib(struct cfs_rq *cfs_rq,
-                                                long load_contrib)
-{
-       if (likely(load_contrib < cfs_rq->blocked_load_avg))
-               cfs_rq->blocked_load_avg -= load_contrib;
-       else
-               cfs_rq->blocked_load_avg = 0;
+       return decayed;
 }

[...]

Quoted 17 lines
kernel/sched/fair.c +3 -7
@@ -4551,18 +4381,34 @@ migrate_task_rq_fair(struct task_struct *p, int next_cpu)
 {
        struct sched_entity *se = &p->se;
        struct cfs_rq *cfs_rq = cfs_rq_of(se);
+       u64 last_update_time;
 
        /*
-        * Load tracking: accumulate removed load so that it can be processed
-        * when we next update owning cfs_rq under rq->lock.  Tasks contribute
-        * to blocked load iff they have a positive decay-count.  It can never
-        * be negative here since on-rq tasks have decay-count == 0.
+        * Task on old CPU catches up with its old cfs_rq, and subtract itself from
+        * the cfs_rq (task must be off the queue now).
         */
-       if (se->avg.decay_count) {
-               se->avg.decay_count = -__synchronize_entity_decay(se);
-               atomic_long_add(se->avg.load_avg_contrib,

kernel/sched/fair.c migrate_task_rq_fair(struct task_struct *p, int next_cpu) · line 4397

Quoted 10 lines
kernel/sched/fair.c +8 -2
-                                               &cfs_rq->removed_load);
-       }
+#ifndef CONFIG_64BIT
+       u64 last_update_time_copy;
+
+       do {
+               last_update_time_copy = cfs_rq->load_last_update_time_copy;
+               smp_rmb();
+               last_update_time = cfs_rq->avg.last_update_time;
+       } while (last_update_time != last_update_time_copy);

Here is the matching read side of the synchronization code. Since reodering might happen on the write side (I believe), I'm not convinced that this works.

Morten

Quoted 19 lines
kernel/sched/fair.c +14 -0
+#else
+       last_update_time = cfs_rq->avg.last_update_time;
+#endif
+       __update_load_avg(last_update_time, &se->avg, 0);
+       atomic_long_add(se->avg.load_avg, &cfs_rq->removed_load_avg);
+
+       /*
+        * We are supposed to update the task to "current" time, then its up to date
+        * and ready to go to new CPU/cfs_rq. But we have difficulty in getting
+        * what current time is, so simply throw away the out-of-date time. This
+        * will result in the wakee task is less decayed, but giving the wakee more
+        * load sounds not bad.
+        */
+       se->avg.last_update_time = 0;
 
        /* We have migrated, no longer consider this task hot */
        se->exec_start = 0;
@@ -5399,36 +5245,6 @@ next:
 }
#4 lore raw
Quoted 39 lines · Morten Rasmussen

On Wed, Jul 16, 2014 at 02:50:47AM +0100, Yuyang Du wrote:

[...]

Quoted 19 lines
kernel/sched/fair.c +11 -6
+/*
+ * Update load_avg of the cfs_rq along with its own se. They should get
+ * synchronized: group se's load_avg is used for task_h_load calc, and
+ * group cfs_rq's load_avg is used for task_h_load (and update_cfs_share
+ * calc).
+ */
+static inline int update_cfs_rq_load_avg(u64 now, struct cfs_rq *cfs_rq)
 {
-       long old_contrib = se->avg.load_avg_contrib;
+       int decayed;
 
-       if (entity_is_task(se)) {
-               __update_task_entity_contrib(se);
-       } else {
-               __update_tg_runnable_avg(&se->avg, group_cfs_rq(se));
-               __update_group_entity_contrib(se);
+       if (atomic_long_read(&cfs_rq->removed_load_avg)) {
+               long r = atomic_long_xchg(&cfs_rq->removed_load_avg, 0);
+               cfs_rq->avg.load_avg = subtract_until_zero(cfs_rq->avg.load_avg, r);

kernel/sched/fair.c static __always_inline int __update_entity_runnable_avg(u64 now, · line 2392

Quoted 10 lines
kernel/sched/fair.c +6 -2
+               r *= LOAD_AVG_MAX;
+               cfs_rq->avg.load_sum = subtract_until_zero(cfs_rq->avg.load_sum, r);
        }
 
-       return se->avg.load_avg_contrib - old_contrib;
-}
+       decayed = __update_load_avg(now, &cfs_rq->avg, cfs_rq->load.weight);
+#ifndef CONFIG_64BIT
+       if (cfs_rq->avg.last_update_time != cfs_rq->load_last_update_time_copy)
+               sa_q->last_update_time_copy = sa_q->last_update_time;

This doesn't build on 32 bit. You need:

(quoted diff) +1 -1
-               sa_q->last_update_time_copy = sa_q->last_update_time;
+               cfs_rq->load_last_update_time_copy = cfs_rq->avg.last_update_time;
Quoted 7 lines

to make it build. But I'm not convinced that this synchronization is right.

First let me say that I'm not an expert on synchronization. It seems to me that there is nothing preventing reordering of the writes in __update_load_avg() which sets cfs_rq->avg.last_update_time and the update of cfs_rq->avg.load_last_update_time_copy.

You're correct, this needs to be if(...) { smp_wmb(); copy = time; }, the same as update_min_vruntime.

author
#5 lore raw
Quoted 39 lines · On Wed, Jul 16, 2014 at 11:53:23AM -0700, [email protected] wrote:
Quoted 37 lines · Morten Rasmussen

On Wed, Jul 16, 2014 at 02:50:47AM +0100, Yuyang Du wrote:

[...]

Quoted 19 lines
kernel/sched/fair.c +11 -6
+/*
+ * Update load_avg of the cfs_rq along with its own se. They should get
+ * synchronized: group se's load_avg is used for task_h_load calc, and
+ * group cfs_rq's load_avg is used for task_h_load (and update_cfs_share
+ * calc).
+ */
+static inline int update_cfs_rq_load_avg(u64 now, struct cfs_rq *cfs_rq)
 {
-       long old_contrib = se->avg.load_avg_contrib;
+       int decayed;
 
-       if (entity_is_task(se)) {
-               __update_task_entity_contrib(se);
-       } else {
-               __update_tg_runnable_avg(&se->avg, group_cfs_rq(se));
-               __update_group_entity_contrib(se);
+       if (atomic_long_read(&cfs_rq->removed_load_avg)) {
+               long r = atomic_long_xchg(&cfs_rq->removed_load_avg, 0);
+               cfs_rq->avg.load_avg = subtract_until_zero(cfs_rq->avg.load_avg, r);

kernel/sched/fair.c static __always_inline int __update_entity_runnable_avg(u64 now, · line 2392

Quoted 10 lines
kernel/sched/fair.c +6 -2
+               r *= LOAD_AVG_MAX;
+               cfs_rq->avg.load_sum = subtract_until_zero(cfs_rq->avg.load_sum, r);
        }
 
-       return se->avg.load_avg_contrib - old_contrib;
-}
+       decayed = __update_load_avg(now, &cfs_rq->avg, cfs_rq->load.weight);
+#ifndef CONFIG_64BIT
+       if (cfs_rq->avg.last_update_time != cfs_rq->load_last_update_time_copy)
+               sa_q->last_update_time_copy = sa_q->last_update_time;

to make it build. But I'm not convinced that this synchronization is right.

Quoted 7 lines
Quoted 4 lines

First let me say that I'm not an expert on synchronization. It seems to me that there is nothing preventing reordering of the writes in __update_load_avg() which sets cfs_rq->avg.last_update_time and the update of cfs_rq->avg.load_last_update_time_copy.

You're correct, this needs to be if(...) { smp_wmb(); copy = time; }, the same as update_min_vruntime.

Ok, I will get the barrier back. Thanks, Morten and Ben.

Yuyang