sched: Rewrite runnable load and utilization average tracking
v10 cover sched: Rewrite runnable load and utilization average tracking
Hi Peter and Ingo,
Changes are made to the 10th version:
1) Include Vincent's fix to update blocked load when CONFIG_FAIR_GROUP_SCHED=n
2) Get runnable_load_avg back
3) Clean up the references to load averages a little bit
Thanks a lot to Dietmar for his review, tests, and suggestions. Thanks a lot for the thoughts and suggestions from Morten and Peter.
The discussion we made is really inspiring...
Thanks,
Yuyang
v9 changes:
1) util_avg fix
2) update_tg_load_avg() in update_blocked_averages(), because cfs_rq's load_avg is updated
3) Add some debug print
v8 changes:
1) Rebase to the latest tip tree
2) scale_load_down the weight when doing the averages
3) change util_sum to u32
Thanks a lot for Ben's comments, which lead to this version. Thanks to Vincent for review.
v7 changes:
The 7th version mostly is to accommodate the utilization load average recently merged into kernel. The general idea is as well to update the cfs_rq as a whole as opposed to only updating an entity at a time and update the cfs_rq with the only updated entity.
1) Rename utilization_load_avg to util_avg to be concise and meaningful
2) To track the cfs_rq util_avg, simply use "cfs_rq->curr != NULL" as the predicate. This should be equivalent to but simpler than aggregating each individual child sched_entity's util_avg when "cfs_rq->curr == se". Because if cfs_rq->curr != NULL, the cfs_rq->curr has to be some se.
3) Remove se's util_avg from its cfs_rq's when migrating it, this was already proposed by Morten and patches sent
3) The group entity's load average is initiated when the entity is created
4) Small nits: the entity's util_avg is removed from switched_from_fair() and task_move_group_fair().
Thanks a lot for Vincent and Morten's help for the 7th version.
Thanks,
Yuyang
v6 changes:
Many thanks to PeterZ for his review, to Dietmar, and to Fengguang for 0Day and LKP.
Rebased on v3.18-rc2.
- Unify decay_load 32 and 64 bits by mul_u64_u32_shr
- Add force option in update_tg_load_avg
- Read real-time cfs's load_avg for calc_tg_weight
- Have tg_load_avg_contrib ifdef CONFIG_FAIR_GROUP_SCHED
- Bug fix
v5 changes:
Thank Peter intensively for reviewing this patchset in detail and all his comments. And Mike for general and cgroup pipe-test. Morten, Ben, and Vincent in the discussion.
- Remove dead task and task group load_avg
- Do not update trivial delta to task_group load_avg (threshold 1/64 old_contrib)
- mul_u64_u32_shr() is used in decay_load, so on 64bit, load_sum can afford about 4353082796 (=2^64/47742/88761) entities with the highest weight (=88761) always runnable, greater than previous theoretical maximum 132845
- Various code efficiency and style changes
We carried out some performance tests (thanks to Fengguang and his LKP). The results are shown as follows. The patchset (including threepatches) is on top of mainline v3.16-rc5. We may report more perf numbers later.
Overall, this rewrite has better performance, and reduced net overhead in load average tracking, flat efficiency in multi-layer cgroup pipe-test.
v4 changes:
Thanks to Morten, Ben, and Fengguang for v4 revision.
- Insert memory barrier before writing cfs_rq->load_last_update_copy.
- Fix typos.
v3 changes:
Many thanks to Ben for v3 revision.
Regarding the overflow issue, we now have for both entity and cfs_rq:
struct sched_avg {
.....
u64 load_sum;
unsigned long load_avg;
.....
};Given the weight for both entity and cfs_rq is:
struct load_weight {
unsigned long weight;
.....
};So, load_sum's max is 47742 * load.weight (which is unsigned long), then on 32bit, it is absolutly safe. On 64bit, with unsigned long being 64bit, but we can afford about 4353082796 (=2^64/47742/88761) entities with the highest weight (=88761) always runnable, even considering we may multiply 1<<15 in decay_load64, we can still support 132845 (=4353082796/2^15) always runnable, which should be acceptible.
load_avg = load_sum / 47742 = load.weight (which is unsigned long), so it should be perfectly safe for both entity (even with arbitrary user group share) and cfs_rq on both 32bit and 64bit. Originally, we saved this division, but have to get it back because of the overflow issue on 32bit (actually load average itself is safe from overflow, but the rest of the code referencing it always uses long, such as cpu_load, etc., which prevents it from saving).
- Fix overflow issue both for entity and cfs_rq on both 32bit and 64bit.
- Track all entities (both task and group entity) due to group entity's clock issue. This actually improves code simplicity.
- Make a copy of cfs_rq sched_avg's last_update_time, to read an intact 64bit variable on 32bit machine when in data race (hope I did it right).
- Minor fixes and code improvement.
v2 changes:
Thanks to PeterZ and Ben for their help in fixing the issues and improving the quality, and Fengguang and his 0Day in finding compile errors in different configurations for version 2.
- Batch update the tg->load_avg, making sure it is up-to-date before update_cfs_shares
- Remove migrating task from the old CPU/cfs_rq, and do so with atomic operations
Yuyang Du (7):
sched: Remove rq's runnable avg
sched: Rewrite runnable load and utilization average tracking
sched: Implement update_blocked_averages() for
CONFIG_FAIR_GROUP_SCHED=n
sched: Init cfs_rq's sched_entity load average
sched: Remove task and group entity load when they are dead
sched: Provide runnable_load_avg back to cfs_rq
sched: Clean up load average references
include/linux/sched.h | 41 ++- kernel/sched/core.c | 5 +- kernel/sched/debug.c | 48 ++-- kernel/sched/fair.c | 736 +++++++++++++++++++------------------------------- kernel/sched/sched.h | 34 +-- 5 files changed, 334 insertions(+), 530 deletions(-)
Signature · 1 line
2.1.4
PATCH v10 1/7 sched: Remove rq's runnable avg
The current rq->avg is not used at all since its merge into kernel, and the code is in the scheduler's hot path, so remove it.
--- kernel/sched/debug.c | 7 +------ kernel/sched/fair.c | 25 ++++--------------------- kernel/sched/sched.h | 2 -- 3 files changed, 5 insertions(+), 29 deletions(-)
kernel/sched/debug.c +1 -6
@@ -68,13 +68,8 @@ static void print_cfs_group_stats(struct seq_file *m, int cpu, struct task_group#define PN (F ) \ SEQ_printf(m, " .%-30s: %lld.%06ld\n", #F, SPLIT_NS((long long)F)) -if (!se) { -struct sched_avg *avg = &cpu_rq (cpu)->avg; -P (avg->runnable_avg_sum); -P (avg->avg_period); +if (!se)return ; - } -PN (se->exec_start);PN (se->vruntime);
kernel/sched/fair.c +4 -21
@@ -2728,19 +2728,12 @@ static inline void __update_group_entity_contrib(struct sched_entity *se) } } -static inline void update_rq_runnable_avg (struct rq *rq,int runnable) -{ -__update_entity_runnable_avg (rq_clock_task (rq),cpu_of (rq), &rq->avg, - runnable, runnable); -__update_tg_runnable_avg (&rq->avg, &rq->cfs); -}#else /* CONFIG_FAIR_GROUP_SCHED */ static inline void __update_cfs_rq_tg_load_contrib (struct cfs_rq *cfs_rq,int force_update) {}static inline void __update_tg_runnable_avg (struct sched_avg *sa,struct cfs_rq *cfs_rq) {}static inline void __update_group_entity_contrib (struct sched_entity *se) {} -static inline void update_rq_runnable_avg (struct rq *rq,int runnable) {}#endif /* CONFIG_FAIR_GROUP_SCHED */ static inline void __update_task_entity_contrib (struct sched_entity *se)
@@ -2944,7 +2937,6 @@ static inline void dequeue_entity_load_avg(struct cfs_rq *cfs_rq,*/ void idle_enter_fair (struct rq *this_rq) { -update_rq_runnable_avg (this_rq,1 ); }/*
@@ -2954,7 +2946,6 @@ void idle_enter_fair(struct rq *this_rq)*/ void idle_exit_fair (struct rq *this_rq) { -update_rq_runnable_avg (this_rq,0 ); }static int idle_balance (struct rq *this_rq);
@@ -2963,7 +2954,6 @@ static int idle_balance(struct rq *this_rq);static inline void update_entity_load_avg (struct sched_entity *se,int update_cfs_rq) {} -static inline void update_rq_runnable_avg (struct rq *rq,int runnable) {}static inline void enqueue_entity_load_avg (struct cfs_rq *cfs_rq,struct sched_entity *se,int wakeup) {}
@@ -4262,10 +4252,9 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags)update_entity_load_avg (se,1 ); } -if (!se) { -update_rq_runnable_avg (rq, rq->nr_running); +if (!se)add_nr_running (rq,1 ); - } +hrtick_update (rq); }
@@ -4323,10 +4312,9 @@ static void dequeue_task_fair(struct rq *rq, struct task_struct *p, int flags)update_entity_load_avg (se,1 ); } -if (!se) { +if (!se)sub_nr_running (rq,1 ); -update_rq_runnable_avg (rq,1 ); - } +hrtick_update (rq); }
@@ -6034,9 +6022,6 @@ static void __update_blocked_averages_cpu(struct task_group *tg, int cpu)*/ if (!se->avg.runnable_avg_sum && !cfs_rq->nr_running)list_del_leaf_cfs_rq (cfs_rq); - }else { -struct rq *rq =rq_of (cfs_rq); -update_rq_runnable_avg (rq, rq->nr_running); } }
@@ -8020,8 +8005,6 @@ static void task_tick_fair(struct rq *rq, struct task_struct *curr, int queued)if (numabalancing_enabled)task_tick_numa (rq, curr); - -update_rq_runnable_avg (rq,1 ); }/*
kernel/sched/sched.h +0 -2
@@ -595,8 +595,6 @@ struct rq {#ifdef CONFIG_FAIR_GROUP_SCHED /* list of leaf cfs_rq on this cpu: */ struct list_head leaf_cfs_rq_list; - -struct sched_avg avg;#endif /* CONFIG_FAIR_GROUP_SCHED */ /*
Signature · 1 line
2.1.4
tip-bot for Yuyang Du
merge bot[tip:sched/core] sched/fair: Remove rq's runnable avg
sched/fair: Remove rq's runnable avg
Commit-ID: cd126afe838d7ea9b971cdea087fd498a7293c7f Gitweb: http://git.kernel.org/tip/cd126afe838d7ea9b971cdea087fd498a7293c7f Author: Yuyang Du <[email protected]> AuthorDate: Wed, 15 Jul 2015 08:04:36 +0800 Committer: Ingo Molnar <[email protected]> CommitDate: Mon, 3 Aug 2015 12:21:28 +0200
sched/fair: Remove rq's runnable avg
The current rq->avg is not used at all since its merge into the kernel, and the code is in the scheduler's hot path, so remove it.
--- kernel/sched/debug.c | 7 +------ kernel/sched/fair.c | 25 ++++--------------------- kernel/sched/sched.h | 2 -- 3 files changed, 5 insertions(+), 29 deletions(-)
kernel/sched/debug.c +1 -6
@@ -68,13 +68,8 @@ static void print_cfs_group_stats(struct seq_file *m, int cpu, struct task_group#define PN (F ) \ SEQ_printf(m, " .%-30s: %lld.%06ld\n", #F, SPLIT_NS((long long)F)) -if (!se) { -struct sched_avg *avg = &cpu_rq (cpu)->avg; -P (avg->runnable_avg_sum); -P (avg->avg_period); +if (!se)return ; - } -PN (se->exec_start);PN (se->vruntime);
kernel/sched/fair.c +4 -21
@@ -2724,19 +2724,12 @@ static inline void __update_group_entity_contrib(struct sched_entity *se) } } -static inline void update_rq_runnable_avg (struct rq *rq,int runnable) -{ -__update_entity_runnable_avg (rq_clock_task (rq),cpu_of (rq), &rq->avg, - runnable, runnable); -__update_tg_runnable_avg (&rq->avg, &rq->cfs); -}#else /* CONFIG_FAIR_GROUP_SCHED */ static inline void __update_cfs_rq_tg_load_contrib (struct cfs_rq *cfs_rq,int force_update) {}static inline void __update_tg_runnable_avg (struct sched_avg *sa,struct cfs_rq *cfs_rq) {}static inline void __update_group_entity_contrib (struct sched_entity *se) {} -static inline void update_rq_runnable_avg (struct rq *rq,int runnable) {}#endif /* CONFIG_FAIR_GROUP_SCHED */ static inline void __update_task_entity_contrib (struct sched_entity *se)
@@ -2940,7 +2933,6 @@ static inline void dequeue_entity_load_avg(struct cfs_rq *cfs_rq,*/ void idle_enter_fair (struct rq *this_rq) { -update_rq_runnable_avg (this_rq,1 ); }/*
@@ -2950,7 +2942,6 @@ void idle_enter_fair(struct rq *this_rq)*/ void idle_exit_fair (struct rq *this_rq) { -update_rq_runnable_avg (this_rq,0 ); }static int idle_balance (struct rq *this_rq);
@@ -2959,7 +2950,6 @@ static int idle_balance(struct rq *this_rq);static inline void update_entity_load_avg (struct sched_entity *se,int update_cfs_rq) {} -static inline void update_rq_runnable_avg (struct rq *rq,int runnable) {}static inline void enqueue_entity_load_avg (struct cfs_rq *cfs_rq,struct sched_entity *se,int wakeup) {}
@@ -4258,10 +4248,9 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags)update_entity_load_avg (se,1 ); } -if (!se) { -update_rq_runnable_avg (rq, rq->nr_running); +if (!se)add_nr_running (rq,1 ); - } +hrtick_update (rq); }
@@ -4319,10 +4308,9 @@ static void dequeue_task_fair(struct rq *rq, struct task_struct *p, int flags)update_entity_load_avg (se,1 ); } -if (!se) { +if (!se)sub_nr_running (rq,1 ); -update_rq_runnable_avg (rq,1 ); - } +hrtick_update (rq); }
@@ -6005,9 +5993,6 @@ static void __update_blocked_averages_cpu(struct task_group *tg, int cpu)*/ if (!se->avg.runnable_avg_sum && !cfs_rq->nr_running)list_del_leaf_cfs_rq (cfs_rq); - }else { -struct rq *rq =rq_of (cfs_rq); -update_rq_runnable_avg (rq, rq->nr_running); } }
@@ -7988,8 +7973,6 @@ static void task_tick_fair(struct rq *rq, struct task_struct *curr, int queued)if (numabalancing_enabled)task_tick_numa (rq, curr); - -update_rq_runnable_avg (rq,1 ); }/*
kernel/sched/sched.h +0 -2
@@ -595,8 +595,6 @@ struct rq {#ifdef CONFIG_FAIR_GROUP_SCHED /* list of leaf cfs_rq on this cpu: */ struct list_head leaf_cfs_rq_list; - -struct sched_avg avg;#endif /* CONFIG_FAIR_GROUP_SCHED */ /*
PATCH v10 2/7 sched: Rewrite runnable load and utilization average tracking
In reply to Yuyang DuThe idea of runnable load average (let runnable time contribute to weight) was proposed by Paul Turner and Ben Segall, and it is still followed by this rewrite. This rewrite aims to solve the following issues:
1. cfs_rq's load average (namely runnable_load_avg and blocked_load_avg) is updated at the granularity of an entity at a time, which results in the cfs_rq's load average is stale or partially updated: at any time, only one entity is up to date, all other entities are effectively lagging behind. This is undesirable.
To illustrate, if we have n runnable entities in the cfs_rq, as time elapses, they certainly become outdated:
t0: cfs_rq { e1_old, e2_old, ..., en_old }
and when we update:
t1: update e1, then we have cfs_rq { e1_new, e2_old, ..., en_old }
t2: update e2, then we have cfs_rq { e1_old, e2_new, ..., en_old }
...
We solve this by combining all runnable entities' load averages together in cfs_rq's avg, and update the cfs_rq's avg as a whole. This is based on the fact that if we regard the update as a function, then:
w * update(e) = update(w * e) and
update(e1) + update(e2) = update(e1 + e2), then
w1 * update(e1) + w2 * update(e2) = update(w1 * e1 + w2 * e2)
therefore, by this rewrite, we have an entirely updated cfs_rq at the time we update it:
t1: update cfs_rq { e1_new, e2_new, ..., en_new }
t2: update cfs_rq { e1_new, e2_new, ..., en_new }
...
2. cfs_rq's load average is different between top rq->cfs_rq and other task_group's per CPU cfs_rqs in whether or not blocked_load_average contributes to the load.
The basic idea behind runnable load average (the same for utilization) is that the blocked state is taken into account as opposed to only accounting for the currently runnable state. Therefore, the average should include both the runnable/running and blocked load averages. This rewrite does that.
In addition, we also combine runnable/running and blocked averages of all entities into the cfs_rq's average, and update it together at once. This is based on the fact that:
update(runnable) + update(blocked) = update(runnable + blocked)
This significantly reduces the codes as we don't need to separately maintain/update runnable/running load and blocked load.
3. How task_group entities' share is calculated is complex and imprecise.
We reduce the complexity in this rewrite to allow a very simple rule:
the task_group's load_avg is aggregated from its per CPU cfs_rqs's load_avgs. Then group entity's weight is simply proportional to its own cfs_rq's load_avg / task_group's load_avg. To illustrate,
if a task_group has { cfs_rq1, cfs_rq2, ..., cfs_rqn }, then,
task_group_avg = cfs_rq1_avg + cfs_rq2_avg + ... + cfs_rqn_avg, then
cfs_rqx's entity's share = cfs_rqx_avg / task_group_avg * task_group's share
To sum up, this rewrite in principle is equivalent to the current one, but fixes the issues described above. Turns out, it significantly reduces the code complexity and hence increases clarity and efficiency. In addition, the new averages are more smooth/continuous (no spurious spikes and valleys) and updated more consistently and quickly to reflect the load dynamics. As a result, we have less load tracking overhead, better performance, and especially better power efficiency due to more balanced load.
--- include/linux/sched.h | 41 ++-- kernel/sched/core.c | 3 - kernel/sched/debug.c | 41 ++-- kernel/sched/fair.c | 630 +++++++++++++++++--------------------------------- kernel/sched/sched.h | 28 +-- 5 files changed, 250 insertions(+), 493 deletions(-)
include/linux/sched.h +18 -23
@@ -1183,29 +1183,24 @@ struct load_weight {u32 inv_weight; }; +/* +* The load_avg/util_avg accumulates an infinite geometric series. +* 1) load_avg factors the amount of time that a sched_entity is +* runnable on a rq into its weight. For cfs_rq, it is the aggregated +* such weights of all runnable and blocked sched_entities. +* 2) util_avg factors frequency scaling into the amount of time +* that a sched_entity is running on a CPU, in the range [0..SCHED_LOAD_SCALE]. +* For cfs_rq, it is the aggregated such times of all runnable and +* blocked sched_entities. +* The 64 bit load_sum can: +* 1) for cfs_rq, afford 4353082796 (=2^64/47742/88761) entities with +* the highest weight (=88761) always runnable, we should not overflow +* 2) for entity, support any load.weight always runnable +*/ struct sched_avg { -u64 last_runnable_update; -s64 decay_count; -/* -* utilization_avg_contrib describes the amount of time that a -* sched_entity is running on a CPU. It is based on running_avg_sum -* and is scaled in the range [0..SCHED_LOAD_SCALE]. -* load_avg_contrib described the amount of time that a sched_entity -* is runnable on a rq. It is based on both runnable_avg_sum and the -* weight of the task. -*/ -unsigned long load_avg_contrib, utilization_avg_contrib; -/* -* These sums represent an infinite geometric series and so are bound -* above by 1024/(1-y). Thus we only need a u32 to store them for all -* choices of y < 1-2^(-32)*1024. -* running_avg_sum reflects the time that the sched_entity is -* effectively running on the CPU. -* runnable_avg_sum represents the amount of time a sched_entity is on -* a runqueue which includes the running time that is monitored by -* running_avg_sum. -*/ -u32 runnable_avg_sum, avg_period, running_avg_sum; +u64 last_update_time, load_sum; +u32 util_sum, period_contrib; +unsigned long load_avg, util_avg; };#ifdef CONFIG_SCHEDSTATS
@@ -1271,7 +1266,7 @@ struct sched_entity {#endif #ifdef CONFIG_SMP -/* Per-entity load-tracking */ +/* Per entity load average tracking */ struct sched_avg avg;#endif };
kernel/sched/core.c +0 -3
@@ -1828,9 +1828,6 @@ static void __sched_fork(unsigned long clone_flags, struct task_struct *p) p->se.prev_sum_exec_runtime =0 ; p->se.nr_migrations =0 ; p->se.vruntime =0 ; -#ifdef CONFIG_SMP - p->se.avg.decay_count =0 ; -#endif INIT_LIST_HEAD (&p->se.group_node);#ifdef CONFIG_SCHEDSTATS
kernel/sched/debug.c +17 -24
62 lines not on this page · raw message
kernel/sched/fair.c +206 -424
964 lines not on this page · raw message
kernel/sched/sched.h +9 -19
@@ -245,7 +245,6 @@ struct task_group {#ifdef CONFIG_SMP atomic_long_t load_avg; -atomic_t runnable_avg;#endif #endif
@@ -366,27 +365,18 @@ struct cfs_rq {#ifdef CONFIG_SMP /* -* CFS Load tracking -* Under CFS, load is tracked on a per-entity basis and aggregated up. -* This allows for the description of both thread and group usage (in -* the FAIR_GROUP_SCHED case). -* runnable_load_avg is the sum of the load_avg_contrib of the -* sched_entities on the rq. -* blocked_load_avg is similar to runnable_load_avg except that its -* the blocked sched_entities on the rq. -* utilization_load_avg is the sum of the average running time of the -* sched_entities on the rq. +* CFS load tracking */ -unsigned long runnable_load_avg, blocked_load_avg, utilization_load_avg; -atomic64_t decay_counter; -u64 last_decay; -atomic_long_t removed_load; - +struct sched_avg avg;#ifdef CONFIG_FAIR_GROUP_SCHED -/* Required to track per-cpu representation of a task_group */ -u32 tg_runnable_contrib; -unsigned long tg_load_contrib; +unsigned long tg_load_avg_contrib; +#endif +atomic_long_t removed_load_avg, removed_util_avg; +#ifndef CONFIG_64BIT +u64 load_last_update_time_copy; +#endif +#ifdef CONFIG_FAIR_GROUP_SCHED /* * h_load = weight * f(tg) *
Signature · 1 line
2.1.4
Dietmar Eggemann
reviewerHi Yuyang,
On 15/07/15 01:04, Yuyang Du wrote:
[...]
kernel/sched/fair.c static long effective_load(struct task_group *tg, int cpu, long wl, long wg) · line 4493
Quoted 9 lines
kernel/sched/fair.c +1
-1@@ -4674,7 +4487,7 @@ static long effective_load(struct task_group *tg, int cpu, long wl, long wg)/* * w = rw_i + @wl */ - w = se->my_q->load.weight + wl; + w = se->my_q->avg.load_avg + wl;/* * wl = S * s'_i; see (2)
There is a comment 'Per the above, wl is the new *se->load.weight* value'. This should be replaced by *se->avg.load_avg*. Also the function header explains the functionality of effective_load() based on weight and not sched_avg::load_avg.
Quoted 13 lines
kernel/sched/fair.c +1
-1@@ -4695,7 +4508,7 @@ static long effective_load(struct task_group *tg, int cpu, long wl, long wg)/* * wl = dw_i = S * (s'_i - s_i); see (3) */ - wl -= se->load.weight; + wl -= se->avg.load_avg;/* * Recursively apply this logic to all parent groups to compute @@ -4769,14 +4582,14 @@ static int wake_affine(struct sched_domain *sd, struct task_struct *p, int sync)*/ if (sync) { tg =task_group (current);
kernel/sched/fair.c static int wake_affine(struct sched_domain *sd, struct task_struct *p, int sync) · line 4592
Quoted 10 lines
kernel/sched/fair.c +2
-2- weight = current->se.load.weight; + weight = current->se.avg.load_avg; this_load +=effective_load (tg, this_cpu, -weight, -weight); load +=effective_load (tg, prev_cpu,0 , -weight); } tg =task_group (p); - weight = p->se.load.weight; + weight = p->se.avg.load_avg;
You changed cfs_rq->load.weight to cfs_rq->avg.load_avg and se->load.weight to se->avg.load_avg in effective_load() and wake_affine() in v2.
I wasn't able to find explanation why you did this. I mean we still have to maintain 'struct load_weight' on cfs_rq's and se's representing tg's.
-- Dietmar
[...]
Hi Dietmar,
Quoted 13 lines · Dietmar Eggemann
Hi Yuyang,
On 15/07/15 01:04, Yuyang Du wrote:
[...]
Quoted 7 lines
kernel/sched/fair.c +1
-1@@ -4674,7 +4487,7 @@ static long effective_load(struct task_group *tg, int cpu, long wl, long wg)/* * w = rw_i + @wl */ - w = se->my_q->load.weight + wl; + w = se->my_q->avg.load_avg + wl;
Quoted 7 lines
kernel/sched/fair.c static long effective_load(struct task_group *tg, int cpu, long wl, long wg) · line 4493
Quoted 2 lines
kernel/sched/fair.c +0
-0/* * wl = S * s'_i; see (2) There is a comment 'Per the above, wl is the new *se->load.weight* value'. This should be replaced by *se->avg.load_avg*. Also the function header explains the functionality of
effective_load()based on weight and notsched_avg::load_avg.
I think it is already replaced when effective_load is called.
About load.weight vs. load_avg, see below.
Quoted 20 lines
Quoted 20 lines
kernel/sched/fair.c +2
-2@@ -4695,7 +4508,7 @@ static long effective_load(struct task_group *tg, int cpu, long wl, long wg)/* * wl = dw_i = S * (s'_i - s_i); see (3) */ - wl -= se->load.weight; + wl -= se->avg.load_avg;/* * Recursively apply this logic to all parent groups to compute @@ -4769,14 +4582,14 @@ static int wake_affine(struct sched_domain *sd, struct task_struct *p, int sync)*/ if (sync) { tg =task_group (current); - weight = current->se.load.weight; + weight = current->se.avg.load_avg; this_load +=effective_load (tg, this_cpu, -weight, -weight); load +=effective_load (tg, prev_cpu,0 , -weight); }
Quoted 9 lines
kernel/sched/fair.c static int wake_affine(struct sched_domain *sd, struct task_struct *p, int sync) · line 4592
Quoted 3 lines
kernel/sched/fair.c +1
-1tg =task_group (p); - weight = p->se.load.weight; + weight = p->se.avg.load_avg;You changed
cfs_rq->load.weighttocfs_rq->avg.load_avgandse->load.weighttose->avg.load_avgineffective_load()andwake_affine()in v2.
I wasn't able to find explanation why you did this. I mean we still have to maintain 'structload_weight' on cfs_rq's and se's representing tg's.
Yes, I might not have explained it specifically, but back then, it was simply motivated/reasoned by consistently expressing the load with load_avg.
As of now, it is sort of the same, adding as I previously stated, as far as group SE is concerned, we use load_avg, instread of runnable_load_avg or load.weight.
As was also suggested by Morten, we need to revisit the bulk of the load balancing code a lot, including rethinking about what to use: load.weight, or runnable_load_avg, or load_avg. I think this patch series is just a starter.
Thanks,
Yuyang
tip-bot for Yuyang Du
merge bot[tip:sched/core] sched/fair: Rewrite runnable load and utilization average tracking
In reply to Yuyang Dusched/fair: Rewrite runnable load and utilization average tracking
Commit-ID: 9d89c257dfb9c51a532d69397f6eed75e5168c35 Gitweb: http://git.kernel.org/tip/9d89c257dfb9c51a532d69397f6eed75e5168c35 Author: Yuyang Du <[email protected]> AuthorDate: Wed, 15 Jul 2015 08:04:37 +0800 Committer: Ingo Molnar <[email protected]> CommitDate: Mon, 3 Aug 2015 12:21:29 +0200
sched/fair: Rewrite runnable load and utilization average tracking
The idea of runnable load average (let runnable time contribute to weight) was proposed by Paul Turner and Ben Segall, and it is still followed by this rewrite. This rewrite aims to solve the following issues:
1. cfs_rq's load average (namely runnable_load_avg and blocked_load_avg) is updated at the granularity of an entity at a time, which results in the cfs_rq's load average is stale or partially updated: at any time, only one entity is up to date, all other entities are effectively lagging behind. This is undesirable.
To illustrate, if we have n runnable entities in the cfs_rq, as time elapses, they certainly become outdated:
t0: cfs_rq { e1_old, e2_old, ..., en_old }and when we update:
t1: update e1, then we have cfs_rq { e1_new, e2_old, ..., en_old }
t2: update e2, then we have cfs_rq { e1_old, e2_new, ..., en_old }
...We solve this by combining all runnable entities' load averages together in cfs_rq's avg, and update the cfs_rq's avg as a whole. This is based on the fact that if we regard the update as a function, then:
w * update(e) = update(w * e) and
update(e1) + update(e2) = update(e1 + e2), then
w1 * update(e1) + w2 * update(e2) = update(w1 * e1 + w2 * e2)
therefore, by this rewrite, we have an entirely updated cfs_rq at the time we update it:
t1: update cfs_rq { e1_new, e2_new, ..., en_new }
t2: update cfs_rq { e1_new, e2_new, ..., en_new }
...2. cfs_rq's load average is different between top rq->cfs_rq and other task_group's per CPU cfs_rqs in whether or not blocked_load_average contributes to the load.
The basic idea behind runnable load average (the same for utilization) is that the blocked state is taken into account as opposed to only accounting for the currently runnable state. Therefore, the average should include both the runnable/running and blocked load averages. This rewrite does that.
In addition, we also combine runnable/running and blocked averages of all entities into the cfs_rq's average, and update it together at once. This is based on the fact that:
update(runnable) + update(blocked) = update(runnable + blocked)
This significantly reduces the code as we don't need to separately maintain/update runnable/running load and blocked load.
3. How task_group entities' share is calculated is complex and imprecise.
We reduce the complexity in this rewrite to allow a very simple rule:
the task_group's load_avg is aggregated from its per CPU cfs_rqs's load_avgs. Then group entity's weight is simply proportional to its own cfs_rq's load_avg / task_group's load_avg. To illustrate,
if a task_group has { cfs_rq1, cfs_rq2, ..., cfs_rqn }, then,
task_group_avg = cfs_rq1_avg + cfs_rq2_avg + ... + cfs_rqn_avg, then
cfs_rqx's entity's share = cfs_rqx_avg / task_group_avg * task_group's share
To sum up, this rewrite in principle is equivalent to the current one, but fixes the issues described above. Turns out, it significantly reduces the code complexity and hence increases clarity and efficiency. In addition, the new averages are more smooth/continuous (no spurious spikes and valleys) and updated more consistently and quickly to reflect the load dynamics.
As a result, we have less load tracking overhead, better performance, and especially better power efficiency due to more balanced load.
--- include/linux/sched.h | 41 ++-- kernel/sched/core.c | 3 - kernel/sched/debug.c | 41 ++-- kernel/sched/fair.c | 630 ++++++++++++++++---------------------------------- kernel/sched/sched.h | 28 +-- 5 files changed, 249 insertions(+), 494 deletions(-)
include/linux/sched.h +18 -23
@@ -1175,29 +1175,24 @@ struct load_weight {u32 inv_weight; }; +/* +* The load_avg/util_avg accumulates an infinite geometric series. +* 1) load_avg factors the amount of time that a sched_entity is +* runnable on a rq into its weight. For cfs_rq, it is the aggregated +* such weights of all runnable and blocked sched_entities. +* 2) util_avg factors frequency scaling into the amount of time +* that a sched_entity is running on a CPU, in the range [0..SCHED_LOAD_SCALE]. +* For cfs_rq, it is the aggregated such times of all runnable and +* blocked sched_entities. +* The 64 bit load_sum can: +* 1) for cfs_rq, afford 4353082796 (=2^64/47742/88761) entities with +* the highest weight (=88761) always runnable, we should not overflow +* 2) for entity, support any load.weight always runnable +*/ struct sched_avg { -u64 last_runnable_update; -s64 decay_count; -/* -* utilization_avg_contrib describes the amount of time that a -* sched_entity is running on a CPU. It is based on running_avg_sum -* and is scaled in the range [0..SCHED_LOAD_SCALE]. -* load_avg_contrib described the amount of time that a sched_entity -* is runnable on a rq. It is based on both runnable_avg_sum and the -* weight of the task. -*/ -unsigned long load_avg_contrib, utilization_avg_contrib; -/* -* These sums represent an infinite geometric series and so are bound -* above by 1024/(1-y). Thus we only need a u32 to store them for all -* choices of y < 1-2^(-32)*1024. -* running_avg_sum reflects the time that the sched_entity is -* effectively running on the CPU. -* runnable_avg_sum represents the amount of time a sched_entity is on -* a runqueue which includes the running time that is monitored by -* running_avg_sum. -*/ -u32 runnable_avg_sum, avg_period, running_avg_sum; +u64 last_update_time, load_sum; +u32 util_sum, period_contrib; +unsigned long load_avg, util_avg; };#ifdef CONFIG_SCHEDSTATS
@@ -1263,7 +1258,7 @@ struct sched_entity {#endif #ifdef CONFIG_SMP -/* Per-entity load-tracking */ +/* Per entity load average tracking */ struct sched_avg avg;#endif };
kernel/sched/core.c +0 -3
@@ -2020,9 +2020,6 @@ static void __sched_fork(unsigned long clone_flags, struct task_struct *p) p->se.prev_sum_exec_runtime =0 ; p->se.nr_migrations =0 ; p->se.vruntime =0 ; -#ifdef CONFIG_SMP - p->se.avg.decay_count =0 ; -#endif INIT_LIST_HEAD (&p->se.group_node);#ifdef CONFIG_SCHEDSTATS
kernel/sched/debug.c +17 -24
62 lines not on this page · raw message
kernel/sched/fair.c +205 -425
965 lines not on this page · raw message
kernel/sched/sched.h +9 -19
@@ -245,7 +245,6 @@ struct task_group {#ifdef CONFIG_SMP atomic_long_t load_avg; -atomic_t runnable_avg;#endif #endif
@@ -366,27 +365,18 @@ struct cfs_rq {#ifdef CONFIG_SMP /* -* CFS Load tracking -* Under CFS, load is tracked on a per-entity basis and aggregated up. -* This allows for the description of both thread and group usage (in -* the FAIR_GROUP_SCHED case). -* runnable_load_avg is the sum of the load_avg_contrib of the -* sched_entities on the rq. -* blocked_load_avg is similar to runnable_load_avg except that its -* the blocked sched_entities on the rq. -* utilization_load_avg is the sum of the average running time of the -* sched_entities on the rq. +* CFS load tracking */ -unsigned long runnable_load_avg, blocked_load_avg, utilization_load_avg; -atomic64_t decay_counter; -u64 last_decay; -atomic_long_t removed_load; - +struct sched_avg avg;#ifdef CONFIG_FAIR_GROUP_SCHED -/* Required to track per-cpu representation of a task_group */ -u32 tg_runnable_contrib; -unsigned long tg_load_contrib; +unsigned long tg_load_avg_contrib; +#endif +atomic_long_t removed_load_avg, removed_util_avg; +#ifndef CONFIG_64BIT +u64 load_last_update_time_copy; +#endif +#ifdef CONFIG_FAIR_GROUP_SCHED /* * h_load = weight * f(tg) *
PATCH v10 3/7 sched: Implement update_blocked_averages() for CONFIG_FAIR_GROUP_SCHED=n
In reply to Yuyang DuThe load and the util of idle cpus must be updated periodically in order to decay the blocked part.
If CONFIG_FAIR_GROUP_SCHED is not set, the load and util of idle cpus are not decayed and stay at the values set before becoming idle.
--- kernel/sched/fair.c | 10 ++++++++++ 1 file changed, 10 insertions(+)
kernel/sched/fair.c +10 -0
@@ -5877,6 +5877,16 @@ static unsigned long task_h_load(struct task_struct *p)#else static inline void update_blocked_averages (int cpu) { +struct rq *rq =cpu_rq (cpu); +struct cfs_rq *cfs_rq = &rq->cfs; +unsigned long flags; + +raw_spin_lock_irqsave (&rq->lock, flags); +update_rq_clock (rq); + +update_cfs_rq_load_avg (cfs_rq_clock_task (cfs_rq), cfs_rq); + +raw_spin_unlock_irqrestore (&rq->lock, flags); }static unsigned long task_h_load (struct task_struct *p)
Signature · 1 line
2.1.4
tip-bot for Vincent Guittot
merge bot[tip:sched/core] sched/fair: Implement update_blocked_averages() for CONFIG_FAIR_GROUP_SCHED=n
sched/fair: Implement update_blocked_averages() for CONFIG_FAIR_GROUP_SCHED=n
Commit-ID: 6c1d47c0827304949e0eb9479f4d587f226fac8b Gitweb: http://git.kernel.org/tip/6c1d47c0827304949e0eb9479f4d587f226fac8b Author: Vincent Guittot <[email protected]> AuthorDate: Wed, 15 Jul 2015 08:04:38 +0800 Committer: Ingo Molnar <[email protected]> CommitDate: Mon, 3 Aug 2015 12:24:28 +0200
sched/fair: Implement update_blocked_averages() for CONFIG_FAIR_GROUP_SCHED=n
The load and the utilization of idle CPUs must be updated periodically in order to decay the blocked part.
If CONFIG_FAIR_GROUP_SCHED is not set, the load and util of idle cpus are not decayed and stay at the values set before becoming idle.
--- kernel/sched/fair.c | 8 ++++++++ 1 file changed, 8 insertions(+)
kernel/sched/fair.c +8 -0
@@ -5846,6 +5846,14 @@ static unsigned long task_h_load(struct task_struct *p)#else static inline void update_blocked_averages (int cpu) { +struct rq *rq =cpu_rq (cpu); +struct cfs_rq *cfs_rq = &rq->cfs; +unsigned long flags; + +raw_spin_lock_irqsave (&rq->lock, flags); +update_rq_clock (rq); +update_cfs_rq_load_avg (cfs_rq_clock_task (cfs_rq), cfs_rq); +raw_spin_unlock_irqrestore (&rq->lock, flags); }static unsigned long task_h_load (struct task_struct *p)
PATCH v10 4/7 sched: Init cfs_rq's sched_entity load average
In reply to Yuyang DuThe runnable load and utilization averages of cfs_rq's sched_entity were not initiated. Like done to a task, give new cfs_rq' sched_entity start values to heavy its load in infant time.
--- kernel/sched/core.c | 2 +- kernel/sched/fair.c | 11 ++++++----- kernel/sched/sched.h | 2 +- 3 files changed, 8 insertions(+), 7 deletions(-)
kernel/sched/core.c +1 -1
@@ -2112,7 +2112,7 @@ void wake_up_new_task(struct task_struct *p)#endif /* Initialize new task's runnable average */ -init_task_runnable_average (p); +init_entity_runnable_average (&p->se); rq =__task_rq_lock (p);activate_task (rq, p,0 ); p->on_rq =TASK_ON_RQ_QUEUED ;
kernel/sched/fair.c +6 -5
@@ -672,10 +672,10 @@ static unsigned long task_h_load(struct task_struct *p);#define LOAD_AVG_MAX 47742/* maximum possible load avg */ #define LOAD_AVG_MAX_N 345/* number of full periods to produce LOAD_MAX_AVG */ -/* Give new task start runnable values to heavy its load in infant time */ -void init_task_runnable_average (struct task_struct *p) +/* Give new sched_entity start runnable values to heavy its load in infant time */ +void init_entity_runnable_average (struct sched_entity *se) { -struct sched_avg *sa = &p->se.avg; +struct sched_avg *sa = &se->avg; sa->last_update_time =0 ;/*
@@ -684,14 +684,14 @@ void init_task_runnable_average(struct task_struct *p)* will definitely be update (after enqueue). */ sa->period_contrib =1023 ; - sa->load_avg =scale_load_down (p->se.load.weight); + sa->load_avg =scale_load_down (se->load.weight); sa->load_sum = sa->load_avg *LOAD_AVG_MAX ; sa->util_avg =scale_load_down (SCHED_LOAD_SCALE ); sa->util_sum =LOAD_AVG_MAX ; /* when this task enqueue'ed, it will contribute to its cfs_rq's load_avg */ }#else -void init_task_runnable_average (struct task_struct *p) +void init_entity_runnable_average (struct sched_entity *se) { }#endif
@@ -8065,6 +8065,7 @@ int alloc_fair_sched_group(struct task_group *tg, struct task_group *parent)init_cfs_rq (cfs_rq);init_tg_cfs_entry (tg, cfs_rq, se, i, parent->se[i]); +init_entity_runnable_average (se); }return 1 ;
kernel/sched/sched.h +1 -1
@@ -1293,7 +1293,7 @@ extern void init_dl_task_timer(struct sched_dl_entity *dl_se);unsigned long to_ratio (u64 period,u64 runtime); -extern void init_task_runnable_average (struct task_struct *p); +extern void init_entity_runnable_average (struct sched_entity *se);static inline void add_nr_running (struct rq *rq,unsigned count) {
Signature · 1 line
2.1.4
tip-bot for Yuyang Du
merge bot[tip:sched/core] sched/fair: Init cfs_rq' s sched_entity load average
sched/fair: Init cfs_rq's sched_entity load average
Commit-ID: 540247fb5ddf6d2364f90387fa1f8f428d15e683 Gitweb: http://git.kernel.org/tip/540247fb5ddf6d2364f90387fa1f8f428d15e683 Author: Yuyang Du <[email protected]> AuthorDate: Wed, 15 Jul 2015 08:04:39 +0800 Committer: Ingo Molnar <[email protected]> CommitDate: Mon, 3 Aug 2015 12:24:29 +0200
sched/fair: Init cfs_rq's sched_entity load average
The runnable load and utilization averages of cfs_rq's sched_entity were not initiated. Like done to a task, give new cfs_rq' sched_entity start values to heavy its load in infant time.
--- kernel/sched/core.c | 2 +- kernel/sched/fair.c | 11 ++++++----- kernel/sched/sched.h | 2 +- 3 files changed, 8 insertions(+), 7 deletions(-)
kernel/sched/core.c +1 -1
@@ -2304,7 +2304,7 @@ void wake_up_new_task(struct task_struct *p)#endif /* Initialize new task's runnable average */ -init_task_runnable_average (p); +init_entity_runnable_average (&p->se); rq =__task_rq_lock (p);activate_task (rq, p,0 ); p->on_rq =TASK_ON_RQ_QUEUED ;
kernel/sched/fair.c +6 -5
@@ -667,10 +667,10 @@ static unsigned long task_h_load(struct task_struct *p);#define LOAD_AVG_MAX 47742/* maximum possible load avg */ #define LOAD_AVG_MAX_N 345/* number of full periods to produce LOAD_MAX_AVG */ -/* Give new task start runnable values to heavy its load in infant time */ -void init_task_runnable_average (struct task_struct *p) +/* Give new sched_entity start runnable values to heavy its load in infant time */ +void init_entity_runnable_average (struct sched_entity *se) { -struct sched_avg *sa = &p->se.avg; +struct sched_avg *sa = &se->avg; sa->last_update_time =0 ;/*
@@ -679,14 +679,14 @@ void init_task_runnable_average(struct task_struct *p)* will definitely be update (after enqueue). */ sa->period_contrib =1023 ; - sa->load_avg =scale_load_down (p->se.load.weight); + sa->load_avg =scale_load_down (se->load.weight); sa->load_sum = sa->load_avg *LOAD_AVG_MAX ; sa->util_avg =scale_load_down (SCHED_LOAD_SCALE ); sa->util_sum =LOAD_AVG_MAX ; /* when this task enqueue'ed, it will contribute to its cfs_rq's load_avg */ }#else -void init_task_runnable_average (struct task_struct *p) +void init_entity_runnable_average (struct sched_entity *se) { }#endif
@@ -8029,6 +8029,7 @@ int alloc_fair_sched_group(struct task_group *tg, struct task_group *parent)init_cfs_rq (cfs_rq);init_tg_cfs_entry (tg, cfs_rq, se, i, parent->se[i]); +init_entity_runnable_average (se); }return 1 ;
kernel/sched/sched.h +1 -1
@@ -1307,7 +1307,7 @@ extern void init_dl_task_timer(struct sched_dl_entity *dl_se);unsigned long to_ratio (u64 period,u64 runtime); -extern void init_task_runnable_average (struct task_struct *p); +extern void init_entity_runnable_average (struct sched_entity *se);static inline void add_nr_running (struct rq *rq,unsigned count) {
PATCH v10 5/7 sched: Remove task and group entity load when they are dead
In reply to Yuyang DuWhen task exits or group is destroyed, the entity's load should be removed from its parent cfs_rq's load. Otherwise, it will take time for the parent cfs_rq to decay the dead entity's load to 0, which is not desired.
--- kernel/sched/fair.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)
kernel/sched/fair.c +10 -1
@@ -4919,6 +4919,11 @@ static void migrate_task_rq_fair(struct task_struct *p, int next_cpu)/* We have migrated, no longer consider this task hot */ p->se.exec_start =0 ; } + +static void task_dead_fair (struct task_struct *p) +{ +remove_entity_load_avg (&p->se); +}#endif /* CONFIG_SMP */ static unsigned long
@@ -8027,8 +8032,11 @@ void free_fair_sched_group(struct task_group *tg)for_each_possible_cpu (i) {if (tg->cfs_rq)kfree (tg->cfs_rq[i]); -if (tg->se) +if (tg->se) { +if (tg->se[i]) +remove_entity_load_avg (tg->se[i]);kfree (tg->se[i]); + } }kfree (tg->cfs_rq);
@@ -8215,6 +8223,7 @@ const struct sched_class fair_sched_class = { .rq_offline = rq_offline_fair, .task_waking = task_waking_fair, + .task_dead = task_dead_fair, #endif .set_curr_task = set_curr_task_fair,
Signature · 1 line
2.1.4
tip-bot for Yuyang Du
merge bot[tip:sched/core] sched/fair: Remove task and group entity load when they are dead
sched/fair: Remove task and group entity load when they are dead
Commit-ID: 1269557889b477e3e43ab99a21035ddf8f7cea4d Gitweb: http://git.kernel.org/tip/1269557889b477e3e43ab99a21035ddf8f7cea4d Author: Yuyang Du <[email protected]> AuthorDate: Wed, 15 Jul 2015 08:04:40 +0800 Committer: Ingo Molnar <[email protected]> CommitDate: Mon, 3 Aug 2015 12:24:30 +0200
sched/fair: Remove task and group entity load when they are dead
When task exits or group is destroyed, the entity's load should be removed from its parent cfs_rq's load. Otherwise, it will take time for the parent cfs_rq to decay the dead entity's load to 0, which is not desired.
--- kernel/sched/fair.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)
kernel/sched/fair.c +10 -1
@@ -4913,6 +4913,11 @@ static void migrate_task_rq_fair(struct task_struct *p, int next_cpu)/* We have migrated, no longer consider this task hot */ p->se.exec_start =0 ; } + +static void task_dead_fair (struct task_struct *p) +{ +remove_entity_load_avg (&p->se); +}#endif /* CONFIG_SMP */ static unsigned long
@@ -7991,8 +7996,11 @@ void free_fair_sched_group(struct task_group *tg)for_each_possible_cpu (i) {if (tg->cfs_rq)kfree (tg->cfs_rq[i]); -if (tg->se) +if (tg->se) { +if (tg->se[i]) +remove_entity_load_avg (tg->se[i]);kfree (tg->se[i]); + } }kfree (tg->cfs_rq);
@@ -8179,6 +8187,7 @@ const struct sched_class fair_sched_class = { .rq_offline = rq_offline_fair, .task_waking = task_waking_fair, + .task_dead = task_dead_fair, #endif .set_curr_task = set_curr_task_fair,
PATCH v10 6/7 sched: Provide runnable_load_avg back to cfs_rq
In reply to Yuyang DuThe cfs_rq's load_avg is composed of runnable_load_avg and blocked_load_avg. Before this series, sometimes the runnable_load_avg is used, and sometimes the load_avg is used. Completely replacing all uses of runnable_load_avg with load_avg may be too big a leap, i.e., the blocked_load_avg is concerned to result in overrated load. Therefore, we get runnable_load_avg back.
The new cfs_rq's runnable_load_avg is improved to be updated with all of the runnable sched_eneities at the same time, so the one sched_entity updated and the others stale problem is solved.
--- kernel/sched/debug.c | 2 ++ kernel/sched/fair.c | 57 +++++++++++++++++++++++++++++++++++++++++----------- kernel/sched/sched.h | 2 ++ 3 files changed, 49 insertions(+), 12 deletions(-)
kernel/sched/debug.c +2 -0
@@ -205,6 +205,8 @@ void print_cfs_rq(struct seq_file *m, int cpu, struct cfs_rq *cfs_rq)#ifdef CONFIG_SMP SEQ_printf (m," .%-30s: %lu\n" ,"load_avg" , cfs_rq->avg.load_avg); +SEQ_printf (m," .%-30s: %lu\n" ,"runnable_load_avg" , + cfs_rq->runnable_load_avg);SEQ_printf (m," .%-30s: %lu\n" ,"util_avg" , cfs_rq->avg.util_avg); SEQ_printf(m, " .%-30s: %ld\n", "removed_load_avg",
kernel/sched/fair.c +45 -12
145 lines not on this page · raw message
kernel/sched/sched.h +2 -0
@@ -368,6 +368,8 @@ struct cfs_rq {* CFS load tracking */ struct sched_avg avg; +u64 runnable_load_sum; +unsigned long runnable_load_avg;#ifdef CONFIG_FAIR_GROUP_SCHED unsigned long tg_load_avg_contrib;#endif
Signature · 1 line
2.1.4
Re: [PATCH v10 6/7] sched: Provide runnable_load_avg back to cfs_rq
Hi Yuyang,
Quoted 10 lines · Yuyang Du
The cfs_rq's
load_avgis composed ofrunnable_load_avgandblocked_load_avg. Before this series, sometimes therunnable_load_avgis used, and sometimes theload_avgis used. Completely replacing all uses ofrunnable_load_avgwithload_avgmay be too big a leap, i.e., theblocked_load_avgis concerned to result in overrated load. Therefore, we getrunnable_load_avgback.The new cfs_rq's
runnable_load_avgis improved to be updated with all of the runnablesched_eneitiesat the same time, so the onesched_entityupdated and the others stale problem is solved.
How about tracking cfs_rq's blocked_load_avg instead of runnable_load_avg, because, AFAICS:
cfs_rq->runnable_load_avg = se->avg.load_avg - cfs_rq->blocked_load_avg.
se is the corresponding sched_entity of cfs_rq. And when we need the runnable_load_avg, we just calculate by the expression above.
This can be thought as a lazy way to update runnable_load_avg, and we don't need to modify __update_load_avg any more.
Regards,
Boqun
Re: [PATCH v10 6/7] sched: Provide runnable_load_avg back to cfs_rq
Quoted 14 lines · Boqun Feng
Hi Yuyang,
Quoted 10 lines · Yuyang Du
The cfs_rq's
load_avgis composed ofrunnable_load_avgandblocked_load_avg. Before this series, sometimes therunnable_load_avgis used, and sometimes theload_avgis used. Completely replacing all uses ofrunnable_load_avgwithload_avgmay be too big a leap, i.e., theblocked_load_avgis concerned to result in overrated load. Therefore, we getrunnable_load_avgback.The new cfs_rq's
runnable_load_avgis improved to be updated with all of the runnablesched_eneitiesat the same time, so the onesched_entityupdated and the others stale problem is solved.
Quoted 4 lines
How about tracking cfs_rq's
blocked_load_avginstead ofrunnable_load_avg, because, AFAICS:
cfs_rq->runnable_load_avg=se->avg.load_avg-cfs_rq->blocked_load_avg.
No, cfs_rq->runnable_load_avg = cfs_rq->avg.load_avg - cfs_rq->blocked_load_avg, without rounding errors and the like.
Quoted 5 lines
se is the corresponding
sched_entityofcfs_rq. And when we need therunnable_load_avg, we just calculate by the expression above.This can be thought as a lazy way to update
runnable_load_avg, and we don't need to modify__update_load_avgany more.
Not lazy at all, but adding (as of now) useless blocked_load_avg and an extra subtraction.
Or did you forget blocked_load_avg also needs to be updated/decayed as time elapses?
Re: [PATCH v10 6/7] sched: Provide runnable_load_avg back to cfs_rq
Quoted 15 lines · Yuyang Du
Quoted 14 lines · Boqun Feng
Hi Yuyang,
Quoted 10 lines · Yuyang Du
The cfs_rq's
load_avgis composed ofrunnable_load_avgandblocked_load_avg. Before this series, sometimes therunnable_load_avgis used, and sometimes theload_avgis used. Completely replacing all uses ofrunnable_load_avgwithload_avgmay be too big a leap, i.e., theblocked_load_avgis concerned to result in overrated load. Therefore, we getrunnable_load_avgback.The new cfs_rq's
runnable_load_avgis improved to be updated with all of the runnablesched_eneitiesat the same time, so the onesched_entityupdated and the others stale problem is solved.
Quoted 8 lines
Quoted 4 lines
How about tracking cfs_rq's
blocked_load_avginstead ofrunnable_load_avg, because, AFAICS:
cfs_rq->runnable_load_avg=se->avg.load_avg-cfs_rq->blocked_load_avg.No,
cfs_rq->runnable_load_avg=cfs_rq->avg.load_avg-cfs_rq->blocked_load_avg, without rounding errors and the like.
Oh, sorry.. yeah, you're right here.
Quoted 8 lines
Quoted 5 lines
se is the corresponding
sched_entityofcfs_rq. And when we need therunnable_load_avg, we just calculate by the expression above.This can be thought as a lazy way to update
runnable_load_avg, and we don't need to modify__update_load_avgany more.Not lazy at all, but adding (as of now) useless
blocked_load_avgand an extra subtraction.
but we can remove runnable_load_avg tracking code in __update_load_avg, as you do in this patch, right?
Quoted 2 lines
Or did you forget
blocked_load_avgalso needs to be updated/decayed as time elapses?
I know we need to update or decay the blocked_load_avg, but we only need to update and decay when 1) entity dequeued/enqueued 2) entity migrated or 3) we need the runnable_load_avg value calcuated by blocked_load_avg, right?
These are more rare than __update_load_avg called, right?
Regards,
Boqun
Re: [PATCH v10 6/7] sched: Provide runnable_load_avg back to cfs_rq
Quoted 19 lines · Boqun Feng
Quoted 18 lines · Yuyang Du
Quoted 17 lines · Boqun Feng
Hi Yuyang,
Quoted 10 lines · Yuyang Du
The cfs_rq's
load_avgis composed ofrunnable_load_avgandblocked_load_avg. Before this series, sometimes therunnable_load_avgis used, and sometimes theload_avgis used. Completely replacing all uses ofrunnable_load_avgwithload_avgmay be too big a leap, i.e., theblocked_load_avgis concerned to result in overrated load. Therefore, we getrunnable_load_avgback.The new cfs_rq's
runnable_load_avgis improved to be updated with all of the runnablesched_eneitiesat the same time, so the onesched_entityupdated and the others stale problem is solved.How about tracking cfs_rq's
blocked_load_avginstead ofrunnable_load_avg, because, AFAICS:
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
cfs_rq->runnable_load_avg=se->avg.load_avg-cfs_rq->blocked_load_avg.No,
cfs_rq->runnable_load_avg=cfs_rq->avg.load_avg-cfs_rq->blocked_load_avg, without rounding errors and the like.Oh, sorry.. yeah, you're right here.
The point is that you have already tracked the sum of runnable_load_avg and blocked_load_avg in cfs_rq->avg.load_avg. If you're going to track part of the sum, you'd better track the one that's updated less frequently, right?
Anyway, this idea just comes into my mind. I wonder which is udpated less frequently myself too. ;-) So I ask to see whether there is something we can improve.
Regards,
Boqun
Re: [PATCH v10 6/7] sched: Provide runnable_load_avg back to cfs_rq
Hi Boqun,
Quoted 8 lines · Boqun Feng
The point is that you have already tracked the sum of
runnable_load_avgandblocked_load_avgincfs_rq->avg.load_avg. If you're going to track part of the sum, you'd better track the one that's updated less frequently, right?Anyway, this idea just comes into my mind. I wonder which is udpated less frequently myself too. ;-) So I ask to see whether there is something we can improve.
Actually, this is not the point.
1) blocked load is more "difficult" to track, hint, migrate.
2) r(t1) - b(t2) is not anything, hint, t1 != t2
Re: [PATCH v10 6/7] sched: Provide runnable_load_avg back to cfs_rq
Hi Yuyang,
Quoted 12 lines · Yuyang Du
Hi Boqun,
Quoted 8 lines · Boqun Feng
The point is that you have already tracked the sum of
runnable_load_avgandblocked_load_avgincfs_rq->avg.load_avg. If you're going to track part of the sum, you'd better track the one that's updated less frequently, right?Anyway, this idea just comes into my mind. I wonder which is udpated less frequently myself too. ;-) So I ask to see whether there is something we can improve.
Quoted 5 lines
Actually, this is not the point.
1) blocked load is more "difficult" to track, hint, migrate.
2) r(t1) - b(t2) is not anything, hint, t1 != t2
Please consider this patch below, which is not tested yet, just for discussion. This patch is based on 1-5 in your patchset and going to replace patch 6. Hope this could make my point clear.
Thanks anyway for being patient with me ;-)
Regards,
Boqun
========================================================================
Subject: [PATCH] sched: lazy blocked load tracking
With this patch, cfs_rq_runnable_load_avg can be implemented as follow:
static inline unsigned long cfs_rq_runnable_load_avg(struct cfs_rq *cfs_rq)
{
u64 now = cfs_rq_clock_task(cfs_rq);
decay_cfs_rq_blocked_load(now, cfs_rq);
return max_t(long, cfs_rq->avg.load_avg - cfs_rq->blocked_load_avg, 0);
}--- kernel/sched/fair.c | 41 +++++++++++++++++++++++++++++++++++++++++ kernel/sched/sched.h | 4 ++++ 2 files changed, 45 insertions(+)
kernel/sched/fair.c +41 -0
@@ -2625,6 +2625,20 @@ static __always_inline int __update_load_avg(u64 now, int cpu,return decayed; } +static inline u64 decay_cfs_rq_blocked_load (u64 now,struct cfs_rq *cfs_rq) +{ +u64 decays; + + now = now >>20 ; + decays = now - cfs_rq->last_blocked_load_decays; + + cfs_rq->blocked_load_sum =decay_load (cfs_rq->blocked_load_sum, decays); + cfs_rq->blocked_load_avg =div_u64 (cfs->blocked_load_sum,LOAD_AVG_MAX ); + cfs_rq->last_blocked_load_update_time = now; + +return decays; +} +#ifdef CONFIG_FAIR_GROUP_SCHED /* * Updating tg's load_avg is necessary before update_cfs_share (which is done)
@@ -2656,6 +2670,12 @@ static inline int update_cfs_rq_load_avg(u64 now, struct cfs_rq *cfs_rq)long r =atomic_long_xchg (&cfs_rq->removed_load_avg,0 ); sa->load_avg =max_t (long, sa->load_avg - r,0 ); sa->load_sum =max_t (s64, sa->load_sum - r *LOAD_AVG_MAX ,0 ); + +decay_cfs_rq_blocked_load (sa->last_update_time, cfs_rq); + cfs_rq->blocked_load_avg =max_t (long, + cfs_rq->blocked_load_avg - r,0 ); + cfs_rq->blocked_load_sum =max_t (s64, + cfs_rq->blocked_load_avg - r *LOAD_AVG_MAX ,0 ); }if (atomic_long_read (&cfs_rq->removed_util_avg)) {
@@ -2719,11 +2739,32 @@ enqueue_entity_load_avg(struct cfs_rq *cfs_rq, struct sched_entity *se) cfs_rq->avg.util_avg += sa->util_avg; cfs_rq->avg.util_sum += sa->util_sum; } +else { +decay_cfs_rq_blocked_load (now, cfs_rq); + + cfs_rq->blocked_load_avg =max_t (long, + cfs_rq->blocked_load_avg - sa->load_avg,0 ); + cfs_rq->blocked_load_sum =max_t (long, + cfs_rq->blocked_load_sum - sa->load_sum,0 ); + }if (decayed || migrated)update_tg_load_avg (cfs_rq,0 ); } +static inline void +dequeue_entity_load_avg (struct cfs_rq *cfs_rq,struct sched_entity *se) +{ +u64 now =cfs_rq_clock_task (cfs_rq); + +update_load_avg (se,1 ); +update_cfs_rq_load_avg (now, cfs_rq); +decay_cfs_rq_blocked_load (now, cfs_rq); + + cfs_rq->blocked_load_sum += se->avg.load_sum; + cfs_rq->blocked_load_avg += se->avg.load_avg; +} +/* * Task first catches up with cfs_rq, and then subtract * itself from the cfs_rq (task must be off the queue now).
kernel/sched/sched.h +4 -0
@@ -368,6 +368,10 @@ struct cfs_rq {* CFS load tracking */ struct sched_avg avg; + +u64 last_blocked_load_decays; +u64 blocked_load_sum; +unsigned long blocked_load_avg;#ifdef CONFIG_FAIR_GROUP_SCHED unsigned long tg_load_avg_contrib;#endif
Signature · 1 line
2.4.6
Re: [PATCH v10 6/7] sched: Provide runnable_load_avg back to cfs_rq
Quoted 57 lines · Boqun Feng
Hi Yuyang,
Quoted 12 lines · Yuyang Du
Hi Boqun,
Quoted 8 lines · Boqun Feng
The point is that you have already tracked the sum of
runnable_load_avgandblocked_load_avgincfs_rq->avg.load_avg. If you're going to track part of the sum, you'd better track the one that's updated less frequently, right?Anyway, this idea just comes into my mind. I wonder which is udpated less frequently myself too. ;-) So I ask to see whether there is something we can improve.
Quoted 5 lines
Actually, this is not the point.
1) blocked load is more "difficult" to track, hint, migrate.
2) r(t1) - b(t2) is not anything, hint, t1 != t2
Please consider this patch below, which is not tested yet, just for discussion. This patch is based on 1-5 in your patchset and going to replace patch 6. Hope this could make my point clear.
Thanks anyway for being patient with me ;-)
Regards,
Boqun========================================================================
Subject: [PATCH] sched: lazy blocked load tracking
With this patch,
cfs_rq_runnable_load_avgcan be implemented as follow:static inline unsigned long
cfs_rq_runnable_load_avg(struct cfs_rq *cfs_rq){ u64 now = cfs_rq_clock_task(cfs_rq); decay_cfs_rq_blocked_load(now, cfs_rq); return max_t(long, cfs_rq->avg.load_avg - cfs_rq->blocked_load_avg, 0); }--- kernel/sched/fair.c | 41 +++++++++++++++++++++++++++++++++++++++++ kernel/sched/sched.h | 4 ++++ 2 files changed, 45 insertions(+)kernel/sched/fair.c +0
-0@@ -2625,6 +2625,20 @@ static __always_inline int __update_load_avg(u64 now, int cpu,return decayed; }
kernel/sched/fair.c static __always_inline int __update_load_avg(u64 now, int cpu, · line 2637
Quoted 10 lines
kernel/sched/fair.c +10
-0+static inline u64 decay_cfs_rq_blocked_load (u64 now,struct cfs_rq *cfs_rq) +{ +u64 decays; + + now = now >>20 ; + decays = now - cfs_rq->last_blocked_load_decays; + + cfs_rq->blocked_load_sum =decay_load (cfs_rq->blocked_load_sum, decays); + cfs_rq->blocked_load_avg =div_u64 (cfs->blocked_load_sum,LOAD_AVG_MAX ); + cfs_rq->last_blocked_load_update_time = now;
Sorry for the typo, should be last_blocked_load_decays here ;-)
Regards,
Boqun
Re: [PATCH v10 6/7] sched: Provide runnable_load_avg back to cfs_rq
In reply to Boqun FengQuoted 17 lines · Boqun Feng
Hi Yuyang,
Quoted 14 lines · Yuyang Du
Hi Boqun,
Quoted 8 lines · Boqun Feng
The point is that you have already tracked the sum of
runnable_load_avgandblocked_load_avgincfs_rq->avg.load_avg. If you're going to track part of the sum, you'd better track the one that's updated less frequently, right?Anyway, this idea just comes into my mind. I wonder which is udpated less frequently myself too. ;-) So I ask to see whether there is something we can improve.
Actually, this is not the point.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Quoted 1 line
Quoted 1 line
1) blocked load is more "difficult" to track, hint, migrate.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Quoted 7 lines
Quoted 1 line
2) r(t1) - b(t2) is not anything, hint, t1 != t2
Please consider this patch below, which is not tested yet, just for discussion. This patch is based on 1-5 in your patchset and going to replace patch 6. Hope this could make my point clear.
Thanks anyway for being patient with me ;-)
Re: [PATCH v10 6/7] sched: Provide runnable_load_avg back to cfs_rq
Hi Yuyang,
Quoted 16 lines · Yuyang Du
Quoted 15 lines · Boqun Feng
Hi Yuyang,
Quoted 12 lines · Yuyang Du
Hi Boqun,
Quoted 8 lines · Boqun Feng
The point is that you have already tracked the sum of
runnable_load_avgandblocked_load_avgincfs_rq->avg.load_avg. If you're going to track part of the sum, you'd better track the one that's updated less frequently, right?Anyway, this idea just comes into my mind. I wonder which is udpated less frequently myself too. ;-) So I ask to see whether there is something we can improve.
Quoted 6 lines
Quoted 2 lines
Quoted 2 lines
Actually, this is not the point.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Quoted 1 line
Quoted 1 line
1) blocked load is more "difficult" to track, hint, migrate.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
I may not get your point here? Are you saying my patch fails to handle the migration or are you just telling me that blocked load tracking need to take migration into consideration?
If it's the latter one, I want to say that, with blocked load or not, we have to handle load_avg in migrations, so *adding* some code to handle blocked load is not a big deal.
Please consider this piece of code in update_cfs_rq_load_avg(), which decays and updates blocked_load_avg.
@@ -2656,6 +2670,12 @@ static inline int update_cfs_rq_load_avg(u64 now, struct cfs_rq *cfs_rq)
long r = atomic_long_xchg(&cfs_rq->removed_load_avg, 0); sa->load_avg = max_t(long, sa->load_avg - r, 0);
sa->load_sum = max_t(s64, sa->load_sum - r * LOAD_AVG_MAX, 0);
+
+ decay_cfs_rq_blocked_load(sa->last_update_time, cfs_rq);
+ cfs_rq->blocked_load_avg = max_t(long,
+ cfs_rq->blocked_load_avg - r, 0);
+ cfs_rq->blocked_load_sum = max_t(s64,
+ cfs_rq->blocked_load_avg - r * LOAD_AVG_MAX, 0);
}
Regards,
Boqun
Re: [PATCH v10 6/7] sched: Provide runnable_load_avg back to cfs_rq
Quoted 7 lines · Boqun Feng
Quoted 3 lines
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Quoted 1 line
Quoted 1 line
1) blocked load is more "difficult" to track, hint, migrate.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
I may not get your point here? Are you saying my patch fails to handle the migration or are you just telling me that blocked load tracking need to take migration into consideration?
Both, is it so difficult to get?
Quoted 6 lines
If it's the latter one, I want to say that, with blocked load or not, we have to handle
load_avgin migrations, so *adding* some code to handle blocked load is not a big deal.Please consider this piece of code in
update_cfs_rq_load_avg(), which decays and updatesblocked_load_avg.
At this point of time, you tell me why exactly you want to track the blocked?
Re: [PATCH v10 6/7] sched: Provide runnable_load_avg back to cfs_rq
Hi Yuyang,
Quoted 11 lines · Yuyang Du
Quoted 7 lines · Boqun Feng
Quoted 3 lines
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Quoted 1 line
Quoted 1 line
1) blocked load is more "difficult" to track, hint, migrate.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
I may not get your point here? Are you saying my patch fails to handle the migration or are you just telling me that blocked load tracking need to take migration into consideration?
Both, is it so difficult to get?
Hmm.. I will appreciate more if you comment on my patch to point out where is wrong ;-)
Quoted 8 lines
Quoted 6 lines
If it's the latter one, I want to say that, with blocked load or not, we have to handle
load_avgin migrations, so *adding* some code to handle blocked load is not a big deal.Please consider this piece of code in
update_cfs_rq_load_avg(), which decays and updatesblocked_load_avg.At this point of time, you tell me why exactly you want to track the blocked?
I want to track the blocked load because you want to track runnable load in your patch, and as you've already tracked load_avg, which is the sum of blocked load and runnable load, so I wonder whether tracking blocked load is *another* way to track runnable load. Because if tracking blocked load costs less than tracking runnable load, it's of course better to track blocked load and calculate runnable load on demand rather than track runnable load directly.
Yes, I do need to decay and update blocked_load_avg in update_cfs_rq_load_avg() if there is a *non-zero* value of remove_load_avg, but:
1. if no entity is migrated and no entity is dequeued(blocked), I
need to do nothing but tracking runnable load directly still needs to update runnable load, for example, in entity_tick().
2. if no entity is migrated and a entity is dequeued(blocked), what
I need to do is similar as tracking runnable load directly does.
and of course:
3. if a entity is migrated, I do need to do more than tracking
runnable load directly.
So,
For #1 situations, tracking blocked load wins
For #2 situations, tie
For #3 situations, tracking runnable load wins
And which are more rare in the system, #1 or #3?
I write that patch to see how much we need to track blocked load *instead* of runnable load, and I don't see that costs a lot. So I basically want to know, is my patch of tracking blocked wrong? If not, does that cost more than or nearly equal to tracking runnable load directly? If not, why not track blocked load instead of tracking runnable load?
However, I admit all the questions should be answered by real benchmarks. I just want to see whether you have thought the same questions as I and could give me a quick answer.
But I think an simple and directly answer is that "because we need runnable load so we track it", code simplicity wins! if your current answer is that, I'm OK with it, and will do some benchmark myself to see whether it's worth to track blocked load rather than runnable load directly. Again, I still hope a quick and convinced answer from you. Thank you ;-)
Regards,
Boqun
Re: [PATCH v10 6/7] sched: Provide runnable_load_avg back to cfs_rq
In reply to Yuyang DuQuoted 12 lines · Yuyang Du
The cfs_rq's
load_avgis composed ofrunnable_load_avgandblocked_load_avg. Before this series, sometimes therunnable_load_avgis used, and sometimes theload_avgis used. Completely replacing all uses ofrunnable_load_avgwithload_avgmay be too big a leap, i.e., theblocked_load_avgis concerned to result in overrated load. Therefore, we getrunnable_load_avgback.The new cfs_rq's
runnable_load_avgis improved to be updated with all of the runnablesched_eneitiesat the same time, so the onesched_entityupdated and the others stale problem is solved.Signed-off-by Yuyang Du <[email protected]>---
<snip>
kernel/sched/fair.c enqueue_entity_load_avg(struct cfs_rq *cfs_rq, struct sched_entity *se) · line 2751
Quoted 6 lines
kernel/sched/fair.c +6
-0+/* Remove the runnable load generated by se from cfs_rq's runnable load average */ +static inline void +dequeue_entity_load_avg (struct cfs_rq *cfs_rq,struct sched_entity *se) +{ +update_load_avg (se,1 ); +
I think we need an update_cfs_rq_load_avg() here? Because the runnable_load_avg may not be up to date when dequeue_entity_load_avg() is called, right?
Quoted 9 lines
kernel/sched/fair.c +6
-0+ cfs_rq->runnable_load_avg = +max_t (long, cfs_rq->runnable_load_avg - se->avg.load_avg,0 ); + cfs_rq->runnable_load_sum = +max_t (s64, cfs_rq->runnable_load_sum - se->avg.load_sum,0 ); +} +/* * Task first catches up with cfs_rq, and then subtract * itself from the cfs_rq (task must be off the queue now).
<snip>
kernel/sched/fair.c dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags) · line 3021
Quoted 9 lines
kernel/sched/fair.c +1
-1@@ -2982,7 +3015,7 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)* Update run-time statistics of the 'current'. */ update_curr (cfs_rq); -update_load_avg (se,1 ); +dequeue_entity_load_avg (cfs_rq, se);update_stats_dequeue (cfs_rq, se);if (flags &DEQUEUE_SLEEP ) {
Thanks and Best Regards,
Boqun
Re: [PATCH v10 6/7] sched: Provide runnable_load_avg back to cfs_rq
Quoted 11 lines · Boqun Feng
kernel/sched/fair.c enqueue_entity_load_avg(struct cfs_rq *cfs_rq, struct sched_entity *se) · line 2751
Quoted 7 lines
kernel/sched/fair.c +6
-0+/* Remove the runnable load generated by se from cfs_rq's runnable load average */ +static inline void +dequeue_entity_load_avg (struct cfs_rq *cfs_rq,struct sched_entity *se) +{ +update_load_avg (se,1 ); +I think we need an
update_cfs_rq_load_avg()here? Because therunnable_load_avgmay not be up to date whendequeue_entity_load_avg()is called, right?
Not in update_load_avg()?
tip-bot for Yuyang Du
merge bot[tip:sched/core] sched/fair: Provide runnable_load_avg back to cfs_rq
In reply to Yuyang Dusched/fair: Provide runnable_load_avg back to cfs_rq
Commit-ID: 139622343ef31941effc6de6a5a9320371a00e62 Gitweb: http://git.kernel.org/tip/139622343ef31941effc6de6a5a9320371a00e62 Author: Yuyang Du <[email protected]> AuthorDate: Wed, 15 Jul 2015 08:04:41 +0800 Committer: Ingo Molnar <[email protected]> CommitDate: Mon, 3 Aug 2015 12:24:31 +0200
sched/fair: Provide runnable_load_avg back to cfs_rq
The cfs_rq's load_avg is composed of runnable_load_avg and blocked_load_avg. Before this series, sometimes the runnable_load_avg is used, and sometimes the load_avg is used. Completely replacing all uses of runnable_load_avg with load_avg may be too big a leap, i.e., the blocked_load_avg is concerned to result in overrated load. Therefore, we get runnable_load_avg back.
The new cfs_rq's runnable_load_avg is improved to be updated with all of the runnable sched_eneities at the same time, so the one sched_entity updated and the others stale problem is solved.
--- kernel/sched/debug.c | 2 ++ kernel/sched/fair.c | 55 ++++++++++++++++++++++++++++++++++++++++++---------- kernel/sched/sched.h | 2 ++ 3 files changed, 49 insertions(+), 10 deletions(-)
kernel/sched/debug.c +2 -0
@@ -207,6 +207,8 @@ void print_cfs_rq(struct seq_file *m, int cpu, struct cfs_rq *cfs_rq)#ifdef CONFIG_SMP SEQ_printf (m," .%-30s: %lu\n" ,"load_avg" , cfs_rq->avg.load_avg); +SEQ_printf (m," .%-30s: %lu\n" ,"runnable_load_avg" , + cfs_rq->runnable_load_avg);SEQ_printf (m," .%-30s: %lu\n" ,"util_avg" , cfs_rq->avg.util_avg); SEQ_printf(m, " .%-30s: %ld\n", "removed_load_avg",
kernel/sched/fair.c +45 -10
143 lines not on this page · raw message
kernel/sched/sched.h +2 -0
@@ -368,6 +368,8 @@ struct cfs_rq {* CFS load tracking */ struct sched_avg avg; +u64 runnable_load_sum; +unsigned long runnable_load_avg;#ifdef CONFIG_FAIR_GROUP_SCHED unsigned long tg_load_avg_contrib;#endif
PATCH v10 7/7 sched: Clean up load average references
In reply to Yuyang DuFor cfs_rq, we have load.weight, runnable_load_avg, and load_avg. We now start to clean up how they are used.
First, as group sched_entity already largely uses load_avg, we now expand to use load_avg in all cases. Second, for CPU-wide load balancing, we choose to use runnable_load_avg in all cases, which is the same as before this series.
--- kernel/sched/fair.c | 44 +++++++++++++++++++++++++++++--------------- 1 file changed, 29 insertions(+), 15 deletions(-)
kernel/sched/fair.c +29 -15
@@ -690,6 +690,9 @@ void init_entity_runnable_average(struct sched_entity *se) sa->util_sum =LOAD_AVG_MAX ; /* when this task enqueue'ed, it will contribute to its cfs_rq's load_avg */ } + +static inline unsigned long cfs_rq_runnable_load_avg (struct cfs_rq *cfs_rq); +static inline unsigned long cfs_rq_load_avg (struct cfs_rq *cfs_rq);#else void init_entity_runnable_average (struct sched_entity *se) {
@@ -2364,7 +2367,7 @@ static inline long calc_tg_weight(struct task_group *tg, struct cfs_rq *cfs_rq)*/ tg_weight =atomic_long_read (&tg->load_avg); tg_weight -= cfs_rq->tg_load_avg_contrib; - tg_weight += cfs_rq->avg.load_avg; + tg_weight +=cfs_rq_load_avg (cfs_rq);return tg_weight; }
@@ -2374,7 +2377,7 @@ static long calc_cfs_shares(struct cfs_rq *cfs_rq, struct task_group *tg)long tg_weight, load, shares; tg_weight =calc_tg_weight (tg, cfs_rq); - load = cfs_rq->avg.load_avg; + load =cfs_rq_load_avg (cfs_rq); shares = (tg->shares * load);if (tg_weight)
@@ -2799,6 +2802,16 @@ void idle_exit_fair(struct rq *this_rq) { } +static inline unsigned long cfs_rq_runnable_load_avg (struct cfs_rq *cfs_rq) +{ +return cfs_rq->runnable_load_avg; +} + +static inline unsigned long cfs_rq_load_avg (struct cfs_rq *cfs_rq) +{ +return cfs_rq->avg.load_avg; +} +static int idle_balance (struct rq *this_rq);#else /* CONFIG_SMP */
@@ -4273,6 +4286,12 @@ static void __update_cpu_load(struct rq *this_rq, unsigned long this_load,sched_avg_update (this_rq); } +/* Used instead of source_load when we know the type == 0 */ +static unsigned long weighted_cpuload (const int cpu) +{ +return cfs_rq_runnable_load_avg (&cpu_rq (cpu)->cfs); +} +#ifdef CONFIG_NO_HZ_COMMON /* * There is no sane way to deal with nohz on smp when using jiffies because the
@@ -4294,7 +4313,7 @@ static void __update_cpu_load(struct rq *this_rq, unsigned long this_load,static void update_idle_cpu_load (struct rq *this_rq) {unsigned long curr_jiffies =READ_ONCE (jiffies); -unsigned long load = this_rq->cfs.avg.load_avg; +unsigned long load =weighted_cpuload (cpu_of (this_rq));unsigned long pending_updates;/*
@@ -4340,7 +4359,7 @@ void update_cpu_load_nohz(void)*/ void update_cpu_load_active (struct rq *this_rq) { -unsigned long load = this_rq->cfs.avg.load_avg; +unsigned long load =weighted_cpuload (cpu_of (this_rq));/* * See the mess around update_idle_cpu_load() / update_cpu_load_nohz(). */
@@ -4348,12 +4367,6 @@ void update_cpu_load_active(struct rq *this_rq)__update_cpu_load (this_rq, load,1 ); } -/* Used instead of source_load when we know the type == 0 */ -static unsigned long weighted_cpuload (const int cpu) -{ -return cpu_rq (cpu)->cfs.avg.load_avg; -} -/* * Return a low guess at the load of a migration-source cpu weighted * according to the scheduling class and "nice" value.
@@ -4401,7 +4414,7 @@ static unsigned long cpu_avg_load_per_task(int cpu) {struct rq *rq =cpu_rq (cpu);unsigned long nr_running =READ_ONCE (rq->cfs.h_nr_running); -unsigned long load_avg = rq->cfs.avg.load_avg; +unsigned long load_avg =weighted_cpuload (cpu);if (nr_running)return load_avg / nr_running;
@@ -4520,7 +4533,7 @@ static long effective_load(struct task_group *tg, int cpu, long wl, long wg)/* * w = rw_i + @wl */ - w = se->my_q->avg.load_avg + wl; + w =cfs_rq_load_avg (se->my_q) + wl;/* * wl = S * s'_i; see (2)
@@ -5891,13 +5904,14 @@ static void update_cfs_rq_h_load(struct cfs_rq *cfs_rq) }if (!se) { - cfs_rq->h_load = cfs_rq->avg.load_avg; + cfs_rq->h_load =cfs_rq_load_avg (cfs_rq); cfs_rq->last_h_load_update = now; }while ((se = cfs_rq->h_load_next) !=NULL ) { load = cfs_rq->h_load; - load =div64_ul (load * se->avg.load_avg, cfs_rq->avg.load_avg +1 ); + load =div64_ul (load * se->avg.load_avg, +cfs_rq_load_avg (cfs_rq) +1 ); cfs_rq =group_cfs_rq (se); cfs_rq->h_load = load; cfs_rq->last_h_load_update = now;
@@ -5910,7 +5924,7 @@ static unsigned long task_h_load(struct task_struct *p)update_cfs_rq_h_load (cfs_rq);return div64_ul (p->se.avg.load_avg * cfs_rq->h_load, - cfs_rq->avg.load_avg + 1); + cfs_rq_load_avg(cfs_rq) + 1); }#else static inline void update_blocked_averages (int cpu)
Signature · 1 line
2.1.4
Dietmar Eggemann
reviewerRe: [PATCH v10 7/7] sched: Clean up load average references
Quoted 5 lines · Yuyang Du
For
cfs_rq, we have load.weight,runnable_load_avg, andload_avg. We now start to clean up how they are used.First, as group
sched_entityalready largely usesload_avg, we now expand to useload_avgin all cases.
You're talking about group se's or cfs_rq owned by the group se's (se->my_q) here or both?
Just asking because both data structures (cfs_rq and se) have a 'struct load_weight load' as well as 'struct sched_avg avg' member.
Second, for CPU-wide load balancing, we
Quoted 2 lines
choose to use
runnable_load_avgin all cases, which is the same as before this series.
With your patch-set there will be still the difference of 'cfs_rq->utilization_load_avg' and your 'cfs_rq->avg.util_avg' in the sense that the former one does not contain the contribution of blocked se's.
The EAS patch-set adds blocked utilization contribution:
https://lkml.org/lkml/2015/7/7/915
The cfs_rq utilization is also used by the load-balancer code via get_cpu_usage() so the blocked utilization contribution to 'cfs_rq->avg.util_avg' can change load-balancing as well.
Since it is not as heavily used as the cfs_rq->runnable_load_avg we might not need to reintroduce cfs_rq->utilization_load_avg but at least mention this here.
-- Dietmar
[...]
Re: [PATCH v10 7/7] sched: Clean up load average references
Hi Dietmar,
Quoted 9 lines · Dietmar Eggemann
Quoted 5 lines · Yuyang Du
For
cfs_rq, we have load.weight,runnable_load_avg, andload_avg. We now start to clean up how they are used.First, as group
sched_entityalready largely usesload_avg, we now expand to useload_avgin all cases.You're talking about group se's or
cfs_rqowned by the group se's (se->my_q) here or both?
Definitely, group SE, and if the cfs_rq owned by group SE is also concerned with group SE, then both. I don't think this is very well calculated to be optimal, but probably this is the right move I can think of now.
We need to revisit all of the codes before we can at least make a final call.
Quoted 14 lines
Just asking because both data structures (
cfs_rqand se) have a 'structload_weightload' as well as 'structsched_avgavg' member.Second, for CPU-wide load balancing, we
Quoted 2 lines
choose to use
runnable_load_avgin all cases, which is the same as before this series.With your patch-set there will be still the difference of '
cfs_rq->utilization_load_avg' and your 'cfs_rq->avg.util_avg' in the sense that the former one does not contain the contribution of blocked se's.The EAS patch-set adds blocked utilization contribution:
https://lkml.org/lkml/2015/7/7/915
Quoted 8 lines
The
cfs_rqutilization is also used by the load-balancer code viaget_cpu_usage()so the blocked utilization contribution to 'cfs_rq->avg.util_avg' can change load-balancing as well.Since it is not as heavily used as the
cfs_rq->runnable_load_avgwe might not need to reintroducecfs_rq->utilization_load_avgbut at least mention this here.
Yes, thanks.
tip-bot for Yuyang Du
merge bot[tip:sched/core] sched/fair: Clean up load average references
In reply to Yuyang Dusched/fair: Clean up load average references
Commit-ID: 7ea241afbf4924c58d41078599f7a32ba49fb985 Gitweb: http://git.kernel.org/tip/7ea241afbf4924c58d41078599f7a32ba49fb985 Author: Yuyang Du <[email protected]> AuthorDate: Wed, 15 Jul 2015 08:04:42 +0800 Committer: Ingo Molnar <[email protected]> CommitDate: Mon, 3 Aug 2015 12:24:32 +0200
sched/fair: Clean up load average references
For cfs_rq, we have load.weight, runnable_load_avg, and load_avg. Clean up how they are used:
- First, as group sched_entity already largely uses load_avg, we now expand to use load_avg in all cases.
- Second, for CPU-wide load balancing, we choose to use runnable_load_avg in all cases, which is the same as before this series.
--- kernel/sched/fair.c | 44 +++++++++++++++++++++++++++++--------------- 1 file changed, 29 insertions(+), 15 deletions(-)
kernel/sched/fair.c +29 -15
@@ -685,6 +685,9 @@ void init_entity_runnable_average(struct sched_entity *se) sa->util_sum =LOAD_AVG_MAX ; /* when this task enqueue'ed, it will contribute to its cfs_rq's load_avg */ } + +static inline unsigned long cfs_rq_runnable_load_avg (struct cfs_rq *cfs_rq); +static inline unsigned long cfs_rq_load_avg (struct cfs_rq *cfs_rq);#else void init_entity_runnable_average (struct sched_entity *se) {
@@ -2360,7 +2363,7 @@ static inline long calc_tg_weight(struct task_group *tg, struct cfs_rq *cfs_rq)*/ tg_weight =atomic_long_read (&tg->load_avg); tg_weight -= cfs_rq->tg_load_avg_contrib; - tg_weight += cfs_rq->avg.load_avg; + tg_weight +=cfs_rq_load_avg (cfs_rq);return tg_weight; }
@@ -2370,7 +2373,7 @@ static long calc_cfs_shares(struct cfs_rq *cfs_rq, struct task_group *tg)long tg_weight, load, shares; tg_weight =calc_tg_weight (tg, cfs_rq); - load = cfs_rq->avg.load_avg; + load =cfs_rq_load_avg (cfs_rq); shares = (tg->shares * load);if (tg_weight)
@@ -2796,6 +2799,16 @@ void idle_exit_fair(struct rq *this_rq) { } +static inline unsigned long cfs_rq_runnable_load_avg (struct cfs_rq *cfs_rq) +{ +return cfs_rq->runnable_load_avg; +} + +static inline unsigned long cfs_rq_load_avg (struct cfs_rq *cfs_rq) +{ +return cfs_rq->avg.load_avg; +} +static int idle_balance (struct rq *this_rq);#else /* CONFIG_SMP */
@@ -4270,6 +4283,12 @@ static void __update_cpu_load(struct rq *this_rq, unsigned long this_load,sched_avg_update (this_rq); } +/* Used instead of source_load when we know the type == 0 */ +static unsigned long weighted_cpuload (const int cpu) +{ +return cfs_rq_runnable_load_avg (&cpu_rq (cpu)->cfs); +} +#ifdef CONFIG_NO_HZ_COMMON /* * There is no sane way to deal with nohz on smp when using jiffies because the
@@ -4291,7 +4310,7 @@ static void __update_cpu_load(struct rq *this_rq, unsigned long this_load,static void update_idle_cpu_load (struct rq *this_rq) {unsigned long curr_jiffies =READ_ONCE (jiffies); -unsigned long load = this_rq->cfs.avg.load_avg; +unsigned long load =weighted_cpuload (cpu_of (this_rq));unsigned long pending_updates;/*
@@ -4337,7 +4356,7 @@ void update_cpu_load_nohz(void)*/ void update_cpu_load_active (struct rq *this_rq) { -unsigned long load = this_rq->cfs.avg.load_avg; +unsigned long load =weighted_cpuload (cpu_of (this_rq));/* * See the mess around update_idle_cpu_load() / update_cpu_load_nohz(). */
@@ -4345,12 +4364,6 @@ void update_cpu_load_active(struct rq *this_rq)__update_cpu_load (this_rq, load,1 ); } -/* Used instead of source_load when we know the type == 0 */ -static unsigned long weighted_cpuload (const int cpu) -{ -return cpu_rq (cpu)->cfs.avg.load_avg; -} -/* * Return a low guess at the load of a migration-source cpu weighted * according to the scheduling class and "nice" value.
@@ -4398,7 +4411,7 @@ static unsigned long cpu_avg_load_per_task(int cpu) {struct rq *rq =cpu_rq (cpu);unsigned long nr_running =READ_ONCE (rq->cfs.h_nr_running); -unsigned long load_avg = rq->cfs.avg.load_avg; +unsigned long load_avg =weighted_cpuload (cpu);if (nr_running)return load_avg / nr_running;
@@ -4517,7 +4530,7 @@ static long effective_load(struct task_group *tg, int cpu, long wl, long wg)/* * w = rw_i + @wl */ - w = se->my_q->avg.load_avg + wl; + w =cfs_rq_load_avg (se->my_q) + wl;/* * wl = S * s'_i; see (2)
@@ -5862,13 +5875,14 @@ static void update_cfs_rq_h_load(struct cfs_rq *cfs_rq) }if (!se) { - cfs_rq->h_load = cfs_rq->avg.load_avg; + cfs_rq->h_load =cfs_rq_load_avg (cfs_rq); cfs_rq->last_h_load_update = now; }while ((se = cfs_rq->h_load_next) !=NULL ) { load = cfs_rq->h_load; - load =div64_ul (load * se->avg.load_avg, cfs_rq->avg.load_avg +1 ); + load =div64_ul (load * se->avg.load_avg, +cfs_rq_load_avg (cfs_rq) +1 ); cfs_rq =group_cfs_rq (se); cfs_rq->h_load = load; cfs_rq->last_h_load_update = now;
@@ -5881,7 +5895,7 @@ static unsigned long task_h_load(struct task_struct *p)update_cfs_rq_h_load (cfs_rq);return div64_ul (p->se.avg.load_avg * cfs_rq->h_load, - cfs_rq->avg.load_avg + 1); + cfs_rq_load_avg(cfs_rq) + 1); }#else static inline void update_blocked_averages (int cpu)