sched: Fix calc_cfs_shares() to consider blocked_load_avg also
PATCH 1/1 sched: Fix calc_cfs_shares() to consider blocked_load_avg also
From: Namhyung Kim <[email protected]>
The calc_tg_weight() and calc_cfs_shares() used cfs_rq->load.weight but this is no longer valid for per-entity load tracking since cfs_rq->tg_load_contrib consists of runnable_load_avg and blocked_ load_avg. Simply using load.weight here will lose blocked_load_avg part so will result in an inaccurate share.
--- kernel/sched/fair.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)
kernel/sched/fair.c +5 -5
@@ -1032,13 +1032,13 @@ static inline long calc_tg_weight(struct task_group *tg, struct cfs_rq *cfs_rq)long tg_weight;/* -* Use this CPU's actual weight instead of the last load_contribution -* to gain a more accurate current total weight. See -* update_cfs_rq_load_contribution(). +* Use this CPU's actual load instead of the last load_contribution +* to gain a more accurate current total load. See +* __update_cfs_rq_tg_load_contrib(). */ tg_weight =atomic64_read (&tg->load_avg); tg_weight -= cfs_rq->tg_load_contrib; - tg_weight += cfs_rq->load.weight; + tg_weight += cfs_rq->runnable_load_avg + cfs_rq->blocked_load_avg;return tg_weight; }
@@ -1048,7 +1048,7 @@ static long calc_cfs_shares(struct cfs_rq *cfs_rq, struct task_group *tg)long tg_weight, load, shares; tg_weight =calc_tg_weight (tg, cfs_rq); - load = cfs_rq->load.weight; + load = cfs_rq->runnable_load_avg + cfs_rq->blocked_load_avg; shares = (tg->shares * load);if (tg_weight)
Signature · 1 line
1.7.11.7
(I'm still in New Zealand and won't be on regular email until March 6th, but I just saw this and wanted to comment quickly)
Quoted 23 lines · Namhyung Kim
From: Namhyung Kim <[email protected]>
The
calc_tg_weight()andcalc_cfs_shares()usedcfs_rq->load.weightbut this is no longer valid for per-entity load tracking sincecfs_rq->tg_load_contribconsists ofrunnable_load_avgandblocked_load_avg. Simply using load.weight here will loseblocked_load_avgpart so will result in an inaccurate share.Cc Paul Turner <[email protected]>Signed-off-by Namhyung Kim <[email protected]>--- kernel/sched/fair.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)kernel/sched/fair.c +0
-1@@ -1032,13 +1032,13 @@ static inline long calc_tg_weight(struct task_group *tg, struct cfs_rq *cfs_rq)long tg_weight;/* -* Use this CPU's actual weight instead of the last load_contribution
kernel/sched/fair.c static inline long calc_tg_weight(struct task_group *tg, struct cfs_rq *cfs_rq) · line 1041
Quoted 10 lines
kernel/sched/fair.c +4
-3-* to gain a more accurate current total weight. See -* update_cfs_rq_load_contribution(). +* Use this CPU's actual load instead of the last load_contribution +* to gain a more accurate current total load. See +* __update_cfs_rq_tg_load_contrib(). */ tg_weight =atomic64_read (&tg->load_avg); tg_weight -= cfs_rq->tg_load_contrib; - tg_weight += cfs_rq->load.weight; + tg_weight += cfs_rq->runnable_load_avg + cfs_rq->blocked_load_avg;
No -- we _really_ do want to use the instantaneous weight, the maintained averages are back-wards looking and do not always predict future usage. We played with allocation strategies like the above during this development and it ends up being a big loss for latency.
In particular:
tasks with a low runnable averages are almost always going to be
_under_ their fair share; using the current runnable average here then harshly penalizes their ability to pre-empt when the calculated weight is subsequently used for emplacement. Stronger: Such tasks are typically interactive (having to wait for us slow humans and all). Consider what the above would do to a while(1) versus interactive thread in the same cgroup; the cpu holding the while(1) thread is always going to wholly dominate share allocation.
Quoted 14 lines
kernel/sched/fair.c +1
-2return tg_weight; }@@ -1048,7 +1048,7 @@ static long calc_cfs_shares(struct cfs_rq *cfs_rq, struct task_group *tg)long tg_weight, load, shares; tg_weight =calc_tg_weight (tg, cfs_rq); - load = cfs_rq->load.weight; + load = cfs_rq->runnable_load_avg + cfs_rq->blocked_load_avg; shares = (tg->shares * load);if (tg_weight) --1.7.11.7