Skip to contentSkip to conversation

sched: Reduce contention in update_cfs_rq_blocked_load

PATCH In review Jason Low 6 participants 10 messages Aug 26, 2014 – Sep 9, 2014 Scheduler

lore
Share & tools
lore

Jason Low

author
#0 lore raw

PATCH v2 1/1 sched: Reduce contention in update_cfs_rq_blocked_load

Based on perf profiles, the update_cfs_rq_blocked_load function constantly shows up as taking up a noticeable % of system run time. This is especially apparent on larger numa systems.

Much of the contention is in __update_cfs_rq_tg_load_contrib when we're updating the tg load contribution stats. However, it was noticed that the values often don't get modified by much. In fact, much of the time, they don't get modified at all. However, the update can always get attempted due to force_update.

In this patch, we remove the force_update in only the __update_cfs_rq_tg_load_contrib. Thus the tg load contrib stats now get modified only if the delta is large enough (in the current code, they get updated when the delta is larger than 12.5%). This is a way to rate-limit the updates while largely keeping the values accurate.

When testing this change with AIM7 workloads, we found that it was able to reduce the overhead of the function by up to a factor of 20x.

Cc Yuyang Du <[email protected]>
Cc Waiman Long <[email protected]>
Cc Mel Gorman <[email protected]>
Cc Mike Galbraith <[email protected]>
Cc Rik van Riel <[email protected]>
Cc Aswin Chandramouleeswaran <[email protected]>
Cc Chegu Vinod <[email protected]>
Cc Scott J Norton <[email protected]>
Signed-off-by Jason Low <[email protected]>
---
 kernel/sched/fair.c |   10 ++++------
 1 files changed, 4 insertions(+), 6 deletions(-)
kernel/sched/fair.c +4 -6
index fea7d33..7a6e18b 100644
@@ -2352,8 +2352,7 @@ static inline u64 __synchronize_entity_decay(struct sched_entity *se)
 }
 
 #ifdef CONFIG_FAIR_GROUP_SCHED
-static inline void __update_cfs_rq_tg_load_contrib(struct cfs_rq *cfs_rq,
-						 int force_update)
+static inline void __update_cfs_rq_tg_load_contrib(struct cfs_rq *cfs_rq)
 {
 	struct task_group *tg = cfs_rq->tg;
 	long tg_contrib;

@@ -2361,7 +2360,7 @@ static inline void __update_cfs_rq_tg_load_contrib(struct cfs_rq *cfs_rq,
 	tg_contrib = cfs_rq->runnable_load_avg + cfs_rq->blocked_load_avg;
 	tg_contrib -= cfs_rq->tg_load_contrib;
 
-	if (force_update || abs(tg_contrib) > cfs_rq->tg_load_contrib / 8) {
+	if (abs(tg_contrib) > cfs_rq->tg_load_contrib / 8) {
 		atomic_long_add(tg_contrib, &tg->load_avg);
 		cfs_rq->tg_load_contrib += tg_contrib;
 	}

@@ -2436,8 +2435,7 @@ static inline void update_rq_runnable_avg(struct rq *rq, int runnable)
 	__update_tg_runnable_avg(&rq->avg, &rq->cfs);
 }
 #else /* CONFIG_FAIR_GROUP_SCHED */
-static inline void __update_cfs_rq_tg_load_contrib(struct cfs_rq *cfs_rq,
-						 int force_update) {}
+static inline void __update_cfs_rq_tg_load_contrib(struct cfs_rq *cfs_rq) {}
 static inline void __update_tg_runnable_avg(struct sched_avg *sa,
 						  struct cfs_rq *cfs_rq) {}
 static inline void __update_group_entity_contrib(struct sched_entity *se) {}

@@ -2537,7 +2535,7 @@ static void update_cfs_rq_blocked_load(struct cfs_rq *cfs_rq, int force_update)
 		cfs_rq->last_decay = now;
 	}
 
-	__update_cfs_rq_tg_load_contrib(cfs_rq, force_update);
+	__update_cfs_rq_tg_load_contrib(cfs_rq);
 }
 
 /* Add the load generated by se into cfs_rq's child load-average */
Signature · 1 line
1.7.1
#1 lore raw
Quoted 10 lines · Jason Low

Based on perf profiles, the update_cfs_rq_blocked_load function constantly shows up as taking up a noticeable % of system run time. This is especially apparent on larger numa systems.

Much of the contention is in __update_cfs_rq_tg_load_contrib when we're updating the tg load contribution stats. However, it was noticed that the values often don't get modified by much. In fact, much of the time, they don't get modified at all. However, the update can always get attempted due to force_update.

Quoted 8 lines

In this patch, we remove the force_update in only the __update_cfs_rq_tg_load_contrib. Thus the tg load contrib stats now get modified only if the delta is large enough (in the current code, they get updated when the delta is larger than 12.5%). This is a way to rate-limit the updates while largely keeping the values accurate.

When testing this change with AIM7 workloads, we found that it was able to reduce the overhead of the function by up to a factor of 20x.

Looks reasonable.

Quoted 29 lines
Cc Yuyang Du <[email protected]>
Cc Waiman Long <[email protected]>
Cc Mel Gorman <[email protected]>
Cc Mike Galbraith <[email protected]>
Cc Rik van Riel <[email protected]>
Cc Aswin Chandramouleeswaran <[email protected]>
Cc Chegu Vinod <[email protected]>
Cc Scott J Norton <[email protected]>
Signed-off-by Jason Low <[email protected]>
---
 kernel/sched/fair.c |   10 ++++------
 1 files changed, 4 insertions(+), 6 deletions(-)
kernel/sched/fair.c +1 -2
index fea7d33..7a6e18b 100644
@@ -2352,8 +2352,7 @@ static inline u64 __synchronize_entity_decay(struct sched_entity *se)
 }
 
 #ifdef CONFIG_FAIR_GROUP_SCHED
-static inline void __update_cfs_rq_tg_load_contrib(struct cfs_rq *cfs_rq,
-                                                int force_update)
+static inline void __update_cfs_rq_tg_load_contrib(struct cfs_rq *cfs_rq)
 {
        struct task_group *tg = cfs_rq->tg;
        long tg_contrib;

kernel/sched/fair.c static inline void __update_cfs_rq_tg_load_contrib(struct cfs_rq *cfs_rq, · line 2364

Quoted 4 lines
kernel/sched/fair.c +0 -1
@@ -2361,7 +2360,7 @@ static inline void __update_cfs_rq_tg_load_contrib(struct cfs_rq *cfs_rq,
        tg_contrib = cfs_rq->runnable_load_avg + cfs_rq->blocked_load_avg;
        tg_contrib -= cfs_rq->tg_load_contrib;
 
-       if (force_update || abs(tg_contrib) > cfs_rq->tg_load_contrib / 8) {

Another option with slightly higher accuracy would be to increase the sensitivity here when force_update == 1.

E.g.:

    abs(tg_contrib) > cfs_rq->tg_load_contrib / (8 * (1 + force_update))) { ...

Alternatively we could bound total inaccuracy:
int divisor = force_update ? NR_CPUS : 8;
if (abs(tg_contrib) > cfs_rq->tg_load_contrib / divisor) { ...

[ And probably rename force_update to want_update ]

Quoted 33 lines
kernel/sched/fair.c +3 -4
+       if (abs(tg_contrib) > cfs_rq->tg_load_contrib / 8) {
                atomic_long_add(tg_contrib, &tg->load_avg);
                cfs_rq->tg_load_contrib += tg_contrib;
        }

@@ -2436,8 +2435,7 @@ static inline void update_rq_runnable_avg(struct rq *rq, int runnable)
        __update_tg_runnable_avg(&rq->avg, &rq->cfs);
 }
 #else /* CONFIG_FAIR_GROUP_SCHED */
-static inline void __update_cfs_rq_tg_load_contrib(struct cfs_rq *cfs_rq,
-                                                int force_update) {}
+static inline void __update_cfs_rq_tg_load_contrib(struct cfs_rq *cfs_rq) {}
 static inline void __update_tg_runnable_avg(struct sched_avg *sa,
                                                  struct cfs_rq *cfs_rq) {}
 static inline void __update_group_entity_contrib(struct sched_entity *se) {}

@@ -2537,7 +2535,7 @@ static void update_cfs_rq_blocked_load(struct cfs_rq *cfs_rq, int force_update)
                cfs_rq->last_decay = now;
        }
 
-       __update_cfs_rq_tg_load_contrib(cfs_rq, force_update);
+       __update_cfs_rq_tg_load_contrib(cfs_rq);
 }
 
 /* Add the load generated by se into cfs_rq's child load-average */
--

1.7.1

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

author
#2 lore raw
Quoted 60 lines · Paul Turner
Quoted 10 lines · Jason Low

Based on perf profiles, the update_cfs_rq_blocked_load function constantly shows up as taking up a noticeable % of system run time. This is especially apparent on larger numa systems.

Much of the contention is in __update_cfs_rq_tg_load_contrib when we're updating the tg load contribution stats. However, it was noticed that the values often don't get modified by much. In fact, much of the time, they don't get modified at all. However, the update can always get attempted due to force_update.

Quoted 8 lines

In this patch, we remove the force_update in only the __update_cfs_rq_tg_load_contrib. Thus the tg load contrib stats now get modified only if the delta is large enough (in the current code, they get updated when the delta is larger than 12.5%). This is a way to rate-limit the updates while largely keeping the values accurate.

When testing this change with AIM7 workloads, we found that it was able to reduce the overhead of the function by up to a factor of 20x.

Looks reasonable.

Quoted 29 lines
Cc Yuyang Du <[email protected]>
Cc Waiman Long <[email protected]>
Cc Mel Gorman <[email protected]>
Cc Mike Galbraith <[email protected]>
Cc Rik van Riel <[email protected]>
Cc Aswin Chandramouleeswaran <[email protected]>
Cc Chegu Vinod <[email protected]>
Cc Scott J Norton <[email protected]>
Signed-off-by Jason Low <[email protected]>
---
 kernel/sched/fair.c |   10 ++++------
 1 files changed, 4 insertions(+), 6 deletions(-)
kernel/sched/fair.c +1 -2
index fea7d33..7a6e18b 100644
@@ -2352,8 +2352,7 @@ static inline u64 __synchronize_entity_decay(struct sched_entity *se)
 }
 
 #ifdef CONFIG_FAIR_GROUP_SCHED
-static inline void __update_cfs_rq_tg_load_contrib(struct cfs_rq *cfs_rq,
-                                                int force_update)
+static inline void __update_cfs_rq_tg_load_contrib(struct cfs_rq *cfs_rq)
 {
        struct task_group *tg = cfs_rq->tg;
        long tg_contrib;

kernel/sched/fair.c static inline void __update_cfs_rq_tg_load_contrib(struct cfs_rq *cfs_rq, · line 2364

Quoted 4 lines
kernel/sched/fair.c +0 -1
@@ -2361,7 +2360,7 @@ static inline void __update_cfs_rq_tg_load_contrib(struct cfs_rq *cfs_rq,
        tg_contrib = cfs_rq->runnable_load_avg + cfs_rq->blocked_load_avg;
        tg_contrib -= cfs_rq->tg_load_contrib;
 
-       if (force_update || abs(tg_contrib) > cfs_rq->tg_load_contrib / 8) {

Another option with slightly higher accuracy would be to increase the sensitivity here when force_update == 1.

E.g.:

Quoted 8 lines
    abs(tg_contrib) > cfs_rq->tg_load_contrib / (8 * (1 + force_update))) { ...

Alternatively we could bound total inaccuracy:

   int divisor = force_update ? NR_CPUS : 8;
   if (abs(tg_contrib) > cfs_rq->tg_load_contrib / divisor) { ...

[ And probably rename force_update to want_update ]

Out of the 2 additional options, I think the first one is better. The other option of using NR_CPUS looks like we're increasing the update rate as the system gets larger, and its the larger systems that are typically more affected by the contention.

Do you prefer either of the 2 other options over this v2 patch? If so, I can test and send out a new patch, otherwise, I'll keep this current v2 patch.

Thanks.

#3 lore raw
Quoted 68 lines · Jason Low
Quoted 67 lines · Paul Turner
Quoted 10 lines · Jason Low

Based on perf profiles, the update_cfs_rq_blocked_load function constantly shows up as taking up a noticeable % of system run time. This is especially apparent on larger numa systems.

Much of the contention is in __update_cfs_rq_tg_load_contrib when we're updating the tg load contribution stats. However, it was noticed that the values often don't get modified by much. In fact, much of the time, they don't get modified at all. However, the update can always get attempted due to force_update.

Quoted 8 lines

In this patch, we remove the force_update in only the __update_cfs_rq_tg_load_contrib. Thus the tg load contrib stats now get modified only if the delta is large enough (in the current code, they get updated when the delta is larger than 12.5%). This is a way to rate-limit the updates while largely keeping the values accurate.

When testing this change with AIM7 workloads, we found that it was able to reduce the overhead of the function by up to a factor of 20x.

Looks reasonable.

Quoted 29 lines
Cc Yuyang Du <[email protected]>
Cc Waiman Long <[email protected]>
Cc Mel Gorman <[email protected]>
Cc Mike Galbraith <[email protected]>
Cc Rik van Riel <[email protected]>
Cc Aswin Chandramouleeswaran <[email protected]>
Cc Chegu Vinod <[email protected]>
Cc Scott J Norton <[email protected]>
Signed-off-by Jason Low <[email protected]>
---
 kernel/sched/fair.c |   10 ++++------
 1 files changed, 4 insertions(+), 6 deletions(-)
kernel/sched/fair.c +1 -2
index fea7d33..7a6e18b 100644
@@ -2352,8 +2352,7 @@ static inline u64 __synchronize_entity_decay(struct sched_entity *se)
 }
 
 #ifdef CONFIG_FAIR_GROUP_SCHED
-static inline void __update_cfs_rq_tg_load_contrib(struct cfs_rq *cfs_rq,
-                                                int force_update)
+static inline void __update_cfs_rq_tg_load_contrib(struct cfs_rq *cfs_rq)
 {
        struct task_group *tg = cfs_rq->tg;
        long tg_contrib;

kernel/sched/fair.c static inline void __update_cfs_rq_tg_load_contrib(struct cfs_rq *cfs_rq, · line 2364

Quoted 4 lines
kernel/sched/fair.c +0 -1
@@ -2361,7 +2360,7 @@ static inline void __update_cfs_rq_tg_load_contrib(struct cfs_rq *cfs_rq,
        tg_contrib = cfs_rq->runnable_load_avg + cfs_rq->blocked_load_avg;
        tg_contrib -= cfs_rq->tg_load_contrib;
 
-       if (force_update || abs(tg_contrib) > cfs_rq->tg_load_contrib / 8) {

Another option with slightly higher accuracy would be to increase the sensitivity here when force_update == 1.

E.g.:

    abs(tg_contrib) > cfs_rq->tg_load_contrib / (8 * (1 + force_update))) { ...

Alternatively we could bound total inaccuracy:

   int divisor = force_update ? NR_CPUS : 8;
   if (abs(tg_contrib) > cfs_rq->tg_load_contrib / divisor) { ...
Quoted 6 lines
Quoted 1 line

[ And probably rename force_update to want_update ]

Out of the 2 additional options, I think the first one is better. The other option of using NR_CPUS looks like we're increasing the update rate as the system gets larger, and its the larger systems that are typically more affected by the contention.

Probably num_present_cpus() will be better than NR_CPUS, which can be much larger than the actual cpus present.

Quoted 4 lines

Do you prefer either of the 2 other options over this v2 patch? If so, I can test and send out a new patch, otherwise, I'll keep this current v2 patch.

If there are multiple non-forced updates, option 1's error seems to accumulate and non-bounded as we do not actually update? Is this a concern?

Thanks.

Tim

author
#4 lore raw
Quoted 71 lines · Tim Chen
Quoted 70 lines · Jason Low
Quoted 68 lines · Paul Turner
Quoted 10 lines · Jason Low

Based on perf profiles, the update_cfs_rq_blocked_load function constantly shows up as taking up a noticeable % of system run time. This is especially apparent on larger numa systems.

Much of the contention is in __update_cfs_rq_tg_load_contrib when we're updating the tg load contribution stats. However, it was noticed that the values often don't get modified by much. In fact, much of the time, they don't get modified at all. However, the update can always get attempted due to force_update.

Quoted 8 lines

In this patch, we remove the force_update in only the __update_cfs_rq_tg_load_contrib. Thus the tg load contrib stats now get modified only if the delta is large enough (in the current code, they get updated when the delta is larger than 12.5%). This is a way to rate-limit the updates while largely keeping the values accurate.

When testing this change with AIM7 workloads, we found that it was able to reduce the overhead of the function by up to a factor of 20x.

Looks reasonable.

Quoted 29 lines
Cc Yuyang Du <[email protected]>
Cc Waiman Long <[email protected]>
Cc Mel Gorman <[email protected]>
Cc Mike Galbraith <[email protected]>
Cc Rik van Riel <[email protected]>
Cc Aswin Chandramouleeswaran <[email protected]>
Cc Chegu Vinod <[email protected]>
Cc Scott J Norton <[email protected]>
Signed-off-by Jason Low <[email protected]>
---
 kernel/sched/fair.c |   10 ++++------
 1 files changed, 4 insertions(+), 6 deletions(-)
kernel/sched/fair.c +1 -2
index fea7d33..7a6e18b 100644
@@ -2352,8 +2352,7 @@ static inline u64 __synchronize_entity_decay(struct sched_entity *se)
 }
 
 #ifdef CONFIG_FAIR_GROUP_SCHED
-static inline void __update_cfs_rq_tg_load_contrib(struct cfs_rq *cfs_rq,
-                                                int force_update)
+static inline void __update_cfs_rq_tg_load_contrib(struct cfs_rq *cfs_rq)
 {
        struct task_group *tg = cfs_rq->tg;
        long tg_contrib;

kernel/sched/fair.c static inline void __update_cfs_rq_tg_load_contrib(struct cfs_rq *cfs_rq, · line 2364

Quoted 4 lines
kernel/sched/fair.c +0 -1
@@ -2361,7 +2360,7 @@ static inline void __update_cfs_rq_tg_load_contrib(struct cfs_rq *cfs_rq,
        tg_contrib = cfs_rq->runnable_load_avg + cfs_rq->blocked_load_avg;
        tg_contrib -= cfs_rq->tg_load_contrib;
 
-       if (force_update || abs(tg_contrib) > cfs_rq->tg_load_contrib / 8) {

Another option with slightly higher accuracy would be to increase the sensitivity here when force_update == 1.

E.g.:

    abs(tg_contrib) > cfs_rq->tg_load_contrib / (8 * (1 + force_update))) { ...

Alternatively we could bound total inaccuracy:

   int divisor = force_update ? NR_CPUS : 8;
   if (abs(tg_contrib) > cfs_rq->tg_load_contrib / divisor) { ...

[ And probably rename force_update to want_update ]

Quoted 7 lines
Quoted 4 lines

Out of the 2 additional options, I think the first one is better. The other option of using NR_CPUS looks like we're increasing the update rate as the system gets larger, and its the larger systems that are typically more affected by the contention.

Probably num_present_cpus() will be better than NR_CPUS, which can be much larger than the actual cpus present.

Yeah, num_present_cpus(), though the same issue would still be there.

Quoted 8 lines
Quoted 4 lines

Do you prefer either of the 2 other options over this v2 patch? If so, I can test and send out a new patch, otherwise, I'll keep this current v2 patch.

If there are multiple non-forced updates, option 1's error seems to accumulate and non-bounded as we do not actually update? Is this a concern?

It should be fine. Once the delta is large enough, we will end up doing the update anyway.

Thanks.

maintainer
#5 lore raw
Quoted 8 lines · Jason Low
Quoted 3 lines · Tim Chen

If there are multiple non-forced updates, option 1's error seems to accumulate and non-bounded as we do not actually update? Is this a concern?

It should be fine. Once the delta is large enough, we will end up doing the update anyway.

Well, the thing is you can have nr_cpus * 12.5% of outstanding delta; that might be a lot, esp on the large machines.

Now there's two problems with all this; the first is the relative threshold, typically such per-cpu things have a fixed update threshold, this makes it much easier to qualify the actual error.

Secondly the indeed the nr_cpus in the error bound. Some things; like the proportion code scale the threshold by log2(nr_cpus) in an attempt to do something sensible there.

But yes, unbounded errors here are a problem, sure relaxing the updates makes things go fast, they also make things go skew.

author Reviewed-by Ben Segall Reviewed-by Waiman Long
#6 lore raw
Quoted 2 lines · Peter Zijlstra

But yes, unbounded errors here are a problem, sure relaxing the updates makes things go fast, they also make things go skew.

Okay. In that case, would you like to take our original patch which avoids unnecessary updates?

-----
Subject: [PATCH] sched: Reduce contention in update_cfs_rq_blocked_load

When running workloads on 2+ socket systems, based on perf profiles, the update_cfs_rq_blocked_load function often shows up as taking up a noticeable % of run time.

Much of the contention is in __update_cfs_rq_tg_load_contrib when we update the tg load contribution stats. However, it turns out that in many cases, they don't need to be updated and "tg_contrib" is 0.

This patch adds a check in __update_cfs_rq_tg_load_contrib to skip updating tg load contribution stats when nothing needs to be updated. This reduces the cacheline contention that would be unnecessary.

Cc Yuyang Du <[email protected]>
Cc Aswin Chandramouleeswaran <[email protected]>
Cc Chegu Vinod <[email protected]>
Cc Scott J Norton <[email protected]>
Reviewed-by Ben Segall <[email protected]>
Reviewed-by Waiman Long <[email protected]>
Signed-off-by Jason Low <[email protected]>
---
 kernel/sched/fair.c |    3 +++
 1 files changed, 3 insertions(+), 0 deletions(-)
kernel/sched/fair.c +3 -0
index d3427a8..45e346c 100644
@@ -2382,6 +2382,9 @@ static inline void __update_cfs_rq_tg_load_contrib(struct cfs_rq *cfs_rq,
 	tg_contrib = cfs_rq->runnable_load_avg + cfs_rq->blocked_load_avg;
 	tg_contrib -= cfs_rq->tg_load_contrib;
 
+	if (!tg_contrib)
+		return;
+
 	if (force_update || abs(tg_contrib) > cfs_rq->tg_load_contrib / 8) {
 		atomic_long_add(tg_contrib, &tg->load_avg);
 		cfs_rq->tg_load_contrib += tg_contrib;
Signature · 1 line
1.7.1
maintainer
#7 lore raw
Quoted 6 lines · Jason Low
Quoted 2 lines · Peter Zijlstra

But yes, unbounded errors here are a problem, sure relaxing the updates makes things go fast, they also make things go skew.

Okay. In that case, would you like to take our original patch which avoids unnecessary updates?

This seems a safe patch; yes I'll queue it while we think of smarter things :-)

merge bot
#8 lore raw

[tip:sched/core] sched: Reduce contention in update_cfs_rq_blocked_load()

In reply to Jason Low
sched: Reduce contention in update_cfs_rq_blocked_load()
Commit-ID:  8236d907ab3411ad452280faa8b26c1347327380
Gitweb:     http://git.kernel.org/tip/8236d907ab3411ad452280faa8b26c1347327380
Author:     Jason Low <[email protected]>
AuthorDate: Tue, 2 Sep 2014 00:41:24 -0700
Committer:  Ingo Molnar <[email protected]>
CommitDate: Tue, 9 Sep 2014 06:47:29 +0200

sched: Reduce contention in update_cfs_rq_blocked_load()

When running workloads on 2+ socket systems, based on perf profiles, the update_cfs_rq_blocked_load() function often shows up as taking up a noticeable % of run time.

Much of the contention is in __update_cfs_rq_tg_load_contrib() when we update the tg load contribution stats. However, it turns out that in many cases, they don't need to be updated and "tg_contrib" is 0.

This patch adds a check in __update_cfs_rq_tg_load_contrib() to skip updating tg load contribution stats when nothing needs to be updated. This reduces the cacheline contention that would be unnecessary.

Reviewed-by Ben Segall <[email protected]>
Reviewed-by Waiman Long <[email protected]>
Signed-off-by Jason Low <[email protected]>
Signed-off-by Peter Zijlstra <[email protected]>
Cc Paul Turner <[email protected]>
Cc [email protected]
Cc Yuyang Du <[email protected]>
Cc Aswin Chandramouleeswaran <[email protected]>
Cc Chegu Vinod <[email protected]>
Cc Scott J Norton <[email protected]>
Cc Tim Chen <[email protected]>
Cc Linus Torvalds <[email protected]>
Link http://lkml.kernel.org/r/1409643684.19197.15.camel@j-VirtualBox open here
Signed-off-by Ingo Molnar <[email protected]>
---
 kernel/sched/fair.c | 3 +++
 1 file changed, 3 insertions(+)
kernel/sched/fair.c +3 -0
index 50d2025..be9e97b 100644
@@ -2382,6 +2382,9 @@ static inline void __update_cfs_rq_tg_load_contrib(struct cfs_rq *cfs_rq,
 	tg_contrib = cfs_rq->runnable_load_avg + cfs_rq->blocked_load_avg;
 	tg_contrib -= cfs_rq->tg_load_contrib;
 
+	if (!tg_contrib)
+		return;
+
 	if (force_update || abs(tg_contrib) > cfs_rq->tg_load_contrib / 8) {
 		atomic_long_add(tg_contrib, &tg->load_avg);
 		cfs_rq->tg_load_contrib += tg_contrib;
#9 lore raw

PATCH 1/1 sched: Rewrite per entity runnable load average tracking v5

In reply to Paul Turner

Hi all,

We finished a round of tests for AIM7 workload for the patch v5 by LKP, the result is as follows:

Hostname: brickland3
Model: Brickland Ivy Bridge-E

CPU: 120

Memory: 512G

v3.16-rc7 rewrite-v5 testbox/testcase/testparams
--------------- ------------------------- ---------------------------

     %stddev        %change               %stddev
            \          |                 /
     26317 ± 3%     +34.7%      35437 ± 1%  brickland3/aim7/2000-fork_test
     43481 ± 0%      +0.5%      43712 ± 0%  brickland3/aim7/3000-brk_test
     80810 ± 0%      -0.2%      80625 ± 1%  brickland3/aim7/3000-sieve
    489830 ± 1%     +14.7%     561691 ± 1%  brickland3/aim7/8000-disk_src
   1672643 ±12%     -13.0%    1455960 ± 6%  brickland3/aim7/8000-disk_wrt
   1193986 ± 0%      -1.5%    1175841 ± 0%  brickland3/aim7/8000-mem_rtns_1
    875113 ±13%     +21.9%    1067191 ±13%  brickland3/aim7/8000-misc_rtns_1
   1188420 ± 1%      -3.2%    1150624 ± 0%  brickland3/aim7/8000-sort_rtns_1
    807986 ± 0%      -0.7%     802275 ± 0%  brickland3/aim7/8000-string_rtns
   6378589 ± 5%      -0.1%    6373359 ± 4%  TOTAL aim7.jobs-per-min

--------------- -------------------------

       339 ± 7%     -22.9%        262 ± 0%  brickland3/aim7/2000-fork_test
       463 ± 0%      +0.4%        465 ± 0%  brickland3/aim7/3000-brk_test
       337 ± 9%      -3.1%        327 ± 6%  brickland3/aim7/3000-disk_rd
       534 ± 0%      +0.9%        539 ± 0%  brickland3/aim7/3000-sieve
       391 ± 0%     -10.7%        349 ± 1%  brickland3/aim7/8000-disk_src
       301 ± 4%     +10.0%        331 ± 5%  brickland3/aim7/8000-disk_wrt
       405 ± 1%      +6.0%        429 ± 0%  brickland3/aim7/8000-mem_rtns_1
       227 ±25%      -0.8%        225 ±12%  brickland3/aim7/8000-misc_rtns_1
       395 ± 1%      +5.1%        415 ± 0%  brickland3/aim7/8000-sort_rtns_1
       419 ± 0%      +4.6%        438 ± 0%  brickland3/aim7/8000-string_rtns
      3815 ± 3%      -0.8%       3784 ± 2%  TOTAL turbostat.Pkg_W

--------------- -------------------------

     90.79 ± 1%     -15.1%      77.04 ± 0%  brickland3/aim7/2000-fork_test
     62.42 ± 0%      -0.0%      62.41 ± 0%  brickland3/aim7/3000-brk_test
     61.69 ± 2%      -1.3%      60.92 ± 2%  brickland3/aim7/3000-disk_rd
    137.13 ± 0%      +0.2%     137.39 ± 0%  brickland3/aim7/3000-sieve
     66.03 ± 1%      +0.6%      66.41 ± 0%  brickland3/aim7/8000-disk_src
     53.94 ± 2%      +6.1%      57.21 ± 0%  brickland3/aim7/8000-disk_wrt
     54.43 ± 0%      +4.8%      57.02 ± 0%  brickland3/aim7/8000-mem_rtns_1
     58.71 ± 2%      +0.0%      58.72 ± 1%  brickland3/aim7/8000-misc_rtns_1
     53.78 ± 1%      +4.8%      56.36 ± 0%  brickland3/aim7/8000-sort_rtns_1
     56.47 ± 2%      +2.5%      57.89 ± 0%  brickland3/aim7/8000-string_rtns
    695.38 ± 1%      -0.6%     691.37 ± 1%  TOTAL turbostat.RAM_W

--------------- -------------------------

     23.40 ±22%     -16.2%      19.60 ± 1%  brickland3/aim7/2000-fork_test
     97.28 ± 0%      -0.0%      97.25 ± 0%  brickland3/aim7/3000-brk_test
     35.09 ±24%      -8.7%      32.05 ±21%  brickland3/aim7/3000-disk_rd
     94.92 ± 0%      -0.0%      94.88 ± 0%  brickland3/aim7/3000-sieve
     41.04 ± 2%     -26.8%      30.06 ± 3%  brickland3/aim7/8000-disk_src
     36.54 ±11%     -10.2%      32.82 ±12%  brickland3/aim7/8000-disk_wrt
     61.39 ± 2%      -7.4%      56.87 ± 1%  brickland3/aim7/8000-mem_rtns_1
     62.57 ± 4%     -11.1%      55.64 ± 3%  brickland3/aim7/8000-sort_rtns_1
     76.06 ± 2%      -3.4%      73.51 ± 0%  brickland3/aim7/8000-string_rtns
    528.30 ± 4%      -6.7%     492.67 ± 3%  TOTAL turbostat.%c0

For performance, I think the overall is flat. For power, we can see some benefits. In perticular, the stddev indicates the variation is smaller than v3.16-rc7 (all subtests are done at least for 5 times).

To Jason, the format or specific metric is not directly comparable with your tests. If you can specify how to get the same look as your tests, we are happy to try.

Thanks,
Yuyang