sched: Reduce contention in update_cfs_rq_blocked_load

When running workloads on 2+ socket systems, based on perf profiles, the update_cfs_rq_blocked_load function constantly shows up as taking up a noticeable % of run time. This is especially apparent on an 8 socket machine. For example, when running the AIM7 custom workload, we see:

4.18% reaim [kernel.kallsyms] [k] update_cfs_rq_blocked_load

Much of the contention is in __update_cfs_rq_tg_load_contrib when we update the tg load contribution stats. However, it turns out that in many cases, they don't need to be updated and "tg_contrib" is 0.

This patch adds a check in __update_cfs_rq_tg_load_contrib to skip updating tg load contribution stats when nothing needs to be updated. This reduces the cacheline contention that would be unnecessary. In the above case, with the patch, perf reports the total time spent in this function went down by more than a factor of 3x:

1.18% reaim [kernel.kallsyms] [k] update_cfs_rq_blocked_load

Signed-off-by Jason Low <[email protected]>

---
 kernel/sched/fair.c |    3 +++
 1 files changed, 3 insertions(+), 0 deletions(-)

kernel/sched/fair.c +3 -0

index bfa3c86..8d4cc72 100644
@@ -2377,6 +2377,9 @@ static inline void __update_cfs_rq_tg_load_contrib(struct cfs_rq *cfs_rq,
 	tg_contrib = cfs_rq->runnable_load_avg + cfs_rq->blocked_load_avg;
 	tg_contrib -= cfs_rq->tg_load_contrib;
 
+	if (!tg_contrib)
+		return;
+
 	if (force_update || abs(tg_contrib) > cfs_rq->tg_load_contrib / 8) {
 		atomic_long_add(tg_contrib, &tg->load_avg);
 		cfs_rq->tg_load_contrib += tg_contrib;

Quoted 19 lines · Jason Low

When running workloads on 2+ socket systems, based on perf profiles, the update_cfs_rq_blocked_load function constantly shows up as taking up a noticeable % of run time. This is especially apparent on an 8 socket machine. For example, when running the AIM7 custom workload, we see:

4.18% reaim [kernel.kallsyms] [k] update_cfs_rq_blocked_load

Much of the contention is in __update_cfs_rq_tg_load_contrib when we update the tg load contribution stats. However, it turns out that in many cases, they don't need to be updated and "tg_contrib" is 0.

This patch adds a check in __update_cfs_rq_tg_load_contrib to skip updating tg load contribution stats when nothing needs to be updated. This reduces the cacheline contention that would be unnecessary. In the above case, with the patch, perf reports the total time spent in this function went down by more than a factor of 3x:

1.18% reaim [kernel.kallsyms] [k] update_cfs_rq_blocked_load

Quoted 1 line

Signed-off-by Jason Low <[email protected]>

Reviewed-by Ben Segall <[email protected]>

That said, it might be better to remove force_update for this function, or make it just reduce the minimum to /64 or something. If the test is easy to run it would be good to see what it's like just removing the force_update param for this function to see if it's worth worrying about or if the zero case catches ~all the perf gain.

Paul, your thoughts?

That said, it might be better to remove force_update for this function, or make it just reduce the minimum to /64 or something. If the test is easy to run it would be good to see what it's like just removing the force_update param for this function to see if it's worth worrying about or if the zero case catches ~all the perf gain.

When running workloads on 2+ socket systems, based on perf profiles, the update_cfs_rq_blocked_load function constantly shows up as taking up a noticeable % of run time. This is especially apparent on an 8 socket machine. For example, when running the AIM7 custom workload, we see:

4.18% reaim [kernel.kallsyms] [k] update_cfs_rq_blocked_load

Much of the contention is in __update_cfs_rq_tg_load_contrib when we update the tg load contribution stats. However, it turns out that in many cases, they don't need to be updated and "tg_contrib" is 0.

This patch adds a check in __update_cfs_rq_tg_load_contrib to skip updating tg load contribution stats when nothing needs to be updated. This reduces the cacheline contention that would be unnecessary. In the above case, with the patch, perf reports the total time spent in this function went down by more than a factor of 3x:

1.18% reaim [kernel.kallsyms] [k] update_cfs_rq_blocked_load

When running workloads on 2+ socket systems, based on perf profiles, the update_cfs_rq_blocked_load function constantly shows up as taking up a noticeable % of run time. This is especially apparent on an 8 socket machine. For example, when running the AIM7 custom workload, we see:

4.18% reaim [kernel.kallsyms] [k] update_cfs_rq_blocked_load

Much of the contention is in __update_cfs_rq_tg_load_contrib when we update the tg load contribution stats. However, it turns out that in many cases, they don't need to be updated and "tg_contrib" is 0.

This patch adds a check in __update_cfs_rq_tg_load_contrib to skip updating tg load contribution stats when nothing needs to be updated. This reduces the cacheline contention that would be unnecessary. In the above case, with the patch, perf reports the total time spent in this function went down by more than a factor of 3x:

1.18% reaim [kernel.kallsyms] [k] update_cfs_rq_blocked_load

Signed-off-by Jason Low <[email protected]>

---
 kernel/sched/fair.c |    3 +++
 1 files changed, 3 insertions(+), 0 deletions(-)

kernel/sched/fair.c +3 -0

index bfa3c86..8d4cc72 100644
@@ -2377,6 +2377,9 @@ static inline void __update_cfs_rq_tg_load_contrib(struct cfs_rq *cfs_rq,
 	tg_contrib = cfs_rq->runnable_load_avg + cfs_rq->blocked_load_avg;
 	tg_contrib -= cfs_rq->tg_load_contrib;
 
+	if (!tg_contrib)
+		return;
+
 	if (force_update || abs(tg_contrib) > cfs_rq->tg_load_contrib / 8) {
 		atomic_long_add(tg_contrib, &tg->load_avg);
 		cfs_rq->tg_load_contrib += tg_contrib;

Signature · 9 lines

1.7.1



--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Hi Jason,

I am not sure whether you noticed my latest work: rewriting per entity load average

http://article.gmane.org/gmane.linux.kernel/1760754
http://article.gmane.org/gmane.linux.kernel/1760755
http://article.gmane.org/gmane.linux.kernel/1760757
http://article.gmane.org/gmane.linux.kernel/1760756

which simply does not track blocked load average at all.

Are you interested in
testing the patchset with the workload you have? The comparison can also help us understand the rewrite. Overall, per our tests, the overhead should be less, and perf should be better.

Quoted 48 lines · Jason Low

When running workloads on 2+ socket systems, based on perf profiles, the update_cfs_rq_blocked_load function constantly shows up as taking up a noticeable % of run time. This is especially apparent on an 8 socket machine. For example, when running the AIM7 custom workload, we see:

4.18% reaim [kernel.kallsyms] [k] update_cfs_rq_blocked_load

Much of the contention is in __update_cfs_rq_tg_load_contrib when we update the tg load contribution stats. However, it turns out that in many cases, they don't need to be updated and "tg_contrib" is 0.

This patch adds a check in __update_cfs_rq_tg_load_contrib to skip updating tg load contribution stats when nothing needs to be updated. This reduces the cacheline contention that would be unnecessary. In the above case, with the patch, perf reports the total time spent in this function went down by more than a factor of 3x:

1.18% reaim [kernel.kallsyms] [k] update_cfs_rq_blocked_load

Signed-off-by Jason Low <[email protected]>

---
 kernel/sched/fair.c |    3 +++
 1 files changed, 3 insertions(+), 0 deletions(-)

kernel/sched/fair.c +3 -0

index bfa3c86..8d4cc72 100644
@@ -2377,6 +2377,9 @@ static inline void __update_cfs_rq_tg_load_contrib(struct cfs_rq *cfs_rq,
 	tg_contrib = cfs_rq->runnable_load_avg + cfs_rq->blocked_load_avg;
 	tg_contrib -= cfs_rq->tg_load_contrib;
 
+	if (!tg_contrib)
+		return;
+
 	if (force_update || abs(tg_contrib) > cfs_rq->tg_load_contrib / 8) {
 		atomic_long_add(tg_contrib, &tg->load_avg);
 		cfs_rq->tg_load_contrib += tg_contrib;

Signature · 9 lines

1.7.1



--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Hi Jason,

I am not sure whether you noticed my latest work: rewriting per entity load average

http://article.gmane.org/gmane.linux.kernel/1760754
http://article.gmane.org/gmane.linux.kernel/1760755
http://article.gmane.org/gmane.linux.kernel/1760757
http://article.gmane.org/gmane.linux.kernel/1760756

which simply does not track blocked load average at all. Are you interested in testing the patchset with the workload you have?

The comparison can also help
us understand the rewrite. Overall, per our tests, the overhead should be less, and perf should be better.

Hi Jason,

I am not sure whether you noticed my latest work: rewriting per entity load average

http://article.gmane.org/gmane.linux.kernel/1760754
http://article.gmane.org/gmane.linux.kernel/1760755
http://article.gmane.org/gmane.linux.kernel/1760757
http://article.gmane.org/gmane.linux.kernel/1760756

which simply does not track blocked load average at all. Are you interested in testing the patchset with the workload you have?

I ran these tests with most of the AIM7 workloads to compare its performance between a 3.16 kernel and the kernel with these patches applied.

The table below contains the percent difference between the baseline kernel and the kernel with the patches at various user counts. A positive percent means the kernel with the patches performed better, while a negative percent means the baseline performed better.

Based on these numbers, for many of the workloads, the change was beneficial in those highly contended, while it had - impact in many of the lightly/moderately contended case (10 to 90 users).

-----------------------------------------------------

              |   10-90   |  100-1000   |  1100-2000
              |   users   |   users     |   users

-----------------------------------------------------
alltests | -3.37% | -10.64% | -2.25%
-----------------------------------------------------
all_utime | +0.33% | +3.73% | +3.33%
-----------------------------------------------------
compute | -5.97% | +2.34% | +3.22%
-----------------------------------------------------
custom | -31.61% | -10.29% | +15.23%
-----------------------------------------------------
disk | +24.64% | +28.96% | +21.28%
-----------------------------------------------------

fserver | -1.35% | +4.82% | +9.35%
-----------------------------------------------------
high_systime | -6.73% | -6.28% | +12.36%
-----------------------------------------------------
shared | -28.31% | -19.99% | -7.10%
-----------------------------------------------------
short | -44.63% | -37.48% | -33.62%
-----------------------------------------------------

Quoted 27 lines · Jason Low

I ran these tests with most of the AIM7 workloads to compare its performance between a 3.16 kernel and the kernel with these patches applied.

The table below contains the percent difference between the baseline kernel and the kernel with the patches at various user counts. A positive percent means the kernel with the patches performed better, while a negative percent means the baseline performed better.

Based on these numbers, for many of the workloads, the change was beneficial in those highly contended, while it had - impact in many of the lightly/moderately contended case (10 to 90 users).

-----------------------------------------------------

              |   10-90   |  100-1000   |  1100-2000
              |   users   |   users     |   users

-----------------------------------------------------
alltests | -3.37% | -10.64% | -2.25%
-----------------------------------------------------
all_utime | +0.33% | +3.73% | +3.33%
-----------------------------------------------------
compute | -5.97% | +2.34% | +3.22%
-----------------------------------------------------
custom | -31.61% | -10.29% | +15.23%
-----------------------------------------------------
disk | +24.64% | +28.96% | +21.28%
-----------------------------------------------------

Quoted 9 lines

fserver | -1.35% | +4.82% | +9.35%
-----------------------------------------------------
high_systime | -6.73% | -6.28% | +12.36%
-----------------------------------------------------
shared | -28.31% | -19.99% | -7.10%
-----------------------------------------------------
short | -44.63% | -37.48% | -33.62%
-----------------------------------------------------

Thanks, Jason. Sorry for late response.

What about the variation of the tests? The machine you test on?

Quoted 39 lines · Yuyang Du

Quoted 27 lines · Jason Low

I ran these tests with most of the AIM7 workloads to compare its performance between a 3.16 kernel and the kernel with these patches applied.

The table below contains the percent difference between the baseline kernel and the kernel with the patches at various user counts. A positive percent means the kernel with the patches performed better, while a negative percent means the baseline performed better.

Based on these numbers, for many of the workloads, the change was beneficial in those highly contended, while it had - impact in many of the lightly/moderately contended case (10 to 90 users).

-----------------------------------------------------

              |   10-90   |  100-1000   |  1100-2000
              |   users   |   users     |   users

-----------------------------------------------------
alltests | -3.37% | -10.64% | -2.25%
-----------------------------------------------------
all_utime | +0.33% | +3.73% | +3.33%
-----------------------------------------------------
compute | -5.97% | +2.34% | +3.22%
-----------------------------------------------------
custom | -31.61% | -10.29% | +15.23%
-----------------------------------------------------
disk | +24.64% | +28.96% | +21.28%
-----------------------------------------------------

Quoted 9 lines

fserver | -1.35% | +4.82% | +9.35%
-----------------------------------------------------
high_systime | -6.73% | -6.28% | +12.36%
-----------------------------------------------------
shared | -28.31% | -19.99% | -7.10%
-----------------------------------------------------
short | -44.63% | -37.48% | -33.62%
-----------------------------------------------------

Thanks, Jason. Sorry for late response.

Quoted 1 line

What about the variation of the tests? The machine you test on?

Hi Yuyang,

These tests were also done on an 8 socket machine (80 cores). In terms of variation between the average throughputs, typically the noise range is about 2% in many of the workloads.

Quoted 14 lines

Quoted 14 lines

Quoted 14 lines

-----------------------------------------------------

              |   10-90   |  100-1000   |  1100-2000
              |   users   |   users     |   users

-----------------------------------------------------
alltests | -3.37% | -10.64% | -2.25%
-----------------------------------------------------
all_utime | +0.33% | +3.73% | +3.33%
-----------------------------------------------------
compute | -5.97% | +2.34% | +3.22%
-----------------------------------------------------
custom | -31.61% | -10.29% | +15.23%
-----------------------------------------------------
disk | +24.64% | +28.96% | +21.28%
-----------------------------------------------------

Quoted 8 lines

Quoted 8 lines

Quoted 8 lines

fserver | -1.35% | +4.82% | +9.35%
-----------------------------------------------------
high_systime | -6.73% | -6.28% | +12.36%
-----------------------------------------------------
shared | -28.31% | -19.99% | -7.10%
-----------------------------------------------------
short | -44.63% | -37.48% | -33.62%
-----------------------------------------------------

Thanks a lot, Jason.

So for this particular set of workloads on a big machine, I think the result is mixed and overall "neutral", but I expected the variation probably could be bigger especially for light workloads.

Any comment from the maintainers and others? Ping Peter and Ben, I haven't heard from you for the 5th version.

Quoted 29 lines · Yuyang Du

Quoted 14 lines

Quoted 14 lines

Quoted 14 lines

-----------------------------------------------------

              |   10-90   |  100-1000   |  1100-2000
              |   users   |   users     |   users

-----------------------------------------------------
alltests | -3.37% | -10.64% | -2.25%
-----------------------------------------------------
all_utime | +0.33% | +3.73% | +3.33%
-----------------------------------------------------
compute | -5.97% | +2.34% | +3.22%
-----------------------------------------------------
custom | -31.61% | -10.29% | +15.23%
-----------------------------------------------------
disk | +24.64% | +28.96% | +21.28%
-----------------------------------------------------

Quoted 8 lines

Quoted 8 lines

Quoted 8 lines

fserver | -1.35% | +4.82% | +9.35%
-----------------------------------------------------
high_systime | -6.73% | -6.28% | +12.36%
-----------------------------------------------------
shared | -28.31% | -19.99% | -7.10%
-----------------------------------------------------
short | -44.63% | -37.48% | -33.62%
-----------------------------------------------------

Thanks a lot, Jason.

So for this particular set of workloads on a big machine, I think the result is mixed and overall "neutral", but I expected the variation probably could be bigger especially for light workloads.

Quoted 2 lines

Any comment from the maintainers and others? Ping Peter and Ben, I haven't heard from you for the 5th version.

Been a bit busy.. but in general I worry about the performance decrease on the lighter loads. I should probably run some workloads on my 2 socket and 4 socket machines, but the coming few weeks will be very busy and I'm afraid I'll not get around to it in a timely manner.

Quoted 30 lines · Peter Zijlstra

Quoted 29 lines · Yuyang Du

Quoted 14 lines

Quoted 14 lines

Quoted 14 lines

-----------------------------------------------------

              |   10-90   |  100-1000   |  1100-2000
              |   users   |   users     |   users

-----------------------------------------------------
alltests | -3.37% | -10.64% | -2.25%
-----------------------------------------------------
all_utime | +0.33% | +3.73% | +3.33%
-----------------------------------------------------
compute | -5.97% | +2.34% | +3.22%
-----------------------------------------------------
custom | -31.61% | -10.29% | +15.23%
-----------------------------------------------------
disk | +24.64% | +28.96% | +21.28%
-----------------------------------------------------

Quoted 8 lines

Quoted 8 lines

Quoted 8 lines

fserver | -1.35% | +4.82% | +9.35%
-----------------------------------------------------
high_systime | -6.73% | -6.28% | +12.36%
-----------------------------------------------------
shared | -28.31% | -19.99% | -7.10%
-----------------------------------------------------
short | -44.63% | -37.48% | -33.62%
-----------------------------------------------------

Thanks a lot, Jason.

So for this particular set of workloads on a big machine, I think the result is mixed and overall "neutral", but I expected the variation probably could be bigger especially for light workloads.

Quoted 7 lines

Quoted 2 lines

Any comment from the maintainers and others? Ping Peter and Ben, I haven't heard from you for the 5th version.

Been a bit busy.. but in general I worry about the performance decrease on the lighter loads. I should probably run some workloads on my 2 socket and 4 socket machines, but the coming few weeks will be very busy and I'm afraid I'll not get around to it in a timely manner.

Ok. I understand. From our part, Fengguang's LKP does not include light loads, we also need some such tests to confirm this and see what is next.

Since typical benchmarks would be heavy ones, what do you suggest for light loads?

Jason, possible you can share some of your workloads?

When running workloads on 2+ socket systems, based on perf profiles, the update_cfs_rq_blocked_load function constantly shows up as taking up a noticeable % of run time. This is especially apparent on an 8 socket machine. For example, when running the AIM7 custom workload, we see:

4.18% reaim [kernel.kallsyms] [k] update_cfs_rq_blocked_load

Much of the contention is in __update_cfs_rq_tg_load_contrib when we update the tg load contribution stats. However, it turns out that in many cases, they don't need to be updated and "tg_contrib" is 0.

This patch adds a check in __update_cfs_rq_tg_load_contrib to skip updating tg load contribution stats when nothing needs to be updated. This reduces the cacheline contention that would be unnecessary. In the above case, with the patch, perf reports the total time spent in this function went down by more than a factor of 3x:

1.18% reaim [kernel.kallsyms] [k] update_cfs_rq_blocked_load

Signed-off-by Jason Low<[email protected]>

---
  kernel/sched/fair.c |    3 +++
  1 files changed, 3 insertions(+), 0 deletions(-)

kernel/sched/fair.c +0 -0

index bfa3c86..8d4cc72 100644