Skip to contentSkip to conversation

sched: Move sched_entity::avg into separate cache line

PATCH In review Jiri Olsa 3 participants 3 messages Dec 8, 2015 – Jan 6, 2016 Scheduler

lore
Share & tools
lore

Jiri Olsa

author
#0 lore raw

PATCH 1/1 sched: Move sched_entity::avg into separate cache line

From: root <[email protected]>

hi,
I tried Joe's and Don's c2c tool and it identified a place with cache line contention. There're more that poped up, this one was just too obvious ;-)

thanks
jirka

---
The sched_entity::avg collides with read-mostly sched_entity data.

The perf c2c tool showed many read HITM accesses across many CPUs for sched_entity's cfs_rq and my_q, while having at the same time tons of stores for avg.

After placing sched_entity::avg into separate cache line, the perf bench sched pipe showed around 20 seconds speedup.

NOTE I cut out all perf events except for cycles and instructions from following output.

Before:

  $ perf stat -r 5 perf bench sched pipe -l 10000000
  # Running 'sched/pipe' benchmark:
  # Executed 10000000 pipe operations between two processes
       Total time: 270.348 [sec]

        27.034805 usecs/op
            36989 ops/sec

...

     245,537,074,035      cycles                    #    1.433 GHz
     187,264,548,519      instructions              #    0.77  insns per cycle
       272.653840535 seconds time elapsed           ( +-  1.31% )

After:

  $ perf stat -r 5 perf bench sched pipe -l 10000000
  # Running 'sched/pipe' benchmark:
  # Executed 10000000 pipe operations between two processes
       Total time: 251.076 [sec]

        25.107678 usecs/op
            39828 ops/sec

...

     244,573,513,928      cycles                    #    1.572 GHz
     187,409,641,157      instructions              #    0.76  insns per cycle
       251.679315188 seconds time elapsed           ( +-  0.31% )
Signed-off-by Jiri Olsa <[email protected]>
---
 include/linux/sched.h | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)
include/linux/sched.h +7 -2
index 3b0de68bce41..80cc1432e6e3 100644
@@ -1268,8 +1268,13 @@ struct sched_entity {
 #endif
 
 #ifdef CONFIG_SMP
-	/* Per entity load average tracking */
-	struct sched_avg	avg;
+	/*
+	 * Per entity load average tracking.
+	 *
+	 * Put into separate cache line so it does not
+	 * collide with read-mostly values above.
+	 */
+	struct sched_avg	avg ____cacheline_aligned_in_smp;
 #endif
 };
Signature · 1 line
2.4.3
#1 lore raw

Em Tue, Dec 08, 2015 at 09:23:59PM +0100, Jiri Olsa escreveu:

Quoted 6 lines

From: root <[email protected]>

hi,
I tried Joe's and Don's c2c tool and it identified a place with cache line contention. There're more that poped up, this one was just too obvious ;-)

Would be interesting to see output from that tool and how you have interpreted it to be for that specific data structure.

- Arnaldo

Quoted 76 lines

thanks
jirka

---
The sched_entity::avg collides with read-mostly sched_entity data.

The perf c2c tool showed many read HITM accesses across many CPUs for sched_entity's cfs_rq and my_q, while having at the same time tons of stores for avg.

After placing sched_entity::avg into separate cache line, the perf bench sched pipe showed around 20 seconds speedup.

NOTE I cut out all perf events except for cycles and instructions from following output.

Before:

  $ perf stat -r 5 perf bench sched pipe -l 10000000
  # Running 'sched/pipe' benchmark:
  # Executed 10000000 pipe operations between two processes
       Total time: 270.348 [sec]

        27.034805 usecs/op
            36989 ops/sec

...

     245,537,074,035      cycles                    #    1.433 GHz
     187,264,548,519      instructions              #    0.77  insns per cycle
       272.653840535 seconds time elapsed           ( +-  1.31% )

After:

  $ perf stat -r 5 perf bench sched pipe -l 10000000
  # Running 'sched/pipe' benchmark:
  # Executed 10000000 pipe operations between two processes
       Total time: 251.076 [sec]

        25.107678 usecs/op
            39828 ops/sec

...

     244,573,513,928      cycles                    #    1.572 GHz
     187,409,641,157      instructions              #    0.76  insns per cycle
       251.679315188 seconds time elapsed           ( +-  0.31% )
Signed-off-by Jiri Olsa <[email protected]>
---
 include/linux/sched.h | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)
include/linux/sched.h +7 -2
index 3b0de68bce41..80cc1432e6e3 100644
@@ -1268,8 +1268,13 @@ struct sched_entity {
 #endif
 
 #ifdef CONFIG_SMP
-	/* Per entity load average tracking */
-	struct sched_avg	avg;
+	/*
+	 * Per entity load average tracking.
+	 *
+	 * Put into separate cache line so it does not
+	 * collide with read-mostly values above.
+	 */
+	struct sched_avg	avg ____cacheline_aligned_in_smp;
 #endif
 };
Signature · 1 line
2.4.3
merge bot
#2 lore raw

[tip:sched/core] sched/core: Move sched_entity:: avg into separate cache line

In reply to Jiri Olsa
sched/core: Move sched_entity::avg into separate cache line
Commit-ID:  5a1078043f844074cbd53981432778a8d5dd56e9
Gitweb:     http://git.kernel.org/tip/5a1078043f844074cbd53981432778a8d5dd56e9
Author:     Jiri Olsa <[email protected]>
AuthorDate: Tue, 8 Dec 2015 21:23:59 +0100
Committer:  Ingo Molnar <[email protected]>
CommitDate: Wed, 6 Jan 2016 11:06:14 +0100

sched/core: Move sched_entity::avg into separate cache line

The sched_entity::avg collides with read-mostly sched_entity data.

The perf c2c tool showed many read HITM accesses across many CPUs for sched_entity's cfs_rq and my_q, while having at the same time tons of stores for avg.

After placing sched_entity::avg into separate cache line, the perf bench sched pipe showed around 20 seconds speedup.

NOTE I cut out all perf events except for cycles and instructions from following output.

Before:

  $ perf stat -r 5 perf bench sched pipe -l 10000000
  # Running 'sched/pipe' benchmark:
  # Executed 10000000 pipe operations between two processes
       Total time: 270.348 [sec]

        27.034805 usecs/op
            36989 ops/sec

...

     245,537,074,035      cycles                    #    1.433 GHz
     187,264,548,519      instructions              #    0.77  insns per cycle
       272.653840535 seconds time elapsed           ( +-  1.31% )

After:

  $ perf stat -r 5 perf bench sched pipe -l 10000000
  # Running 'sched/pipe' benchmark:
  # Executed 10000000 pipe operations between two processes
       Total time: 251.076 [sec]

        25.107678 usecs/op
            39828 ops/sec

...

     244,573,513,928      cycles                    #    1.572 GHz
     187,409,641,157      instructions              #    0.76  insns per cycle
       251.679315188 seconds time elapsed           ( +-  0.31% )
Signed-off-by Jiri Olsa <[email protected]>
Signed-off-by Peter Zijlstra (Intel) <[email protected]>
Cc Arnaldo Carvalho de Melo <[email protected]>
Cc Don Zickus <[email protected]>
Cc Joe Mario <[email protected]>
Cc Linus Torvalds <[email protected]>
Cc Mike Galbraith <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Thomas Gleixner <[email protected]>
Link http://lkml.kernel.org/r/[email protected] open here
Signed-off-by Ingo Molnar <[email protected]>
---
 include/linux/sched.h | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)
include/linux/sched.h +7 -2
index 791b47e..0c0e781 100644
@@ -1268,8 +1268,13 @@ struct sched_entity {
 #endif
 
 #ifdef CONFIG_SMP
-	/* Per entity load average tracking */
-	struct sched_avg	avg;
+	/*
+	 * Per entity load average tracking.
+	 *
+	 * Put into separate cache line so it does not
+	 * collide with read-mostly values above.
+	 */
+	struct sched_avg	avg ____cacheline_aligned_in_smp;
 #endif
 };