perf: Cure the context switch optimization

Hi all,

Aside from the very cheery changelog on the below patch, it does something very weird indeed, I've been staring at it on and off for the past months but can't seem to spot the (probably trivial) mistake.

A kernel with the patch applied offers the following 'feature':

$ taskset 1 perf stat --repeat 10 make kernel -j128

Performance counter stats for 'make O=westmere-build kernel -j128' (10 runs):

        154.128844 task-clock                #    0.101 CPUs utilized            ( +- 97.87% )
               100 context-switches          #    0.001 M/sec                    ( +- 92.92% )
                 0 CPU-migrations            #    0.000 M/sec                    ( +-100.00% )
            17,315 page-faults               #    0.112 M/sec                    ( +- 95.25% )
       449,946,976 cycles                    #    2.919 GHz                      ( +- 97.92% )
       158,294,954 stalled-cycles-frontend   #   35.18% frontend cycles idle     ( +- 97.09% )
        81,976,126 stalled-cycles-backend    #   18.22% backend  cycles idle     ( +- 96.40% )
       607,676,028 instructions              #    1.35  insns per cycle        
                                             #    0.26  stalled cycles per insn  ( +- 98.56% )
       126,479,710 branches                  #  820.610 M/sec                    ( +- 98.49% )
         3,381,265 branch-misses             #    2.67% of all branches          ( +- 97.92% )

1.522814732 seconds time elapsed ( +- 0.19% )

See those totally ridiculous stdev numbers, these are caused by (obtained from perf stat -v output):

$ grep task-clock stat.log
task-clock: 3556075 3556075 3556075
task-clock: 3247072 3247072 3247072
task-clock: 3224954 3224954 3224954
task-clock: 3300664 3300664 3300664
task-clock: 3246115 3246115 3246115
task-clock: 3250695 3250695 3250695
task-clock: 3279908 3279908 3279908
task-clock: 3235312 3235312 3235312
task-clock: 1511715951 1511715951 1511715951
task-clock: 3231690 3231690 3231690

Which suggests there's something going going wrong with the folding back the inherited counter value into the parent counter.

It takes a bit to reproduce on my machine, but Ingo is king at reproducing this and gets results like:

task-clock: 11511929686 11511929686 11511929686
task-clock: 11558456160 11558456160 11558456160
task-clock: 96314987 96314987 96314987
task-clock: 11626224609 11626224609 11626224609
task-clock: 11617826792 11617826792 11617826792
task-clock: 108555025 108555025 108555025
task-clock: 11394346454 11394346454 11394346454
task-clock: 61045813 61045813 61045813
task-clock: 11555885469 11555885469 11555885469
task-clock: 11542959135 11542959135 11542959135

On a very regular basis.

---
Subject: perf: Cure the context switch optimization
From: Peter Zijlstra <[email protected]>
Date: Sat May 21 09:20:36 2011 +0200

The generation counts were mismanaged, this caused us to not do the lazy PMU context switch optimization in many cases.

This caused overhead in heavily context switching workloads.

For example:

$ perf stat perf bench sched pipe
# Running sched/pipe benchmark...
# Executed 1000000 pipe operations between two tasks

Total time: 17.724 [sec]

17.724428 usecs/op

      56419 ops/sec

Performance counter stats for 'perf bench sched pipe':

   9395.469517 task-clock                #    0.530 CPUs utilized
     2,000,043 context-switches          #    0.213 M/sec

Baseline scheduling overhead (without PMU overhead) is:

$ perf bench sched pipe
# Running sched/pipe benchmark...
# Executed 1000000 pipe operations between two tasks

Total time: 6.908 [sec]

6.908246 usecs/op

     144754 ops/sec

With the fix applied the overhead goes away and we are fast again:

$ perf stat perf bench sched pipe
# Running sched/pipe benchmark...
# Executed 1000000 pipe operations between two tasks

Total time: 6.909 [sec]

6.909234 usecs/op

     144733 ops/sec

Performance counter stats for 'perf bench sched pipe':

   9438.132883 task-clock                #    0.531 CPUs utilized
     2,000,042 context-switches          #    0.213 M/sec

---
 include/linux/perf_event.h |    5 ++
 kernel/events/core.c       |   78 +++++++++++++++++++++++++++++++++++++++------
 2 files changed, 73 insertions(+), 10 deletions(-)

Index: linux-2.6/include/linux/perf_event.h
===================================================================
@@ -729,6 +729,11 @@ struct swevent_hlist {
 #define PERF_ATTACH_GROUP	0x02
 #define PERF_ATTACH_TASK	0x04
 
+#define PERF_INHERITED_EVENT	0x08
+#define PERF_COLLECTED_EVENT	0x10
+#define PERF_READ_EVENT		0x20
+
+
 #ifdef CONFIG_CGROUP_PERF
 /*
  * perf_cgroup_info keeps track of time_enabled for a cgroup.