Skip to contentSkip to conversation

8ad8e9d3fd6: -7.5% aim7.2000.jobs-per-min, -45.7% turbostat.%c1

uprobes/x86 Jet Chen 3 participants 3 messages May 15, 2014

lore
Share & tools
lore

Jet Chen

author
#0 lore raw

Hi Oleg,

we noticed the below changes on

git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git master
commit 8ad8e9d3fd64f101eed6652964670672d699e563 ("uprobes/x86: Introduce uprobe_xol_ops and arch_uprobe->ops")

Test case: lkp-snb01/aim7/signal_test

34e7317d6ae8f61 8ad8e9d3fd64f101eed665296
--------------- -------------------------

    230689 ~ 0%      -7.5%     213485 ~ 0%  TOTAL aim7.2000.jobs-per-min
      0.51 ~30%     -45.7%       0.28 ~10%  TOTAL turbostat.%c1
       430 ~17%     +27.8%        549 ~15%  TOTAL vmstat.procs.r
      0.83 ~14%     +38.1%       1.15 ~12%  TOTAL perf-profile.cpu-cycles.copy_pte_range.copy_page_range.copy_process.do_fork.sys_clone
    106076 ~ 4%     +22.4%     129816 ~ 3%  TOTAL softirqs.RCU
     12117 ~ 6%      -9.9%      10914 ~ 6%  TOTAL slabinfo.kmalloc-256.active_objs
     32163 ~17%     +39.4%      44824 ~12%  TOTAL time.voluntary_context_switches
    276487 ~ 1%     +16.5%     322091 ~ 1%  TOTAL time.involuntary_context_switches
     83.14 ~ 0%     +13.3%      94.21 ~ 0%  TOTAL time.user_time
    108800 ~ 2%      +9.4%     119014 ~ 3%  TOTAL time.minor_page_faults
      1255 ~ 0%      +9.9%       1379 ~ 0%  TOTAL time.system_time
      6774 ~ 1%      +8.8%       7373 ~ 1%  TOTAL vmstat.system.cs
     52.15 ~ 0%      +8.0%      56.34 ~ 0%  TOTAL time.elapsed_time
     25185 ~ 0%      +2.4%      25784 ~ 0%  TOTAL vmstat.system.in
     79.23 ~ 0%      +2.0%      80.80 ~ 0%  TOTAL turbostat.%c0
      2567 ~ 0%      +1.9%       2615 ~ 0%  TOTAL time.percent_of_cpu_this_job_got
       112 ~ 0%      +1.3%        113 ~ 0%  TOTAL turbostat.Cor_W
       138 ~ 0%      +1.0%        139 ~ 0%  TOTAL turbostat.Pkg_W

Legend:
~XX% - stddev percent
[+-]XX% - change percent

Thanks,
Jet

echo performance > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu1/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu10/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu11/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu12/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu13/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu14/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu15/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu16/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu17/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu18/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu19/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu2/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu20/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu21/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu22/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu23/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu24/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu25/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu26/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu27/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu28/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu29/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu3/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu30/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu31/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu4/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu5/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu6/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu7/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu8/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu9/cpufreq/scaling_governor

#1 lore raw

Hi Jet,

Quoted 20 lines · Jet Chen

we noticed the below changes on

git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git master
commit 8ad8e9d3fd64f101eed6652964670672d699e563 ("uprobes/x86: Introduce uprobe_xol_ops and arch_uprobe->ops")

Test case: lkp-snb01/aim7/signal_test

34e7317d6ae8f61 8ad8e9d3fd64f101eed665296
--------------- -------------------------

    230689 ~ 0%      -7.5%     213485 ~ 0%  TOTAL aim7.2000.jobs-per-min
      0.51 ~30%     -45.7%       0.28 ~10%  TOTAL turbostat.%c1
       430 ~17%     +27.8%        549 ~15%  TOTAL vmstat.procs.r
      0.83 ~14%     +38.1%       1.15 ~12%  TOTAL perf-profile.cpu-cycles.copy_pte_range.copy_page_range.copy_process.do_fork.sys_clone
    106076 ~ 4%     +22.4%     129816 ~ 3%  TOTAL softirqs.RCU
     12117 ~ 6%      -9.9%      10914 ~ 6%  TOTAL slabinfo.kmalloc-256.active_objs
     32163 ~17%     +39.4%      44824 ~12%  TOTAL time.voluntary_context_switches
    276487 ~ 1%     +16.5%     322091 ~ 1%  TOTAL time.involuntary_context_switches
     83.14 ~ 0%     +13.3%      94.21 ~ 0%  TOTAL time.user_time
    108800 ~ 2%      +9.4%     119014 ~ 3%  TOTAL time.minor_page_faults
Quoted 8 lines
      1255 ~ 0%      +9.9%       1379 ~ 0%  TOTAL time.system_time
      6774 ~ 1%      +8.8%       7373 ~ 1%  TOTAL vmstat.system.cs
     52.15 ~ 0%      +8.0%      56.34 ~ 0%  TOTAL time.elapsed_time
     25185 ~ 0%      +2.4%      25784 ~ 0%  TOTAL vmstat.system.in
     79.23 ~ 0%      +2.0%      80.80 ~ 0%  TOTAL turbostat.%c0
      2567 ~ 0%      +1.9%       2615 ~ 0%  TOTAL time.percent_of_cpu_this_job_got
       112 ~ 0%      +1.3%        113 ~ 0%  TOTAL turbostat.Cor_W
       138 ~ 0%      +1.0%        139 ~ 0%  TOTAL turbostat.Pkg_W

Cough... this looks simply impossible ;)

Not only this patch "obviously can not" cause any noticable difference performance-wise, the code changed by this patch is not executed unless you play with uprobes?

Oleg.

#2 lore raw

Hi Oleg,

Quoted 32 lines · Oleg Nesterov

Hi Jet,

Quoted 20 lines · Jet Chen

we noticed the below changes on

git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git master
commit 8ad8e9d3fd64f101eed6652964670672d699e563 ("uprobes/x86: Introduce uprobe_xol_ops and arch_uprobe->ops")

Test case: lkp-snb01/aim7/signal_test

34e7317d6ae8f61 8ad8e9d3fd64f101eed665296
--------------- -------------------------

    230689 ~ 0%      -7.5%     213485 ~ 0%  TOTAL aim7.2000.jobs-per-min
      0.51 ~30%     -45.7%       0.28 ~10%  TOTAL turbostat.%c1
       430 ~17%     +27.8%        549 ~15%  TOTAL vmstat.procs.r
      0.83 ~14%     +38.1%       1.15 ~12%  TOTAL perf-profile.cpu-cycles.copy_pte_range.copy_page_range.copy_process.do_fork.sys_clone
    106076 ~ 4%     +22.4%     129816 ~ 3%  TOTAL softirqs.RCU
     12117 ~ 6%      -9.9%      10914 ~ 6%  TOTAL slabinfo.kmalloc-256.active_objs
     32163 ~17%     +39.4%      44824 ~12%  TOTAL time.voluntary_context_switches
    276487 ~ 1%     +16.5%     322091 ~ 1%  TOTAL time.involuntary_context_switches
     83.14 ~ 0%     +13.3%      94.21 ~ 0%  TOTAL time.user_time
    108800 ~ 2%      +9.4%     119014 ~ 3%  TOTAL time.minor_page_faults
Quoted 8 lines
      1255 ~ 0%      +9.9%       1379 ~ 0%  TOTAL time.system_time
      6774 ~ 1%      +8.8%       7373 ~ 1%  TOTAL vmstat.system.cs
     52.15 ~ 0%      +8.0%      56.34 ~ 0%  TOTAL time.elapsed_time
     25185 ~ 0%      +2.4%      25784 ~ 0%  TOTAL vmstat.system.in
     79.23 ~ 0%      +2.0%      80.80 ~ 0%  TOTAL turbostat.%c0
      2567 ~ 0%      +1.9%       2615 ~ 0%  TOTAL time.percent_of_cpu_this_job_got
       112 ~ 0%      +1.3%        113 ~ 0%  TOTAL turbostat.Cor_W
       138 ~ 0%      +1.0%        139 ~ 0%  TOTAL turbostat.Pkg_W
Quoted 5 lines

Cough... this looks simply impossible ;)

Not only this patch "obviously can not" cause any noticable difference performance-wise, the code changed by this patch is not executed unless you play with uprobes?

Yes this is interesting. From bisect POV, we do find kernels before/after this commit have some clearly different behavior, as you can see in some of the below graphs. For example, the "time.user_time" numbers are clearly separable before/after the first bad commit.

Legend:

[*] bisect-good sample
[O] bisect-bad sample

time.user_time

98 ++--------------------------------O------------------------------------+

     |     O   O O O O O   O     O   O    O   O                 O           |
  96 O+O O   O           O   O O   O        O                     O         |
  94 ++                                           O O O O O   O       O O   O
     |                                          O           O       O     O |
  92 ++                                                                     |
     |                                                                      |
  90 ++                                                                     |
     |                                                                      |
  88 ++                                                                     |
  86 ++                                                                     |
     |              .*                                                      |
  84 *+ .*.   .*. .*  +                                                     |
     | *   *.*   *     *.*.*.*.*.*.*.*.*..*.*                               |

82 ++---------------------------------------------------------------------+

time.system_time

1420 ++-------------------------------------------------------------------+

       |                                                        O           |
  1400 O+O O O O                                                  O         |
  1380 ++                                OO     O O         O       O O O O O
       |         O O   O O O   O O O O O            O O O O   O             |
  1360 ++            O                                                      |
  1340 ++                    O              O O                             |
       |                                                                    |
  1320 ++                                                                   |
  1300 ++                                                                   |
       |                                                                    |
  1280 ++                                                                   |
  1260 ++            *.                                                     |
       *.*.*.*.*.   +  *.*.*.*.*.*.*.*.*.**.*                               |

1240 ++--------*-*--------------------------------------------------------+

time.percent_of_cpu_this_job_got

2630 ++-------------------------------------------------------------------+

       |                                                        O O         |
  2620 O+O O O O                         OO                           O O   O
  2610 ++        O O     O     O O O O O        O O O O O O O O     O     O |
       |             O O   O O                O                             |
  2600 ++                                   O                               |
       |                                                                    |
  2590 ++                                                                   |
       |                                                                    |
  2580 ++                                                                   |
  2570 ++                                                                   |
       |            .*.*.*.*.*.*. .*.*.*.**.*                               |
  2560 *+*.*.*.*.*.*             *                                          |
       |                                                                    |

2550 ++-------------------------------------------------------------------+

time.elapsed_time

58 ++---------------------------------------------------------------------+

     |                                                                      |
  57 ++O O O                                                    O O         |
     O       O                         O  O                 O       O O O O O
  56 ++        O O   O O O   O O O O O          O O O O O O   O             |
     |             O       O                O O                             |
  55 ++                                                                     |
     |                                                                      |
  54 ++                                                                     |
     |                                                                      |
  53 ++                                                                     |
     |            .*.*.                                                     |
  52 *+*.*.*.*.*.*     *.*.*.*.*.*.*.*.*..*.*                               |
     |                                                                      |

51 ++---------------------------------------------------------------------+

time.involuntary_context_switches

340000 ++-----------------------------------------------------------------+
330000 ++ O O OO O O |
O O O O O O O O O |
320000 ++ O O O O O O

  310000 ++  O   O  O   O O     O                      O   O                |
  300000 ++O                                   O               O            |
  290000 ++                 O                O               O              |
         |            O         *      .*                                   |
  280000 ++           *.    *  : :   .*  + *.                               |
  270000 ++          :  *  : + : : .*     *  *                              |
  260000 ++       .* :   + :  *   *                                         |
  250000 ++   .*.*  *     *                                                 |
         |.*.*                                                              |
  240000 *+                                                                 |

230000 ++-----------------------------------------------------------------+

softirqs.RCU

140000 ++-------------------------O---O-----------------------------------+

         |                    O                        O   O          O     |
  130000 ++                                      O                  O     O |
         O                                 O                                O
         |   O   O                                 O              O     O   |
  120000 ++    O   O    O       O                    O   O   O              |
         |          O     O       O   O   O    O                 O          |
  110000 ++                 O   *.   .*. .**                                |
         | O          O        :  *.*   *   :                  O            |
  100000 ++             *      :            :O                              |
         |     *.*  *. + + .*.*              *                              |
         |    :   + : *   *                                                 |
   90000 ++*. :    *                                                        |
         |+  *                                                              |

80000 *+-----------------------------------------------------------------+

                               aim7.2000.jobs-per-min

235000 ++-----------------------------------------------------------------+

         |                                                                  |
         *.*.*.*.*.**.   .*.*. .*.*.*.*.*. *.*                              |
  230000 ++           *.*     *           *                                 |
         |                                                                  |
         |                                                                  |
  225000 ++                                                                 |
         |                                                                  |
  220000 ++                                                                 |
         |                                                                  |
         |                    O              O O                            |
  215000 ++        OO O O O O   O                                           |
         |                        O O O O        O O O O O O O O    O O O O O
         O O O O O                        OO                      O         |

210000 ++------------------------------------------------------O----------+

vmstat.system.in

26200 ++O-----------------------------------------------------------------+

        O   O O O                                                 O         |
  26000 ++          O    O     O         O O                                |
        |         O                                             O           |
  25800 ++                 O       O   O                            O O O O O
        |             O O    O   O   O       O   O O O   O  O               |
  25600 ++                                     O       O   O  O             |
        |                  *        .*                                      |
  25400 ++*. .*.*. .*      :       *  :                                     |
        | : *     *  +    : :      :  :                                     |
  25200 ++            *.* : :     :    :.*.                                 |
        |:               :   :.*. :    *   *.*                              |
  25000 ++               *   *   *                                          |
        *                                                                   |

24800 ++------------------------------------------------------------------+

turbostat.%c0

81.5 ++-------------------------------------------------------------------+

       |                                                                    |
    81 ++                                                       O O         |
       O O O O O                 O O O O OO     O O                   O O O O
       |         O O   O O O   O                    O O O O O O     O       |
  80.5 ++            O       O              O O                             |
       |                                                                    |
    80 ++                                                                   |
       |                                                                    |
  79.5 ++                                                                   |
       |             *.       .*.   .*.   *.                                |
       *. .*. .*.*. +  *.*.*.*   *.*   *.*  *                               |
    79 ++*   *     *                                                        |
       |                                                                    |

78.5 ++-------------------------------------------------------------------+

turbostat.Pkg_W

141 ++------------------------------------------------------------------+
| O |

  140.5 ++                                             O   O                |
    140 ++    O              O   O O O O O O                O   O           |
        O O O   O O O O        O               O O            O   O     O   |
  139.5 ++              O  O         *             O O   O            O   O O
        |                O          : :                             O       |
    139 ++                          : :                                     |
        |                  *      .*   :                                    |
  138.5 ++                + :  *.*     *.                                   |
    138 ++*.*.*      .*. *  : +          *                                  |
        *      + .*.*   *    *            +                                 |
  137.5 ++      *                          *.*                              |
        |                                                                   |
    137 ++------------------------------------------------------------------+


                                   turbostat.Cor_W

114.5 ++-----------------------------------O------------------------------+

        |                                              O   O                |
    114 ++                                                      O           |
        | O O O                  O   O   O O     O          O     O     O   |
        O       O O O        O     O   O       O     O   O    O     O     O O
  113.5 ++            O O  O   O                   O                  O     |
        |                O           *                                      |
    113 ++                           ::                                     |
        |                  *        : :                                     |
  112.5 ++                : :    *. :  :                                    |
        |                 : :   +  *   *.                                   |
        |     *          *   :.*         *                                  |
    112 *+   + :     .*. :   *            +                                 |
        | *.*  : .*.*   *                  *.*                              |

111.5 ++------*-----------------------------------------------------------+

proc-vmstat.numa_pte_updates

7000 ++--O----------------------------------------------------------------+

       |                                                                    |
  6000 ++O                                                                  |
       |       O                                                  O         |
  5000 ++            O                                                      |
       |                   O                          O   O               O |
  4000 ++                                     O             O               |
       O                                 O                    O             |
  3000 ++    O                                                              |
       |                                  O                     O   O       |
  2000 ++                *   * O * O * O                              O     |
       |         * O    :O: : : :O+ +O:     O     O O   O               O   |
  1000 ++       +O+ .*.O: : :O: :  *  : .*      O                           O
       |  .*. .*   *   *   *   *       *  *.                                |
     0 *+*---*------------------------------*-------------------------------+


                           proc-vmstat.numa_hint_faults

2500 ++-------------------------------------------------------------------+

       |                                                                    |
       |   O                                                                |
  2000 ++O                                                                  |
       |       O                                                  O         |
       |             O                                                      |
  1500 ++                  O                          O   O O             O |
       O                                 O    O                             |
  1000 ++                                                     O             |
       |     O                            O                     O   O       |
       |                 *   * O   O   O                                    |
   500 ++        * O    :O: : :  O. .O      O     O O   O             O O   |
       |        +O+    O: : :O: +  *  + .*      O                           O
       |  .*. .*   *.*.*   *   *       *  *.                                |
     0 *+*---*------------------------------*-------------------------------+

Thanks,
Fengguang