Skip to contentSkip to conversation

b84c4e08143: +3.1% will-it-scale.per_thread_ops

sched,rcu Fengguang Wu 2 participants 4 messages Apr 17, 2014 – Apr 22, 2014

lore
Share & tools
lore
#0 lore raw

Hi Paul,

FYI, this improves will-it-scale/open1 throughput.

git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git dev.2014.04.14a
commit b84c4e08143c98dad4b4d139f08db0b98b0d3ec4 ("sched,rcu: Make cond_resched() report RCU quiescent states")

ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------

    563496 ~ 0%      +3.1%     581059 ~ 0%  nhm4/micro/will-it-scale/open1
    563496 ~ 0%      +3.1%     581059 ~ 0%  TOTAL will-it-scale.per_thread_ops

ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------

    756894 ~ 0%      +2.8%     778452 ~ 0%  nhm4/micro/will-it-scale/open1
    756894 ~ 0%      +2.8%     778452 ~ 0%  TOTAL will-it-scale.per_process_ops

ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------

      0.57 ~ 0%      -2.7%       0.55 ~ 0%  nhm4/micro/will-it-scale/open1
      0.57 ~ 0%      -2.7%       0.55 ~ 0%  TOTAL will-it-scale.scalability

ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------

    346764 ~ 2%     -74.0%      90164 ~ 1%  nhm4/micro/will-it-scale/open1
    346764 ~ 2%     -74.0%      90164 ~ 1%  TOTAL slabinfo.kmalloc-256.active_objs

ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------

     10837 ~ 2%     -73.9%       2824 ~ 1%  nhm4/micro/will-it-scale/open1
     10837 ~ 2%     -73.9%       2824 ~ 1%  TOTAL slabinfo.kmalloc-256.active_slabs

ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------

     10837 ~ 2%     -73.9%       2824 ~ 1%  nhm4/micro/will-it-scale/open1
     10837 ~ 2%     -73.9%       2824 ~ 1%  TOTAL slabinfo.kmalloc-256.num_slabs

ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------

    346821 ~ 2%     -73.9%      90393 ~ 1%  nhm4/micro/will-it-scale/open1
    346821 ~ 2%     -73.9%      90393 ~ 1%  TOTAL slabinfo.kmalloc-256.num_objs

ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------

    105961 ~ 1%     -63.0%      39153 ~ 1%  nhm4/micro/will-it-scale/open1
    105961 ~ 1%     -63.0%      39153 ~ 1%  TOTAL meminfo.SUnreclaim

ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------

     26432 ~ 1%     -62.9%       9814 ~ 1%  nhm4/micro/will-it-scale/open1
     26432 ~ 1%     -62.9%       9814 ~ 1%  TOTAL proc-vmstat.nr_slab_unreclaimable

ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------

     50298 ~ 0%    +194.3%     148011 ~ 0%  nhm4/micro/will-it-scale/open1
     37020 ~ 0%     +42.6%      52798 ~ 1%  nhm4/micro/will-it-scale/signal1
     87318 ~ 0%    +130.0%     200809 ~ 0%  TOTAL softirqs.RCU

ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------

    140354 ~ 1%     -47.6%      73490 ~ 0%  nhm4/micro/will-it-scale/open1
    140354 ~ 1%     -47.6%      73490 ~ 0%  TOTAL meminfo.Slab

ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------

     77391 ~ 1%     -46.7%      41235 ~ 2%  nhm4/micro/will-it-scale/signal1
     77391 ~ 1%     -46.7%      41235 ~ 2%  TOTAL cpuidle.C6-NHM.usage

ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------

     19871 ~ 2%     -37.6%      12397 ~ 2%  nhm4/micro/will-it-scale/open1
     18497 ~ 1%     -37.5%      11556 ~ 1%  nhm4/micro/will-it-scale/signal1
     38368 ~ 2%     -37.6%      23954 ~ 2%  TOTAL softirqs.SCHED

ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------

      1.24 ~ 4%     -35.4%       0.80 ~ 3%  nhm4/micro/will-it-scale/open1
      1.24 ~ 4%     -35.4%       0.80 ~ 3%  TOTAL perf-profile.cpu-cycles.do_notify_resume.int_signal.close

ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------

      1.43 ~ 4%     +41.9%       2.03 ~ 4%  nhm4/micro/will-it-scale/open1
      1.43 ~ 4%     +41.9%       2.03 ~ 4%  TOTAL perf-profile.cpu-cycles.rcu_process_callbacks.__do_softirq.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt

ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------

      1.27 ~ 3%     -30.0%       0.89 ~ 6%  nhm4/micro/will-it-scale/open1
      1.27 ~ 3%     -30.0%       0.89 ~ 6%  TOTAL perf-profile.cpu-cycles.setup_object.isra.46.new_slab.__slab_alloc.kmem_cache_alloc.get_empty_filp

ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------

      1.54 ~ 7%     +35.6%       2.09 ~ 8%  nhm4/micro/will-it-scale/open1
      1.54 ~ 7%     +35.6%       2.09 ~ 8%  TOTAL perf-profile.cpu-cycles.kmem_cache_alloc.getname_flags.getname.do_sys_open.sys_open

ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------

      4.21 ~ 2%     -29.1%       2.98 ~ 3%  nhm4/micro/will-it-scale/open1
      4.21 ~ 2%     -29.1%       2.98 ~ 3%  TOTAL perf-profile.cpu-cycles.link_path_walk.path_openat.do_filp_open.do_sys_open.sys_open

ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------

      1.37 ~ 4%     -23.1%       1.05 ~ 7%  nhm4/micro/will-it-scale/open1
      1.37 ~ 4%     -23.1%       1.05 ~ 7%  TOTAL perf-profile.cpu-cycles.__d_lookup_rcu.lookup_fast.link_path_walk.path_openat.do_filp_open

ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------

      0.88 ~17%     +29.1%       1.14 ~ 9%  nhm4/micro/will-it-scale/open1
      0.88 ~17%     +29.1%       1.14 ~ 9%  TOTAL perf-profile.cpu-cycles.path_init.path_openat.do_filp_open.do_sys_open.sys_open

ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------

      0.67 ~16%     +33.6%       0.90 ~10%  nhm4/micro/will-it-scale/signal1
      0.67 ~16%     +33.6%       0.90 ~10%  TOTAL perf-profile.cpu-cycles.restore_sigcontext.sys_rt_sigreturn.stub_rt_sigreturn.raise

ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------

      3.19 ~ 1%     +17.4%       3.74 ~ 5%  nhm4/micro/will-it-scale/open1
      3.19 ~ 1%     +17.4%       3.74 ~ 5%  TOTAL perf-profile.cpu-cycles.file_free_rcu.rcu_process_callbacks.__do_softirq.irq_exit.smp_apic_timer_interrupt

ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------

      4329 ~ 7%     +15.2%       4986 ~ 5%  nhm4/micro/will-it-scale/open1
      4329 ~ 7%     +15.2%       4986 ~ 5%  TOTAL slabinfo.vm_area_struct.active_objs

ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------

      2536 ~ 1%     -75.8%        614 ~ 9%  nhm4/micro/will-it-scale/open1
      2536 ~ 1%     -75.8%        614 ~ 9%  TOTAL time.involuntary_context_switches

ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------

     32593 ~ 1%     -62.1%      12349 ~ 2%  nhm4/micro/will-it-scale/signal1
     32593 ~ 1%     -62.1%      12349 ~ 2%  TOTAL interrupts.0:IO-APIC-edge.timer

ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------

      4374 ~14%    +124.7%       9830 ~ 8%  nhm4/micro/will-it-scale/open1
      2559 ~ 2%     +20.3%       3078 ~ 3%  nhm4/micro/will-it-scale/signal1
      6934 ~ 9%     +86.2%      12908 ~ 7%  TOTAL interrupts.RES

ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------

       490 ~ 1%     -37.3%        307 ~ 1%  nhm4/micro/will-it-scale/signal1
       490 ~ 1%     -37.3%        307 ~ 1%  TOTAL vmstat.system.cs

ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------

      1639 ~ 0%      -8.8%       1495 ~ 0%  nhm4/micro/will-it-scale/signal1
      1639 ~ 0%      -8.8%       1495 ~ 0%  TOTAL vmstat.system.in

ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------

    419046 ~ 0%      -1.5%     412661 ~ 0%  nhm4/micro/will-it-scale/open1
    400635 ~ 0%      -5.9%     376866 ~ 0%  nhm4/micro/will-it-scale/signal1
    819681 ~ 0%      -3.7%     789527 ~ 0%  TOTAL interrupts.LOC

Legend:
~XX% - stddev percent
[+-]XX% - change percent

time.involuntary_context_switches

3500 ++------------------------------------------------------------------+

        |             .*..                                                  |
   3000 ++         .*.    *..*..  .*..*.. .*..                              |
        *..*..*..*.             *.       *                                  |
        |                                     *..*..     .*..     .*..*     |
   2500 ++                                          *..*.    *..*.          |
        |                                                                   |
   2000 ++                                                                  |
        |                                                                   |
   1500 ++                                                                  |
        |                                                                   |
        |                                                                   |
   1000 ++                                                                  |
        |     O  O  O              O  O                   O  O     O        O

500 O+-O-----------O--O--O--O--------O-O--O--O--O--O--------O-----O--O--+

[*] bisect-good sample
[O] bisect-bad sample

Thanks,
Fengguang

#1 lore raw
Quoted 3 lines · Fengguang Wu

Hi Paul,

FYI, this improves will-it-scale/open1 throughput.

Cool! Not a planned benefit, but I will take it. ;-)

Quoted 2 lines

git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git dev.2014.04.14a
commit b84c4e08143c98dad4b4d139f08db0b98b0d3ec4 ("sched,rcu: Make cond_resched() report RCU quiescent states")

But how should I read the data below? I see lots of positive percentages and lots of negative percentages for the delta, and all near zero for standard deviation. Is the overall improvement an average of these or some such? What is being measured?

Quoted 171 lines

ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------

    563496 ~ 0%      +3.1%     581059 ~ 0%  nhm4/micro/will-it-scale/open1
    563496 ~ 0%      +3.1%     581059 ~ 0%  TOTAL will-it-scale.per_thread_ops

ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------

    756894 ~ 0%      +2.8%     778452 ~ 0%  nhm4/micro/will-it-scale/open1
    756894 ~ 0%      +2.8%     778452 ~ 0%  TOTAL will-it-scale.per_process_ops

ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------

      0.57 ~ 0%      -2.7%       0.55 ~ 0%  nhm4/micro/will-it-scale/open1
      0.57 ~ 0%      -2.7%       0.55 ~ 0%  TOTAL will-it-scale.scalability

ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------

    346764 ~ 2%     -74.0%      90164 ~ 1%  nhm4/micro/will-it-scale/open1
    346764 ~ 2%     -74.0%      90164 ~ 1%  TOTAL slabinfo.kmalloc-256.active_objs

ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------

     10837 ~ 2%     -73.9%       2824 ~ 1%  nhm4/micro/will-it-scale/open1
     10837 ~ 2%     -73.9%       2824 ~ 1%  TOTAL slabinfo.kmalloc-256.active_slabs

ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------

     10837 ~ 2%     -73.9%       2824 ~ 1%  nhm4/micro/will-it-scale/open1
     10837 ~ 2%     -73.9%       2824 ~ 1%  TOTAL slabinfo.kmalloc-256.num_slabs

ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------

    346821 ~ 2%     -73.9%      90393 ~ 1%  nhm4/micro/will-it-scale/open1
    346821 ~ 2%     -73.9%      90393 ~ 1%  TOTAL slabinfo.kmalloc-256.num_objs

ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------

    105961 ~ 1%     -63.0%      39153 ~ 1%  nhm4/micro/will-it-scale/open1
    105961 ~ 1%     -63.0%      39153 ~ 1%  TOTAL meminfo.SUnreclaim

ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------

     26432 ~ 1%     -62.9%       9814 ~ 1%  nhm4/micro/will-it-scale/open1
     26432 ~ 1%     -62.9%       9814 ~ 1%  TOTAL proc-vmstat.nr_slab_unreclaimable

ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------

     50298 ~ 0%    +194.3%     148011 ~ 0%  nhm4/micro/will-it-scale/open1
     37020 ~ 0%     +42.6%      52798 ~ 1%  nhm4/micro/will-it-scale/signal1
     87318 ~ 0%    +130.0%     200809 ~ 0%  TOTAL softirqs.RCU

ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------

    140354 ~ 1%     -47.6%      73490 ~ 0%  nhm4/micro/will-it-scale/open1
    140354 ~ 1%     -47.6%      73490 ~ 0%  TOTAL meminfo.Slab

ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------

     77391 ~ 1%     -46.7%      41235 ~ 2%  nhm4/micro/will-it-scale/signal1
     77391 ~ 1%     -46.7%      41235 ~ 2%  TOTAL cpuidle.C6-NHM.usage

ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------

     19871 ~ 2%     -37.6%      12397 ~ 2%  nhm4/micro/will-it-scale/open1
     18497 ~ 1%     -37.5%      11556 ~ 1%  nhm4/micro/will-it-scale/signal1
     38368 ~ 2%     -37.6%      23954 ~ 2%  TOTAL softirqs.SCHED

ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------

      1.24 ~ 4%     -35.4%       0.80 ~ 3%  nhm4/micro/will-it-scale/open1
      1.24 ~ 4%     -35.4%       0.80 ~ 3%  TOTAL perf-profile.cpu-cycles.do_notify_resume.int_signal.close

ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------

      1.43 ~ 4%     +41.9%       2.03 ~ 4%  nhm4/micro/will-it-scale/open1
      1.43 ~ 4%     +41.9%       2.03 ~ 4%  TOTAL perf-profile.cpu-cycles.rcu_process_callbacks.__do_softirq.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt

ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------

      1.27 ~ 3%     -30.0%       0.89 ~ 6%  nhm4/micro/will-it-scale/open1
      1.27 ~ 3%     -30.0%       0.89 ~ 6%  TOTAL perf-profile.cpu-cycles.setup_object.isra.46.new_slab.__slab_alloc.kmem_cache_alloc.get_empty_filp

ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------

      1.54 ~ 7%     +35.6%       2.09 ~ 8%  nhm4/micro/will-it-scale/open1
      1.54 ~ 7%     +35.6%       2.09 ~ 8%  TOTAL perf-profile.cpu-cycles.kmem_cache_alloc.getname_flags.getname.do_sys_open.sys_open

ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------

      4.21 ~ 2%     -29.1%       2.98 ~ 3%  nhm4/micro/will-it-scale/open1
      4.21 ~ 2%     -29.1%       2.98 ~ 3%  TOTAL perf-profile.cpu-cycles.link_path_walk.path_openat.do_filp_open.do_sys_open.sys_open

ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------

      1.37 ~ 4%     -23.1%       1.05 ~ 7%  nhm4/micro/will-it-scale/open1
      1.37 ~ 4%     -23.1%       1.05 ~ 7%  TOTAL perf-profile.cpu-cycles.__d_lookup_rcu.lookup_fast.link_path_walk.path_openat.do_filp_open

ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------

      0.88 ~17%     +29.1%       1.14 ~ 9%  nhm4/micro/will-it-scale/open1
      0.88 ~17%     +29.1%       1.14 ~ 9%  TOTAL perf-profile.cpu-cycles.path_init.path_openat.do_filp_open.do_sys_open.sys_open

ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------

      0.67 ~16%     +33.6%       0.90 ~10%  nhm4/micro/will-it-scale/signal1
      0.67 ~16%     +33.6%       0.90 ~10%  TOTAL perf-profile.cpu-cycles.restore_sigcontext.sys_rt_sigreturn.stub_rt_sigreturn.raise

ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------

      3.19 ~ 1%     +17.4%       3.74 ~ 5%  nhm4/micro/will-it-scale/open1
      3.19 ~ 1%     +17.4%       3.74 ~ 5%  TOTAL perf-profile.cpu-cycles.file_free_rcu.rcu_process_callbacks.__do_softirq.irq_exit.smp_apic_timer_interrupt

ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------

      4329 ~ 7%     +15.2%       4986 ~ 5%  nhm4/micro/will-it-scale/open1
      4329 ~ 7%     +15.2%       4986 ~ 5%  TOTAL slabinfo.vm_area_struct.active_objs

ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------

      2536 ~ 1%     -75.8%        614 ~ 9%  nhm4/micro/will-it-scale/open1
      2536 ~ 1%     -75.8%        614 ~ 9%  TOTAL time.involuntary_context_switches

ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------

     32593 ~ 1%     -62.1%      12349 ~ 2%  nhm4/micro/will-it-scale/signal1
     32593 ~ 1%     -62.1%      12349 ~ 2%  TOTAL interrupts.0:IO-APIC-edge.timer

ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------

      4374 ~14%    +124.7%       9830 ~ 8%  nhm4/micro/will-it-scale/open1
      2559 ~ 2%     +20.3%       3078 ~ 3%  nhm4/micro/will-it-scale/signal1
      6934 ~ 9%     +86.2%      12908 ~ 7%  TOTAL interrupts.RES

ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------

       490 ~ 1%     -37.3%        307 ~ 1%  nhm4/micro/will-it-scale/signal1
       490 ~ 1%     -37.3%        307 ~ 1%  TOTAL vmstat.system.cs

ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------

      1639 ~ 0%      -8.8%       1495 ~ 0%  nhm4/micro/will-it-scale/signal1
      1639 ~ 0%      -8.8%       1495 ~ 0%  TOTAL vmstat.system.in

ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------

    419046 ~ 0%      -1.5%     412661 ~ 0%  nhm4/micro/will-it-scale/open1
    400635 ~ 0%      -5.9%     376866 ~ 0%  nhm4/micro/will-it-scale/signal1
    819681 ~ 0%      -3.7%     789527 ~ 0%  TOTAL interrupts.LOC

Legend:
~XX% - stddev percent
[+-]XX% - change percent

time.involuntary_context_switches

3500 ++------------------------------------------------------------------+

        |             .*..                                                  |
   3000 ++         .*.    *..*..  .*..*.. .*..                              |
        *..*..*..*.             *.       *                                  |
        |                                     *..*..     .*..     .*..*     |
   2500 ++                                          *..*.    *..*.          |
        |                                                                   |
   2000 ++                                                                  |
        |                                                                   |
   1500 ++                                                                  |
        |                                                                   |
        |                                                                   |
   1000 ++                                                                  |
        |     O  O  O              O  O                   O  O     O        O
Quoted 5 lines

500 O+-O-----------O--O--O--O--------O-O--O--O--O--O--------O-----O--O--+

[*] bisect-good sample
[O] bisect-bad sample

So the good case increases involuntary context switches, but helps something else? Or does the benefit stem from increased involuntary context switches and thus less time spinning or some such?

Thanx, Paul

Quoted 3 lines

Thanks,
Fengguang

#2 lore raw
Quoted 7 lines · Paul E. McKenney
Quoted 3 lines · Fengguang Wu

Hi Paul,

FYI, this improves will-it-scale/open1 throughput.

Cool! Not a planned benefit, but I will take it. ;-)

Quoted 7 lines
Quoted 2 lines

git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git dev.2014.04.14a
commit b84c4e08143c98dad4b4d139f08db0b98b0d3ec4 ("sched,rcu: Make cond_resched() report RCU quiescent states")

But how should I read the data below? I see lots of positive percentages and lots of negative percentages for the delta, and all near zero for standard deviation. Is the overall improvement an average of these or some such? What is being measured?

There are a lot of things being measured, which are shown after each "TOTAL". For example, to get an overview of the report:

grep "TOTAL" this_email

     563496 ~ 0%      +3.1%     581059 ~ 0%  TOTAL will-it-scale.per_thread_ops
     756894 ~ 0%      +2.8%     778452 ~ 0%  TOTAL will-it-scale.per_process_ops
       0.57 ~ 0%      -2.7%       0.55 ~ 0%  TOTAL will-it-scale.scalability
     346764 ~ 2%     -74.0%      90164 ~ 1%  TOTAL slabinfo.kmalloc-256.active_objs
      10837 ~ 2%     -73.9%       2824 ~ 1%  TOTAL slabinfo.kmalloc-256.active_slabs
      10837 ~ 2%     -73.9%       2824 ~ 1%  TOTAL slabinfo.kmalloc-256.num_slabs
     346821 ~ 2%     -73.9%      90393 ~ 1%  TOTAL slabinfo.kmalloc-256.num_objs
     105961 ~ 1%     -63.0%      39153 ~ 1%  TOTAL meminfo.SUnreclaim
      26432 ~ 1%     -62.9%       9814 ~ 1%  TOTAL proc-vmstat.nr_slab_unreclaimable
      87318 ~ 0%    +130.0%     200809 ~ 0%  TOTAL softirqs.RCU
     140354 ~ 1%     -47.6%      73490 ~ 0%  TOTAL meminfo.Slab
      77391 ~ 1%     -46.7%      41235 ~ 2%  TOTAL cpuidle.C6-NHM.usage
      38368 ~ 2%     -37.6%      23954 ~ 2%  TOTAL softirqs.SCHED
       1.24 ~ 4%     -35.4%       0.80 ~ 3%  TOTAL perf-profile.cpu-cycles.do_notify_resume.int_signal.close
       1.43 ~ 4%     +41.9%       2.03 ~ 4%  TOTAL perf-profile.cpu-cycles.rcu_process_callbacks.__do_softirq.irq_exit.smp_apic_timer_in
rupt.apic_timer_interrupt
       1.27 ~ 3%     -30.0%       0.89 ~ 6%  TOTAL perf-profile.cpu-cycles.setup_object.isra.46.new_slab.__slab_alloc.kmem_cache_alloc.g
empty_filp
       1.54 ~ 7%     +35.6%       2.09 ~ 8%  TOTAL perf-profile.cpu-cycles.kmem_cache_alloc.getname_flags.getname.do_sys_open.sys_open
       4.21 ~ 2%     -29.1%       2.98 ~ 3%  TOTAL perf-profile.cpu-cycles.link_path_walk.path_openat.do_filp_open.do_sys_open.sys_open
       1.37 ~ 4%     -23.1%       1.05 ~ 7%  TOTAL perf-profile.cpu-cycles.__d_lookup_rcu.lookup_fast.link_path_walk.path_openat.do_filp
en
       0.88 ~17%     +29.1%       1.14 ~ 9%  TOTAL perf-profile.cpu-cycles.path_init.path_openat.do_filp_open.do_sys_open.sys_open
       0.67 ~16%     +33.6%       0.90 ~10%  TOTAL perf-profile.cpu-cycles.restore_sigcontext.sys_rt_sigreturn.stub_rt_sigreturn.raise
       3.19 ~ 1%     +17.4%       3.74 ~ 5%  TOTAL perf-profile.cpu-cycles.file_free_rcu.rcu_process_callbacks.__do_softirq.irq_exit.smp
ic_timer_interrupt
       4329 ~ 7%     +15.2%       4986 ~ 5%  TOTAL slabinfo.vm_area_struct.active_objs
       2536 ~ 1%     -75.8%        614 ~ 9%  TOTAL time.involuntary_context_switches
      32593 ~ 1%     -62.1%      12349 ~ 2%  TOTAL interrupts.0:IO-APIC-edge.timer
       6934 ~ 9%     +86.2%      12908 ~ 7%  TOTAL interrupts.RES
        490 ~ 1%     -37.3%        307 ~ 1%  TOTAL vmstat.system.cs
       1639 ~ 0%      -8.8%       1495 ~ 0%  TOTAL vmstat.system.in
     819681 ~ 0%      -3.7%     789527 ~ 0%  TOTAL interrupts.LOC
Quoted 24 lines
Quoted 24 lines

Legend:
~XX% - stddev percent
[+-]XX% - change percent

time.involuntary_context_switches

3500 ++------------------------------------------------------------------+

        |             .*..                                                  |
   3000 ++         .*.    *..*..  .*..*.. .*..                              |
        *..*..*..*.             *.       *                                  |
        |                                     *..*..     .*..     .*..*     |
   2500 ++                                          *..*.    *..*.          |
        |                                                                   |
   2000 ++                                                                  |
        |                                                                   |
   1500 ++                                                                  |
        |                                                                   |
        |                                                                   |
   1000 ++                                                                  |
        |     O  O  O              O  O                   O  O     O        O
    500 O+-O-----------O--O--O--O--------O-O--O--O--O--O--------O-----O--O--+
Quoted 6 lines
Quoted 2 lines

[*] bisect-good sample
[O] bisect-bad sample

So the good case increases involuntary context switches, but helps something else? Or does the benefit stem from increased involuntary context switches and thus less time spinning or some such?

In bisect POV, branch BASE is good and HEAD is bad. Which has nothing to do with the improvement/regression in performance POV.

Here the HEAD(bisect bad) commit has less involuntary_context_switches which indicates an improvement over BASE. It does look like close to the root cause of improved will-it-scale throughput.

Thanks,
Fengguang

#3 lore raw
Quoted 11 lines · Fengguang Wu
Quoted 10 lines · Paul E. McKenney
Quoted 3 lines · Fengguang Wu

Hi Paul,

FYI, this improves will-it-scale/open1 throughput.

Cool! Not a planned benefit, but I will take it. ;-)

Quoted 2 lines

git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git dev.2014.04.14a
commit b84c4e08143c98dad4b4d139f08db0b98b0d3ec4 ("sched,rcu: Make cond_resched() report RCU quiescent states")

Quoted 7 lines
Quoted 4 lines

But how should I read the data below? I see lots of positive percentages and lots of negative percentages for the delta, and all near zero for standard deviation. Is the overall improvement an average of these or some such? What is being measured?

There are a lot of things being measured, which are shown after each "TOTAL". For example, to get an overview of the report:

OK, let me see if I understand this...

Quoted 4 lines

grep "TOTAL" this_email

     563496 ~ 0%      +3.1%     581059 ~ 0%  TOTAL will-it-scale.per_thread_ops
     756894 ~ 0%      +2.8%     778452 ~ 0%  TOTAL will-it-scale.per_process_ops

These two look like improvements, albeit small ones.

Quoted 1 line
       0.57 ~ 0%      -2.7%       0.55 ~ 0%  TOTAL will-it-scale.scalability

This one looks like a degradation, again small.

Quoted 6 lines
     346764 ~ 2%     -74.0%      90164 ~ 1%  TOTAL slabinfo.kmalloc-256.active_objs
      10837 ~ 2%     -73.9%       2824 ~ 1%  TOTAL slabinfo.kmalloc-256.active_slabs
      10837 ~ 2%     -73.9%       2824 ~ 1%  TOTAL slabinfo.kmalloc-256.num_slabs
     346821 ~ 2%     -73.9%      90393 ~ 1%  TOTAL slabinfo.kmalloc-256.num_objs
     105961 ~ 1%     -63.0%      39153 ~ 1%  TOTAL meminfo.SUnreclaim
      26432 ~ 1%     -62.9%       9814 ~ 1%  TOTAL proc-vmstat.nr_slab_unreclaimable

The above all look great -- about the same amount of work done with a lot less work on the part of the Linux kernel's memory-management system.

Quoted 1 line
      87318 ~ 0%    +130.0%     200809 ~ 0%  TOTAL softirqs.RCU

This one is not so good, as RCU is consuming more time an energy for roughly the same amount of work.

Quoted 4 lines
     140354 ~ 1%     -47.6%      73490 ~ 0%  TOTAL meminfo.Slab
      77391 ~ 1%     -46.7%      41235 ~ 2%  TOTAL cpuidle.C6-NHM.usage
      38368 ~ 2%     -37.6%      23954 ~ 2%  TOTAL softirqs.SCHED
       1.24 ~ 4%     -35.4%       0.80 ~ 3%  TOTAL perf-profile.cpu-cycles.do_notify_resume.int_signal.close

These look like improvements, again less work for a given level of throughput.

Quoted 1 line
       1.43 ~ 4%     +41.9%       2.03 ~ 4%  TOTAL perf-profile.cpu-cycles.rcu_process_callbacks.__do_softirq.irq_exit.smp_apic_timer_in

This is another aspect of RCU working harder. One good thing is that there is less work per invocation. This could be consistent with shorter RCU grace-period latencies, which would generally be a good thing. (And yes, I still owe you RCU grace-period-latency measurements!)

Quoted 2 lines

rupt.apic_timer_interrupt

       1.27 ~ 3%     -30.0%       0.89 ~ 6%  TOTAL perf-profile.cpu-cycles.setup_object.isra.46.new_slab.__slab_alloc.kmem_cache_alloc.g

Less memory-allocation work, goodness.

Quoted 2 lines

empty_filp

       1.54 ~ 7%     +35.6%       2.09 ~ 8%  TOTAL perf-profile.cpu-cycles.kmem_cache_alloc.getname_flags.getname.do_sys_open.sys_open

But more of whatever this turns out to be. More allocations from within the open() system call?

Quoted 4 lines
       4.21 ~ 2%     -29.1%       2.98 ~ 3%  TOTAL perf-profile.cpu-cycles.link_path_walk.path_openat.do_filp_open.do_sys_open.sys_open
       1.37 ~ 4%     -23.1%       1.05 ~ 7%  TOTAL perf-profile.cpu-cycles.__d_lookup_rcu.lookup_fast.link_path_walk.path_openat.do_filp
en
       0.88 ~17%     +29.1%       1.14 ~ 9%  TOTAL perf-profile.cpu-cycles.path_init.path_openat.do_filp_open.do_sys_open.sys_open

Mixed results on other aspects of the open() system call?

Quoted 1 line
       0.67 ~16%     +33.6%       0.90 ~10%  TOTAL perf-profile.cpu-cycles.restore_sigcontext.sys_rt_sigreturn.stub_rt_sigreturn.raise

I confess to being lost on this one.

Quoted 1 line
       3.19 ~ 1%     +17.4%       3.74 ~ 5%  TOTAL perf-profile.cpu-cycles.file_free_rcu.rcu_process_callbacks.__do_softirq.irq_exit.smp

More RCU softirqs means more softirq exit overhead, presumably.

Quoted 1 line

ic_timer_interrupt

???

Quoted 1 line
       4329 ~ 7%     +15.2%       4986 ~ 5%  TOTAL slabinfo.vm_area_struct.active_objs

But increased number of active objects would seem to hint at longer RCU grace-period latencies, in contradiction with the previous hints in the other direction.

Quoted 2 lines
       2536 ~ 1%     -75.8%        614 ~ 9%  TOTAL time.involuntary_context_switches
      32593 ~ 1%     -62.1%      12349 ~ 2%  TOTAL interrupts.0:IO-APIC-edge.timer

These look like improvements.

Quoted 1 line
       6934 ~ 9%     +86.2%      12908 ~ 7%  TOTAL interrupts.RES

Looks like a degradation. Caused by the increase in RCU softirqs?

Quoted 3 lines
        490 ~ 1%     -37.3%        307 ~ 1%  TOTAL vmstat.system.cs
       1639 ~ 0%      -8.8%       1495 ~ 0%  TOTAL vmstat.system.in
     819681 ~ 0%      -3.7%     789527 ~ 0%  TOTAL interrupts.LOC

These all look like improvements.

Quoted 31 lines
Quoted 24 lines
Quoted 24 lines

Legend:
~XX% - stddev percent
[+-]XX% - change percent

time.involuntary_context_switches

3500 ++------------------------------------------------------------------+

        |             .*..                                                  |
   3000 ++         .*.    *..*..  .*..*.. .*..                              |
        *..*..*..*.             *.       *                                  |
        |                                     *..*..     .*..     .*..*     |
   2500 ++                                          *..*.    *..*.          |
        |                                                                   |
   2000 ++                                                                  |
        |                                                                   |
   1500 ++                                                                  |
        |                                                                   |
        |                                                                   |
   1000 ++                                                                  |
        |     O  O  O              O  O                   O  O     O        O
    500 O+-O-----------O--O--O--O--------O-O--O--O--O--O--------O-----O--O--+
Quoted 6 lines
Quoted 2 lines

[*] bisect-good sample
[O] bisect-bad sample

So the good case increases involuntary context switches, but helps something else? Or does the benefit stem from increased involuntary context switches and thus less time spinning or some such?

Quoted 6 lines

In bisect POV, branch BASE is good and HEAD is bad. Which has nothing to do with the improvement/regression in performance POV.

Here the HEAD(bisect bad) commit has less involuntary_context_switches which indicates an improvement over BASE. It does look like close to the root cause of improved will-it-scale throughput.

Ah, that explains it!

Would it make sense to add "(HEAD)" to the legend for "[O]"?

Thanx, Paul