Skip to contentSkip to conversation

perf-stat.cpu-migrations changes in dd write test

sched [email protected] 2 participants 6 messages Dec 22, 2013 – Dec 25, 2013

lore
Share & tools
lore
#0 lore raw

Alex,

We noticed some perf-stat.cpu-migrations changes by your commit

6eb8b571283e64e96ca1a91aad085f9e08f857f0 is the first bad commit commit 6eb8b571283e64e96ca1a91aad085f9e08f857f0

Author: Alex Shi <[email protected]>
Date:   Tue Nov 19 20:29:51 2013 +0800

sched: remove rq->cpu_load[load_idx] array

Since load_idx effect removed in load balance, we don't need the load_idx decays in scheduler. that will save some process in sched_tick and others places.

Here are the changed stats comparing to its parent commit:

1de157d718f6494 6eb8b571283e64e96ca1
--------------- -------------------------

     37779 ~40%    +115.2%      81288 ~50%  kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd
     37779         +115.2%      81288       TOTAL proc-vmstat.pgscan_direct_normal

kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: proc-vmstat.pgscan_direct_normal

160000 ++----------------------------------------------------------------+

          |                                                                 |
   140000 ++                                                                O
          |                                                                 |
   120000 ++                                                                |
          |                O                                                |
   100000 ++                                                                |
          |                                                                 |
    80000 ++                               O                                |
          |                                                               ..*
    60000 ++                                                          ....  |
          |                                                       ....      |
    40000 *+...............                                 O ....          |
          O                *...............         ........*.              |
    20000 ++-------------------------------*--------------------------------+

1de157d718f6494 6eb8b571283e64e96ca1
--------------- -------------------------

    121579 ~ 6%     -50.8%      59852 ~13%  kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd
    121579          -50.8%      59852       TOTAL softirqs.SCHED

kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: softirqs.SCHED

140000 ++----------------------------------------------------------------+

   130000 ++           ....*...........                 ....*.......        |
          |    ........                ....     ........            ......  |
   120000 *+...                            *....                          ..|
   110000 ++                                                                *
          |                                                                 |
   100000 ++                                                                |
    90000 ++                                                                |
    80000 ++                                                                |
          |                                                                 |
    70000 ++                               O                O               |
    60000 ++               O                                                |
          |                                                                 O
    50000 O+                                                                |
    40000 ++----------------------------------------------------------------+

1de157d718f6494 6eb8b571283e64e96ca1
--------------- -------------------------

      5636 ~ 4%     -22.1%       4392 ~17%  kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd
      5636          -22.1%       4392       TOTAL proc-vmstat.nr_tlb_remote_flush

kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: proc-vmstat.nr_tlb_remote_flush

6500 ++------------------------------------------------------------------+

        |                                                                   |
   6000 ++       ........*............                                      |
        *........                     ....                                  |
   5500 ++               O                *................*................|
        |                                                                   *
   5000 ++                                O                                 |
        |                                                                   |
   4500 ++                                                 O                |
        |                                                                   |
   4000 ++                                                                  |
        |                                                                   O
   3500 O+                                                                  |
        |                                                                   |

3000 ++------------------------------------------------------------------+

1de157d718f6494 6eb8b571283e64e96ca1
--------------- -------------------------

      5679 ~ 4%     -21.9%       4437 ~17%  kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd
      5679          -21.9%       4437       TOTAL proc-vmstat.nr_tlb_remote_flush_received

6500 ++------------------------------------------------------------------+

        |                                                                   |
   6000 ++       ........*........                                          |
        *........                 ......                                    |
        |                               ..         ........*................|
   5500 ++               O                *........                         *
        |                                                                   |
   5000 ++                                O                                 |
        |                                                                   |
   4500 ++                                                 O                |
        |                                                                   |
        |                                                                   |
   4000 ++                                                                  |
        |                                                                   O

3500 O+------------------------------------------------------------------+

1de157d718f6494 6eb8b571283e64e96ca1
--------------- -------------------------

       167 ~11%     +19.4%        200 ~ 4%  kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd
       167          +19.4%        200       TOTAL uptime.idle

kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: uptime.idle

220 ++-------------------------------------------------------------------+

       |                                                                    O
   210 ++                                                                   |
   200 O+                                 O                                 |
       |              ..*.....                             O                |
   190 ++         ....  O     ...                                           |
   180 ++     ....               ....                                       |
       |  ....                       ....                                   |
   170 *+.                               .                                 .*
   160 ++                                 *......                      .... |
       |                                         ....               ...     |
   150 ++                                            ....        ...        |
   140 ++                                                ..  ....           |
       |                                                   *.               |

130 ++-------------------------------------------------------------------+

1de157d718f6494 6eb8b571283e64e96ca1
--------------- -------------------------

    154781 ~ 5%     -66.8%      51449 ~ 8%  kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd
    154781          -66.8%      51449       TOTAL perf-stat.cpu-migrations

kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: perf-stat.cpu-migrations

180000 ++----------------------------------------------------------------+

          |            ....*...........                                     |
   160000 ++   ........                ....         ........*...............|
          *....                            *........                        *
   140000 ++                                                                |
          |                                                                 |
   120000 ++                                                                |
          |                                                                 |
   100000 ++                                                                |
          |                                                                 |
    80000 ++                                                                |
          |                                                                 |
    60000 ++                                                O               |
          |                O               O                                O
    40000 O+----------------------------------------------------------------+

1de157d718f6494 6eb8b571283e64e96ca1
--------------- -------------------------

      1943 ~ 3%      -7.8%       1792 ~ 4%  kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd
      1943           -7.8%       1792       TOTAL vmstat.system.cs

kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: vmstat.system.cs

2050 ++------------------------------------------------------------------+

        |                         ........*................*.....           |
   2000 ++              .*........                               ...        |
   1950 ++          ....                                            ...     |
        |        ...                                                   .... |
   1900 ++    ...                                                          .|
   1850 ++....                            O                O                *
        *.                                                                  |
   1800 ++               O                                                  O
   1750 ++                                                                  |
        |                                                                   |
   1700 ++                                                                  |
   1650 O+                                                                  |
        |                                                                   |

1600 ++------------------------------------------------------------------+

1de157d718f6494 6eb8b571283e64e96ca1
--------------- -------------------------

   1216177 ~ 3%      -9.4%    1101605 ~ 7%  kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd
   1216177           -9.4%    1101605       TOTAL perf-stat.context-switches
      kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: perf-stat.context-switches

    1.3e+06 ++--------------------------------------------------------------+
            |             ..*...............                                |
   1.25e+06 ++      ......                  *...............*.......        |
            |  .....                                                ......  |
    1.2e+06 *+.                             O                             ..|
   1.15e+06 ++                                              O               *
            |               O                                               |
    1.1e+06 ++                                                              |
            |                                                               O
   1.05e+06 ++                                                              |
      1e+06 ++                                                              |
            |                                                               |
     950000 O+                                                              |
            |                                                               |
     900000 ++--------------------------------------------------------------+
#1 lore raw
Quoted 3 lines · On 12/22/2013 08:30 AM, [email protected] wrote:

Alex,

We noticed some perf-stat.cpu-migrations changes by your commit

Thanks for your wonder testing, Fengguang!
How many CPU in your system and how many dd task running?

Quoted 209 lines

6eb8b571283e64e96ca1a91aad085f9e08f857f0 is the first bad commit commit 6eb8b571283e64e96ca1a91aad085f9e08f857f0

Author: Alex Shi <[email protected]>
Date:   Tue Nov 19 20:29:51 2013 +0800

sched: remove rq->cpu_load[load_idx] array

Since load_idx effect removed in load balance, we don't need the load_idx decays in scheduler. that will save some process in sched_tick and others places.

Here are the changed stats comparing to its parent commit:

1de157d718f6494 6eb8b571283e64e96ca1
--------------- -------------------------

     37779 ~40%    +115.2%      81288 ~50%  kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd
     37779         +115.2%      81288       TOTAL proc-vmstat.pgscan_direct_normal

kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: proc-vmstat.pgscan_direct_normal

160000 ++----------------------------------------------------------------+

          |                                                                 |
   140000 ++                                                                O
          |                                                                 |
   120000 ++                                                                |
          |                O                                                |
   100000 ++                                                                |
          |                                                                 |
    80000 ++                               O                                |
          |                                                               ..*
    60000 ++                                                          ....  |
          |                                                       ....      |
    40000 *+...............                                 O ....          |
          O                *...............         ........*.              |
    20000 ++-------------------------------*--------------------------------+

1de157d718f6494 6eb8b571283e64e96ca1
--------------- -------------------------

    121579 ~ 6%     -50.8%      59852 ~13%  kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd
    121579          -50.8%      59852       TOTAL softirqs.SCHED

kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: softirqs.SCHED

140000 ++----------------------------------------------------------------+

   130000 ++           ....*...........                 ....*.......        |
          |    ........                ....     ........            ......  |
   120000 *+...                            *....                          ..|
   110000 ++                                                                *
          |                                                                 |
   100000 ++                                                                |
    90000 ++                                                                |
    80000 ++                                                                |
          |                                                                 |
    70000 ++                               O                O               |
    60000 ++               O                                                |
          |                                                                 O
    50000 O+                                                                |
    40000 ++----------------------------------------------------------------+

1de157d718f6494 6eb8b571283e64e96ca1
--------------- -------------------------

      5636 ~ 4%     -22.1%       4392 ~17%  kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd
      5636          -22.1%       4392       TOTAL proc-vmstat.nr_tlb_remote_flush

kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: proc-vmstat.nr_tlb_remote_flush

6500 ++------------------------------------------------------------------+

        |                                                                   |
   6000 ++       ........*............                                      |
        *........                     ....                                  |
   5500 ++               O                *................*................|
        |                                                                   *
   5000 ++                                O                                 |
        |                                                                   |
   4500 ++                                                 O                |
        |                                                                   |
   4000 ++                                                                  |
        |                                                                   O
   3500 O+                                                                  |
        |                                                                   |

3000 ++------------------------------------------------------------------+

1de157d718f6494 6eb8b571283e64e96ca1
--------------- -------------------------

      5679 ~ 4%     -21.9%       4437 ~17%  kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd
      5679          -21.9%       4437       TOTAL proc-vmstat.nr_tlb_remote_flush_received

6500 ++------------------------------------------------------------------+

        |                                                                   |
   6000 ++       ........*........                                          |
        *........                 ......                                    |
        |                               ..         ........*................|
   5500 ++               O                *........                         *
        |                                                                   |
   5000 ++                                O                                 |
        |                                                                   |
   4500 ++                                                 O                |
        |                                                                   |
        |                                                                   |
   4000 ++                                                                  |
        |                                                                   O

3500 O+------------------------------------------------------------------+

1de157d718f6494 6eb8b571283e64e96ca1
--------------- -------------------------

       167 ~11%     +19.4%        200 ~ 4%  kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd
       167          +19.4%        200       TOTAL uptime.idle

kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: uptime.idle

220 ++-------------------------------------------------------------------+

       |                                                                    O
   210 ++                                                                   |
   200 O+                                 O                                 |
       |              ..*.....                             O                |
   190 ++         ....  O     ...                                           |
   180 ++     ....               ....                                       |
       |  ....                       ....                                   |
   170 *+.                               .                                 .*
   160 ++                                 *......                      .... |
       |                                         ....               ...     |
   150 ++                                            ....        ...        |
   140 ++                                                ..  ....           |
       |                                                   *.               |

130 ++-------------------------------------------------------------------+

1de157d718f6494 6eb8b571283e64e96ca1
--------------- -------------------------

    154781 ~ 5%     -66.8%      51449 ~ 8%  kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd
    154781          -66.8%      51449       TOTAL perf-stat.cpu-migrations

kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: perf-stat.cpu-migrations

180000 ++----------------------------------------------------------------+

          |            ....*...........                                     |
   160000 ++   ........                ....         ........*...............|
          *....                            *........                        *
   140000 ++                                                                |
          |                                                                 |
   120000 ++                                                                |
          |                                                                 |
   100000 ++                                                                |
          |                                                                 |
    80000 ++                                                                |
          |                                                                 |
    60000 ++                                                O               |
          |                O               O                                O
    40000 O+----------------------------------------------------------------+

1de157d718f6494 6eb8b571283e64e96ca1
--------------- -------------------------

      1943 ~ 3%      -7.8%       1792 ~ 4%  kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd
      1943           -7.8%       1792       TOTAL vmstat.system.cs

kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: vmstat.system.cs

2050 ++------------------------------------------------------------------+

        |                         ........*................*.....           |
   2000 ++              .*........                               ...        |
   1950 ++          ....                                            ...     |
        |        ...                                                   .... |
   1900 ++    ...                                                          .|
   1850 ++....                            O                O                *
        *.                                                                  |
   1800 ++               O                                                  O
   1750 ++                                                                  |
        |                                                                   |
   1700 ++                                                                  |
   1650 O+                                                                  |
        |                                                                   |

1600 ++------------------------------------------------------------------+

1de157d718f6494 6eb8b571283e64e96ca1
--------------- -------------------------

   1216177 ~ 3%      -9.4%    1101605 ~ 7%  kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd
   1216177           -9.4%    1101605       TOTAL perf-stat.context-switches
      kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: perf-stat.context-switches

    1.3e+06 ++--------------------------------------------------------------+
            |             ..*...............                                |
   1.25e+06 ++      ......                  *...............*.......        |
            |  .....                                                ......  |
    1.2e+06 *+.                             O                             ..|
   1.15e+06 ++                                              O               *
            |               O                                               |
    1.1e+06 ++                                                              |
            |                                                               O
   1.05e+06 ++                                                              |
      1e+06 ++                                                              |
            |                                                               |
     950000 O+                                                              |
            |                                                               |
     900000 ++--------------------------------------------------------------+
Signature · 2 lines
Thanks
    Alex
#2 lore raw
Quoted 7 lines · Alex Shi
Quoted 3 lines · On 12/22/2013 08:30 AM, [email protected] wrote:

Alex,

We noticed some perf-stat.cpu-migrations changes by your commit

Thanks for your wonder testing, Fengguang!
How many CPU in your system and how many dd task running?

For the case

        kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd

It's a KVM with 4 CPUs, 16G memory. There are 4 virtual disks, on top of each disk there runs 10 dd tasks.

Thanks,
Fengguang

Quoted 214 lines
Quoted 209 lines

6eb8b571283e64e96ca1a91aad085f9e08f857f0 is the first bad commit commit 6eb8b571283e64e96ca1a91aad085f9e08f857f0

Author: Alex Shi <[email protected]>
Date:   Tue Nov 19 20:29:51 2013 +0800

sched: remove rq->cpu_load[load_idx] array

Since load_idx effect removed in load balance, we don't need the load_idx decays in scheduler. that will save some process in sched_tick and others places.

Here are the changed stats comparing to its parent commit:

1de157d718f6494 6eb8b571283e64e96ca1
--------------- -------------------------

     37779 ~40%    +115.2%      81288 ~50%  kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd
     37779         +115.2%      81288       TOTAL proc-vmstat.pgscan_direct_normal

kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: proc-vmstat.pgscan_direct_normal

160000 ++----------------------------------------------------------------+

          |                                                                 |
   140000 ++                                                                O
          |                                                                 |
   120000 ++                                                                |
          |                O                                                |
   100000 ++                                                                |
          |                                                                 |
    80000 ++                               O                                |
          |                                                               ..*
    60000 ++                                                          ....  |
          |                                                       ....      |
    40000 *+...............                                 O ....          |
          O                *...............         ........*.              |
    20000 ++-------------------------------*--------------------------------+

1de157d718f6494 6eb8b571283e64e96ca1
--------------- -------------------------

    121579 ~ 6%     -50.8%      59852 ~13%  kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd
    121579          -50.8%      59852       TOTAL softirqs.SCHED

kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: softirqs.SCHED

140000 ++----------------------------------------------------------------+

   130000 ++           ....*...........                 ....*.......        |
          |    ........                ....     ........            ......  |
   120000 *+...                            *....                          ..|
   110000 ++                                                                *
          |                                                                 |
   100000 ++                                                                |
    90000 ++                                                                |
    80000 ++                                                                |
          |                                                                 |
    70000 ++                               O                O               |
    60000 ++               O                                                |
          |                                                                 O
    50000 O+                                                                |
    40000 ++----------------------------------------------------------------+

1de157d718f6494 6eb8b571283e64e96ca1
--------------- -------------------------

      5636 ~ 4%     -22.1%       4392 ~17%  kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd
      5636          -22.1%       4392       TOTAL proc-vmstat.nr_tlb_remote_flush

kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: proc-vmstat.nr_tlb_remote_flush

6500 ++------------------------------------------------------------------+

        |                                                                   |
   6000 ++       ........*............                                      |
        *........                     ....                                  |
   5500 ++               O                *................*................|
        |                                                                   *
   5000 ++                                O                                 |
        |                                                                   |
   4500 ++                                                 O                |
        |                                                                   |
   4000 ++                                                                  |
        |                                                                   O
   3500 O+                                                                  |
        |                                                                   |

3000 ++------------------------------------------------------------------+

1de157d718f6494 6eb8b571283e64e96ca1
--------------- -------------------------

      5679 ~ 4%     -21.9%       4437 ~17%  kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd
      5679          -21.9%       4437       TOTAL proc-vmstat.nr_tlb_remote_flush_received

6500 ++------------------------------------------------------------------+

        |                                                                   |
   6000 ++       ........*........                                          |
        *........                 ......                                    |
        |                               ..         ........*................|
   5500 ++               O                *........                         *
        |                                                                   |
   5000 ++                                O                                 |
        |                                                                   |
   4500 ++                                                 O                |
        |                                                                   |
        |                                                                   |
   4000 ++                                                                  |
        |                                                                   O

3500 O+------------------------------------------------------------------+

1de157d718f6494 6eb8b571283e64e96ca1
--------------- -------------------------

       167 ~11%     +19.4%        200 ~ 4%  kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd
       167          +19.4%        200       TOTAL uptime.idle

kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: uptime.idle

220 ++-------------------------------------------------------------------+

       |                                                                    O
   210 ++                                                                   |
   200 O+                                 O                                 |
       |              ..*.....                             O                |
   190 ++         ....  O     ...                                           |
   180 ++     ....               ....                                       |
       |  ....                       ....                                   |
   170 *+.                               .                                 .*
   160 ++                                 *......                      .... |
       |                                         ....               ...     |
   150 ++                                            ....        ...        |
   140 ++                                                ..  ....           |
       |                                                   *.               |

130 ++-------------------------------------------------------------------+

1de157d718f6494 6eb8b571283e64e96ca1
--------------- -------------------------

    154781 ~ 5%     -66.8%      51449 ~ 8%  kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd
    154781          -66.8%      51449       TOTAL perf-stat.cpu-migrations

kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: perf-stat.cpu-migrations

180000 ++----------------------------------------------------------------+

          |            ....*...........                                     |
   160000 ++   ........                ....         ........*...............|
          *....                            *........                        *
   140000 ++                                                                |
          |                                                                 |
   120000 ++                                                                |
          |                                                                 |
   100000 ++                                                                |
          |                                                                 |
    80000 ++                                                                |
          |                                                                 |
    60000 ++                                                O               |
          |                O               O                                O
    40000 O+----------------------------------------------------------------+

1de157d718f6494 6eb8b571283e64e96ca1
--------------- -------------------------

      1943 ~ 3%      -7.8%       1792 ~ 4%  kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd
      1943           -7.8%       1792       TOTAL vmstat.system.cs

kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: vmstat.system.cs

2050 ++------------------------------------------------------------------+

        |                         ........*................*.....           |
   2000 ++              .*........                               ...        |
   1950 ++          ....                                            ...     |
        |        ...                                                   .... |
   1900 ++    ...                                                          .|
   1850 ++....                            O                O                *
        *.                                                                  |
   1800 ++               O                                                  O
   1750 ++                                                                  |
        |                                                                   |
   1700 ++                                                                  |
   1650 O+                                                                  |
        |                                                                   |

1600 ++------------------------------------------------------------------+

1de157d718f6494 6eb8b571283e64e96ca1
--------------- -------------------------

   1216177 ~ 3%      -9.4%    1101605 ~ 7%  kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd
   1216177           -9.4%    1101605       TOTAL perf-stat.context-switches
      kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: perf-stat.context-switches

    1.3e+06 ++--------------------------------------------------------------+
            |             ..*...............                                |
   1.25e+06 ++      ......                  *...............*.......        |
            |  .....                                                ......  |
    1.2e+06 *+.                             O                             ..|
   1.15e+06 ++                                              O               *
            |               O                                               |
    1.1e+06 ++                                                              |
            |                                                               O
   1.05e+06 ++                                                              |
      1e+06 ++                                                              |
            |                                                               |
     950000 O+                                                              |
            |                                                               |
     900000 ++--------------------------------------------------------------+
Signature · 2 lines
Thanks
    Alex
#3 lore raw
Quoted 9 lines · Fengguang Wu
Quoted 7 lines · Alex Shi
Quoted 3 lines · On 12/22/2013 08:30 AM, [email protected] wrote:

Alex,

We noticed some perf-stat.cpu-migrations changes by your commit

Thanks for your wonder testing, Fengguang!
How many CPU in your system and how many dd task running?

Quoted 6 lines

For the case

        kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd

It's a KVM with 4 CPUs, 16G memory. There are 4 virtual disks, on top of each disk there runs 10 dd tasks.

I don't know much of kvm. Where the data get from, VM guest or VM host? Looks like any guest cpu usage will be counted as user usage in vmstat. and Does the CS and cpu migration reducing cause dd performance increase?

Quoted 218 lines

Thanks,
Fengguang

Quoted 214 lines
Quoted 209 lines

6eb8b571283e64e96ca1a91aad085f9e08f857f0 is the first bad commit commit 6eb8b571283e64e96ca1a91aad085f9e08f857f0

Author: Alex Shi <[email protected]>
Date:   Tue Nov 19 20:29:51 2013 +0800

sched: remove rq->cpu_load[load_idx] array

Since load_idx effect removed in load balance, we don't need the load_idx decays in scheduler. that will save some process in sched_tick and others places.

Here are the changed stats comparing to its parent commit:

1de157d718f6494 6eb8b571283e64e96ca1
--------------- -------------------------

     37779 ~40%    +115.2%      81288 ~50%  kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd
     37779         +115.2%      81288       TOTAL proc-vmstat.pgscan_direct_normal

kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: proc-vmstat.pgscan_direct_normal

160000 ++----------------------------------------------------------------+

          |                                                                 |
   140000 ++                                                                O
          |                                                                 |
   120000 ++                                                                |
          |                O                                                |
   100000 ++                                                                |
          |                                                                 |
    80000 ++                               O                                |
          |                                                               ..*
    60000 ++                                                          ....  |
          |                                                       ....      |
    40000 *+...............                                 O ....          |
          O                *...............         ........*.              |
    20000 ++-------------------------------*--------------------------------+

1de157d718f6494 6eb8b571283e64e96ca1
--------------- -------------------------

    121579 ~ 6%     -50.8%      59852 ~13%  kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd
    121579          -50.8%      59852       TOTAL softirqs.SCHED

kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: softirqs.SCHED

140000 ++----------------------------------------------------------------+

   130000 ++           ....*...........                 ....*.......        |
          |    ........                ....     ........            ......  |
   120000 *+...                            *....                          ..|
   110000 ++                                                                *
          |                                                                 |
   100000 ++                                                                |
    90000 ++                                                                |
    80000 ++                                                                |
          |                                                                 |
    70000 ++                               O                O               |
    60000 ++               O                                                |
          |                                                                 O
    50000 O+                                                                |
    40000 ++----------------------------------------------------------------+

1de157d718f6494 6eb8b571283e64e96ca1
--------------- -------------------------

      5636 ~ 4%     -22.1%       4392 ~17%  kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd
      5636          -22.1%       4392       TOTAL proc-vmstat.nr_tlb_remote_flush

kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: proc-vmstat.nr_tlb_remote_flush

6500 ++------------------------------------------------------------------+

        |                                                                   |
   6000 ++       ........*............                                      |
        *........                     ....                                  |
   5500 ++               O                *................*................|
        |                                                                   *
   5000 ++                                O                                 |
        |                                                                   |
   4500 ++                                                 O                |
        |                                                                   |
   4000 ++                                                                  |
        |                                                                   O
   3500 O+                                                                  |
        |                                                                   |

3000 ++------------------------------------------------------------------+

1de157d718f6494 6eb8b571283e64e96ca1
--------------- -------------------------

      5679 ~ 4%     -21.9%       4437 ~17%  kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd
      5679          -21.9%       4437       TOTAL proc-vmstat.nr_tlb_remote_flush_received

6500 ++------------------------------------------------------------------+

        |                                                                   |
   6000 ++       ........*........                                          |
        *........                 ......                                    |
        |                               ..         ........*................|
   5500 ++               O                *........                         *
        |                                                                   |
   5000 ++                                O                                 |
        |                                                                   |
   4500 ++                                                 O                |
        |                                                                   |
        |                                                                   |
   4000 ++                                                                  |
        |                                                                   O

3500 O+------------------------------------------------------------------+

1de157d718f6494 6eb8b571283e64e96ca1
--------------- -------------------------

       167 ~11%     +19.4%        200 ~ 4%  kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd
       167          +19.4%        200       TOTAL uptime.idle

kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: uptime.idle

220 ++-------------------------------------------------------------------+

       |                                                                    O
   210 ++                                                                   |
   200 O+                                 O                                 |
       |              ..*.....                             O                |
   190 ++         ....  O     ...                                           |
   180 ++     ....               ....                                       |
       |  ....                       ....                                   |
   170 *+.                               .                                 .*
   160 ++                                 *......                      .... |
       |                                         ....               ...     |
   150 ++                                            ....        ...        |
   140 ++                                                ..  ....           |
       |                                                   *.               |

130 ++-------------------------------------------------------------------+

1de157d718f6494 6eb8b571283e64e96ca1
--------------- -------------------------

    154781 ~ 5%     -66.8%      51449 ~ 8%  kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd
    154781          -66.8%      51449       TOTAL perf-stat.cpu-migrations

kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: perf-stat.cpu-migrations

180000 ++----------------------------------------------------------------+

          |            ....*...........                                     |
   160000 ++   ........                ....         ........*...............|
          *....                            *........                        *
   140000 ++                                                                |
          |                                                                 |
   120000 ++                                                                |
          |                                                                 |
   100000 ++                                                                |
          |                                                                 |
    80000 ++                                                                |
          |                                                                 |
    60000 ++                                                O               |
          |                O               O                                O
    40000 O+----------------------------------------------------------------+

1de157d718f6494 6eb8b571283e64e96ca1
--------------- -------------------------

      1943 ~ 3%      -7.8%       1792 ~ 4%  kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd
      1943           -7.8%       1792       TOTAL vmstat.system.cs

kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: vmstat.system.cs

2050 ++------------------------------------------------------------------+

        |                         ........*................*.....           |
   2000 ++              .*........                               ...        |
   1950 ++          ....                                            ...     |
        |        ...                                                   .... |
   1900 ++    ...                                                          .|
   1850 ++....                            O                O                *
        *.                                                                  |
   1800 ++               O                                                  O
   1750 ++                                                                  |
        |                                                                   |
   1700 ++                                                                  |
   1650 O+                                                                  |
        |                                                                   |

1600 ++------------------------------------------------------------------+

1de157d718f6494 6eb8b571283e64e96ca1
--------------- -------------------------

   1216177 ~ 3%      -9.4%    1101605 ~ 7%  kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd
   1216177           -9.4%    1101605       TOTAL perf-stat.context-switches
      kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: perf-stat.context-switches

    1.3e+06 ++--------------------------------------------------------------+
            |             ..*...............                                |
   1.25e+06 ++      ......                  *...............*.......        |
            |  .....                                                ......  |
    1.2e+06 *+.                             O                             ..|
   1.15e+06 ++                                              O               *
            |               O                                               |
    1.1e+06 ++                                                              |
            |                                                               O
   1.05e+06 ++                                                              |
      1e+06 ++                                                              |
            |                                                               |
     950000 O+                                                              |
            |                                                               |
     900000 ++--------------------------------------------------------------+
Signature · 2 lines
Thanks
    Alex
Signature · 2 lines
Thanks
    Alex
#4 lore raw
Quoted 10 lines · Alex Shi
Quoted 9 lines · Fengguang Wu
Quoted 7 lines · Alex Shi
Quoted 3 lines · On 12/22/2013 08:30 AM, [email protected] wrote:

Alex,

We noticed some perf-stat.cpu-migrations changes by your commit

Thanks for your wonder testing, Fengguang!
How many CPU in your system and how many dd task running?

Quoted 8 lines
Quoted 6 lines

For the case

        kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd

It's a KVM with 4 CPUs, 16G memory. There are 4 virtual disks, on top of each disk there runs 10 dd tasks.

I don't know much of kvm. Where the data get from, VM guest or VM host?

The data are collected in the VM guest.

Thanks,
Fengguang

Quoted 225 lines

Looks like any guest cpu usage will be counted as user usage in vmstat. and Does the CS and cpu migration reducing cause dd performance increase?

Quoted 218 lines

Thanks,
Fengguang

Quoted 214 lines
Quoted 209 lines

6eb8b571283e64e96ca1a91aad085f9e08f857f0 is the first bad commit commit 6eb8b571283e64e96ca1a91aad085f9e08f857f0

Author: Alex Shi <[email protected]>
Date:   Tue Nov 19 20:29:51 2013 +0800

sched: remove rq->cpu_load[load_idx] array

Since load_idx effect removed in load balance, we don't need the load_idx decays in scheduler. that will save some process in sched_tick and others places.

Here are the changed stats comparing to its parent commit:

1de157d718f6494 6eb8b571283e64e96ca1
--------------- -------------------------

     37779 ~40%    +115.2%      81288 ~50%  kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd
     37779         +115.2%      81288       TOTAL proc-vmstat.pgscan_direct_normal

kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: proc-vmstat.pgscan_direct_normal

160000 ++----------------------------------------------------------------+

          |                                                                 |
   140000 ++                                                                O
          |                                                                 |
   120000 ++                                                                |
          |                O                                                |
   100000 ++                                                                |
          |                                                                 |
    80000 ++                               O                                |
          |                                                               ..*
    60000 ++                                                          ....  |
          |                                                       ....      |
    40000 *+...............                                 O ....          |
          O                *...............         ........*.              |
    20000 ++-------------------------------*--------------------------------+

1de157d718f6494 6eb8b571283e64e96ca1
--------------- -------------------------

    121579 ~ 6%     -50.8%      59852 ~13%  kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd
    121579          -50.8%      59852       TOTAL softirqs.SCHED

kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: softirqs.SCHED

140000 ++----------------------------------------------------------------+

   130000 ++           ....*...........                 ....*.......        |
          |    ........                ....     ........            ......  |
   120000 *+...                            *....                          ..|
   110000 ++                                                                *
          |                                                                 |
   100000 ++                                                                |
    90000 ++                                                                |
    80000 ++                                                                |
          |                                                                 |
    70000 ++                               O                O               |
    60000 ++               O                                                |
          |                                                                 O
    50000 O+                                                                |
    40000 ++----------------------------------------------------------------+

1de157d718f6494 6eb8b571283e64e96ca1
--------------- -------------------------

      5636 ~ 4%     -22.1%       4392 ~17%  kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd
      5636          -22.1%       4392       TOTAL proc-vmstat.nr_tlb_remote_flush

kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: proc-vmstat.nr_tlb_remote_flush

6500 ++------------------------------------------------------------------+

        |                                                                   |
   6000 ++       ........*............                                      |
        *........                     ....                                  |
   5500 ++               O                *................*................|
        |                                                                   *
   5000 ++                                O                                 |
        |                                                                   |
   4500 ++                                                 O                |
        |                                                                   |
   4000 ++                                                                  |
        |                                                                   O
   3500 O+                                                                  |
        |                                                                   |

3000 ++------------------------------------------------------------------+

1de157d718f6494 6eb8b571283e64e96ca1
--------------- -------------------------

      5679 ~ 4%     -21.9%       4437 ~17%  kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd
      5679          -21.9%       4437       TOTAL proc-vmstat.nr_tlb_remote_flush_received

6500 ++------------------------------------------------------------------+

        |                                                                   |
   6000 ++       ........*........                                          |
        *........                 ......                                    |
        |                               ..         ........*................|
   5500 ++               O                *........                         *
        |                                                                   |
   5000 ++                                O                                 |
        |                                                                   |
   4500 ++                                                 O                |
        |                                                                   |
        |                                                                   |
   4000 ++                                                                  |
        |                                                                   O

3500 O+------------------------------------------------------------------+

1de157d718f6494 6eb8b571283e64e96ca1
--------------- -------------------------

       167 ~11%     +19.4%        200 ~ 4%  kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd
       167          +19.4%        200       TOTAL uptime.idle

kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: uptime.idle

220 ++-------------------------------------------------------------------+

       |                                                                    O
   210 ++                                                                   |
   200 O+                                 O                                 |
       |              ..*.....                             O                |
   190 ++         ....  O     ...                                           |
   180 ++     ....               ....                                       |
       |  ....                       ....                                   |
   170 *+.                               .                                 .*
   160 ++                                 *......                      .... |
       |                                         ....               ...     |
   150 ++                                            ....        ...        |
   140 ++                                                ..  ....           |
       |                                                   *.               |

130 ++-------------------------------------------------------------------+

1de157d718f6494 6eb8b571283e64e96ca1
--------------- -------------------------

    154781 ~ 5%     -66.8%      51449 ~ 8%  kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd
    154781          -66.8%      51449       TOTAL perf-stat.cpu-migrations

kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: perf-stat.cpu-migrations

180000 ++----------------------------------------------------------------+

          |            ....*...........                                     |
   160000 ++   ........                ....         ........*...............|
          *....                            *........                        *
   140000 ++                                                                |
          |                                                                 |
   120000 ++                                                                |
          |                                                                 |
   100000 ++                                                                |
          |                                                                 |
    80000 ++                                                                |
          |                                                                 |
    60000 ++                                                O               |
          |                O               O                                O
    40000 O+----------------------------------------------------------------+

1de157d718f6494 6eb8b571283e64e96ca1
--------------- -------------------------

      1943 ~ 3%      -7.8%       1792 ~ 4%  kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd
      1943           -7.8%       1792       TOTAL vmstat.system.cs

kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: vmstat.system.cs

2050 ++------------------------------------------------------------------+

        |                         ........*................*.....           |
   2000 ++              .*........                               ...        |
   1950 ++          ....                                            ...     |
        |        ...                                                   .... |
   1900 ++    ...                                                          .|
   1850 ++....                            O                O                *
        *.                                                                  |
   1800 ++               O                                                  O
   1750 ++                                                                  |
        |                                                                   |
   1700 ++                                                                  |
   1650 O+                                                                  |
        |                                                                   |

1600 ++------------------------------------------------------------------+

1de157d718f6494 6eb8b571283e64e96ca1
--------------- -------------------------

   1216177 ~ 3%      -9.4%    1101605 ~ 7%  kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd
   1216177           -9.4%    1101605       TOTAL perf-stat.context-switches
      kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: perf-stat.context-switches

    1.3e+06 ++--------------------------------------------------------------+
            |             ..*...............                                |
   1.25e+06 ++      ......                  *...............*.......        |
            |  .....                                                ......  |
    1.2e+06 *+.                             O                             ..|
   1.15e+06 ++                                              O               *
            |               O                                               |
    1.1e+06 ++                                                              |
            |                                                               O
   1.05e+06 ++                                                              |
      1e+06 ++                                                              |
            |                                                               |
     950000 O+                                                              |
            |                                                               |
     900000 ++--------------------------------------------------------------+
Signature · 2 lines
Thanks
    Alex
Signature · 2 lines
Thanks
    Alex
#5 lore raw
Quoted 13 lines · Fengguang Wu
Quoted 12 lines · Alex Shi
Quoted 11 lines · Fengguang Wu
Quoted 7 lines · Alex Shi
Quoted 3 lines · On 12/22/2013 08:30 AM, [email protected] wrote:

Alex,

We noticed some perf-stat.cpu-migrations changes by your commit

Thanks for your wonder testing, Fengguang!
How many CPU in your system and how many dd task running?

For the case

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
        kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd

It's a KVM with 4 CPUs, 16G memory. There are 4 virtual disks, on top of each disk there runs 10 dd tasks.

I don't know much of kvm. Where the data get from, VM guest or VM host?

The data are collected in the VM guest.

do you have similar testing on native linux?

Quoted 229 lines

Thanks,
Fengguang

Quoted 225 lines

Looks like any guest cpu usage will be counted as user usage in vmstat. and Does the CS and cpu migration reducing cause dd performance increase?

Quoted 218 lines

Thanks,
Fengguang

Quoted 214 lines
Quoted 209 lines

6eb8b571283e64e96ca1a91aad085f9e08f857f0 is the first bad commit commit 6eb8b571283e64e96ca1a91aad085f9e08f857f0

Author: Alex Shi <[email protected]>
Date:   Tue Nov 19 20:29:51 2013 +0800

sched: remove rq->cpu_load[load_idx] array

Since load_idx effect removed in load balance, we don't need the load_idx decays in scheduler. that will save some process in sched_tick and others places.

Here are the changed stats comparing to its parent commit:

1de157d718f6494 6eb8b571283e64e96ca1
--------------- -------------------------

     37779 ~40%    +115.2%      81288 ~50%  kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd
     37779         +115.2%      81288       TOTAL proc-vmstat.pgscan_direct_normal

kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: proc-vmstat.pgscan_direct_normal

160000 ++----------------------------------------------------------------+

          |                                                                 |
   140000 ++                                                                O
          |                                                                 |
   120000 ++                                                                |
          |                O                                                |
   100000 ++                                                                |
          |                                                                 |
    80000 ++                               O                                |
          |                                                               ..*
    60000 ++                                                          ....  |
          |                                                       ....      |
    40000 *+...............                                 O ....          |
          O                *...............         ........*.              |
    20000 ++-------------------------------*--------------------------------+

1de157d718f6494 6eb8b571283e64e96ca1
--------------- -------------------------

    121579 ~ 6%     -50.8%      59852 ~13%  kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd
    121579          -50.8%      59852       TOTAL softirqs.SCHED

kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: softirqs.SCHED

140000 ++----------------------------------------------------------------+

   130000 ++           ....*...........                 ....*.......        |
          |    ........                ....     ........            ......  |
   120000 *+...                            *....                          ..|
   110000 ++                                                                *
          |                                                                 |
   100000 ++                                                                |
    90000 ++                                                                |
    80000 ++                                                                |
          |                                                                 |
    70000 ++                               O                O               |
    60000 ++               O                                                |
          |                                                                 O
    50000 O+                                                                |
    40000 ++----------------------------------------------------------------+

1de157d718f6494 6eb8b571283e64e96ca1
--------------- -------------------------

      5636 ~ 4%     -22.1%       4392 ~17%  kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd
      5636          -22.1%       4392       TOTAL proc-vmstat.nr_tlb_remote_flush

kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: proc-vmstat.nr_tlb_remote_flush

6500 ++------------------------------------------------------------------+

        |                                                                   |
   6000 ++       ........*............                                      |
        *........                     ....                                  |
   5500 ++               O                *................*................|
        |                                                                   *
   5000 ++                                O                                 |
        |                                                                   |
   4500 ++                                                 O                |
        |                                                                   |
   4000 ++                                                                  |
        |                                                                   O
   3500 O+                                                                  |
        |                                                                   |

3000 ++------------------------------------------------------------------+

1de157d718f6494 6eb8b571283e64e96ca1
--------------- -------------------------

      5679 ~ 4%     -21.9%       4437 ~17%  kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd
      5679          -21.9%       4437       TOTAL proc-vmstat.nr_tlb_remote_flush_received

6500 ++------------------------------------------------------------------+

        |                                                                   |
   6000 ++       ........*........                                          |
        *........                 ......                                    |
        |                               ..         ........*................|
   5500 ++               O                *........                         *
        |                                                                   |
   5000 ++                                O                                 |
        |                                                                   |
   4500 ++                                                 O                |
        |                                                                   |
        |                                                                   |
   4000 ++                                                                  |
        |                                                                   O

3500 O+------------------------------------------------------------------+

1de157d718f6494 6eb8b571283e64e96ca1
--------------- -------------------------

       167 ~11%     +19.4%        200 ~ 4%  kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd
       167          +19.4%        200       TOTAL uptime.idle

kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: uptime.idle

220 ++-------------------------------------------------------------------+

       |                                                                    O
   210 ++                                                                   |
   200 O+                                 O                                 |
       |              ..*.....                             O                |
   190 ++         ....  O     ...                                           |
   180 ++     ....               ....                                       |
       |  ....                       ....                                   |
   170 *+.                               .                                 .*
   160 ++                                 *......                      .... |
       |                                         ....               ...     |
   150 ++                                            ....        ...        |
   140 ++                                                ..  ....           |
       |                                                   *.               |

130 ++-------------------------------------------------------------------+

1de157d718f6494 6eb8b571283e64e96ca1
--------------- -------------------------

    154781 ~ 5%     -66.8%      51449 ~ 8%  kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd
    154781          -66.8%      51449       TOTAL perf-stat.cpu-migrations

kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: perf-stat.cpu-migrations

180000 ++----------------------------------------------------------------+

          |            ....*...........                                     |
   160000 ++   ........                ....         ........*...............|
          *....                            *........                        *
   140000 ++                                                                |
          |                                                                 |
   120000 ++                                                                |
          |                                                                 |
   100000 ++                                                                |
          |                                                                 |
    80000 ++                                                                |
          |                                                                 |
    60000 ++                                                O               |
          |                O               O                                O
    40000 O+----------------------------------------------------------------+

1de157d718f6494 6eb8b571283e64e96ca1
--------------- -------------------------

      1943 ~ 3%      -7.8%       1792 ~ 4%  kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd
      1943           -7.8%       1792       TOTAL vmstat.system.cs

kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: vmstat.system.cs

2050 ++------------------------------------------------------------------+

        |                         ........*................*.....           |
   2000 ++              .*........                               ...        |
   1950 ++          ....                                            ...     |
        |        ...                                                   .... |
   1900 ++    ...                                                          .|
   1850 ++....                            O                O                *
        *.                                                                  |
   1800 ++               O                                                  O
   1750 ++                                                                  |
        |                                                                   |
   1700 ++                                                                  |
   1650 O+                                                                  |
        |                                                                   |

1600 ++------------------------------------------------------------------+

1de157d718f6494 6eb8b571283e64e96ca1
--------------- -------------------------

   1216177 ~ 3%      -9.4%    1101605 ~ 7%  kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd
   1216177           -9.4%    1101605       TOTAL perf-stat.context-switches
      kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: perf-stat.context-switches

    1.3e+06 ++--------------------------------------------------------------+
            |             ..*...............                                |
   1.25e+06 ++      ......                  *...............*.......        |
            |  .....                                                ......  |
    1.2e+06 *+.                             O                             ..|
   1.15e+06 ++                                              O               *
            |               O                                               |
    1.1e+06 ++                                                              |
            |                                                               O
   1.05e+06 ++                                                              |
      1e+06 ++                                                              |
            |                                                               |
     950000 O+                                                              |
            |                                                               |
     900000 ++--------------------------------------------------------------+
Signature · 2 lines
Thanks
    Alex
Signature · 2 lines
Thanks
    Alex
Signature · 2 lines
Thanks
    Alex