perf-stat.cpu-migrations changes in dd write test
Alex,
We noticed some perf-stat.cpu-migrations changes by your commit
6eb8b571283e64e96ca1a91aad085f9e08f857f0 is the first bad commit commit 6eb8b571283e64e96ca1a91aad085f9e08f857f0
Author: Alex Shi <[email protected]> Date: Tue Nov 19 20:29:51 2013 +0800
sched: remove rq->cpu_load[load_idx] array
Since load_idx effect removed in load balance, we don't need the load_idx decays in scheduler. that will save some process in sched_tick and others places.
Here are the changed stats comparing to its parent commit:
1de157d718f6494 6eb8b571283e64e96ca1
--------------- -------------------------
37779 ~40% +115.2% 81288 ~50% kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd
37779 +115.2% 81288 TOTAL proc-vmstat.pgscan_direct_normalkbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: proc-vmstat.pgscan_direct_normal
160000 ++----------------------------------------------------------------+
| |
140000 ++ O
| |
120000 ++ |
| O |
100000 ++ |
| |
80000 ++ O |
| ..*
60000 ++ .... |
| .... |
40000 *+............... O .... |
O *............... ........*. |
20000 ++-------------------------------*--------------------------------+1de157d718f6494 6eb8b571283e64e96ca1
--------------- -------------------------
121579 ~ 6% -50.8% 59852 ~13% kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd
121579 -50.8% 59852 TOTAL softirqs.SCHEDkbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: softirqs.SCHED
140000 ++----------------------------------------------------------------+
130000 ++ ....*........... ....*....... |
| ........ .... ........ ...... |
120000 *+... *.... ..|
110000 ++ *
| |
100000 ++ |
90000 ++ |
80000 ++ |
| |
70000 ++ O O |
60000 ++ O |
| O
50000 O+ |
40000 ++----------------------------------------------------------------+1de157d718f6494 6eb8b571283e64e96ca1
--------------- -------------------------
5636 ~ 4% -22.1% 4392 ~17% kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd
5636 -22.1% 4392 TOTAL proc-vmstat.nr_tlb_remote_flushkbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: proc-vmstat.nr_tlb_remote_flush
6500 ++------------------------------------------------------------------+
| |
6000 ++ ........*............ |
*........ .... |
5500 ++ O *................*................|
| *
5000 ++ O |
| |
4500 ++ O |
| |
4000 ++ |
| O
3500 O+ |
| |3000 ++------------------------------------------------------------------+
1de157d718f6494 6eb8b571283e64e96ca1
--------------- -------------------------
5679 ~ 4% -21.9% 4437 ~17% kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd
5679 -21.9% 4437 TOTAL proc-vmstat.nr_tlb_remote_flush_received6500 ++------------------------------------------------------------------+
| |
6000 ++ ........*........ |
*........ ...... |
| .. ........*................|
5500 ++ O *........ *
| |
5000 ++ O |
| |
4500 ++ O |
| |
| |
4000 ++ |
| O3500 O+------------------------------------------------------------------+
1de157d718f6494 6eb8b571283e64e96ca1
--------------- -------------------------
167 ~11% +19.4% 200 ~ 4% kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd
167 +19.4% 200 TOTAL uptime.idlekbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: uptime.idle
220 ++-------------------------------------------------------------------+
| O
210 ++ |
200 O+ O |
| ..*..... O |
190 ++ .... O ... |
180 ++ .... .... |
| .... .... |
170 *+. . .*
160 ++ *...... .... |
| .... ... |
150 ++ .... ... |
140 ++ .. .... |
| *. |130 ++-------------------------------------------------------------------+
1de157d718f6494 6eb8b571283e64e96ca1
--------------- -------------------------
154781 ~ 5% -66.8% 51449 ~ 8% kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd
154781 -66.8% 51449 TOTAL perf-stat.cpu-migrationskbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: perf-stat.cpu-migrations
180000 ++----------------------------------------------------------------+
| ....*........... |
160000 ++ ........ .... ........*...............|
*.... *........ *
140000 ++ |
| |
120000 ++ |
| |
100000 ++ |
| |
80000 ++ |
| |
60000 ++ O |
| O O O
40000 O+----------------------------------------------------------------+1de157d718f6494 6eb8b571283e64e96ca1
--------------- -------------------------
1943 ~ 3% -7.8% 1792 ~ 4% kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd
1943 -7.8% 1792 TOTAL vmstat.system.cskbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: vmstat.system.cs
2050 ++------------------------------------------------------------------+
| ........*................*..... |
2000 ++ .*........ ... |
1950 ++ .... ... |
| ... .... |
1900 ++ ... .|
1850 ++.... O O *
*. |
1800 ++ O O
1750 ++ |
| |
1700 ++ |
1650 O+ |
| |1600 ++------------------------------------------------------------------+
1de157d718f6494 6eb8b571283e64e96ca1
--------------- -------------------------
1216177 ~ 3% -9.4% 1101605 ~ 7% kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd 1216177 -9.4% 1101605 TOTAL perf-stat.context-switches
kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: perf-stat.context-switches
1.3e+06 ++--------------------------------------------------------------+
| ..*............... |
1.25e+06 ++ ...... *...............*....... |
| ..... ...... |
1.2e+06 *+. O ..|
1.15e+06 ++ O *
| O |
1.1e+06 ++ |
| O
1.05e+06 ++ |
1e+06 ++ |
| |
950000 O+ |
| |
900000 ++--------------------------------------------------------------+Quoted 3 lines · On 12/22/2013 08:30 AM, [email protected] wrote:
Alex,
We noticed some perf-stat.cpu-migrations changes by your commit
Thanks for your wonder testing, Fengguang!
How many CPU in your system and how many dd task running?
Quoted 209 lines
6eb8b571283e64e96ca1a91aad085f9e08f857f0is the first bad commit commit6eb8b571283e64e96ca1a91aad085f9e08f857f0Author: Alex Shi <[email protected]> Date: Tue Nov 19 20:29:51 2013 +0800sched: remove rq->cpu_load[load_idx] array
Since
load_idxeffect removed in load balance, we don't need theload_idxdecays in scheduler. that will save some process insched_tickand others places.Signed-off-by Alex Shi <[email protected]>Here are the changed stats comparing to its parent commit:
1de157d718f64946eb8b571283e64e96ca1
--------------- -------------------------37779 ~40% +115.2% 81288 ~50% kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd 37779 +115.2% 81288 TOTAL proc-vmstat.pgscan_direct_normalkbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: proc-vmstat.pgscan_direct_normal
160000 ++----------------------------------------------------------------+
| | 140000 ++ O | | 120000 ++ | | O | 100000 ++ | | | 80000 ++ O | | ..* 60000 ++ .... | | .... | 40000 *+............... O .... | O *............... ........*. | 20000 ++-------------------------------*--------------------------------+
1de157d718f64946eb8b571283e64e96ca1
--------------- -------------------------121579 ~ 6% -50.8% 59852 ~13% kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd 121579 -50.8% 59852 TOTAL softirqs.SCHEDkbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: softirqs.SCHED
140000 ++----------------------------------------------------------------+
130000 ++ ....*........... ....*....... | | ........ .... ........ ...... | 120000 *+... *.... ..| 110000 ++ * | | 100000 ++ | 90000 ++ | 80000 ++ | | | 70000 ++ O O | 60000 ++ O | | O 50000 O+ | 40000 ++----------------------------------------------------------------+
1de157d718f64946eb8b571283e64e96ca1
--------------- -------------------------5636 ~ 4% -22.1% 4392 ~17% kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd 5636 -22.1% 4392 TOTAL proc-vmstat.nr_tlb_remote_flushkbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: proc-vmstat.nr_tlb_remote_flush
6500 ++------------------------------------------------------------------+
| | 6000 ++ ........*............ | *........ .... | 5500 ++ O *................*................| | * 5000 ++ O | | | 4500 ++ O | | | 4000 ++ | | O 3500 O+ | | |3000 ++------------------------------------------------------------------+
1de157d718f64946eb8b571283e64e96ca1
--------------- -------------------------5679 ~ 4% -21.9% 4437 ~17% kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd 5679 -21.9% 4437 TOTAL proc-vmstat.nr_tlb_remote_flush_received6500 ++------------------------------------------------------------------+
| | 6000 ++ ........*........ | *........ ...... | | .. ........*................| 5500 ++ O *........ * | | 5000 ++ O | | | 4500 ++ O | | | | | 4000 ++ | | O3500 O+------------------------------------------------------------------+
1de157d718f64946eb8b571283e64e96ca1
--------------- -------------------------167 ~11% +19.4% 200 ~ 4% kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd 167 +19.4% 200 TOTAL uptime.idlekbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: uptime.idle
220 ++-------------------------------------------------------------------+
| O 210 ++ | 200 O+ O | | ..*..... O | 190 ++ .... O ... | 180 ++ .... .... | | .... .... | 170 *+. . .* 160 ++ *...... .... | | .... ... | 150 ++ .... ... | 140 ++ .. .... | | *. |130 ++-------------------------------------------------------------------+
1de157d718f64946eb8b571283e64e96ca1
--------------- -------------------------154781 ~ 5% -66.8% 51449 ~ 8% kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd 154781 -66.8% 51449 TOTAL perf-stat.cpu-migrationskbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: perf-stat.cpu-migrations
180000 ++----------------------------------------------------------------+
| ....*........... | 160000 ++ ........ .... ........*...............| *.... *........ * 140000 ++ | | | 120000 ++ | | | 100000 ++ | | | 80000 ++ | | | 60000 ++ O | | O O O 40000 O+----------------------------------------------------------------+
1de157d718f64946eb8b571283e64e96ca1
--------------- -------------------------1943 ~ 3% -7.8% 1792 ~ 4% kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd 1943 -7.8% 1792 TOTAL vmstat.system.cskbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: vmstat.system.cs
2050 ++------------------------------------------------------------------+
| ........*................*..... | 2000 ++ .*........ ... | 1950 ++ .... ... | | ... .... | 1900 ++ ... .| 1850 ++.... O O * *. | 1800 ++ O O 1750 ++ | | | 1700 ++ | 1650 O+ | | |1600 ++------------------------------------------------------------------+
1de157d718f64946eb8b571283e64e96ca1
--------------- -------------------------1216177 ~ 3% -9.4% 1101605 ~ 7% kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd 1216177 -9.4% 1101605 TOTAL perf-stat.context-switcheskbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: perf-stat.context-switches 1.3e+06 ++--------------------------------------------------------------+ | ..*............... | 1.25e+06 ++ ...... *...............*....... | | ..... ...... | 1.2e+06 *+. O ..| 1.15e+06 ++ O * | O | 1.1e+06 ++ | | O 1.05e+06 ++ | 1e+06 ++ | | | 950000 O+ | | | 900000 ++--------------------------------------------------------------+
Signature · 2 lines
Thanks
AlexQuoted 7 lines · Alex Shi
Quoted 3 lines · On 12/22/2013 08:30 AM, [email protected] wrote:
Alex,
We noticed some perf-stat.cpu-migrations changes by your commit
Thanks for your wonder testing, Fengguang!
How many CPU in your system and how many dd task running?
For the case
kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd
It's a KVM with 4 CPUs, 16G memory. There are 4 virtual disks, on top of each disk there runs 10 dd tasks.
Thanks,
Fengguang
Quoted 214 lines
Quoted 209 lines
6eb8b571283e64e96ca1a91aad085f9e08f857f0is the first bad commit commit6eb8b571283e64e96ca1a91aad085f9e08f857f0Author: Alex Shi <[email protected]> Date: Tue Nov 19 20:29:51 2013 +0800sched: remove rq->cpu_load[load_idx] array
Since
load_idxeffect removed in load balance, we don't need theload_idxdecays in scheduler. that will save some process insched_tickand others places.Signed-off-by Alex Shi <[email protected]>Here are the changed stats comparing to its parent commit:
1de157d718f64946eb8b571283e64e96ca1
--------------- -------------------------37779 ~40% +115.2% 81288 ~50% kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd 37779 +115.2% 81288 TOTAL proc-vmstat.pgscan_direct_normalkbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: proc-vmstat.pgscan_direct_normal
160000 ++----------------------------------------------------------------+
| | 140000 ++ O | | 120000 ++ | | O | 100000 ++ | | | 80000 ++ O | | ..* 60000 ++ .... | | .... | 40000 *+............... O .... | O *............... ........*. | 20000 ++-------------------------------*--------------------------------+
1de157d718f64946eb8b571283e64e96ca1
--------------- -------------------------121579 ~ 6% -50.8% 59852 ~13% kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd 121579 -50.8% 59852 TOTAL softirqs.SCHEDkbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: softirqs.SCHED
140000 ++----------------------------------------------------------------+
130000 ++ ....*........... ....*....... | | ........ .... ........ ...... | 120000 *+... *.... ..| 110000 ++ * | | 100000 ++ | 90000 ++ | 80000 ++ | | | 70000 ++ O O | 60000 ++ O | | O 50000 O+ | 40000 ++----------------------------------------------------------------+
1de157d718f64946eb8b571283e64e96ca1
--------------- -------------------------5636 ~ 4% -22.1% 4392 ~17% kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd 5636 -22.1% 4392 TOTAL proc-vmstat.nr_tlb_remote_flushkbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: proc-vmstat.nr_tlb_remote_flush
6500 ++------------------------------------------------------------------+
| | 6000 ++ ........*............ | *........ .... | 5500 ++ O *................*................| | * 5000 ++ O | | | 4500 ++ O | | | 4000 ++ | | O 3500 O+ | | |3000 ++------------------------------------------------------------------+
1de157d718f64946eb8b571283e64e96ca1
--------------- -------------------------5679 ~ 4% -21.9% 4437 ~17% kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd 5679 -21.9% 4437 TOTAL proc-vmstat.nr_tlb_remote_flush_received6500 ++------------------------------------------------------------------+
| | 6000 ++ ........*........ | *........ ...... | | .. ........*................| 5500 ++ O *........ * | | 5000 ++ O | | | 4500 ++ O | | | | | 4000 ++ | | O3500 O+------------------------------------------------------------------+
1de157d718f64946eb8b571283e64e96ca1
--------------- -------------------------167 ~11% +19.4% 200 ~ 4% kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd 167 +19.4% 200 TOTAL uptime.idlekbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: uptime.idle
220 ++-------------------------------------------------------------------+
| O 210 ++ | 200 O+ O | | ..*..... O | 190 ++ .... O ... | 180 ++ .... .... | | .... .... | 170 *+. . .* 160 ++ *...... .... | | .... ... | 150 ++ .... ... | 140 ++ .. .... | | *. |130 ++-------------------------------------------------------------------+
1de157d718f64946eb8b571283e64e96ca1
--------------- -------------------------154781 ~ 5% -66.8% 51449 ~ 8% kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd 154781 -66.8% 51449 TOTAL perf-stat.cpu-migrationskbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: perf-stat.cpu-migrations
180000 ++----------------------------------------------------------------+
| ....*........... | 160000 ++ ........ .... ........*...............| *.... *........ * 140000 ++ | | | 120000 ++ | | | 100000 ++ | | | 80000 ++ | | | 60000 ++ O | | O O O 40000 O+----------------------------------------------------------------+
1de157d718f64946eb8b571283e64e96ca1
--------------- -------------------------1943 ~ 3% -7.8% 1792 ~ 4% kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd 1943 -7.8% 1792 TOTAL vmstat.system.cskbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: vmstat.system.cs
2050 ++------------------------------------------------------------------+
| ........*................*..... | 2000 ++ .*........ ... | 1950 ++ .... ... | | ... .... | 1900 ++ ... .| 1850 ++.... O O * *. | 1800 ++ O O 1750 ++ | | | 1700 ++ | 1650 O+ | | |1600 ++------------------------------------------------------------------+
1de157d718f64946eb8b571283e64e96ca1
--------------- -------------------------1216177 ~ 3% -9.4% 1101605 ~ 7% kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd 1216177 -9.4% 1101605 TOTAL perf-stat.context-switcheskbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: perf-stat.context-switches 1.3e+06 ++--------------------------------------------------------------+ | ..*............... | 1.25e+06 ++ ...... *...............*....... | | ..... ...... | 1.2e+06 *+. O ..| 1.15e+06 ++ O * | O | 1.1e+06 ++ | | O 1.05e+06 ++ | 1e+06 ++ | | | 950000 O+ | | | 900000 ++--------------------------------------------------------------+Signature · 2 lines
Thanks Alex
Quoted 9 lines · Fengguang Wu
Quoted 7 lines · Alex Shi
Quoted 3 lines · On 12/22/2013 08:30 AM, [email protected] wrote:
Alex,
We noticed some perf-stat.cpu-migrations changes by your commit
Thanks for your wonder testing, Fengguang!
How many CPU in your system and how many dd task running?
Quoted 6 lines
For the case
kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10ddIt's a KVM with 4 CPUs, 16G memory. There are 4 virtual disks, on top of each disk there runs 10 dd tasks.
I don't know much of kvm. Where the data get from, VM guest or VM host? Looks like any guest cpu usage will be counted as user usage in vmstat. and Does the CS and cpu migration reducing cause dd performance increase?
Quoted 218 lines
Thanks,
FengguangQuoted 214 lines
Quoted 209 lines
6eb8b571283e64e96ca1a91aad085f9e08f857f0is the first bad commit commit6eb8b571283e64e96ca1a91aad085f9e08f857f0Author: Alex Shi <[email protected]> Date: Tue Nov 19 20:29:51 2013 +0800sched: remove rq->cpu_load[load_idx] array
Since
load_idxeffect removed in load balance, we don't need theload_idxdecays in scheduler. that will save some process insched_tickand others places.Signed-off-by Alex Shi <[email protected]>Here are the changed stats comparing to its parent commit:
1de157d718f64946eb8b571283e64e96ca1
--------------- -------------------------37779 ~40% +115.2% 81288 ~50% kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd 37779 +115.2% 81288 TOTAL proc-vmstat.pgscan_direct_normalkbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: proc-vmstat.pgscan_direct_normal
160000 ++----------------------------------------------------------------+
| | 140000 ++ O | | 120000 ++ | | O | 100000 ++ | | | 80000 ++ O | | ..* 60000 ++ .... | | .... | 40000 *+............... O .... | O *............... ........*. | 20000 ++-------------------------------*--------------------------------+
1de157d718f64946eb8b571283e64e96ca1
--------------- -------------------------121579 ~ 6% -50.8% 59852 ~13% kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd 121579 -50.8% 59852 TOTAL softirqs.SCHEDkbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: softirqs.SCHED
140000 ++----------------------------------------------------------------+
130000 ++ ....*........... ....*....... | | ........ .... ........ ...... | 120000 *+... *.... ..| 110000 ++ * | | 100000 ++ | 90000 ++ | 80000 ++ | | | 70000 ++ O O | 60000 ++ O | | O 50000 O+ | 40000 ++----------------------------------------------------------------+
1de157d718f64946eb8b571283e64e96ca1
--------------- -------------------------5636 ~ 4% -22.1% 4392 ~17% kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd 5636 -22.1% 4392 TOTAL proc-vmstat.nr_tlb_remote_flushkbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: proc-vmstat.nr_tlb_remote_flush
6500 ++------------------------------------------------------------------+
| | 6000 ++ ........*............ | *........ .... | 5500 ++ O *................*................| | * 5000 ++ O | | | 4500 ++ O | | | 4000 ++ | | O 3500 O+ | | |3000 ++------------------------------------------------------------------+
1de157d718f64946eb8b571283e64e96ca1
--------------- -------------------------5679 ~ 4% -21.9% 4437 ~17% kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd 5679 -21.9% 4437 TOTAL proc-vmstat.nr_tlb_remote_flush_received6500 ++------------------------------------------------------------------+
| | 6000 ++ ........*........ | *........ ...... | | .. ........*................| 5500 ++ O *........ * | | 5000 ++ O | | | 4500 ++ O | | | | | 4000 ++ | | O3500 O+------------------------------------------------------------------+
1de157d718f64946eb8b571283e64e96ca1
--------------- -------------------------167 ~11% +19.4% 200 ~ 4% kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd 167 +19.4% 200 TOTAL uptime.idlekbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: uptime.idle
220 ++-------------------------------------------------------------------+
| O 210 ++ | 200 O+ O | | ..*..... O | 190 ++ .... O ... | 180 ++ .... .... | | .... .... | 170 *+. . .* 160 ++ *...... .... | | .... ... | 150 ++ .... ... | 140 ++ .. .... | | *. |130 ++-------------------------------------------------------------------+
1de157d718f64946eb8b571283e64e96ca1
--------------- -------------------------154781 ~ 5% -66.8% 51449 ~ 8% kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd 154781 -66.8% 51449 TOTAL perf-stat.cpu-migrationskbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: perf-stat.cpu-migrations
180000 ++----------------------------------------------------------------+
| ....*........... | 160000 ++ ........ .... ........*...............| *.... *........ * 140000 ++ | | | 120000 ++ | | | 100000 ++ | | | 80000 ++ | | | 60000 ++ O | | O O O 40000 O+----------------------------------------------------------------+
1de157d718f64946eb8b571283e64e96ca1
--------------- -------------------------1943 ~ 3% -7.8% 1792 ~ 4% kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd 1943 -7.8% 1792 TOTAL vmstat.system.cskbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: vmstat.system.cs
2050 ++------------------------------------------------------------------+
| ........*................*..... | 2000 ++ .*........ ... | 1950 ++ .... ... | | ... .... | 1900 ++ ... .| 1850 ++.... O O * *. | 1800 ++ O O 1750 ++ | | | 1700 ++ | 1650 O+ | | |1600 ++------------------------------------------------------------------+
1de157d718f64946eb8b571283e64e96ca1
--------------- -------------------------1216177 ~ 3% -9.4% 1101605 ~ 7% kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd 1216177 -9.4% 1101605 TOTAL perf-stat.context-switcheskbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: perf-stat.context-switches 1.3e+06 ++--------------------------------------------------------------+ | ..*............... | 1.25e+06 ++ ...... *...............*....... | | ..... ...... | 1.2e+06 *+. O ..| 1.15e+06 ++ O * | O | 1.1e+06 ++ | | O 1.05e+06 ++ | 1e+06 ++ | | | 950000 O+ | | | 900000 ++--------------------------------------------------------------+Signature · 2 lines
Thanks Alex
Signature · 2 lines
Thanks
AlexQuoted 10 lines · Alex Shi
Quoted 9 lines · Fengguang Wu
Quoted 7 lines · Alex Shi
Quoted 3 lines · On 12/22/2013 08:30 AM, [email protected] wrote:
Alex,
We noticed some perf-stat.cpu-migrations changes by your commit
Thanks for your wonder testing, Fengguang!
How many CPU in your system and how many dd task running?
Quoted 8 lines
Quoted 6 lines
For the case
kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10ddIt's a KVM with 4 CPUs, 16G memory. There are 4 virtual disks, on top of each disk there runs 10 dd tasks.
I don't know much of kvm. Where the data get from, VM guest or VM host?
The data are collected in the VM guest.
Thanks,
Fengguang
Quoted 225 lines
Looks like any guest cpu usage will be counted as user usage in vmstat. and Does the CS and cpu migration reducing cause dd performance increase?
Quoted 218 lines
Thanks,
FengguangQuoted 214 lines
Quoted 209 lines
6eb8b571283e64e96ca1a91aad085f9e08f857f0is the first bad commit commit6eb8b571283e64e96ca1a91aad085f9e08f857f0Author: Alex Shi <[email protected]> Date: Tue Nov 19 20:29:51 2013 +0800sched: remove rq->cpu_load[load_idx] array
Since
load_idxeffect removed in load balance, we don't need theload_idxdecays in scheduler. that will save some process insched_tickand others places.Signed-off-by Alex Shi <[email protected]>Here are the changed stats comparing to its parent commit:
1de157d718f64946eb8b571283e64e96ca1
--------------- -------------------------37779 ~40% +115.2% 81288 ~50% kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd 37779 +115.2% 81288 TOTAL proc-vmstat.pgscan_direct_normalkbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: proc-vmstat.pgscan_direct_normal
160000 ++----------------------------------------------------------------+
| | 140000 ++ O | | 120000 ++ | | O | 100000 ++ | | | 80000 ++ O | | ..* 60000 ++ .... | | .... | 40000 *+............... O .... | O *............... ........*. | 20000 ++-------------------------------*--------------------------------+
1de157d718f64946eb8b571283e64e96ca1
--------------- -------------------------121579 ~ 6% -50.8% 59852 ~13% kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd 121579 -50.8% 59852 TOTAL softirqs.SCHEDkbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: softirqs.SCHED
140000 ++----------------------------------------------------------------+
130000 ++ ....*........... ....*....... | | ........ .... ........ ...... | 120000 *+... *.... ..| 110000 ++ * | | 100000 ++ | 90000 ++ | 80000 ++ | | | 70000 ++ O O | 60000 ++ O | | O 50000 O+ | 40000 ++----------------------------------------------------------------+
1de157d718f64946eb8b571283e64e96ca1
--------------- -------------------------5636 ~ 4% -22.1% 4392 ~17% kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd 5636 -22.1% 4392 TOTAL proc-vmstat.nr_tlb_remote_flushkbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: proc-vmstat.nr_tlb_remote_flush
6500 ++------------------------------------------------------------------+
| | 6000 ++ ........*............ | *........ .... | 5500 ++ O *................*................| | * 5000 ++ O | | | 4500 ++ O | | | 4000 ++ | | O 3500 O+ | | |3000 ++------------------------------------------------------------------+
1de157d718f64946eb8b571283e64e96ca1
--------------- -------------------------5679 ~ 4% -21.9% 4437 ~17% kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd 5679 -21.9% 4437 TOTAL proc-vmstat.nr_tlb_remote_flush_received6500 ++------------------------------------------------------------------+
| | 6000 ++ ........*........ | *........ ...... | | .. ........*................| 5500 ++ O *........ * | | 5000 ++ O | | | 4500 ++ O | | | | | 4000 ++ | | O3500 O+------------------------------------------------------------------+
1de157d718f64946eb8b571283e64e96ca1
--------------- -------------------------167 ~11% +19.4% 200 ~ 4% kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd 167 +19.4% 200 TOTAL uptime.idlekbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: uptime.idle
220 ++-------------------------------------------------------------------+
| O 210 ++ | 200 O+ O | | ..*..... O | 190 ++ .... O ... | 180 ++ .... .... | | .... .... | 170 *+. . .* 160 ++ *...... .... | | .... ... | 150 ++ .... ... | 140 ++ .. .... | | *. |130 ++-------------------------------------------------------------------+
1de157d718f64946eb8b571283e64e96ca1
--------------- -------------------------154781 ~ 5% -66.8% 51449 ~ 8% kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd 154781 -66.8% 51449 TOTAL perf-stat.cpu-migrationskbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: perf-stat.cpu-migrations
180000 ++----------------------------------------------------------------+
| ....*........... | 160000 ++ ........ .... ........*...............| *.... *........ * 140000 ++ | | | 120000 ++ | | | 100000 ++ | | | 80000 ++ | | | 60000 ++ O | | O O O 40000 O+----------------------------------------------------------------+
1de157d718f64946eb8b571283e64e96ca1
--------------- -------------------------1943 ~ 3% -7.8% 1792 ~ 4% kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd 1943 -7.8% 1792 TOTAL vmstat.system.cskbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: vmstat.system.cs
2050 ++------------------------------------------------------------------+
| ........*................*..... | 2000 ++ .*........ ... | 1950 ++ .... ... | | ... .... | 1900 ++ ... .| 1850 ++.... O O * *. | 1800 ++ O O 1750 ++ | | | 1700 ++ | 1650 O+ | | |1600 ++------------------------------------------------------------------+
1de157d718f64946eb8b571283e64e96ca1
--------------- -------------------------1216177 ~ 3% -9.4% 1101605 ~ 7% kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd 1216177 -9.4% 1101605 TOTAL perf-stat.context-switcheskbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: perf-stat.context-switches 1.3e+06 ++--------------------------------------------------------------+ | ..*............... | 1.25e+06 ++ ...... *...............*....... | | ..... ...... | 1.2e+06 *+. O ..| 1.15e+06 ++ O * | O | 1.1e+06 ++ | | O 1.05e+06 ++ | 1e+06 ++ | | | 950000 O+ | | | 900000 ++--------------------------------------------------------------+Signature · 2 lines
Thanks AlexSignature · 2 lines
Thanks Alex
Quoted 13 lines · Fengguang Wu
Quoted 12 lines · Alex Shi
Quoted 11 lines · Fengguang Wu
Quoted 7 lines · Alex Shi
Quoted 3 lines · On 12/22/2013 08:30 AM, [email protected] wrote:
Alex,
We noticed some perf-stat.cpu-migrations changes by your commit
Thanks for your wonder testing, Fengguang!
How many CPU in your system and how many dd task running?For the case
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10ddIt's a KVM with 4 CPUs, 16G memory. There are 4 virtual disks, on top of each disk there runs 10 dd tasks.
I don't know much of kvm. Where the data get from, VM guest or VM host?
The data are collected in the VM guest.
do you have similar testing on native linux?
Quoted 229 lines
Thanks,
FengguangQuoted 225 lines
Looks like any guest cpu usage will be counted as user usage in vmstat. and Does the CS and cpu migration reducing cause dd performance increase?
Quoted 218 lines
Thanks,
FengguangQuoted 214 lines
Quoted 209 lines
6eb8b571283e64e96ca1a91aad085f9e08f857f0is the first bad commit commit6eb8b571283e64e96ca1a91aad085f9e08f857f0Author: Alex Shi <[email protected]> Date: Tue Nov 19 20:29:51 2013 +0800sched: remove rq->cpu_load[load_idx] array
Since
load_idxeffect removed in load balance, we don't need theload_idxdecays in scheduler. that will save some process insched_tickand others places.Signed-off-by Alex Shi <[email protected]>Here are the changed stats comparing to its parent commit:
1de157d718f64946eb8b571283e64e96ca1
--------------- -------------------------37779 ~40% +115.2% 81288 ~50% kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd 37779 +115.2% 81288 TOTAL proc-vmstat.pgscan_direct_normalkbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: proc-vmstat.pgscan_direct_normal
160000 ++----------------------------------------------------------------+
| | 140000 ++ O | | 120000 ++ | | O | 100000 ++ | | | 80000 ++ O | | ..* 60000 ++ .... | | .... | 40000 *+............... O .... | O *............... ........*. | 20000 ++-------------------------------*--------------------------------+
1de157d718f64946eb8b571283e64e96ca1
--------------- -------------------------121579 ~ 6% -50.8% 59852 ~13% kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd 121579 -50.8% 59852 TOTAL softirqs.SCHEDkbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: softirqs.SCHED
140000 ++----------------------------------------------------------------+
130000 ++ ....*........... ....*....... | | ........ .... ........ ...... | 120000 *+... *.... ..| 110000 ++ * | | 100000 ++ | 90000 ++ | 80000 ++ | | | 70000 ++ O O | 60000 ++ O | | O 50000 O+ | 40000 ++----------------------------------------------------------------+
1de157d718f64946eb8b571283e64e96ca1
--------------- -------------------------5636 ~ 4% -22.1% 4392 ~17% kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd 5636 -22.1% 4392 TOTAL proc-vmstat.nr_tlb_remote_flushkbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: proc-vmstat.nr_tlb_remote_flush
6500 ++------------------------------------------------------------------+
| | 6000 ++ ........*............ | *........ .... | 5500 ++ O *................*................| | * 5000 ++ O | | | 4500 ++ O | | | 4000 ++ | | O 3500 O+ | | |3000 ++------------------------------------------------------------------+
1de157d718f64946eb8b571283e64e96ca1
--------------- -------------------------5679 ~ 4% -21.9% 4437 ~17% kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd 5679 -21.9% 4437 TOTAL proc-vmstat.nr_tlb_remote_flush_received6500 ++------------------------------------------------------------------+
| | 6000 ++ ........*........ | *........ ...... | | .. ........*................| 5500 ++ O *........ * | | 5000 ++ O | | | 4500 ++ O | | | | | 4000 ++ | | O3500 O+------------------------------------------------------------------+
1de157d718f64946eb8b571283e64e96ca1
--------------- -------------------------167 ~11% +19.4% 200 ~ 4% kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd 167 +19.4% 200 TOTAL uptime.idlekbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: uptime.idle
220 ++-------------------------------------------------------------------+
| O 210 ++ | 200 O+ O | | ..*..... O | 190 ++ .... O ... | 180 ++ .... .... | | .... .... | 170 *+. . .* 160 ++ *...... .... | | .... ... | 150 ++ .... ... | 140 ++ .. .... | | *. |130 ++-------------------------------------------------------------------+
1de157d718f64946eb8b571283e64e96ca1
--------------- -------------------------154781 ~ 5% -66.8% 51449 ~ 8% kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd 154781 -66.8% 51449 TOTAL perf-stat.cpu-migrationskbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: perf-stat.cpu-migrations
180000 ++----------------------------------------------------------------+
| ....*........... | 160000 ++ ........ .... ........*...............| *.... *........ * 140000 ++ | | | 120000 ++ | | | 100000 ++ | | | 80000 ++ | | | 60000 ++ O | | O O O 40000 O+----------------------------------------------------------------+
1de157d718f64946eb8b571283e64e96ca1
--------------- -------------------------1943 ~ 3% -7.8% 1792 ~ 4% kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd 1943 -7.8% 1792 TOTAL vmstat.system.cskbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: vmstat.system.cs
2050 ++------------------------------------------------------------------+
| ........*................*..... | 2000 ++ .*........ ... | 1950 ++ .... ... | | ... .... | 1900 ++ ... .| 1850 ++.... O O * *. | 1800 ++ O O 1750 ++ | | | 1700 ++ | 1650 O+ | | |1600 ++------------------------------------------------------------------+
1de157d718f64946eb8b571283e64e96ca1
--------------- -------------------------1216177 ~ 3% -9.4% 1101605 ~ 7% kbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd 1216177 -9.4% 1101605 TOTAL perf-stat.context-switcheskbuildx/micro/dd-write/4HDD-JBOD-cfq-ext4-10dd: perf-stat.context-switches 1.3e+06 ++--------------------------------------------------------------+ | ..*............... | 1.25e+06 ++ ...... *...............*....... | | ..... ...... | 1.2e+06 *+. O ..| 1.15e+06 ++ O * | O | 1.1e+06 ++ | | O 1.05e+06 ++ | 1e+06 ++ | | | 950000 O+ | | | 900000 ++--------------------------------------------------------------+Signature · 2 lines
Thanks AlexSignature · 2 lines
Thanks Alex
Signature · 2 lines
Thanks
Alex