Compile Option Os versus O2 on latest x86 platform
RFC PATCH 1/1 Compile Option Os versus O2 on latest x86 platform
From: Ma Ling <[email protected]>
Hi All
In current kernel compile original option we prefer Os to O2. Os will reduce compiled kernel code size obviously, and O2 pay more attention to performance than code size, so in real environment O2 will bring more i-cache miss than Os, totally performance should slowdown.
In our system test machine kernel code size from Os is 12M, and that from O2 is 14M.
But we have two questions about it on latest platform:
1. 10% * current kernel code size from Os(CPU execution path)
is far more L1 i-cache size, the difference of i-cache-miss counts from both options should become little.
2. our latest platform should has excellent prefetch capability by adjusting predication execution path.
Based on above reasons we re-compiled linux kernel with O2 option on below platform. CPU type: 2P Quad-core Core i7(2 socket*4 core *2 hyper threads)
CPU frequency: 2670MHz
Memory: 6 x 1GBMb
We mainly tested common and stable benchmarks two times, results show O2 performance is better than Os (linux kernel version 2.6.32-rc8)
Benchmarks: improvement
volano 8% netperf 6.7% tbench 6.45% Kbuild 5.5% (3 time test, average improvement) specjbb2000 2% fio 2% specjbb2005 No change cpu2000 No change aim7 No change hackbench No Change oltp No Change
This patch try to enable O2 option and disable Os option.
Appreciate any comments.
Thanks
Ling
--- arch/x86/configs/x86_64_defconfig | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
arch/x86/configs/x86_64_defconfig +1 -1
@@ -126,7 +126,7 @@ CONFIG_INITRAMFS_SOURCE="" CONFIG_RD_GZIP=y CONFIG_RD_BZIP2=y CONFIG_RD_LZMA=y -CONFIG_CC_OPTIMIZE_FOR_SIZE=y +CONFIG_CC_OPTIMIZE_FOR_SIZE=n CONFIG_SYSCTL=y CONFIG_ANON_INODES=y # CONFIG_EMBEDDED is not set
Signature · 1 line
1.6.2.5
Quoted 5 lines · * [email protected] <[email protected]> wrote:
Benchmarks: improvement
volano 8% netperf 6.7% tbench 6.45% Kbuild 5.5% (3 time test, average improvement)
that Kbuild result looks suspicious. A kbuild only uses 25% of system time, so an 5.5% improvement means that system utilization dropped from 25% to 19.5%, a 28% improvement in the kernel! That looks rather unlikely.
Could you please post before/after 'perf stat --repeat 3' results so that we can see the noise level?
Thanks,
Ingo
Hi Ingo
Thanks for your correction, so we use perf stat --repeat 3 to test volano, tbench, and kbuild, Because netperf has multiple items we may send out later.
volano_Os:
Performance counter stats for '/bm/bin/runs -t volano -r /bm/recipes/lkp-ne02.recipe' (3 runs):
6386111.436735 task-clock-msecs # 13.554 CPUs ( +- 0.336% )
914192633 context-switches # 0.143 M/sec ( +- 0.046% )
49186605 CPU-migrations # 0.008 M/sec ( +- 0.962% )
768344 page-faults # 0.000 M/sec ( +- 0.338% )
18680627716893 cycles # 2925.196 M/sec ( +- 0.339% )
7247421283541 instructions # 0.388 IPC ( +- 0.124% )
226838591574 cache-references # 35.521 M/sec ( +- 0.971% )
9420427393 cache-misses # 1.475 M/sec ( +- 0.897% )471.172398867 seconds time elapsed ( +- 1.292% )
volano_O2:
Performance counter stats for '/bm/bin/runs -t volano -r /bm/recipes/lkp-ne02.recipe' (3 runs):
5873675.998422 task-clock-msecs # 13.447 CPUs ( +- 0.338% )
916070728 context-switches # 0.156 M/sec ( +- 0.050% )
48759104 CPU-migrations # 0.008 M/sec ( +- 0.614% )
738964 page-faults # 0.000 M/sec ( +- 0.082% )
17145170491943 cycles # 2918.985 M/sec ( +- 0.288% )
7324126478801 instructions # 0.427 IPC ( +- 0.090% )
219064318074 cache-references # 37.296 M/sec ( +- 0.792% )
9491237013 cache-misses # 1.616 M/sec ( +- 0.439% )436.806579899 seconds time elapsed ( +- 0.392% )
O2 is better than Os for volano
tbench_Os:
Performance counter stats for '/bm/bin/runs -t tbench -r /bm/recipes/lkp-ne02.recipe' (3 runs):
11630970.099215 task-clock-msecs # 15.476 CPUs ( +- 1.285% )
1162148139 context-switches # 0.100 M/sec ( +- 0.372% )
39772 CPU-migrations # 0.000 M/sec ( +- 0.502% )
1536289 page-faults # 0.000 M/sec ( +- 0.020% )
33408973681696 cycles # 2872.415 M/sec ( +- 0.028% )
14229765107716 instructions # 0.426 IPC ( +- 0.113% )
290717607018 cache-references # 24.995 M/sec ( +- 10.425% )
2525058529 cache-misses # 0.217 M/sec ( +- 1.798% )751.537009428 seconds time elapsed ( +- 0.173% )
tbench_O2:
Performance counter stats for '/bm/bin/runs -t tbench -r /bm/recipes/lkp-ne02.recipe' (3 runs):
12093825.537708 task-clock-msecs # 16.084 CPUs ( +- 6.363% )
1235837814 context-switches # 0.102 M/sec ( +- 0.857% )
42363 CPU-migrations # 0.000 M/sec ( +- 3.968% )
1535481 page-faults # 0.000 M/sec ( +- 0.350% )
33028312063911 cycles # 2731.006 M/sec ( +- 0.908% )
15535465986643 instructions # 0.470 IPC ( +- 0.058% )
280118529329 cache-references # 23.162 M/sec ( +- 0.695% )
2866275183 cache-misses # 0.237 M/sec ( +- 0.893% )751.921568581 seconds time elapsed ( +- 0.182% )
O2 is not different with Os for tbench
kbuild_Os:
Performance counter stats for '/bm/bin/runs -t kbuild -r /bm/recipes/lkp-ne02.recipe' (3 runs):
886426.102100 task-clock-msecs # 1.053 CPUs ( +- 1.712% )
980944 context-switches # 0.001 M/sec ( +- 1.149% )
285613 CPU-migrations # 0.000 M/sec ( +- 1.543% )
81244856 page-faults # 0.092 M/sec ( +- 1.611% )
2610381816839 cycles # 2944.839 M/sec ( +- 1.696% )
2907701964460 instructions # 1.114 IPC ( +- 1.726% )
14758764510 cache-references # 16.650 M/sec ( +- 1.581% )
3212068899 cache-misses # 3.624 M/sec ( +- 1.729% )841.492770793 seconds time elapsed ( +- 0.209% )
kbuild_O2:
Performance counter stats for '/bm/bin/runs -t kbuild -r /bm/recipes/lkp-ne02.recipe' (3 runs):
897281.428095 task-clock-msecs # 1.062 CPUs ( +- 0.524% )
964812 context-switches # 0.001 M/sec ( +- 1.630% )
287443 CPU-migrations # 0.000 M/sec ( +- 0.532% )
82509345 page-faults # 0.092 M/sec ( +- 0.071% )
2635837258275 cycles # 2937.581 M/sec ( +- 0.150% )
2955626723788 instructions # 1.121 IPC ( +- 0.117% )
14939108242 cache-references # 16.649 M/sec ( +- 0.609% )
3267365744 cache-misses # 3.641 M/sec ( +- 0.066% ) 844.891541856 seconds time elapsed ( +- 0.468% )
O2 is not different with Os for kbuild
Thanks
Ling
Quoted 29 lines
-----Original Message-----
From: Ingo Molnar [mailto:[email protected]]
Sent: Thursday, November 26, 2009 5:50 PM
To: Ma, Ling; Arjan van de Ven; Dave Jones
Cc: [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH RFC] [X86] Compile Option Os versus O2 on latest x86 platformQuoted 5 lines · * [email protected] <[email protected]> wrote:
Benchmarks: improvement
volano 8% netperf 6.7% tbench 6.45% Kbuild 5.5% (3 time test, averageimprovement)
that
Kbuildresult looks suspicious. A kbuild only uses 25% of system time, so an 5.5% improvement means that system utilization dropped from 25% to 19.5%, a 28% improvement in the kernel! That looks rather unlikely.Could you please post before/after 'perf stat --repeat 3' results so that we can see the noise level?
Thanks,
Ingo
Quoted 5 lines · Ling
Hi Ingo
Thanks for your correction, so we use perf stat --repeat 3 to test volano, tbench, and kbuild, Because netperf has multiple items we may send out later.
a key question is.. how much more memory do you have free due to -Os? (because memory is cache is performance on a system level as well) and how much less icache pressure is there?
Signature · 3 lines
Arjan van de Ven Intel Open Source Technology Centre For development, discussion and tips for power savings, visit http://www.lesswatts.org
H. Peter Anvin
reviewerQuoted 9 lines · Arjan van de Ven
Quoted 5 lines · Ling
Hi Ingo
Thanks for your correction, so we use perf stat --repeat 3 to test volano, tbench, and kbuild, Because netperf has multiple items we may send out later.
Quoted 6 lines
a key question is.. how much more memory do you have free due to -Os? (because memory is cache is performance on a system level as well) and how much less icache pressure is there?
From the re-run, it sounds like the only test that actually shows a
significant difference is volano. From reading the numbers, it looks like the improvements are almost exclusively in IPC i.e. better scheduling -- all the other metrics are substantially worse; including a 10% increase in cache misses.
It would be interesting to see what functions are hot in volano. It might very well be that we could get a boost without significantly bloat the kernel as a whole by picking out a couple of hot object files and compiling those with -O2 or -O3.
-hpa
Signature · 2 lines
H. Peter Anvin, Intel Open Source Technology Center I work for Intel. I don't speak on their behalf.
Quoted 2 lines
a key question is.. how much more memory do you have free due to -Os? (because memory is cache is performance on a system level as well)
The kernel code size from Os is 12M, that from O2 is 14M.
Quoted 2 lines
and how much less icache pressure is there?
From perf stat report, cache reference(unified cache) from O2 is almost the same with Os.
Thanks
Ling
H. Peter Anvin
reviewerQuoted 5 lines · Ling
Quoted 2 lines
a key question is.. how much more memory do you have free due to -Os? (because memory is cache is performance on a system level as well)
The kernel code size from Os is 12M, that from O2 is 14M.
Quoted 1 line
and how much less icache pressure is there?
From perf stat report, cache reference(unified cache) from O2 is almost the same with Os.
The icache pressure was substantially higher (by ~10%) in the reports that I saw.
-hpa
Signature · 2 lines
H. Peter Anvin, Intel Open Source Technology Center I work for Intel. I don't speak on their behalf.
Quoted 9 lines · H. Peter Anvin
Quoted 5 lines · Ling
Quoted 2 lines
a key question is.. how much more memory do you have free due to -Os? (because memory is cache is performance on a system level as well)
The kernel code size from Os is 12M, that from O2 is 14M.
Quoted 1 line
and how much less icache pressure is there?
From perf stat report, cache reference(unified cache) from O2 is almost the same with Os.
The icache pressure was substantially higher (by ~10%) in the reports that I saw.
hm, icache numbers are not included in perf stat runs by default. Are there some icache numbers i missed perhaps?
Ingo
H. Peter Anvin
reviewerQuoted 7 lines · Ingo Molnar
Quoted 4 lines · H. Peter Anvin
Quoted 3 lines · Ling
Quoted 2 lines
a key question is.. how much more memory do you have free due to -Os? (because memory is cache is performance on a system level as well)
The kernel code size from Os is 12M, that from O2 is 14M.
Quoted 9 lines
Quoted 5 lines
Quoted 2 lines
Quoted 1 line
and how much less icache pressure is there?
From perf stat report, cache reference(unified cache) from O2 is almost the same with Os.
The icache pressure was substantially higher (by ~10%) in the reports that I saw.
hm, icache numbers are not included in perf stat runs by default. Are there some icache numbers i missed perhaps?
Sorry, you're right; cache references and cache misses. Furthermore, I'm wrong, I was looking at references *per unit time*, which just show that roughly the same number was squeezed into a shorter time.
Never mind me... :-/
-hpa
Signature · 2 lines
H. Peter Anvin, Intel Open Source Technology Center I work for Intel. I don't speak on their behalf.
Quoted 15 lines · Ma, Ling
Hi Ingo
Thanks for your correction, so we use perf stat --repeat 3 to test volano, tbench, and kbuild, Because netperf has multiple items we may send out later.
volano_Os:18680627716893 cycles # 2925.196 M/sec ( +- 0.339% ) 7247421283541 instructions # 0.388 IPC ( +- 0.124% ) 226838591574 cache-references # 35.521 M/sec ( +- 0.971% ) 9420427393 cache-misses # 1.475 M/sec ( +- 0.897% )
volano_O2:
Quoted 8 lines
17145170491943 cycles # 2918.985 M/sec ( +- 0.288% ) 7324126478801 instructions # 0.427 IPC ( +- 0.090% ) 219064318074 cache-references # 37.296 M/sec ( +- 0.792% ) 9491237013 cache-misses # 1.616 M/sec ( +- 0.439% )O2 is better than Os for volano
O2 is not different with Os for tbench
O2 is not different with Os for kbuild
Ok, this looks pretty credible, thanks for going through it.
For Volano, the difference is 8.9%, well above the 0.3% noise level, so it's significant.
Would it be possible to do a 'perf record' and 'perf report' comparison between two volano runs, to see where the nearly 10% overhead comes from? It might be one or two functions mis-optimized by GCC perhaps. Or it could be across-the-spectrum slowdown.
Note that the number of instructions increased only by 1%, but the overhead by 9%. So we might be hitting some nasty corner case - or it might be some caching effect. (which does not seem to be supported by the numbers though - the LLC cache-misses does not look significantly higher in the Os case)
'perf annotate fn_name' will also help you see where the overhead hot-spots are. If you build the vmlinux via CONFIG_DEBUG_INFO the perf annotate output will interleave assembly and source code output. (otherwise it will be assembly output only)
You probably want to use the latest version of 'perf' for all that analysis, from:
http://people.redhat.com/mingo/tip.git/README
Thanks,
Ingo